Filtering lines efficiently

Whenever you are dealing with log lines or that you're program is filtering data you always have to handle 'escaping' efficiently.

While developing a log module using a gen_event, I needed to escape simple quotes.
Sometime thoses quotes were already escaped...

I've found this regexp to handle gracefully the case:

re:replace( Bin, "(?<!\\\\)'", "\\\\'", [ global ] ).

erlang: testing many conditions easily with lists of funs...

Sometimes you have to test many things before being able to choose the next action...

In many languages, you'll end up using a bunch of "if then else".
But in erlang, and the power of fun()s, you can efficiently write a simple function that will do all the job for you :p

Here is our purpose: call many functions with one argument.
For this example, we need to determine a file type with its filename.

Let's say that:
- the filename could be a valid 'word' temporary file,
- or a 'excel' temporary file or
- a known file type.

Firts let's define a simple fun that takes a list of fun and stop evaluating those fun as soon as a result is found:

% the simple case where the list is empty
any(_, []) -> undefined;

% the general case when the list contains funs...
any(Arg, [ {F, PrepareFun} | Funs] ) ->
        case F( PrepareFun(Arg) ) of
                undefined ->
                        any(Arg, Funs);

                _V ->
                        _V
        end.

In this code you'll notice that there are two fun()s:
- the 'F',
- the 'PrepareFun'.
The idea is that 'PrepareFun' will be called before calling 'F' to filter the argument 'Arg'.
Imagine that sometimes you need to extract the basename from the filename, or whatever else...

The code is a simple list iteration that recurse only if the result of the function call is 'undefined'.

Now that we have a valid fun that can iterate over a list of funs and stop whenever a valid result is found (or end of list), let's get back to our example, and build our 'filetype' function:

fileType(File) ->
        any( File, [ 
                        {fun word_temp/1, fun filename:basename/1}, 
                        {fun db_extension/1, fun lists:reverse/1},
                        {fun excel_temp/1, fun filename:basename/1}
                ]).

You can read this code like this:
"any of the funs from the list may determine the type of the file".
And once found, stops.

Let's describe those called functions 'db_extension/1', 'word_temp/1', 'excel_temp/1'...

First 'db_extension':
You'll notice that we test only the end of the filename, that's why the filename is
reversed before being passed to the function:

db_extension( "pmt."  ++ _ ) -> temp;
db_extension( "PMT."  ++ _ ) -> temp;
db_extension( "xcod." ++ _ ) -> doc;
db_extension( "cod."  ++ _ ) -> doc;
db_extension( "xslx." ++ _ ) -> xls;
db_extension( "slx."  ++ _ ) -> xls;
db_extension( "xtpp." ++ _ ) -> ppt;
db_extension( "tpp." ++ _ ) -> ppt;
db_extension( _ ) -> undefined.

The 'word_temp/1' need to call the basename of the file but we don't need the full path, so 'PrepareFun' is simply 'filename:basename/1' in this case:

word_temp( "~$"   ++ _) -> temp;
word_temp( "~WRD" ++ _) -> temp;
word_temp( "~WRL" ++ _) -> temp;
word_temp( _ ) -> undefined.

For 'excel_temp/1', the temp file is determined by a number written as 8 hexadecimal values. We use the re module to easily match this with the filename. In this case the 'PrepareFun' is also the 'filename:basename/1':

excel_temp( File ) ->
        ReList = [ <<"^[0-9A-Z]{8}$">> ],
        do_re(File, ReList).
        
% We are able to test many re but in the specific 

% case the list contains only one element...
do_re(_, []) -> undefined;
do_re(Subject, [ Re | Rest ]) ->
        case re:run(Subject, Re, [{capture,none}]) of
                nomatch ->
                        do_re(Subject, Rest);

                match ->
                        temp
        end.

From the re module, options "capture none" is used to only returns if the re match, and not the part that successfully match...
(this is simple optimisation, since we don't care about the matching part)

If we look at back at what we've done here, we can see that

fileType(File) ->
        any( File, [ 
                        {fun word_temp/1, fun filename:basename/1}, 
                        {fun db_extension/1, fun lists:reverse/1},
                        {fun excel_temp/1, fun filename:basename/1}
                ]).

can really easily extended with other functions, as long as those new functions take only one parameter...

fileType(File) ->
        any( File, [ 
                        {fun word_temp/1, fun filename:basename/1}, 
                        {fun db_extension/1, fun lists:reverse/1},
                        {fun excel_temp/1, fun filename:basename/1},
                        {fun firefox_temp/1, fun filename:basename/1},
                        {fun directory_temp/1, fun(X) -> X end}
                ]).

Conclusion:
Building list of functions is an efficient way of "testing many conditions".

An gen_server for mass regexp computing... (LibTre)

This is the first test session of my 'tregex_srv' that provides some nice regexp features:

266> l(tregex_srv).                                                 
{module,tregex_srv}
267> tregex_srv:start_link().                                       
{ok,<0.4045.0>}
268> tregex_srv:store( [<<"[0-9+] pid">>, <<"[a-z]+.tmp">>]).       
ok
269> tregex_srv:grep(<<"test 9405904.tmp acuu.tmpmulaor 10+ pid">>).
[[[{34,39,<<"+ pid">>}],[{17,25,<<"acuu.tmp">>}]]]
270> tregex_srv:store( [{ <<"test">>, fun(X) -> io:format("found: ~p~n", [X]) end},  <<"[0-9][0-9]">>]).
ok
271> tregex_srv:grep(<<"test 9405904.tmp acuu.tmpmulaor 10+ pid">>).                                    
found: [{0,4,<<"test">>}]
[[[{34,39,<<"+ pid">>}],[{17,25,<<"acuu.tmp">>}]],
 [[{0,4,<<"test">>}],[{5,7,<<"94">>}]]]
272> tregex_srv:store( [{ <<"SRC=[^ ]+">>, fun(X) -> 
 [{_,_,M}] = X, io:format("Source: ~p~n", [M]) 
end}]).                   
ok
273> tregex_srv:grep(<<"test 9405904.tmp acuu.tmpmulaor 10+ pid">>).                                      
found: [{0,4,<<"test">>}]
[[[{34,39,<<"+ pid">>}],[{17,25,<<"acuu.tmp">>}]],
 [[{0,4,<<"test">>}],[{5,7,<<"94">>}]],
 []]
274> tregex_srv:grep(<<"tst SRC=192.135.15.1 pid">>).               
Source: <<"SRC=192.135.15.1">>
[[[{19,24,<<"1 pid">>}]],
 [[{8,10,<<"19">>}]],
 [[{4,20,<<"SRC=192.135.15.1">>}]]]
275> tregex_srv:store( [{ <<"SRC=([^ ]+)">>, fun(X) -> 
 [{_,_,_}, {_,_,M}] = X, io:format("Source IP: ~p~n", [M]) 
 end}]).
ok
276> tregex_srv:grep(<<"tst SRC=192.135.15.1 pid">>).                                                                   
Source IP: <<"192.135.15.1">>
Source: <<"SRC=192.135.15.1">>
[[[{19,24,<<"1 pid">>}]],
 [[{8,10,<<"19">>}]],
 [[{4,20,<<"SRC=192.135.15.1">>}]],
 [[{4,20,<<"SRC=192.135.15.1">>},{8,20,<<"192.135.15.1">>}]]]

As you can see, you can associate Funs with regexp Matches. This means that you can bind action to regexp...
First we store (in fact add regexp to the existing regexp list) new tuples {RE, Fun}:

275> tregex_srv:store( [{ <<"SRC=([^ ]+)">>, fun(X) -> 
 [{_,_,_}, {_,_,M}] = X, io:format("Source IP: ~p~n", [M]) 
 end}]).
ok

Now the exec does call already registered funs, but call the new one since our regexp matches and you can see that the IP number is only printed, the "submatches" feature works as expected:

276> tregex_srv:grep(<<"tst SRC=192.135.15.1 pid">>).                                                                   
Source IP: <<"192.135.15.1">>
Source: <<"SRC=192.135.15.1">>
...

The gen_server state is the following:

-record(state, {
                requests,
                reindex,
                re = [],
                pids = []
                }).

Its init function is:

init(_Args) ->
        process_flag(trap_exit, true),
        {ok, #state{  
 re = ets:new(?MODULE, [set,private]), 
 requests = 0, 
 reindex = 1 }}.

Internally the module calls 'treregex:compile' to compile regexp and store the resulting #port into a list that is stored in the 'ets' table. Every call to 'tregex_srv:store' create a new entry in the ets table.

%% Storing RE and Funs
%% Creating simple fun when there's none provided...
%%
store([], Res, State) ->
        ets:insert(State#state.re, { State#state.reindex, Res});
store([ { Regexp, Fun } | List ], Res, State) ->
        {ok, Re } = treregex:compile(iolist_to_binary(Regexp), [extended]),
        store(List, [ { Re, Fun } | Res ], State);
store([ Regexp | List ], Res, State) ->
        {ok, Re } = treregex:compile(iolist_to_binary(Regexp), [extended]),
        store(List, [ { Re, fun(_) -> false end} | Res ], State).

The 'tregex_srv:grep' just uses 'ets:foldl' to compute results:

handle_call({grep, Line}, _Node, State) ->
        Requests = State#state.requests,
        Grep = fun({_Reindex, ReList}, Acc) ->
                        [ exec(ReList, Line, []) | Acc]
                end,
        {reply, ets:foldl(Grep, [], State#state.re), State#state{ requests = Requests + 1} }.

%% exec, using a List of {Re, Funs}
exec([], _Line, Acc) ->
        Acc;
exec([ { Re, Fun } | ReList ], Line, Acc) ->
        case treregex:exec(Re, Line) of
                {ok, Matches} ->
                        Fun(Matches),
                        exec(ReList, Line, [ Matches | Acc ]);

                {error, nomatch} ->
                        exec(ReList, Line, Acc)
        end;
exec([ _Any | ReList ], Line, Acc) ->
        exec(ReList, Line, Acc).

The code is still young, but seems to work.

The main purpose here, is to be able to massively process lines of logs. I want to be able to
spawn multiple process on multiples nodes that will be able to extract valuable content from
various lines. This is the first step forward :-)

I may cleanup the 'grep' fun since it will returns empty list whenever a regexp didn't match anything from the supplied line...

I'm really excited to think that I will be able to use the 'gen_server:multi_call' with this module :)

LibTre for Fast regular expression

While reading the almost famous article about regular expressions, I tried to use TRE.
Since TRE is posix Compliant, but unfortunately don't have any erlang driver I downloaded ''posregex'' and hack it a little to make it use TRE.

Now here's some experiment with it:

22> Init = fun() ->  erl_ddll:load_driver(code:priv_dir(treregex)++"/bin", "TRE_drv") end.
#Fun
23> Init().
ok
24> f(List), List = treutils:build([<<"test">>, <<"toto">>, <<"[a-z][0-9]$">>, <<"^[a-zA-Z][a-zA-Z0-9_]+">>]).
[{<<"test">>,#Port<0.84>},
{<<"toto">>,#Port<0.85>},
{<<"[a-z][0-9]$">>,#Port<0.86>},
{<<"^[a-zA-Z][a-zA-Z0-9_]+">>,#Port<0.87>}]

25> treutils:exec(List, <<"alkjlskdjflskjglksjflakgjlkfgjl;dkgjklsdjglkdsjfglksd
jlkgjsdlkfgjlsdk fg989t9sgdkgj lkyrdy sjd gyrdsl;gkj test asl;dksdf">>).
{<<"alkjlskdjflskjglksjflakgjlkfgjl;dkgjklsdjglkdsjfglksdjlkgjsdlkfgjlsdk 
fg989t9sgdkgj lkyrdy sjd gyrdsl;gkj test a"...>>,

[{ok,<<"^[a-zA-Z][a-zA-Z0-9_]+">>},{ok,<<"test">>}]}

26> treutils:exec(List, <<"10alkjlskdjflskjglksjflakgjlkfgjl;dkgjklsdjglkdsjfglksdjlkgjsdlkfgjlsdk 
fg989t9sgdkgj lkyrdy sjd gyrdsl;gkj test asl;dksdf">>).                   
{<<"10alkjlskdjflskjglksjflakgjlkfgjl;dkgjklsdjglkdsjfglksdjlkgjsdlkfgjlsdk 
fg989t9sgdkgj lkyrdy sjd gyrdsl;gkj test"...>>,

[{ok,<<"test">>}]}

I build a List of Port that are compiled regular expressions, then I iterate thru the list matching "Line".

Here's the TreUtils module (BTW, you can replace treregex with posregex if you want...)

-module(treutils).
-export([build/1, exec/2, destroy/1]).

build(List) ->
       Fun = fun(X) ->
               {ok, RE} = treregex:compile(X, [extended]),
               {X, RE}
       end,
       lists:map(Fun, List).

destroy(List) ->
       lists:foreach( fun({_Name, RE}) -> treregex:free(RE) end, List).

exec(List, Line) ->
       exec(List, Line, []).

exec([], Line, Acc) ->
       {Line, Acc};
exec([H|Rest], Line, Acc) ->
       {Name, RE} = H,
       case treregex:match(RE, Line) of
               ok ->
                       exec(Rest, Line, [ {ok, Name} | Acc ]);

               {error, nomatch} ->
                       exec(Rest, Line, Acc)
       end.

Other libraries are also available.

Unfortunately I'm unable to make this module rock solid since it segfault if I ever call the 'exec' method two times... I think that there's a problem in the TRV_drv.c (heavily copied from the RE_drv.c) in the 'RE_from_erlang' function. The regexec call may garbage some of its internal...

typedef struct _desc {
    ErlDrvPort     port;
    ErlDrvTermData dport;          /* the port identifier as ErlDrvTermData */
    regex_t re;
    regmatch_t pm[16];
    int compiled;
} Desc;

... snip ...

static void RE_from_erlang(ErlDrvData drv_data, char *buf, int len)
{
        int status;
        unsigned int op = get_int32(buf);
        unsigned int flags = get_int32(buf+4);
        Desc *d = (Desc*) drv_data;

        switch(op) {

... snip ...

                case EXEC: 
                        status = regexec(&d->re, buf+8, (size_t) 16, &d->pm[0], flags);
                        if (status != 0) {
                                driver_send_status(d, status);
                                return;
                        }
                        driver_send_pm(d);
                        break;

I hope I'll come back soon with good news, since this TRE library looks very very promising...
If anyone have any clue ;p, please comment !