Monday, May 19, 2008

Monitoring log files with 'tail'

When you need to look for specific events from logfiles, your first idea is to use 'tail'. Tail is obviously the number one command that any sysadmin knows about.

From the first version of Tail and nowadays, some really nice feature have been implemented, one of those is the "follow=name" feature...

Since your erlang node will stay alive for many days, you'll end up meeting some logrotation tool that will replace the file you're lurking... So "follow=name" is for you !

Extract from a manual page:

There are two ways to specify how you'd like to track files with
this option, but that difference is noticeable only when a
followed file is removed or renamed. If you'd like to continue to
track the end of a growing file even after it has been unlinked,
use `--follow=descriptor'. This is the default behavior, but it
is not useful if you're tracking a log file that may be rotated
(removed or renamed, then reopened). In that case, use
`--follow=name' to track the named file by reopening it
periodically to see if it has been removed and recreated by some
other program.

Implementing this feature in pure erlang is of course possible, but why loose time when you can directly use the "tail" binary already installed on your system ?

-export([start/1, start/2, start/3, stop/1, snapshot/1, display/1, init/3]).

start(File) ->
start(File, fun display/1, "/var/log").

start(File, Callback) ->
Dir = "/var/log",
start(File, Callback, Dir).

start(File, Callback, Dir) ->
spawn_link(?MODULE, init, [File, Callback, Dir]).

snapshot(Pid) ->
Pid ! {snap, self() },
{Port, Callback} ->
{Port, erlang:fun_info(Callback)};
_Any ->

stop(Pid) ->
Pid ! stop.

init(File, Callback, Dir) ->
Cmd = "/usr/bin/tail --follow=name "++ File,
Port = open_port({spawn, Cmd}, [ {cd, Dir}, stderr_to_stdout, {line, 256}, exit_status, binary]),
tail_loop(Port, Callback).

tail_loop(Port, Callback) ->
{Port, {data, {eol, Bin}}} ->
tail_loop(Port, Callback);

{Port, {data, {noeol, Bin}}} ->
tail_loop(Port, Callback);

{Port, {data, Bin}} ->
tail_loop(Port, Callback);

{Port, {exit_status, Status}} ->
{ok, Status};
%tail_loop(Port, Callback);

{Port, eof} ->
{ok, eof};

{snap, Who} ->
Who ! { Port, Callback},
tail_loop(Port, Callback);

stop ->
{ok, stop};

_Any ->
tail_loop(Port, Callback)

display(Bin) ->
Content = iolist_to_binary(Bin),
io:format("[INFO] ~s~n", [Content]).

Let's say you want to monitor "/var/log/messages", here's how you can do it:

Shell> Tail = tail:start("messages").

This will display every new line (running in background) in your shell session.

Now let's say you want to do some tricky things with every line, you can pass as a parameter a callback fun:

Shell> Pid = logger_new(). % an example
Shell> Callback = fun(X) -> Pid ! {line, X} end. % sending a tuple to Pid
Shell> Tail = tail:start("message", Callback).

Finally, you'll be able to hack the code and transform this method to "tail" multiple files since "tail" is able to watch more than one file...

Quick tip :

init(ListOfFiles, Callback, Dir) ->
Args = [ [ X, $ ] || X <- ListOfFiles ]
Cmd = "/usr/bin/tail --follow=name "++ lists:flatten(Args),

Happy Tailing !

Wednesday, May 14, 2008

Following your directories

While designing some "integrity checker" tool, I've found myself in trouble whenever I need to
manage directories...

It seems that erlang, for now, is not able to detect "symbolic links" (Unix definition). Opening "kernel/include/file.hrl" holds the truth... The module "filelib" doesn't have any thing related to 'links' neither.

So if you have some links that point to "." you'll observe that "filelib:fold_files" follow the link many times (hopefully its stop somewhere) but you loose some precious time and increase disk accesses...

Then I've rewrote the fold_files to detect links by searching in the path some identical elements:

checktree([]) ->
checktree([_Elem,_Elem|_Rest]) ->
checktree([_|Rest]) ->

This is not really high quality but seems to work as expected...

I've also added some "maxdeep" functionality that prevent the script to go too many deeper.

I've named this module "wfile" for no particular reason :) and here's the full code:

-export([list/1, list/2]).

list(Dir) ->
Fun = fun(X, _Acc) -> io:format("+ ~s~n", [X]) end,
list(Dir, Fun).

list(Dir, Fun) ->
fold_files(Dir, Fun, []).

fold_files(Dir, Fun, Acc) ->
fold_files(Dir, true, Fun, Acc).

fold_files(Dir, Recursive, Fun, Acc) ->
fold_files1(Dir, Recursive, Fun, Acc).

fold_files1(Dir, Recursive, Fun, Acc) ->
case file:list_dir(Dir) of
{ok, Files} -> fold_files2(Files, Dir, Recursive, Fun, Acc);
{error, _} -> Acc

fold_files2([], _Dir, _Recursive, _Fun, Acc) ->
fold_files2([File|T], Dir, Recursive, Fun, Acc0) ->
FullName = filename:join(Dir, File),
case filelib:is_regular(FullName) of
true ->
Acc = Fun(FullName, Acc0),
fold_files2(T, Dir, Recursive, Fun, Acc);

false ->
case Recursive and filelib:is_dir(FullName) and maxdeep(FullName, 6) of
true ->
Acc1 = fold_files1(FullName, Recursive, Fun, Acc0),
fold_files2(T, Dir, Recursive, Fun, Acc1);
false ->
fold_files2(T, Dir, Recursive, Fun, Acc0)

maxdeep(Filename, Max) ->
Elems = filename:split(Filename),
( Max > length(Elems) ) and checktree(Elems).

checktree([]) ->
checktree([_Elem,_Elem|_Rest]) ->
checktree([_|Rest]) ->

To end to story, here's how I used this module:

erl> IC = integrity_checker:start().
erl> wfile:list("/home/rolphin/tmp", fun(X, Acc) -> IC ! {add, X}, io:format("added: ~s~n", [X]) end).

Thursday, May 8, 2008

EasyErl goes Mobile Friendly

Hi, I'm experiencing Mobile Tagging.
You can see it at the left side of this page.
Tags are generated dynamically, I'm using of course Erlang, Mochiweb, Ghostscript (rendering backend) and iec16022 to build the semacode.

Since I own a Nokia N95, I can efficiently scan my own codes, but I don't know if it's the case for everyone :)

So leave me a comment if you find those code hard to read...