Monday, October 1, 2007

High Order Functions, filtering lists...

I have a list of collected cpu and network values, from eth0 and eth1:
 L =
[{cpu,user,<<"3.05">>},
{cpu,nice,<<"0.00">>},
{cpu,system,<<"0.72">>},
{cpu,iowait,<<"0.03">>},
{cpu,steal,<<"0.00">>},
{cpu,idle,<<"96.20">>},
{eth0,rxpck,<<"2.52">>},
{eth0,txpck,<<"0.15">>},
{eth0,rxbyt,<<"173.80">>},
{eth0,txbyt,<<"44.68">>},
{eth0,rxcmp,<<"0.00">>},
{eth0,txcmp,<<"0.00">>},
{eth0,rxmcst,<<"1.25">>},
{eth1,rxpck,<<"0.00">>},
{eth1,txpck,<<"0.02">>},
{eth1,rxbyt,<<"0.00">>},
{eth1,txbyt,<<"1.00">>},
{eth1,rxcmp,<<"0.00">>},
{eth1,txcmp,<<"0.00">>},
{eth1,rxmcst,<<"0.00">>}]

If I want to manipulate such data set I'll need some filter functions that will help me to extract values, I need cpu values and eth0 values. High order function can do that for me !

First we need to select tuples, 'cpu' tuples:

Cpu = fun({X, Y, Z}) ->
if X == cpu ->
true;
true ->
false
end
end.


A better and far more erlangish method (thanks Zvi):

Cpu = fun({cpu,_,_}) -> true;
(_) -> false
end.


Okay this fun will return true whenever the first element of the tuple is 'cpu'.
But this fun is static, since 'cpu' is written in the function body. Let's make it dynamic:

Filter = fun(Motif) ->
fun({X, Y, Z}) ->
if X == Motif ->
true;
true ->
false
end
end
end.


The new High order function 'Filter' is generated by 'fun(Motif)' and takes as argument a tuple '{X, Y, Z}', this is a fun that return a fun...
This function can be used like this:

List = {cpu, test, dummy}. % a sample list
Cpu = Filter(cpu). % generate the Cpu fun
Cpu(List). %executing the Cpu fun
true.
List2 = {test, cpu, dummy}. %Other dummy list
Cpu(List2).
false.


Come back to our initial data set, and realize that we must iterate thru the list to extract possible values. Iterate and apply a fun to every element is what we need to do, futhermore we need to retrieve matching values... In fact we need to build the list of extracted values, and this can be accomplished by 'lists:foldl':

extractor(Motif) ->
fun(L) ->
lists:foldl(
fun({X, Y, Z}, List) ->
if X == Motif
-> [ {Y,Z} | List ];
true -> List
end
end,
[], L)
end.


In details, 'List' is the accumulator list, the one that will grow with valid tuple, the one we will return. The fun is the same as described before. To make things clear 'extractor/1' returns a fun that will parse a list of tuple extracting values that matches 'Motif'.

Another easier method, using only list comprehension:

extractor(Motif) ->
fun(L) when is_list(L) ->
[ {Y, Z} || {X, Y, Z} <- L, X == Motif]
end.

Usage:

38> Cpu = module:extractor(cpu).
#Fun<module.1.131615259>
39> Cpu(L).
[{idle,<<"96.20">>},
{steal,<<"0.00">>},
{iowait,<<"0.03">>},
{system,<<"0.72">>},
{nice,<<"0.00">>},
{user,<<"3.05">>}]
40> Eth0 = module:extractor(eth0).
41> Eth0(L).
[{rxmcst,<<"1.25">>},
{txcmp,<<"0.00">>},
{rxcmp,<<"0.00">>},
{txbyt,<<"44.68">>},
{rxbyt,<<"173.80">>},
{txpck,<<"0.15">>},
{rxpck,<<"2.52">>}]


I use such code for my monitoring project, I use the sar output (sysstat package), and I graph using rrdtool... This is a part of a bigger project that will compose a 'scheduler'.

6 comments:

Mamut said...

I guess you could rewrite your CPU function like this:

Cpu = fun({X, Y, Z}) -> X =:= cpu end.

rolphin said...

Correct, but that's not the purpose :p
Thank you.

Zvi said...

or even better:

Cpu = fun({cpu,_,_}) -> true;
(_) -> false
end.

Marc said...

or
extractor(Motif) ->
fun(L) when is_list(L) ->
[ {Y, Z} | {X, Y, Z} <- L, X == Motif]
end.

rolphin said...

Nice, I didn't realize that this was so easy :p

extractor2(Motif) ->
fun(L) when is_list(L) ->
[ {Y, Z} || {X, Y, Z} <- L, X == Motif]
end.

Marc said...

yes one should check one's code before posting... :D

Sticky