Wednesday, October 12, 2011

"Pseudo Randomly" retrieve data...

When you want to crawl some websites and you want to hide yourself a little, here's a simple trick to change randomly your user-agent string.
There's a very convenient function in erlang called 'uniform' from the module 'random', you can use it by calling 'random:uniform(MaxValue)' where MaxValue is the high limit.

So if you want to generate a random value from 1 to 5 you can simple use, 'random:uniform(5)'...

Now that you know how to generate random values, here's some code that do just what's the title say:

peek_rand(List) ->
 Rand = random:uniform(length(List)),
 lists:nth(Rand, List).

This 'peek_rand/1' fun returns the random'th element from the list List.

Now we can make a simple server loop that will send back a random string to whoever call him:
loop(List) ->
  {get, Who} ->
   Rand = random:uniform(length(List)),
   Who ! {ua, lists:nth(Rand, List), Rand},

  {add, Ua} ->
  stop ->
  _E -> 

We have also implemented a message that let you store new strings into the main list, look at the message '{add, Ua}' to understand...

To summup here, we are able to randomly peek a string from a list, and give it back to some other process...
Wouldn't be nice if we were able to store this list of strings ?
Of course yes !

So let's look at the 'io:format/3' fun. We know that the format string can contain the '~p' that basically express in a simple form any erlang term, and more precisely a List...

dump(List) ->
 {ok, Fd} = file:open("ua.list", [write]),
 io:format(Fd, "~p.~n", [List]),

And this is a convenient way to retrieve the data written in the 'ua.list':
load() ->
 {ok, [List|_]} = file:consult("ua.list"),

No comments: