Angelo’s blog

I’ve been doing some analysis on some log files I got from a client. Since I have to propose an infrastructure that can support these web applications, I’d like to do some reconnaisance first, and see what the traffic is, and how many requests/sec are actually happening. Are we talking 10 req/s or 1000 req/s?

So, I wrote some scripts to graph out the requests/sec from an apache logfile, and I want to share this with you.

First of all, I want to merge the logfiles, so I get one big log file of all cluster nodes. For this, I use the logresolvemerge.pl script that comes with AWStats.

perl logresolvemerge.pl example.node1.log example.node2.log > example.log

Then basicly what I do, is I sort the log file (aparently apache log files are not really chronologic, probably because of the difference between the time a request is made, and the time a request is completed), then I run a script to get the number of requests per minute using this requests.rb script. I then create an rrd file, post all the information about the requests per minute into it, and then I export graphs

To sort the logfile, save this script as _sortlog.sh (you might want to use another location than ~/tmp as temporary sorting dir:

Here’s a complete script. Keep in mind I use FreeBSD, so on other *nix flavours you might have to change the script a bit (use expr instead of gexpr, etc). Requirements are the _sortlog.sh script and the requests.rb script mentioned above, and on FreeBSD the packages gexpr and rrdtool12 (or maybe rrdtool 1.3 wil word as well). The 4 timestamps mentioned in the script are for the exporting of the graphs. (My blog will replace straigt double quotes with the curly ones as well, you should replace them back):

You can run this script using the command:

$ ./go.sh example.log

And the output would look like:

Wed Mar 18 17:30:30 CET 2009 started script for log example.log..
Wed Mar 18 17:30:30 CET 2009 sorting log file example.log to example_sorted.log..
Sorting example.log
Wed Mar 18 17:54:23 CET 2009 calculating requests per minute in file example_requests.txt..
Thu Mar 19 03:22:42 CET 2009 making script example_createrrd.sh..
Thu Mar 19 03:22:42 CET 2009 deleting example.rrd..
rm: example.rrd: No such file or directory
Thu Mar 19 03:22:42 CET 2009 executing script example_createrrd.sh..
Thu Mar 19 03:22:42 CET 2009 making script example_updaterrd.sh..
rm: example_updaterrd.sh: No such file or directory
Thu Mar 19 03:23:10 CET 2009 executing script example_updaterrd.sh..
Thu Mar 19 03:23:27 CET 2009 making script example_graph_week.sh..
Thu Mar 19 03:23:27 CET 2009 executing script example_graph_week.sh..
595×199
Thu Mar 19 03:23:27 CET 2009 making script example_graph_day.sh..
Thu Mar 19 03:23:27 CET 2009 executing script example_graph_day.sh..
595×199
Thu Mar 19 03:23:27 CET 2009 done

Note the requests.rb file can take long, the above output was for an 18GB file. The graphs will perhaps look like this:

If you are ever in need of a new background pattern, this is the place to be.. They have around a thousand of free background patterns, and with a single click you can change the page’s background pattern. Sweet, I found one for this blog as well

For the first time in a while I’m really, really angry again over something I saw.

What you see in the next video, is members of the European parliament, coming into the office in the morning to sign in to work, and then leave immediately. This way, they can get up to 14.000 EUR per month, by doing nothing..

They are clearly committing fraud, stealing MY money, and all the other tax payers’. And what makes me even more angry, is the way these people respond. They do not take responsibility like politicians should, they act like 3 year-olds caught with their hand in the cookie jar.

Irish MEP Kathy Sinnott threatens the reporter, and German MEP Hiltrud Breyer almost hits her head in a wall, because she wants to put her head in the sand so badly. Look at her, she’s a total disgrace to the EU. These people are our politicians! Where’s the pride, where’s the backbone?

And of course, even though the reporter, Herr Meier, apparently has the right to film in the hall, he’s thrown out nonetheless..

I really hope the MEP’s in this footage will be fired, arrested and charged with fraud, but of course they won’t. Most likely they’ll just get another warning, and nothing happens.

Here’s the video.

Finally, it’s on the MSDN! They first put the x64 version of Standard, Enterprise and Datacenter on the MSDN, and they will be publishing the other versions later.

Looks like MS really did a great job on server capacity,I downloaded this DVD at 4.2MB/s (yes, thats 33.6Mbps).

Because I’m so busy I don’t have time to upgrade machines here and there (and I do want to wait a while), but new projects will run on Windows Server 2008 and IIS7!

My ISP Tiscali (now together with the other dutch provider Telfort) have set up a new VoIP solution, called DigiTalk. They previously used SIPExpress. Now, my asterisk box would not register anymore. Well, that was partly my fault, because I put up records in my hosts file, staticly addressing their SIP Proxy. So now, when I called, I got a recording saying I should reboot my modem, bla bla.

After an evening of trial-and-error, and searching on the net, I have a working configuration again. My sip.conf: (my asterisk box is not behind NAT, and this is not my real phone number. My hosts file has no more related records anymore)

[general]
bindport=5060
bindaddr=0.0.0.0
srvlookup=yes
nat=no    

register => 0131234567:password:0131234567@tiscali    

[tiscali]
type=friend
insecure=very
context=tiscali-in
username=0131234567
secret=password
fromuser=0131234567
authuser=0131234567
fromdomain=tel.telefoniedienst.nl
qualify=no
host=tel.telefoniedienst.nl
dtmfmode = rfc2833
canreinvite=no
canrevisit=no
disallow=all
allow=alaw
allow=gsm
allowsubscribe=no

It’s here, and ready to download on the MSDN site for subscribers. I just downloaded and installed the professional edtion (can’t find a document that describes the differences betheen express, standard and professional though).

I don’t notice many differences except for some nicer graphics, but hey, I’m a sysadmin, not a developer. I’ll have to dive in deeper to see the nice new things I can do with it. I don’t even know how to use LINQ and such.. That will come later on, I still have a bachelor thesis to write, and code a lot of software along with that.

Geweldig, must see:

video

I just found out the great English dictionary me and my girlfriend often use (in book-form and on cd-rom), the ‘Longman Dictionary of Contemporary English’, is available online. And best of all, it’s free! The are some features that are only available from cd-rom, but still! Also great to find out the difference between American en British spelling.

http://www.ldoceonline.com

To my homepage. There’s not much to tell right now, but I hope that’ll graduatly change.