Angelo’s blog

I’ve been doing some analysis on some log files I got from a client. Since I have to propose an infrastructure that can support these web applications, I’d like to do some reconnaisance first, and see what the traffic is, and how many requests/sec are actually happening. Are we talking 10 req/s or 1000 req/s?

So, I wrote some scripts to graph out the requests/sec from an apache logfile, and I want to share this with you.

First of all, I want to merge the logfiles, so I get one big log file of all cluster nodes. For this, I use the logresolvemerge.pl script that comes with AWStats.

perl logresolvemerge.pl example.node1.log example.node2.log > example.log

Then basicly what I do, is I sort the log file (aparently apache log files are not really chronologic, probably because of the difference between the time a request is made, and the time a request is completed), then I run a script to get the number of requests per minute using this requests.rb script. I then create an rrd file, post all the information about the requests per minute into it, and then I export graphs

To sort the logfile, save this script as _sortlog.sh (you might want to use another location than ~/tmp as temporary sorting dir:

Here’s a complete script. Keep in mind I use FreeBSD, so on other *nix flavours you might have to change the script a bit (use expr instead of gexpr, etc). Requirements are the _sortlog.sh script and the requests.rb script mentioned above, and on FreeBSD the packages gexpr and rrdtool12 (or maybe rrdtool 1.3 wil word as well). The 4 timestamps mentioned in the script are for the exporting of the graphs. (My blog will replace straigt double quotes with the curly ones as well, you should replace them back):

You can run this script using the command:

$ ./go.sh example.log

And the output would look like:

Wed Mar 18 17:30:30 CET 2009 started script for log example.log..
Wed Mar 18 17:30:30 CET 2009 sorting log file example.log to example_sorted.log..
Sorting example.log
Wed Mar 18 17:54:23 CET 2009 calculating requests per minute in file example_requests.txt..
Thu Mar 19 03:22:42 CET 2009 making script example_createrrd.sh..
Thu Mar 19 03:22:42 CET 2009 deleting example.rrd..
rm: example.rrd: No such file or directory
Thu Mar 19 03:22:42 CET 2009 executing script example_createrrd.sh..
Thu Mar 19 03:22:42 CET 2009 making script example_updaterrd.sh..
rm: example_updaterrd.sh: No such file or directory
Thu Mar 19 03:23:10 CET 2009 executing script example_updaterrd.sh..
Thu Mar 19 03:23:27 CET 2009 making script example_graph_week.sh..
Thu Mar 19 03:23:27 CET 2009 executing script example_graph_week.sh..
595×199
Thu Mar 19 03:23:27 CET 2009 making script example_graph_day.sh..
Thu Mar 19 03:23:27 CET 2009 executing script example_graph_day.sh..
595×199
Thu Mar 19 03:23:27 CET 2009 done

Note the requests.rb file can take long, the above output was for an 18GB file. The graphs will perhaps look like this:

Sometimes you just have a single public IP address (unfortunately ipv6 is not that widespread yet), and you still want to publish stuff like Outlook Web Access and other applications to the net in a secure way. If you want that, the easiest way to do so, is to just pass port 443 to the Exchange server. But this means that if you have other web apps, you have to run them on the Exchange server as well. And besides, not everybody wants to put an IIS machine directly out on the net..

One way to solve that, is by putting a reverse proxy like Apache or Squid in front of it. Read more ..

Quote from Slashdot:

“The popular open source privacy tool, TrueCrypt, has just received a major update. The most exciting new feature provides the ability to encrypt an entire drive, prompting the user for a password during boot up; this makes TrueCrypt the perfect tool for non-technical laptop users (the kind who are likely to lose all of that sensitive customer data). The Linux version receives a GUI and independence from the kernel internals, and a Mac version is at last available too.”

My firewall/asterisk/mail scanner is a Compaq Deskpro EN/SFF, a very small Pentium III 500 box, with 256MB RAM and a 13GB ATA disk. I am very happy with it, since it scans inbound and outbound email all day using amavisd-new, spamassassin and ClamAV, and at the same time I can call through asterisk without a glitch. And it also runs OpenVPN server, DHCP server, BIND, NTP daemon, IPv6 tunnel, PF firewall with QoS, etc. And all that with only 30W power consumption! On auction sites, they go for around 25-30 EUR a piece. You don’t get a Cisco router with that flexibility and features for that price

The thing that annoyed me, was the fact that the disk made an annoying high-pitch sound. I could replace it with another drive, perhaps a 2.5″ laptop harddisk, to reduce the noise, but eventually, I decided to go for a CF-card. The CF thing was kind of new to me, but after reading for a while, it looks like the CF card with an adapter just shows up as an IDE disk, and that’s it, bootable and all.So I went out and bought a CF to IDE adapter and an 8GB Sandisk CompactFlash Extreme III card. The adapter is quite cool, you can mount it in a floppy bay, or you can mount it at the back of you PC between the PCI slots, so you can just eject the card from the back or the front. (Well, looking up the product I bought on John’s site reveals more similar adapters to me, I think afterwards this adapter would have been cheaper, even though I’m not sure if it would fit in the small form factor PC.)

I installed the card, and after some fiddling with cables and jumpers, FreeBSD saw it as my second disk. I then stopped all network services, created a new slice on it and created partitions/labels with sysinstall. I then did a dump/restore to copy the partitions from the old disk to the new disk. After that was done (took quite a while, played some CoD4 while waiting), I rebooted the machine using only the CF card, fixed the missing /tmp filesystem, and rebooted again, and that’s it, the machine was op and running again! The only way to tell it’s running is the light on the front of the machine, because it’s now dead silent

And the performance is not as bad as I thought. I used diskinfo to do some simple and naive tests. The old disk (12407MB <Maxtor 91301U3 FA570480> at ata0-master UDMA33) has seeks time of up to 22ms, and a tranfer rate of up to 24MB/s. The new disk (7815MB <SanDisk SDCFX3-8192 HDX 4.03> at ata1-master WDMA2) had seek times of under 0.5ms, and a steady transfer rate of 15MB/s. The CF card feels fast and snappy as well, probably because of the low seek times..

I was trying to get Apache2 to authenticate using the local user database, and I would expect it to be quite easy. I was wrong.

But thank god for blogs This user described a way to use pwauth and mod_athnz_external, that works like a charm for me:
http://blog.innerewut.de/2007/6/26/apache-2-2-authentication-with-mod_authnz_external

Thanks Jonathan!

FreeBSD RELENG_7 was born on 2007-10-11 04:28:08 UTC. I wonder when the official announcement will be made.

http://lists.freebsd.org/pipermail/cvs-src/2007-October/082249.html

Update: http://lists.freebsd.org/pipermail/freebsd-current/2007-October/078172.html

A message from Ken Smith, stating that this was done for development reasons, and in a few days, it’ll get beta status, and there will be a schedule for going stable. I hope they go stable soon!

Quote from original article: “The inventors of Sun Microsystems’ ZFS, Jeff Bonwick, distinguished engineer and storage CTO, and Bill Moore, hardware/software architect, tell me what was behind coming up with ZFS. It’s a file system. That sounds boring, right? But it’s not, since this technology is used in many of the world’s datacenters to keep files safely stored even when bad things happen. The conversation takes us all over the place, since these guys are experts on storage systems. Hard drives and all that. What does ZFS stand for? Zettabyte File System.”

I’ve got a MS ISA 2004 server, and some colleagues that connect to the office remotely, using the built-in windows PPTP client.

After using Microsoft’s PPTP server for about 6 years, I’ve totally had it. I’m tired and sick of problems with networking infrastructue (passing GRE through firewalls), the lack of easy error reporting functionality, etc. Every now and then connections don’t work, and rebooting a firewall here and there usually solves the problem.

After thinking about it for a while, I decided to ditch the MS PPTP server, and implement an OpenVPN server for my collegues on the road. I’ve been using OpenVPN for a while now as well, and I’m a big fan.. I’ve hooked up several data centres and remote locations using OpenVPN, and it has always been a rock solid solution. It can get a little complex at times, but it’s definitly worth the effort.

read more

At work, we have some workstations varying in age between 4 years and 4 months. But when is a workstation written off? We have the default policy of writing a PC off after 3 years, but what if the PC is still working like a charm?

 To have some extra numbers to take into consideration, I wanted to create a speed rating that would define the performance of a PC in a single number. I searched for quite some time to find a decent benchmark that would run on any PC (or server), and eventually, I cooked something up.

I downloaded the knoppix cd to boot from in text mode (type ‘knoppix 2′ at the boot screen) on each PC, download the script below, and run it.

It downloads and compiles ubench and seeker. Ubench is a tool that does some mathematical calculations, and benchmarks the CPU and memory. I then use hdparm to get the maximum sequential read in MB/s, then I run seeker to get the number of seeks per second, and the average access time in milliseconds. I run all these tests 3 times, to make sure one isn’t off.

I enter all data in an Excel sheet, and create a rating for the disk, by averaging the 3 disk numbers (using the PERCENTRANK function to place them on a scale from very bad to very good). I then create a average between the scaled CPU/Mem rating (60% weighed) and disk rating (40% weighed) to create a final ranking score on a scale from 0 to 100. Still have to find out the exact weights though, but the rating gives a global idea of the system’s performance.

# angelo@hongens.nl 2007/5/14
#SET VARIABLE TO CORRECT DISK!
DISK=/dev/sda

#download and compile ubench
cd ~
wget http://www.phystech.com/ftp/ubench-0.32.tar.gz
tar xzf ubench-0.32.tar.gz
cd ~/ubench-0.32
./configure
make

#download and compile seeker
wget -O ~/seeker.c http://www.linuxinsight.com/files/seeker.c.txt
gcc -O2 ~/seeker.c -o ~/seeker
rm ~/seeker.c

~/ubench-0.32/ubench
hdparm -t $DISK
~/seeker $DISK

~/ubench-0.32/ubench
hdparm -t $DISK
~/seeker $DISK

~/ubench-0.32/ubench
hdparm -t $DISK
~/seeker $DISK

Three Mac-ad spoofs from Novell:

Links to the Hi-Res Novell Videos

1. Meet Linux (.mpg) (.ogg)
2. New Duds (.mpg) (.ogg)
3. Running Linux (.mpg) (.ogg)