Had an interesting problem today with one of our servers at work. First thing I noticed was yesterday that an upgrade of Apache2 didn't complete properly because /etc/init.d/apache2 stop didn't return. Killing it and starting apache allowed the upgrade to finish. I noticed there was a zombie process but didn't think too much off it.

Then this morning got an email from the MD saying that various internal services websites were down (webmail, wiki etc). My manager noticed that it was due to logrotate hanging, again because restarting apache had hung. Looking at the server I noticed a few more zombie processes. One thing I'd noticed was that all these processes had reparented themselves under init and a quick web search later confirmed that init(1) should be reaping these processes. I thought maybe restarting init would clear the zombies. I tried running telinit q to reread the config file, but that returned an error message about timing out on /dev/initctl named pipe. I checked that file existed and everything looked fine. The next thing I checked was the other end of the named pipe by running lsof -p 1. This showed that init had /dev/console rather than /dev/initctl as fd 0. I tried running kill -SIGHUP 1, but that didn't do anything. Then I tried kill -SIGUSR1 1, but that didn't do anything either. I checked the console, but there wasn't enough scrollback to see the system booting and decided to schedule a reboot for this evening.

Rebooting the server presented me with an interesting challenge. Normally the shutdown command signals to init to change to runlevel 0 or 6 to shutdown or reboot using /dev/initctl. Of course init wasn't listening on this file, so that was out. Sending it an SIGINT signal (the same signal init gets on ctrl-alt-delete) had no response. Obviously telinit 0 wasn't going to work either. I decided to start shutting services down manually with the help of Brett Parker. The idea was to stop all non-essential services, unexporting nfs exports, remounting disks read-only and then either using sysrq or a hardware reset. Unfortunately someone accidentally ran /etc/init.d/halt stop, hanging the server, but he is suffering from a bad cold today so I forgive him. The server restarted without a hitch (thank god for ext3) and running lsof -p 1 showed init having /dev/initctl open. I don't know what happened to init the last reboot on Monday, but a reboot seemed to fix it. Odd bug, but thankfully it was a nice simple fix. I could have spent the evening debugged init. :)

I'm currently mourning the loss of my laptop's hard disk. I don't think there was massive amounts of data on there that I needed, but it's still upsetting. Looks like I'll have to buy a new 2.5" drive. At least it gives me a reason to reinstall Debian.

Update: The bad news is that it is actually a 1.8" drive, which means I need to spend 93GBP including tax and postage for a new drive. Should get it by Friday. The good news is that having left the laptop off during the day, I managed to get the laptop booting and am not rsyncing the data off as fast as I can. Shame about the money, but at least I haven't lost much in the way of data.

I'd like to announce the initial release of Eddie, a feed parser library written in Java. It's taken me over 100 hours, but it now correctly parses 90% of the FeedParser unit tests, including all the rss and atom tests. It's GPLed, with an exception allowing you to use it in any open sourced program. Get it at my website. Need to add documentation and character set and encoding support. Also need to separate the testing infrastructure from the rest of the code.

This is the first time I've done any java programming in anger, and I have to say I'm surprised to discover I quite like it. In many ways it seems a very quick language to program in. It seems almost like programming in a scripting language, but stronger typed. This is probably due to not having to worry about memory management. Certainly I don't think I could have written this quite so quickly in C++.

Having said that, there are a couple things that I don't like about Java. Everything is a pointer. This is useful at times, but it means that every time you want to call a method on an object you have to test whether it is null or you run the risk of getting the dreaded NullPointerException. Java also doesn't have keywords for and, or and not. I know not everyone likes these, but I keep finding myself trying to use them.

I'm sure there are other things I hated, but I can't remember them now. I think I'll end up doing more java programming in the future.