I should have mentioned that my previous blog posting was using MySQL 4.0(4.0.23_Debian-3ubuntu2-log). It seems that in 5.0.2 they changed the precedence of the NOT operator to be lower than LIKE. From the manual:

The precedence shown for NOT is as of MySQL 5.0.2. For earlier versions, or from 5.0.2 on if the HIGH_NOT_PRECEDENCE SQL mode is enabled, the precedence of NOT is the same as that of the ! operator.

Using 5.0 (5.0.22-Debian_0ubuntu6.06.8-log), and a slightly smaller dataset, I get:

mysql> select count(*) from Table where blobid is null or not blobid like '%-%';
+----------+
| count(*) |
+----------+
|   199057 | 
+----------+
1 row in set (3.26 sec)

mysql> select count(*) from Table where blobid is null or blobid not like '%-%';
+----------+
| count(*) |
+----------+
|   199057 | 
+----------+
1 row in set (0.96 sec)

Jim Kingdon experimented with other databases and was unable to reproduce this problem. My test with PostgreSQL 8.3:

quux=> create table foo (blobid varchar(255));
CREATE TABLE
quux=> insert into foo (blobid) values 
   ('5cd1237469cc4b52ca094e215156c582-9ef460ac4134c600a4d2382c4b0acee7'), 
   (NULL), 
   ('d20cb4037f8f9ab1de5de264660f005c-2c34209dcfb39251cf7c16bb6754bbd2'), 
   ('845a8d06719d8bad521455a8dd47745c-095d9a0831433c92cd269e14e717b3a9'),
   ('9580ed23f34dd68d35da82f7b2a293d6-bf39df7509d977a1de767340536ebe80'), 
   ('06c9521472cdac02a2d4b2a18f8bec0f-0a8a28d3b63df54860055f1d1de92969'), 
   ('ed3cd0dd9b55f76db7544eeb64f3cfa0-80a6a3eb6d73c0a58f88b7c332866d5c'),
   (NULL),
   ('b339f6545651fbfa49fa500b7845c4ce-6defb5ffc188b8f72f1aa10bbd5c6bec'),
   ('642075963d6f69bb11c35a110dd07c2c8db54ac2d2accae7fa4a22db1d6caae9');
INSERT 0 10
quux=> select count(*) from foo 
   where blobid is null or blobid not like '%-%';
 count 
-------
     3
(1 row)

quux=> select count(*) from foo 
   where blobid is null or not blobid like '%-%';
 count 
-------
     3
(1 row)

quux=> select not blobid from foo limit 10;
ERROR:  argument of NOT must be type boolean, not type character varying

This appears to have been the case since at least 7.4

Problems like this is going to make the transition from MySQL 4.0 to 5.x all the more fun when we get around to doing it.

Dear lazyweb,

I'm possibly being stupid, but can someone explain the differences between these two queries?

mysql> select count(*) from Table 
   where blobid is null or not blobid like '%-%';
+----------+
| count(*) |
+----------+
| 15262487 |
+----------+
1 row in set (25 min 4.18 sec)

mysql> select count(*) from Table 
   where blobid is null or blobid not like '%-%';
+----------+
| count(*) |
+----------+
| 20044216 |
+----------+
1 row in set (24 min 54.06 sec)

For reference:

mysql> select count(*) from Table where blobid is null;
+----------+
| count(*) |
+----------+
| 15262127 |
+----------+
1 row in set (24 min 7.15 sec)

Update: It turns out that the former was doing (not blobid) like '%-%' which turns out to not do anything sensible:

mysql> select not blobid from Table limit 10;
+------------+
| not blobid |
+------------+
|          0 |
|       NULL |
|          1 |
|          0 |
|          0 |
|          0 |
|          1 |
|       NULL |
|          1 |
|          0 |
+------------+
10 rows in set (0.02 sec)

mysql> select blobid from Table limit 10;
+-------------------------------------------------------------------+
| blobid                                                            |
+-------------------------------------------------------------------+
| 5cd1237469cc4b52ca094e215156c582-9ef460ac4134c600a4d2382c4b0acee7 |
| NULL                                                              |
| d20cb4037f8f9ab1de5de264660f005c-2c34209dcfb39251cf7c16bb6754bbd2 |
| 845a8d06719d8bad521455a8dd47745c-095d9a0831433c92cd269e14e717b3a9 |
| 9580ed23f34dd68d35da82f7b2a293d6-bf39df7509d977a1de767340536ebe80 |
| 06c9521472cdac02a2d4b2a18f8bec0f-0a8a28d3b63df54860055f1d1de92969 |
| ed3cd0dd9b55f76db7544eeb64f3cfa0-80a6a3eb6d73c0a58f88b7c332866d5c |
| NULL                                                              |
| b339f6545651fbfa49fa500b7845c4ce-6defb5ffc188b8f72f1aa10bbd5c6bec |
| 642075963d6f69bb11c35a110dd07c2c-8db54ac2d2accae7fa4a22db1d6caae9 |
+-------------------------------------------------------------------+
10 rows in set (0.00 sec)

The documentation says Logical NOT. Evaluates to 1 if the operand is 0, to 0 if the operand is non-zero, and NOT NULL returns NULL. but doesn't describe the behaviour of NOT 'string'. It would appear that a string starting with a number returns 0 and a string starting with a letter returns 1. Either way, neither has a hyphen in.

A while ago I published an article on PostgreSQL user administration. Typically, things have changed since I wrote that article. I thought I'd detail a couple of the differences since I wrote that guide.

The major difference is that you now have roles rather than users and you use the CREATE ROLE command to create them instead of CREATE USER, although the latter command still works. The command line options for the createuser command have changed as a result too. Before superuser and the ability to create new users were the same thing. Now you can give a role permissions to create new roles without giving them superuser powers. The options are now -s for superuser and -S for not superuser, -d to allow them to create databases and -D to disallow database creation and -r to allow the new role to create other roles and -R to prevent them. for a standard user you probably want somethig like:

createuser -S -D -R -P user

The -P makes createuser ask you for a password for the new role.

You can find out more information about the new role system in PostgreSQL in the user management and CREATE ROLE reference sections of the manual.

I'm currently upgrading my blog to PyBlosxom 1.4.3. I apologise for any broken links or entry flooding.

Update: I've finished playing now. I've upgraded to 1.4.3 and I don't think I've broken anything yet.

I've also taken the opportunity to add a couple of plugins to add tagging to entries and added the obligatory tag cloud to the side bar rather than the list of months. I'm going to make some changes to the comment plugin later to add OpenID support. I'd be interested to know of any other pyBlosxom plugins you find useful.

I did manage to make a mistake by using vim to edit entries to add some tags rather than my wrapper script to keep timestamps the same. This is where I'm glad I have a database table with the metadata from all my entries to hand. A quick touch foo.txt -d 2006-06-07 19:02:57+01 later and everything was fixed. Hopefully not too many people got bitten by the few entries that had new dates for a few minutes. Please let me know if you notice anything broken.

Perl doesn't enforce access to modules' namespaces. This would usually be considered a bad thing, but sometimes it allows us to work around problems in modules without changing their code. Here's a perfect example:

I've been writing a script to talk to an XML-RPC endpoint, using Frontier::Client but for one of the requests, the script throws the following error:

wanted a data type, got `ex:i8'

Turning on debugging showed the response type was indeed ex:i8, which isn't one of the types that Frontier::Client supports.

<?xml version="1.0" encoding="UTF-8"?>
<methodResponse xmlns:ex="http://ws.apache.org/xmlrpc/namespaces/extensions">
  <params>
    <param>
      <value>
        <ex:i8>161</ex:i8>
      </value>
    </param>
  </params>
</methodResponse>

Searching through the code shows Frontier::Client is a wrapper around Frontier::RPC2 and the error message happens at the following section:

   } elsif ($scalars{$tag}) {
       $expat->{'rpc_text'} = "";
       push @{ $expat->{'rpc_state'} }, 'cdata';
   } else {
       Frontier::RPC2::die($expat, "wanted a data type, got \`$tag'\n");
   }

So we can see that it's looking up the tag into a hash called %scalars to see if the type is a scalar type, otherwise throws the error we saw. Looking at the top, we can see this hash:

%scalars = (
    'base64' => 1,
    'boolean' => 1,
    'dateTime.iso8601' => 1,
    'double' => 1,
    'int' => 1,
    'i4' => 1,
    'string' => 1,
);

So, if we could add ex:i8 to this scalar, we could fix the problem. We could fix the module, but that would require every user of the script to patch their copy of the module. The alternative is to inject something into that hash across module boundaries, which we can do by just refering to the hash by it's complete name including the package name. We can use:

$Frontier::RPC2::scalars{'ex:i8'} = 1;

Now when we run the script, everything works. It's not nice and it's dependent on Frontier::RPC2 not changing. but it allows us to get on with our script.

I just emailed my MP the following letter:

Dear David Lepper,

I would just like to thank you for signing Auston Mitchell's Early Day Motion 1155 Photography In Public Areas. I have been increasingly concerned with reports of police action against innocent photographers, including most recently a man assaulted by several security guards in Stoke (http://www.flickr.com/photos/happyaslarry/2420960125/). I'm sure you appreciate Brighton's reputation as an artistic city and your support for this motion shows your continued support for the photography community in Brighton.

Yours sincerely, David Pashley

If your MP hasn't signed this EDM, I recommend you contact them to urge them to sign it and if they have, contact them again to thank them.

Archive::Tar is a useful library for working with tar archives from Perl. Unfortunately, one thing it doesn't allow is using data from memory as the archive. From the TODO section:

Allow archives to be passed in as string

Currently, we only allow opened filehandles or filenames, but not strings. The internals would need some reworking to facilitate stringified archives.

Fortunately, it does allow you to use a filehandle. I've previously mentioned about how useful the IO::Handle subsystem in perl is. And we should be able to use it in this case. The module we'll want is IO::String, which is a IO::Handle over a perl scalar. We can use it:

my $tar = new Archive::Tar(new IO::String($data));

Unfortunately when we run this now we get:

Cannot read compressed format in tar-mode at Foo.pm line 41
No data could be read from file at Foo.pm line 41

It turns out that this is because Archive::Tar uses IO::Zlib internally if the file isn't uncompressed, but this doesn't provide the ability to uncompress from a filehandle. The answer is to uncompress the data before passing it to Archive::Tar and the easiest way to do this is to use the IO::Uncompress::Gunzip module, so we can rewrite our code to:

my $tar = new Archive::Tar(new IO::Uncompress::Gunzip(new IO::String($data)));

Now when you run the script, Archive::Tar has an uncompressed tar stream. Yet another situation where IO::Handles comes to the rescue.

Because I always forget when I need to create a new class in perl:

package Foo::Bar;

use strict;
use warnings;

sub new {
   my $this = shift;
   my $class = ref($this) || $this;
   my $self = {};
   bless $self, $class;
   $self->initialize(@_);
   return $self;
}

sub initialize {
   my $self = shift;
}

1;

If you have any useful additions I'd love to know.

Regular viewers will know that I don't think favourably of MySQL. Here is yet another reason. Let's create an InnoDB table:

mysql> CREATE TABLE `User_` (
mysql> ...
mysql> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Query OK, 0 rows affected, 1 warning (0.04 sec) 

One warning, but we're running this as part of an import, so we'll fail to spot this and even if we did, we wouldn't be able to get it back out of mysql because SHOW WARNINGS only shows the last command. So let's look at the table we just created:

mysql> show create table User_\G
*************************** 1. row ***************************
       Table: User_
Create Table: CREATE TABLE `User_` (
...
) ENGINE=MyISAM DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

Eh? what's going on? We asked for InnoDB, but have got a MyISAM table. Lets look at the engines available.

mysql> show engines;
+------------+----------+----------------------------------------------------------------+
| Engine     | Support  | Comment                                                        |
+------------+----------+----------------------------------------------------------------+
| MyISAM     | DEFAULT  | Default engine as of MySQL 3.23 with great performance         | 
| MEMORY     | YES      | Hash based, stored in memory, useful for temporary tables      | 
| InnoDB     | DISABLED | Supports transactions, row-level locking, and foreign keys     | 
| BerkeleyDB | NO       | Supports transactions and page-level locking                   | 
| BLACKHOLE  | NO       | /dev/null storage engine (anything you write to it disappears) | 
| EXAMPLE    | NO       | Example storage engine                                         | 
| ARCHIVE    | YES      | Archive storage engine                                         | 
| CSV        | YES      | CSV storage engine                                             | 
| ndbcluster | DISABLED | Clustered, fault-tolerant, memory-based tables                 | 
| FEDERATED  | YES      | Federated MySQL storage engine                                 | 
| MRG_MYISAM | YES      | Collection of identical MyISAM tables                          | 
| ISAM       | NO       | Obsolete storage engine                                        | 
+------------+----------+----------------------------------------------------------------+
12 rows in set (0.00 sec)

Oh, so innodb has been disabled. We can fix that easily by removing skip-innodb from my.cnf.

root@cmsdb01:/var/log# grep skip-innodb /etc/mysql/my.cnf
root@cmsdb01:/var/log#

But hang on a second, that's not in the config file. What's going on? It turns out that the reason InnoDB is disabled is because of the innodb_log_file_size setting not matching the files on disk.

root@cmsdb01:/var/log# grep innodb_log_file_size /etc/mysql/my.cnf
innodb_log_file_size            = 512M
root@cmsdb01:/var/log# ls -lh /var/lib/mysql/ib_logfile*
-rw-rw---- 1 mysql mysql 5.0M 2006-12-19 18:39 /var/lib/mysql/ib_logfile0
-rw-rw---- 1 mysql mysql 5.0M 2006-12-19 18:39 /var/lib/mysql/ib_logfile1

Rumour has it that you can just stop MySQL, delete these log files and start MySQL again. I'm yet to try this as the server in question is in production use. The alternative is to change the innodb_log_file_size setting to match the file.

So in summary the problems with MySQL are:

• Not logging warnings anywhere useful.
• Converting engine types with a warning rather than throwing an error. This can be fixed by setting sql_mode to include NO_ENGINE_SUBSTITUTION.
• Starting up and disabling InnoDB when there is a problem rather than failing to start, giving a false impression that everything is working.

Unfortunately I live in the UK, where 6 months of the year, the time is GMT. Now is the time of year when I discover which of my servers don't have the right timezone configuration and show the wrong time during daylight saving. For future reference, here's how to set the timezone to Europe/London rather than UTC.

root@cms01:/tmp/openssl-0.9.8g# date
Mon Mar 31 08:23:35 GMT 2008
root@cms01:/tmp/openssl-0.9.8g# tzconfig
Your current time zone is set to GMT
Do you want to change that? [n]: y

Please enter the number of the geographic area in which you live:


   1) Africa         7) Australia

   2) America        8) Europe

   3) US time zones     9) Indian Ocean

   4) Canada time zones    10) Pacific Ocean

   5) Asia           11) Use System V style time zones

   6) Atlantic Ocean    12) None of the above


Then you will be shown a list of cities which represent the time zone
in which they are located. You should choose a city in your time zone.

Number: 8

Amsterdam Andorra Athens Belfast Belgrade Berlin Bratislava Brussels
Bucharest Budapest Chisinau Copenhagen Dublin Gibraltar Guernsey Helsinki
Isle_of_Man Istanbul Jersey Kaliningrad Kiev Lisbon Ljubljana London
Luxembourg Madrid Malta Mariehamn Minsk Monaco Moscow Nicosia Oslo Paris
Podgorica Prague Riga Rome Samara San_Marino Sarajevo Simferopol Skopje
Sofia Stockholm Tallinn Tirane Tiraspol Uzhgorod Vaduz Vatican Vienna
Vilnius Volgograd Warsaw Zagreb Zaporozhye Zurich

Please enter the name of one of these cities or zones
You just need to type enough letters to resolve ambiguities
Press Enter to view all of them again
Name: [] London
Your default time zone is set to 'Europe/London'.
Local time is now:      Mon Mar 31 09:23:48 BST 2008.
Universal Time is now:  Mon Mar 31 08:23:48 UTC 2008.

More information is available in the Debian System Administrator Manual.

Recently, we rolled out a Shibboleth Single Sign On service to protect one of our services. However, we started recieving intermittant login failures, both on our automated monitoring and from customers. Curiously these failures tended to happen mostly in the evening, which isn't a peak time for us. Debugging showed that the authentication worked, but the authorisaton was failing. Shibboleth works as an apache module and daemom that protects a service, which communicates with a webservice that does the authenication processing. The log files were showing an occasional SSL error in this communcation link.

INFO shibtarget.SessionCache [43005] sessionGet: trying to get new attributes 
      for session (ID=_d0cd2f93840bb92050b28fa73d19ce4f)
INFO SAML.SAMLSOAPHTTPBinding [43005] sessionGet: sending SOAP message to 
      https://login.example.com/shibboleth/AA
ERROR SAML.SAMLSOAPHTTPBinding [43005] sessionGet: failed while contacting   
      SAML responder: error:1408F06B:SSL routines:SSL3_GET_RECORD:bad 
      decompression
ERROR shibtarget.SessionCache [43005] sessionGet: caught SAML exception 
      during SAML attribute query: SOAPHTTPBindingProvider::send() failed 
      while contacting SAML responder: error:1408F06B:SSL 
      routines:SSL3_GET_RECORD:bad decompression
ERROR shibtarget.SessionCache [43005] sessionGet: no response obtained

We didn't manage to find any suitable solutions on the internet, so we pulled out the trusty wireshark and started looking to see what was going on. We could see that the client was advertising deflate and null compression, and that the server was responding by asking for deflate compression. However the client would then claim that there was a decompression error in the servers response. This opened a few lines of enquiry. I made sure that both ends of the connection were running the same version of OpenSSL and they were both using 0.9.8a from Ubuntu Dapper. Interestingly 0.9.8a is the first version that had compression support. We found a couple of suggestions including forcing connections to be SSL2, which lacked compression or recompiling openssl without zlib support. As the former was easier, we tried that first by putting

SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA

in /etc/apache2/mods-enabled/ssl.conf as suggested by Debian bug #338008, and this seemed to work for around an hour. Packet sniffing showed that it was still negotiating SSL3 including deflate compression. Clearly we had to try something else.

Rather than recompile OpenSSL without zlib support, I thought I'd try upgrading the version of OpenSSL to something later in case that fixed the decompression bug. the version in Hardy is 0.9.8g, which sadly required recompiling and disabling the Ubuntu change to enable -Bsymbolic-functions during linking. Installing this on the client end didn't fix the problem, however installing it on the server end seemed to fix it. So far it's been running for 24 hours without an error, so fingers crossed that this has fixed it for good.

Installing the Sun Java packages on Debian or Ubuntu require to you accept Sun's license before you can install them. This means that it's not easy to install non-interactively, for example when using pbuilder. Fortunately the license uses debconf to check to see if you have already accepted the license. This means you can use debconf to accept the license before you install the packages. Create a file containing the following lines:

sun-java5-jdk shared/accepted-sun-dlj-v1-1 select true
sun-java5-jre shared/accepted-sun-dlj-v1-1 select true
sun-java6-jdk shared/accepted-sun-dlj-v1-1 select true
sun-java6-jre shared/accepted-sun-dlj-v1-1 select true

Then run /usr/bin/debconf-set-selections <file> and when you install the java packages, you should find it doesn't prompt for the license any more.

By default, warnquota sends out emails with the device name in the message, which probably doesn't make much sense to most non-technical users.

Hi,

We noticed that you are in violation with the quotasystem
used on this system. We have found the following violations:


/dev/mapper/Ubuntu-home

                        Block limits               File limits
Filesystem           used    soft    hard  grace    used  soft  hard  grace
/dev/mapper/Ubuntu-home
               +- 1044404 1000000 1200000  6days    1781     0     0

You can improve this by using /etc/quotatab to assign a more meaningful name to the partition:

/dev/mapper/Ubuntu-home:user directory
/dev/mapper/Ubuntu-shared:shared area

I've been running Hardy on my workstation for a while and had recently noticed that I was failing to type a space after "I". I was doing it far too much for it to just be me failing to press the space bar properly, and it wasn't happening after any other letter. After a little bit of experiementing, I discovered that something was eating shift-space. What was happening was that I was failing to release the shift key quick enough after typing "I" and before I hit the space bar, so it wasn't getting passed on.

Turns out that the problem was a recent update of Hardy installed SCIM, which uses shift-space as a keyboard shortcut. To turn it off, load the SCIM Setup program and go to the FrontEnd Global Setup screen and remove "Shift+Space" from the Trigger hotkey.

Caused confusion for a few minutes. :)

Subject: FELICITATION !!!!!  VOUS VENEZ DE GAGNER (ceci n'est pas un spam)

It's in French; of course it's a spam.

dpkg has a very useful feature where if you delete a conffile (pretty much everything under /etc and a few other files) it isn't replaced when you upgrade the package[0]. This behaviour was confusing me for a while until I realised what was happening. I was attempting to reinstall a package to get the default configuration files back that had been accidentally deleted, but no matter what I tried, the files didn't exist after running dpkg. Once I figured out that dpkg had this behaviour the solution was simple; use the --force-confmiss command line argument.

root@quux:~# dpkg --force-confmiss -i /tmp/foo_2.0.0-build.14_all.deb 
(Reading database ... 33418 files and directories currently installed.)
Preparing to replace foo 2.0.0-build.14 (using .../foo_2.0.0-build.14_all.deb) ...
Unpacking replacement foo ...
Setting up foo (2.0.0-build.14) ...

Configuration file `/etc/foo/foo.xml', does not exist on system.
Installing new config file as you request.
root@quux:~#

[0] If the file didn't exist in the previously installed version, it is installed, so you get new configuration files.

Phishers aren't even trying these days:

The following things stand out:

• The date header is +0900. Suspicion rating: 2/10
• The recorded log in time is in EST. The Halifax and myself are in GMT. Suspicion rating: 6/10
• The recorded log in time hadn't occured by the time I get the email. Suspicion rating: 8/10
• I don't bank with the Halifax. Suspicion rating: 10 million/10

If you've got various indenting and text wrapping options turned on in vim, pasting text into the editor results in screwed up results. You can get around this by turning on paste mode using :set paste and off with :set nopaste. To make things a little easier, you can use the following snippet in your .vimrc to allow you to toggle paste on and off using a single keypress:

nmap <F4> :set invpaste paste?<CR>
imap <F4> <C-O>:set invpaste<CR>
set pastetoggle=<F4>

(Warning: my vim settings have organically grown over the last 10 years, so they may not be the best or modern way of achieving an effect.)

If you're trying to import a dump file created using mysqldump and you get an error like:

ERROR 1005 (HY000): Can't create table './Database/Table.frm' (errno: 150)

Then you've just been bitten by mysqldump being far too stupid. The problem occurs because mysqldump includes foreign key constraints in the initial CREATE TABLE command, so if a table refers to a table that doesn't currently exist, it throws an error. mysqldump does correctly disable the contraints when inserting data into the tables. The correct way for this would be for mysqldump to create all the tables without the constraints, use ALTER TABLE to add the constraints to the tables, and then importing the data into the tables.

The workaround for this problem is to use:

SET FOREIGN_KEY_CHECKS = 0;
source dump.sql
SET FOREIGN_KEY_CHECKS = 1;

Update: Someone has pointed out that it appears that mysql 5 has fixed this problem by including the above statements in the dump.

Breaking news on BBC New24. Just confirmed in the last few minutes. Very few details.... Last night, Tony Blair converted to Catholism

Why is this news? Who cares? Why is the BBC treating this like it's the biggest news item of the year? Why have they rolled out Anne Widdecomme to do a phone interview? His wife is a catholic, his children are catholic, it's been on the cards for a while. He isn't in power any more. It remains to be seen if he has any relevance any more. So why does it matter what denomination he is.

Dear Lazyweb,

I'm possibly asking for the moon on a stick here, but in the office we have VoIP phones, which talk to our Asterisk server. Unfortunately, the ringtone on them are incredibly quiet and I tend to listen to music and don't notice either the ring or the small green flashing light when a call comes in.

The question then is does anyone know of a program which will talk SIP to the asterisk server and notice when a call comes in and turn my music down and display a notification?

In my article on Perl's IO::Handle objects I talked briefly about IO::AtomicFile and IO::Digest. I've just had reason to use these very useful modules to create a script which edits a file in place. These modules allowed me to do the rewrite atomically and optionally make a backup if the contents have changed. The example assumes you have a function called perform_rewrite that takes two file handles as the first two parameters.

use File::Copy;
use IO::File;
use IO::AtomicFile;
use IO::Digest;

sub rewrite_file {
   my $file = shift;
   my $sub = shift;
   my $input = new IO::File($file,'r');
   my $input_md5 = new IO::Digest($input, 'MD5');
   my $output = new IO::AtomicFile($file,'w');
   my $output_md5 = new IO::Digest($output, 'MD5');

   $sub->($input, $output, @_);

   if ($input_md5->hexdigest ne $output_md5->hexdigest) {
           copy ("$file", "$file.bak");
           $output->close();
   } else {
           # we haven't changed so don't bother updating
           $output->delete();
   }
   $input->close();
}

rewrite_file("/foo/bar", \&perform_rewrite, $baz, $quux);

Currently listening to Biffy Clyro's new album, Puzzle, and I have to say it is definitely their best album yet. It's up there with 65daysofstatic for best album of the year so far. It's made a change from all the recent albums where it's taken repeated listenings to like the album. With the first listen I love this album. I am definitely going to have to see them live more often. I've only seen them twice and the gig a fortnight ago was possibly one of my best gigs ever.

I'm looking to buy either a Samsung E900 or D900 and would like some horror stories on either phone from the LazyWeb. I'm tended towards the E900 as it's a smaller phone, but then the D900 has a better camera. I haven't really felt the need for a camera phone, but the quality is reaching the same as my digital camera, with the exception of not having an optical zoom.

Particularly interested in tales of slow or confusing UI and success stories of running an ssh client on it. Not really interested in recommendations of other phones at the moment. That might come later when I discover how bad these phones are.

Tim Bray just posted an entry about including a comment count in his atom feed. He said that he soon disabled it as people complained that it meant they saw the entry again as an updated entry. As I've recently done the same for my atom feed, I'm confused as to why this is happening. In my case I don't update either <published> or <updated>, just the <content> element. As the guid never changes, readers shouldn't consider it updated. Do readers really take a hash of the contents and consider it updated if it changes? Why do readers ignore the date fields. Is this affecting anyone? Certainly, it doesn't seem to affect Planet or the few readers I've tried it on.

After my last post on music players capable of Ogg Vorbis playback, I was recommended:

• Anything by Cowan (iAudio)
• iRiver t10
• iRiver iHP 120
• Samsung's YP-U2
• Kingston K-PEX

I can discount the iHP120 straight away as it's a HD based player and will be too large for what I want.

Update: I'm intentional avoiding any Apple products.

iRiver t10

iRiver T10

90GBP will get me a 2GB device. Several people have claimed it works as USB mass storage, although I've seen someone say American versions wouldn't work. iRiver have a firmware updater to convert it to UMS, so I might need to find a Windows installation. It has good battery life of around 45-50 hours from a single AA battery, which is good. It's got good reviews for its sound quality and its rugged construction might make it good in the gym.

Kingston K-PEX100

Kingston K-PEX100

A 2GB model will set me back around 65GBP. It's a touch larger, but thinner than the T10. Looks like a UMS device. It can be expanded with miniSD cards, which seems a nice feature. It only has a 17 hour battery time using the internal rechargable, which seems a bit low, considering my CD player will do 120 hours off two AA. Reviews have mentioned that the build quality is a little budget and a couple of firmware bugs, but hopefully these have been fixed now.

Samsung's YP-U2

Samsung YP-U2

Can only find a 1GB unit for 60GBP, although there is a 2GB unit available. Battery life is only 14 hours from the internal rechargable battery. It's smaller than the iRiver T10 and is a UMS device. I've seen reviews complain about the audio quality of this device. I think the lack of battery life and audio problems will rule this one out.

Cowon iAudio U3 and T2

I wasn't recommended any particular models, but the two I'm considering is the T2 and U3.

iAudio T2

The T2 is a necklace style player, which is rather unusual, but I can pick up a 2GB model for around 100GBP. Battery is around 12 hours using the internal rechargable battery. Looking at the reviews, the supplied headphones make up the actual necklace part and you have to remove that to use your own headphones, which is what I'll be doing, having just bought some Shure E2Cs. They also say the controls are a little confusing.

iAudio U3

This is the most expensive of the products I've looked at at 115GBP for the 2GB model. Battery life is around 20 hours. Reviews have been very positive. Not sure if it's worth the extra price though.

Update: Apparently the 2GB model can only be charged via USB, where as the 1GB model comes with a charger too, Also, I read in a review that it has doesn't do gapless playback, which is very annoying. Why don't players do gapless playback?

Summary

I think my top choice is the iRiver T10, although each of the others have their plus points. The K-PEX is expandable, the T2 is small and the U3 gets good reviews. I think I've discounted the Samsung. I will try and find each of the players in a local shop to have a play before I go out and buy one. If you have any comments on any of these players, or you have another product to recommend I'd be very interested.

Oh, and continuing my trend of discovering bands that have split up, I've been getting into At The Drive-In. :S

Firefox 2 is an improvement on previous versions, but one thing annoys me is the new tab style. I don't like having a close button on each tab and I don't like it hiding tabs after you have a certain number open. Fortunately you can fix this. Go to about:config in the URL and then set browser.tabs.closeButtons to 3 and browser.tabs.tabMinWidth to 0 and now you should have a close button on the right and all tabs displayed.

Content-Type: multipart/alternative;
        boundary="----=25532899_4522_4927_1140_664401643181"

This is a multi-part message in MIME format.
------=25532899_4522_4927_1140_664401643181
Content-Type: text/plain; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

This message is in MIME format. Since your mail reader does not
understand =
this format, some or all of this message may not be legible.
------=25532899_4522_4927_1140_664401643181
Content-Type: text/html; charset="iso-8859-1"
Content-Transfer-Encoding: quoted-printable

No, my mail reader understands it perfectly. It's your crappy mail client that sends out multipart/alternative mails which don't contain alternatives. Fuckers.

If you use Last.fm, you may have got around to using their events feature where you can search for events near you and mark yourself attending those events. I was manually entering the information into my google calendar, which was a pain.

Fortunately, being one of those hip, friendly web2.0 websites, Last.fm allows you to get your data out in all sorts of useful ways, including you event calendar as an iCal file. Google being useful, allow you to display external iCal files in your calendar. Can you see where I'm going with this? To get this up, log into you Google calendar, go to "Manage Calendars", then "Add Calendar", then "Public Calendar Address". Finally type in the url of your last.fm iCal file, which will be http://ws.audioscrobbler.com/1.0/user/dpash/events.ics, although obviously you need to change dpash to your particular last.fm username. Now, you only need to mark yourself attending a gig to have it show up in your Google calendar.

You can use something like this to get it into your Evolution calendar.

In the continuing saga I like to call "Dave, what's wrong with your ex-employer today?", I'm still missing large chunks of my pension. Finally had enough of my former employers not giving me any answers or claiming to be looking into it, as I've been asking since around last March. As per this guide, I've started making efforts to resolve the problem myself. The first step has been to formally request a copy of the pension scheme's dispute resolution procedure. I sent them an email last wednesday and hand delivered a letter last night, giving them until the 9th February to supply me with the procedure. If by then they haven't I'll contact the The Pension Advisory Service. I'm hoping it won't come to that, but given the lack of any progress in the past, I'm not holding out much hope.

Do you ever feel you should implement equals(), hashCode() and toString, but just can't be bothered to do it for every class? Well, if you aren't bothered by speed, you can use Jakarta Commons Lang to do it for you. Just add this to your class:

import org.apache.commons.lang.builder.ToStringBuilder;
import org.apache.commons.lang.builder.EqualsBuilder;
import org.apache.commons.lang.builder.HashCodeBuilder;

class Foo {
   public int hashCode() {
      return HashCodeBuilder.reflectionHashCode(this);
   }
   public boolean equals(Object other) {
      return EqualsBuilder.reflectionEquals(this,other);
   }
   public String toString() {
      return ToStringBuilder.reflectionToString(this);
   }
}

And that's it. Your class will just do the right thing. As you can probably guess from the function names, it uses reflection, so may be suboptimal. If you need performance, you can use tell it to use particular members, but I think I'll leave that up to a future article. I also recommend you don't use this technique if you are using something like Hibernate, which does things behind the scenes on member access; you may find it does undesirable things. :)

I noticed today that Mark Pilgrim linked to Eddie, my liberal RSS and Atom parsing library for Java, so I figured I should make a new release. It's been a few months since I did any serious work on the parser, but in the last few days I've reduced the number of test case failures to less than 100 out of 3502 test cases which come as part of Mark's Feedparser parser for python. The majority of the failures are in the date parsing routines and due to bugs in the Jython library which cause literal dictionaries not to match with classes inherited fro PyDictionary.

Improvements in this version include:

• Massively improved support for different character encodings. With Java 6, it also has support for UTF32 feeds.
• CDF Support.
• Optional support of TagSoup for sanitizing of HTML in entries.
• Improved support for different input sources including String, InputStream and byte[].
• Numerous bug fixes, with 97% of test cases passing, up from 90%

If you use Eddie, drop me an email. I'd like to thank Mark Pilgrim again for providing the community with a fantastic and comprehensive suite of test cases, extensive documentation and a first class Python library.

I was quietly minding my own business, fixing some encoding bugs in Eddie, my liberal RSS and Atom parser, when I noticed that Java 6 included support for UTF-32, which is one of the encoding tests that was failing. I downloaded and installed the Ubuntu packages and installed it, and decided to run a quick benchmark using my unit tests.

First up was the Sun Java 5 JVM. I'd been running the unit tests all night, but timed it this time,and got these results:

Ran 3502 tests
Passed 3322 tests
Failed 180 tests

real    1m10.293s
user    0m40.375s
sys     0m3.632s

Next I tried the Sun Java 6 JVM, using the same jar files and got;

Ran 3502 tests
Passed 3326 tests
Failed 176 tests

real    0m56.059s
user    0m39.198s
sys     0m4.212s

One thing to note was that it spend a couple of seconds noticing new jars to read, so I decided to run it again and got:

Ran 3502 tests
Passed 3326 tests
Failed 176 tests

real    0m45.317s
user    0m34.770s
sys     0m3.516s

Wow, I'd gone from 70 seconds to 45 seconds using the new runtime, and interestingly enough, past 4 more tests in the process. I'm assuming they are the UTF-32 tests, although I have't checked yet. The other thing for me to try is recompiling the code to see if that has any additional benefits.

Update: Got around to checking what Java 6 fixed and it turned out it was the additional support for koi-u and cspc862latinhebrew encodings. After I fixed the UTF32 support in Eddie, it passed an additional 16 tests. Down to just 160 out of 3502. I just wish they would add support for some of the stranger encodings. Maybe this will happen when it's open source.

Dear Lazyweb,

I'm after recommendations of a portable music player, which is small and can play ogg vorbis files. I'm not sure I need something with a large capacity; 2GB should be fine. Basically want something I can use in the gym, so I don't have to listen to the god-awful dance music they keep playing on MTV Dance. Reasonable audio quality a bonus. Last.fm support would be amazing. I would be tempted by something that could run Rockbox, but I suspect they are going to be on the top end of the size scale.

Spent some time checking upcoming gigs and I've settled on And You Will Know Us By The Trail Of The Dead, The Killers, 65daysofstatic, and The Barfly's Great Escape mini-festival, 3 days of gigs over 20 venues. Trying to decide if I want to go to see Inspiral Carpets and Electric Six in the world's dirtiest club.

MySQL cleverly maps

CREATE INDEX foo_bar ON Foo(Bar);

to

LOCK TABLE Foo WRITE;
CREATE TEMPORARY TABLE A-Foo ( .... INDEX foo_bar (Bar));
INSERT INTO A-Foo SELECT * FROM Foo;
ALTER TABLE Foo RENAME TO B-Foo;
ALTER TABLE A-Foo RENAME TO Foo;
DROP TABLE B-Foo;

If you have a very large table, expect this operation to take a) a lot of disk space, b) a very very long time and c) block any writes to the table in the process. I don't recommend adding indexes or altering any very large tables that are in production on MySQL, because you won't be in production for quite some time.

Update: Tom Haddon asked me if this applied to recent versions of MySQL or to PostgreSQL. Looking at the docs, it appears to still apply to 5.1:

• http://dev.mysql.com/doc/refman/5.1/en/create-index.html
• http://dev.mysql.com/doc/refman/5.1/en/alter-table.html

  In some cases, no temporary table is necessary:

  • If you use ALTER TABLE tbl_name RENAME TO new_tbl_name without any other options, MySQL simply renames any files that correspond to the table tbl_name. (You can also use the RENAME TABLE statement to rename tables. See Section 13.1.16, “RENAME TABLE Syntax”.)

  • ALTER TABLE ... ADD PARTITION creates no temporary table except for MySQL Cluster. ADD or DROP operations for RANGE or LIST partitions are immediate operations or nearly so. ADD or COALESCE operations for HASH or KEY partitions copy data between changed partitions; unless LINEAR HASH/KEY was used, this is much the same as creating a new table (although the operation is done partition by partition). REORGANIZE operations copy only changed partitions and do not touch unchanged ones.

  If other cases, MySQL creates a temporary table, even if the data wouldn't strictly need to be copied (such as when you change the name of a column).

• http://dev.mysql.com/doc/refman/5.1/en/alter-table-problems.html

As far as PostgreSQL is concerned, it doesn't mention anything about doing the same thing, but does mention that it does a full sequential scan of the table. During this time writes are blocked. You can use the CONCURRENTLY keyword to allow writes to happen, but it does two scans and will take longer, but you can still use your database.

http://www.postgresql.org/docs/8.2/interactive/sql-createindex.html

There is currently a petition on the Prime Minister's website calling for a clear ban on software patents. I was hesitant to sign it, not because I want software patents, but due to the langauge of the petition.

Software patents are used by convicted monopolists to threaten customers who consider using rival software. As a result, patents stifle innovation.

Patents are supposed to increase the rate of innovation by publicising how inventions work. Reading a software patent gives no useful information for creating or improving software. All patents are writen in a sufficiently cryptic language to prevent them from being of any use. Once decoded, the patents turn out to be for something so obvious that programmers find them laughable.

It is not funny because the cost of defending against nuicance lawsuites is huge.

The UK patent office grants software patents against the letter and the spirit of the law. They do this by pretending that there is a difference between software and 'computer implemented inventions'.

Some companies waste money on 'defensive patents'. These have no value against pure litigation companies and do not counter threats made directly to customers.

The aggressive and ad-hominem language doesn't do anything to help the cause. It looks unprofessional and will result in the authorities ignoring it as a fanatic incoherent rant and will put off people from signing the petition. I'd be interested to know how many people didn't sign because of the text.

It's been a while since I last wrote an article, but I've just published a new one on troubleshooting networking. It's based on a couple of emails I sent to a mailing list helping someone with their networking problem. I decided to clean it up a bit and publish it. It is mostly for finding where the problem lies, rather than fixing the issue, but I would be grateful for any comments you have, particularly if you think I've missed any obvious steps.

mojo-jojo david% ant clean
Buildfile: build.xml

clean:
     [echo] Deleting all project files
   [delete] Deleting directory /home/david/tomcat5.5-5.5.20/servletapi
   [delete] Deleting directory /home/david/tomcat5.5-5.5.20/container
   [delete] Deleting directory /home/david/tomcat5.5-5.5.20/jasper
   [delete] Deleting directory /home/david/tomcat5.5-5.5.20/connectors
   [delete] Deleting directory /home/david/tomcat5.5-5.5.20/build

BUILD SUCCESSFUL
Total time: 46 seconds
mojo-jojo david% ls
build.xml  debian/
mojo-jojo david% 

*blink*

I decided it was probably time that I started backing up my PostgreSQL databases and ended up writing a quick script to do just that. I did have a look at pg_dumpall, but it appears to only write to stdout, meaning I couldn't have a file per database like I wanted.

#!/bin/bash

set -u
set -e

dumpdir=/var/backups/postgresql 

date=$(date +"%Y-%m-%d")
mkdir -p $dumpdir
find $dumpdir -name "*.sql" -mtime +10 -exec rm {} \; 
for database in $(su - postgres -c "psql -l -t" | cut -d'|' -f 1 | grep -v template0); 
do 
        su - postgres -c "pg_dump $database" > $dumpdir/$database-$date.sql; 
done

su - postgres -c "pg_dumpall -g" > $dumpdir/global-$date.sql

Just stick that in /etc/cron.daily/postgresql-backups and you should end up with 10 days worth of backups. I'm not sure if postgreSQL can have spaces in database names. I'm assuming it can't. The only other issue to deal with is not storing it on the same machine as the database server, let alone on the same physical hardware or even worse, the same partition.

Update: Corrected typo and added dumping global objects to the script.

Sorry to anyone that got spammed by my change to pyblosxom earlier today. I changed the base url, which changed the link of the entries. I should sort out having proper guids.

It can be quite discouraging to type "yum update" and have yum simply go off forever. Among other things, one must wait a great long time to distinguish this behavior from yum's normal mode of operation. Other times, it comes back very quickly with a message saying, for all practical purposes, "RPM crashed, you lose, sorry."

via LWN.net (subscription required)

Jonathan Corbet normally manages to amuse me on a weekly basis, but this time he's outdone himself. Consider my LWN.net subcription renewed for another year.

Update: I;ve been convinced to include a subscriber link to the article for those without LWN.net subscriptions. http://lwn.net/SubscriberLink/210373/8badbe4f9c463fb8/