When you want to copy files from one machine to another, you might think about using scp to copy them. You might think about using rsync. If, however, you're trying to copy a large amount of data between two machines, here's a better, quicker, way to do it is using netcat.

On the receiving machine, run:

# cd /dest/dir && nc -l -p 12345 | tar -xf -

On the sending machine you can now run:

# cd /src/dr && tar -xf - | nc -q 0 remote-server 12345

You should find that everything works nicely, and a lot quicker. If bandwidth is more constrained than CPU, then you can add "z" or "j" to the tar options ("tar -xzf -" etc) to compress the data before it sends it over the network. If you're on gigabit, I wouldn't bother with the compression. If it dies, you'll have to start from the beginning, but then you might find you can get away with using rsync if you've copied enough. It's also worth pointing out that the recieving netcat will die as soon as the connection closes, so you'll need to restart it if you want to copy the data again using this method.

It's worth pointing out that this does not have the security that scp or rsync-over-ssh has, so make sure you trust the end points and everything in between if you don't want anyone else to see the data.

Why not use scp? because it's incredibly slow in comparison. God knows what scp is doing, but it doesn't copy data at wire speed. It aint the encyption and decryption, because that'd just use CPU and when I've done it it's hasn't been CPU bound. I can only assume that the scp process has a lot of handshaking and ssh protocol overhead.

Why not rsync? Rsync doesn't really buy you that much on the first copy. It's only the subsequent runs where rsync really shines. However, rsync requires the source to send a complete file list to the destination before it starts copying any data. If you've got a filesystem with a large number of files, that's an awfully large overhead, especially as the destination host has to hold it in memory.

Ever wanted to find out how much diskspace each table was taking in a database? Here's how:

database=# SELECT 
   tablename, 
   pg_size_pretty(pg_relation_size(tablename)) AS table_size, 
   pg_size_pretty(pg_total_relation_size(tablename)) AS total_table_size 
FROM 
   pg_tables 
WHERE 
   schemaname = 'public';
 tablename  | table_size | total_table_size 
------------+------------+------------------
 deferrals  | 205 MB     | 486 MB
 errors     | 58 MB      | 137 MB
 deliveries | 2646 MB    | 10096 MB
 queue      | 7464 kB    | 22 MB
 unknown    | 797 MB     | 2644 MB
 messages   | 1933 MB    | 6100 MB
 rejects    | 25 GB      | 75 GB
(7 rows)

Table size is the size for the current data. Total table size includes indexes and data that is too large to fix in the main table store (things like large BLOB fields). You can find more information in the PostgreSQL manual.

Edit: changed to use pg_size_pretty(), which I thought existed, but couldn't find in the docs. Brett Parker reminded me it did exist after all and I wasn't just imagining it.

One really nice feature of maven is the dependency resolution stuff that it does. The dependency plugin also has an analyse goal that can detect a number of problems with your dependencies. It can detect libraries you use but haven't declared in your POM, but work through transitive dependencies. This can cause build problems when you remove the library that was dragging in the undeclared dependency. It can also work out which dependencies you are no longer using, but have a declared dependency.

mojo-jojo david% mvn dependency:analyze
[INFO] Scanning for projects...
...
[INFO] [dependency:analyze]
[WARNING] Used undeclared dependencies found:
[WARNING]    commons-collections:commons-collections:jar:3.2:compile
[WARNING]    commons-validator:commons-validator:jar:1.3.1:compile
[WARNING]    org.apache.myfaces.core:myfaces-api:jar:1.2.6:compile
[WARNING] Unused declared dependencies found:
[WARNING]    javax.faces:jsf-api:jar:1.2_02:compile
...

How not to configure your DNS

david% dig -x 190.208.19.230

; <<>> DiG 9.4.2-P2 <<>> -x 190.208.19.230
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 35398
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;230.19.208.190.in-addr.arpa.   IN      PTR

;; ANSWER SECTION:
230.19.208.190.in-addr.arpa. 3600 IN    PTR     190.208.19.230.

;; Query time: 253 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Fri Apr 10 10:00:21 2009
;; MSG SIZE  rcvd: 73

Whoops

On off the biggest problems with developing servlets under a container like Tomcat is the amount of time taken to build your code, deploy it to the container and restart it to pick up any changes. Maven and the Jetty plugin allow you to cut down on this cycle considerably. The first step is to allow you to start your application in maven by running:

mvn jetty:run

We do this by configuring the jetty plugin inside our pom.xml:

<plugin>
   <groupId>org.mortbay.jetty</groupId>
   <artifactId>maven-jetty-plugin</artifactId>
   <version>6.1.10</version>
</plugin>

Now when you run mvn jetty:run your application will start up. But we can improve on this. The Jetty plugin can be configured to scan your project every so often and rebuild it and reload it if anything changes. We do this by changing our pom.xml to read:

<plugin>
   <groupId>org.mortbay.jetty</groupId>
   <artifactId>maven-jetty-plugin</artifactId>
   <version>6.1.10</version>
   <configuration>
      <scanIntervalSeconds>10</scanIntervalSeconds>
   </configuration>
</plugin>

Now when you save a file in your IDE, by the time you've switched to your web browser, Jetty is already running your updated code. Your development cycle is almost up to the same speed as Perl or PHP.

You can find more information at the plugin page.

After my entry yesterday about MySQL truncating data, several people have pointed out that MySQL 4.1 or later gives you a warning. Yes, this is true. You can even see it in the example I gave:

Query OK, 1 row affected, 1 warning (0.00 sec)

I ignored mentioning this, but perhaps should have said something about it. I reason I didn't mention it was because I didn't feel that a warning really helped anyone. Developers have enough problems remembering to check for errors, let along remembering to check in case there was a warning as well. Plus, they'd then have to work out if the warning was something serious or something they could ignore. There's also the question of how well the language bindings present this information. Take for example, PHP. The mysqli extension gained support for checking for warnings in PHP5 and gives the following code as an example of getting warnings:

mysqli_query($link, $query);

if (mysqli_warning_count($link)) {
   if ($result = mysqli_query($link, "SHOW WARNINGS")) {
      $row = mysqli_fetch_row($result);
      printf("%s (%d): %s\n", $row[0], $row[1], $row[2]);
      mysqli_free_result($result);
   }
}

Hardly concise code. As of 5.1.0, there is also mysqli_get_warnings(), but is undocumented beyond noting its existence. The MySQL extension does not support getting warning information. The PDO wrapper doesn't provide any way to get this information.

In perl, DBD::mysql has a mysql_warning_count() function, but presumably would have to call "SHOW WARNINGS" like in the PHP example. Seems Python's MYSQLdb module will raise an exception on warnings in certain cases. Mostly using the Cursor object.

In java, you can set the jdbcCompliantTruncation connection parameter to make the driver throw java.sql.DataTruncation exceptions, as per the JDBC spec, which makes you wonder why this isn't set by default. Unfortunately this setting is usually outside the programmer's control. There is also the java.sql.Statement.getWarnings(), but once again, you need to check this after every statement. Not sure if ORM tools like Hibernate check this or not.

So, yes MySQL does give you a warning, but in practice is useless.

MySQL in its standard configuration has this wonderful "feature" of truncating your data if it can't fit in the field.

mysql> create table foo (bar varchar(4));
Query OK, 0 rows affected (0.00 sec)

mysql> insert into foo (bar) values ("12345");
Query OK, 1 row affected, 1 warning (0.00 sec)

In comparison, PostgeSQL does:

psql=> create table foo (bar varchar(4));
CREATE TABLE
psql=> insert into foo (bar) values ('12345');
ERROR:  value too long for type character varying(4)

You can make MySQL do the right thing by setting the SQL Mode option to include STRICT_TRANS_TABLES or STRICT_ALL_TABLES. The difference is that the former will only enable it for transactional data storage engines. As much as I'm loathed to say it, I don't recommend using STRICT_ALL_TABLES, as an error during updating a non-transational table will result in a partial update, which is probably worse than a truncated field. Setting the mode to TRADITIONAL includes both these and a couple of related issues (NO_ZERO_IN_DATE, NO_ZERO_DATE, ERROR_FOR_DIVISION_BY_ZERO) You can set the mode using:

• On the command line:

  --sql-mode="TRADITIONAL"

• In /etc/mysql/my.cnf:

  sql-mode="TRADITIONAL"

• At runtime:

  SET GLOBAL sql_mode="TRADITIONAL"
  SET SESSION sql_mode="TRADITIONAL"

Just say no to databases that happily throw away your data

Was attempting to merge a branch in one of my projects and upon committing the merge, I kept getting this error:

mojo-jojo david% svn commit -m "merge in the maven branch"
Sending        trunk
Sending        trunk/.classpath
Sending        trunk/.project
Adding         trunk/.settings
svn: Commit failed (details follow):
svn: Server sent unexpected return value (502 Bad Gateway) in response
to COPY request for '/svn/eddie/!svn/bc/314/branches/maven/.settings'

A quick search found several other people having the same problem. Seems it only happens for https repositories using mod_dav_svn. The solution is to make sure that your virtual host in apache has explicit SSL config options, even if you are using an SSL config from a default virtual host. For example, I added the following to my subversion vhost, which was just copied from my default vhost:

SSLEngine on
SSLCertificateFile /etc/apache2/ssl/catnip.org.uk.crt
SSLCertificateKeyFile /etc/apache2/ssl/catnip.org.uk.key

I keep writing code to talk to databases in perl and I'm forever forgetting the correct runes for talking to databases, so I thought I'd stick it here for easy reference.

use DBI;

my $db_driver = "Pg" # Pg or mysql (or others)
my $db_name = "database";
my $db_host = "localhost";
my $db_user = "username";
my $db_pass = "password";


my $dbh = DBI->connect("dbi:$db_driver:dbname=$db_name;host=$db_host", 
   $db_user, $db_pass);

It's probably handy to give an example of a common database read operation

my $sth = $dbh->prepare( "SELECT * FROM table WHERE id  = ?") 
      or die $dbh->errstr;

$sth->execute($id) or die $dbh->errstr;

while (my $hashref = $sth->fetchrow_hashref) {
   print $hashref->{id};
}

I've just set up syntax highlighting for Puppet manifest files, and thought I'd share the simple steps. The first thing to do is download the syntax file from http://www.reductivelabs.com/downloads/puppet/puppet.vim and save this to ~/.vim/syntax/puppet.vim. Now when the filetype is set to "puppet", vim will use this syntax file.

That's useful, it it would be even nicer if we could make vim know that files ending in .pp were puppet files. Turns out this is very easy to do. You need to create a file to detect the correct filetype when you open a file. You need to put the following lines in ~/.vim/ftdetect/puppet.vim:

au BufRead,BufNewFile *.pp   setfiletype puppet

Now when you load a file ending in .pp, you should get nice syntax highlighting. You can also make vim use special settings for the puppet filetype by creating a vim script file in one of ~/.vim/ftplugin/puppet.vim, ~/.vim/ftplugin/puppet_*.vim and/or ~/.vim/ftplugin/puppet/*.vim. Vim has a lot of flexible hooks to enable file type specific configuration; hopefully it should be fairly easy to modify these examples for other file formats.

I've recently had to set up a new machine, but didn't have an install cdrom available, so I decided to use the easiest method for installing Ubuntu; PXE booting. Here's how I did it. PXE involves setting up two simple technologies, DHCP and TFTP. We start by setting up TFTP.

TFTP is Trivial File Transfer Protocol, a cut down version of FTP. There are a number of TFTP servers in Debian and Ubuntu, but not all of them support the extensions that the pxelinux bootloader used by debian-installer need. Experience has shown that tftpd-hpa works correctly, so we'll want to install that.

ace root% apt-get install tftpd-hpa

Note: If this installs an inetd at the same time, you may need to restart the inetd so it enables the tftpd service.

The tftpd will serve files out of /var/lib/tftpboot, so we need to add some files for it to serve. You can use this script to fetch various netboot installers from Ubuntu's servers.

#!/bin/bash

set -u
set -e

cd /var/lib/tftpboot

for dist in dapper feisty gutsy hardy intrepid; do
    mkdir -p $dist
    for arch in amd64 i386; do
        mkdir -p $dist/$arch/
        (cd $dist/$arch/ && ncftpget -RT \
           ftp://archive.ubuntu.com/ubuntu/dists/$dist/main/installer-$arch/current/images/netboot/)
    done
done

Download ubuntu-tftp-update.sh

Now we need to alter our dhcpd configuration. (You are using DHCP aren't you?) All we need to add is a group declaration to your subnet declaration, adding a next-server and a filename parameter. You can then add a host declaration for any machine you want to netboot into the installer.

group { # intrepid amd64
     next-server 10.0.0.1;
     filename "intrepid/amd64/pxelinux.0";
     host foobar { hardware ethernet 00:22:15:45:cc:fa; fixed-address foobar.example.com; }
}

You'll need to restart the dhcp server so it picks up the new setting. The next-server parameter is the name or IP address of your tftp server. filename is the path to the bootloader. Obviously, you can use this to pick which version of the installer you want to run. If you do a lot of installations, it might be worth configuring every installer you're likely to use and then move hosts in and out of the suitable group as and when you need to install them.

All that's left to do now is to boot the computer and set it to boot from the network and enjoy medialess installation.

Dear Lazyweb,

We have a web application that has quite a large database and reasonable usage. Back in the dim and distant past, we scaled the application by the age-old method of using several read-only slave databases to prevent reads on the master swamping writes. This worked well for several years, and then we introduced memcached into the mix to improve performance by reducing the number of reads from the database. This improved our database capacity even further.

Now the question has arisen about reducing or even removing the code to read from the slaves. I'm trying to come up with some compelling reasons to keep the application reading from the slaves. The pros and cons I currently have for removing the code are:

Pros

• Reduces code complexity
• Removes consistency problems due to latency in the replication. This is less of a problem than it used to be after we solved a problem with our replication

Cons

• Reduces our existing capacity
• Cache flushes would cause huge spikes on our master server until the cache filled up again
• Caches wouldn't help queries with unique critera

I would appreciate any additional reasons, pro or cons. We already have an existing non-live slave for backups and slow queries by developers. We would retain a slave for redundancy in the case of master failure. I'm only looking for issues that would affect the application.

We've recently installed a couple of Juniper J-Series routers that have the new JUNOS with Enhanced Services installed on them. During the transition from our existing Linux routers, we started moving internal subnets to the new routers, but when we moved the first subnet, we discovered a problem with hosts that had addresses on two different subnets. Connections would either connect for a minute and work and then get a connection reset, or packets would come in to the server and leave again, but then get swallowed in the ether.

I spent quite a bit of time this week reading about the security features of the new routers, and finally came up with a solution. The first clue was that I was getting connection reset from something on the network, but carrying out packet sniffing on our existing routers and the end points showed that they weren't generating it. I eventually found the tcp-rst option, which generates a reset packet for any non-SYN packet that doesn't match an existing flow session. JUNOS ES does stateful packet inspection by creating a session when it sees an initial SYN packet and then does filtering and routing based on that flow session so it doesn't have to do it for every packet. When I turned off the tcp-rst option on the trust zone, my connection that worked for a minute worked again only for a minute, but this time, it just hung, rather than dying with a connection reset. This cemented the idea that the Juniper routers were the cause.

It turned out that the problem was that there was asynchronous routing going on. A packet was coming in to 10.0.0.1/24, but the server was also on 10.0.1.1/24 and the default route nexthop was 10.0.1.2. Depending on which subnet we moved depended on the resulting behavour. If we moved 10.0.0.0/24 to the new routers, they would only see the incoming side of the conversation. If we moved 10.0.1.0/24, they would only see the outgoing side of the conversation. If we then think how this would work with the session-based routing and firewalling, in the first case, the router would see the initial SYN packet, but would never see the returning SYN-ACK packet, and after an initial timeout, decide the flow never established and destroy the session info, resulting in further incoming packets to be dropped. In the second case, it would never see the SYN packet, only the SYN-ACK. This packet wouldn't belong to an existing session, so would be blocked. The solution is to turn off the SYN check, using:

[edit]
user@host# set security flow tcp-session no-syn-check

After commiting that, sessions work correctly, even without the router seeing both sides of the connection:

user@host> show security flow session destination-prefix 10.0.0.1    
Session ID: 201341, Policy name: default-permit/4, Timeout: 1798
  In: 192.168.0.1/61136 --> 10.0.0.1/22;tcp, If: ge-0/0/0.7
  Out: 10.0.0.1/22 --> 192.168.0.1/61136;tcp, If: ge-0/0/1.7

1 sessions displayed

Sadly, I couldn't find much information on the no-syn-check option, so hopefully people will find this explaination useful.

I recently had a failed drive in my RAID1 array. I've just installed the replacement drive and thought I'd share the method.

Let's look at the current situation:

root@ace:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 sda3[1]
      483403776 blocks [2/1] [_U]
      
md0 : active raid1 sda1[1]
      96256 blocks [2/1] [_U]
      
unused devices: <none>

So we can see we have two mirrored arrays with one drive missing in both.

Let's see that we've recognised the second drive:

root@ace:~# dmesg | grep sd
[   21.465395] Driver 'sd' needs updating - please use bus_type methods
[   21.465486] sd 2:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB)
[   21.465496] sd 2:0:0:0: [sda] Write Protect is off
[   21.465498] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   21.465512] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   21.465562] sd 2:0:0:0: [sda] 976773168 512-byte hardware sectors (500108 MB)
[   21.465571] sd 2:0:0:0: [sda] Write Protect is off
[   21.465573] sd 2:0:0:0: [sda] Mode Sense: 00 3a 00 00
[   21.465587] sd 2:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   21.465590]  sda: sda1 sda2 sda3
[   21.487248] sd 2:0:0:0: [sda] Attached SCSI disk
[   21.487303] sd 2:0:1:0: [sdb] 976773168 512-byte hardware sectors (500108 MB)
[   21.487314] sd 2:0:1:0: [sdb] Write Protect is off
[   21.487317] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[   21.487331] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   21.487371] sd 2:0:1:0: [sdb] 976773168 512-byte hardware sectors (500108 MB)
[   21.487381] sd 2:0:1:0: [sdb] Write Protect is off
[   21.487382] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[   21.487403] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[   21.487407]  sdb: unknown partition table
[   21.502763] sd 2:0:1:0: [sdb] Attached SCSI disk
[   21.506690] sd 2:0:0:0: Attached scsi generic sg0 type 0
[   21.506711] sd 2:0:1:0: Attached scsi generic sg1 type 0
[   21.793835] md: bind<sda1>
[   21.858027] md: bind<sda3>

So, sda has three partitions, sda1, sda2 and sda3, and sdb has no partition table. Let's give it one the same as sda. The easiest way to do this is using sfdisk:

root@ace:~# sfdisk -d /dev/sda | sfdisk /dev/sdb
Checking that no-one is using this disk right now ...
OK

Disk /dev/sdb: 60801 cylinders, 255 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an MSDOS signature
 /dev/sdb: unrecognised partition table type
Old situation:
No partitions found
New situation:
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
/dev/sdb1   *        63    192779     192717  fd  Linux RAID autodetect
/dev/sdb2        192780   9960299    9767520  82  Linux swap / Solaris
/dev/sdb3       9960300 976768064  966807765  fd  Linux RAID autodetect
/dev/sdb4             0         -          0   0  Empty
Successfully wrote the new partition table

Re-reading the partition table ...

If you created or changed a DOS partition, /dev/foo7, say, then use dd(1)
to zero the first 512 bytes:  dd if=/dev/zero of=/dev/foo7 bs=512 count=1
(See fdisk(8).)

If we check dmesg now to check it's worked, we'll see:

root@ace:~# dmesg | grep sd
...
[  224.246102] sd 2:0:1:0: [sdb] 976773168 512-byte hardware sectors (500108 MB)
[  224.246322] sd 2:0:1:0: [sdb] Write Protect is off
[  224.246325] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[  224.246547] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  224.246686]  sdb: unknown partition table
[  227.326278] sd 2:0:1:0: [sdb] 976773168 512-byte hardware sectors (500108 MB)
[  227.326504] sd 2:0:1:0: [sdb] Write Protect is off
[  227.326507] sd 2:0:1:0: [sdb] Mode Sense: 00 3a 00 00
[  227.326703] sd 2:0:1:0: [sdb] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA
[  227.326708]  sdb: sdb1 sdb2 sdb3

So, now we have identical partition tables. The next thing to do is to add the new partitions to the array:

root@ace:~# mdadm /dev/md0 --add /dev/sdb1
mdadm: added /dev/sdb1
root@ace:~# mdadm /dev/md1 --add /dev/sdb3
mdadm: added /dev/sdb3

Everything looks good. Let's check dmesg:

[  323.941542] md: bind<sdb1>
[  324.038183] RAID1 conf printout:
[  324.038189]  --- wd:1 rd:2
[  324.038192]  disk 0, wo:1, o:1, dev:sdb1
[  324.038195]  disk 1, wo:0, o:1, dev:sda1
[  324.038300] md: recovery of RAID array md0
[  324.038303] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  324.038305] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[  324.038310] md: using 128k window, over a total of 96256 blocks.
[  325.417219] md: md0: recovery done.
[  325.453629] RAID1 conf printout:
[  325.453632]  --- wd:2 rd:2
[  325.453634]  disk 0, wo:0, o:1, dev:sdb1
[  325.453636]  disk 1, wo:0, o:1, dev:sda1
[  347.970105] md: bind<sdb3>
[  348.004566] RAID1 conf printout:
[  348.004571]  --- wd:1 rd:2
[  348.004573]  disk 0, wo:1, o:1, dev:sdb3
[  348.004574]  disk 1, wo:0, o:1, dev:sda3
[  348.004657] md: recovery of RAID array md1
[  348.004659] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  348.004660] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for recovery.
[  348.004664] md: using 128k window, over a total of 483403776 blocks.

Everything still looks good. Let's sit back and watch it rebuild using the wonderfully useful watch command:

root@ace:~# watch -n 1 cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 sdb3[2] sda3[1]
      483403776 blocks [2/1] [_U]
      [=====>...............]  recovery = 26.0% (126080960/483403776) finish=96.2min speed=61846K/sec
      
md0 : active raid1 sdb1[0] sda1[1]
      96256 blocks [2/2] [UU]
      
unused devices: <none>

The Ubuntu and Debian installers will allow you create RAID1 arrays with less drives than you actually have, so you can use this technique if you plan to add an additional drive after you've installed the system. Just tell it the eventual number of drives, but only select the available partitions during RAID setup. I used this method when a new machine recent didn't have enough SATA power cables and had to wait for an adaptor to be delivered.

(Why did no one tell me about watch until recently. I wonder how many more incredibly useful programs I've not discovered even after 10 years of using Linux)

If you get the following error:

/etc/ldap/slapd.conf: line 127: substr index of attribute "sambaSID" disallowed

when you run slapindex, then you haven't updated your samba.schema to the version from Samba 3.0.23. Dapper and Edgy had 3.0.22, so if you've recently upgraded to Hardy, you will see this problem. The file should have an MD5 of 0e23b3ad05cd2b38a302fe61c921f300. I'm hoping this resolves problems I have with samba not picking up group membership changes. I'll update if it does.

Update: Having installed the new schema and run slapindex, net rpc info shows I have twelve groups when previously it showed zero. This may not solve my group membership problems, but it can't be a step backwards.

Spamassassin 3.2, which is available in Gutsy and Lenny, comes with a new feature to increase performance by compiling its regular expressions using re2c. It's very quick to enable. First, you need to install the required packages:

apt-get install re2c libc6-dev gcc make

Next, edit /etc/spamassassin/v320.pre and uncomment the line that says:

loadplugin Mail::SpamAssassin::Plugin::Rule2XSBody

Next pre-compile the regular expressions using sa-compile:

femme:/etc/logcheck# sa-compile
[18741] info: generic: base extraction starting. this can take a while...
[18741] info: generic: extracting from rules of type body_0
100% [===========================] 3293.83 rules/sec 00m00s DONE
100% [===========================] 650.12 bases/sec 00m01s DONE
[18741] info: body_0: 647 base strings extracted in 2 seconds
[snip compiler output]
make install
Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /var/lib/spamassassin/compiled/3.002004/auto/Mail/SpamAssassin/CompiledRegexps/body_0/body_0.so
Installing /tmp/.spamassassin18741hDrlUQtmp/ignored/man/man3/Mail::SpamAssassin::CompiledRegexps::body_0.3pm
Writing /var/lib/spamassassin/compiled/3.002004/auto/Mail/SpamAssassin/CompiledRegexps/body_0/.packlist
Appending installation info to /var/lib/spamassassin/compiled/3.002004/perllocal.pod
cp /tmp/.spamassassin18741hDrlUQtmp/bases_body_0.pl /var/lib/spamassassin/compiled/3.002004/bases_body_0.pl
cd /
rm -rf /tmp/.spamassassin18741hDrlUQtmp

Finally, restart spamassassin, and you should find it runs faster. You will need to run sa-compile every time you update your rules, or they won't take effect.

If you get the following warning:

Can't locate Mail/SpamAssassin/CompiledRegexps/body_0.pm in @INC

you forgot to run sa-compile; re-run it and the error should go away.

Apache 2.2 changed the way you configure LDAP authentication. mod_auth_ldap was replaced with mod_authnz_ldap, so don't forget to enable the new module and disable the old one. Because I'll always forget, here's the new style config.

AuthType basic
AuthName "admin"
AuthBasicProvider ldap
AuthLDAPUrl ldap://ldap.example.com:389/ou=people,dc=example,dc=com?uid?sub
AuthLDAPGroupAttributeIsDN off
Require ldap-group cn=systems,ou=groups,dc=example,dc=com
AuthLDAPGroupAttribute memberUid

The sections in bold are the sections I had to change from the 2.0 config.

I should have mentioned that my previous blog posting was using MySQL 4.0(4.0.23_Debian-3ubuntu2-log). It seems that in 5.0.2 they changed the precedence of the NOT operator to be lower than LIKE. From the manual:

The precedence shown for NOT is as of MySQL 5.0.2. For earlier versions, or from 5.0.2 on if the HIGH_NOT_PRECEDENCE SQL mode is enabled, the precedence of NOT is the same as that of the ! operator.

Using 5.0 (5.0.22-Debian_0ubuntu6.06.8-log), and a slightly smaller dataset, I get:

mysql> select count(*) from Table where blobid is null or not blobid like '%-%';
+----------+
| count(*) |
+----------+
|   199057 | 
+----------+
1 row in set (3.26 sec)

mysql> select count(*) from Table where blobid is null or blobid not like '%-%';
+----------+
| count(*) |
+----------+
|   199057 | 
+----------+
1 row in set (0.96 sec)

Jim Kingdon experimented with other databases and was unable to reproduce this problem. My test with PostgreSQL 8.3:

quux=> create table foo (blobid varchar(255));
CREATE TABLE
quux=> insert into foo (blobid) values 
   ('5cd1237469cc4b52ca094e215156c582-9ef460ac4134c600a4d2382c4b0acee7'), 
   (NULL), 
   ('d20cb4037f8f9ab1de5de264660f005c-2c34209dcfb39251cf7c16bb6754bbd2'), 
   ('845a8d06719d8bad521455a8dd47745c-095d9a0831433c92cd269e14e717b3a9'),
   ('9580ed23f34dd68d35da82f7b2a293d6-bf39df7509d977a1de767340536ebe80'), 
   ('06c9521472cdac02a2d4b2a18f8bec0f-0a8a28d3b63df54860055f1d1de92969'), 
   ('ed3cd0dd9b55f76db7544eeb64f3cfa0-80a6a3eb6d73c0a58f88b7c332866d5c'),
   (NULL),
   ('b339f6545651fbfa49fa500b7845c4ce-6defb5ffc188b8f72f1aa10bbd5c6bec'),
   ('642075963d6f69bb11c35a110dd07c2c8db54ac2d2accae7fa4a22db1d6caae9');
INSERT 0 10
quux=> select count(*) from foo 
   where blobid is null or blobid not like '%-%';
 count 
-------
     3
(1 row)

quux=> select count(*) from foo 
   where blobid is null or not blobid like '%-%';
 count 
-------
     3
(1 row)

quux=> select not blobid from foo limit 10;
ERROR:  argument of NOT must be type boolean, not type character varying

This appears to have been the case since at least 7.4

Problems like this is going to make the transition from MySQL 4.0 to 5.x all the more fun when we get around to doing it.

Dear lazyweb,

I'm possibly being stupid, but can someone explain the differences between these two queries?

mysql> select count(*) from Table 
   where blobid is null or not blobid like '%-%';
+----------+
| count(*) |
+----------+
| 15262487 |
+----------+
1 row in set (25 min 4.18 sec)

mysql> select count(*) from Table 
   where blobid is null or blobid not like '%-%';
+----------+
| count(*) |
+----------+
| 20044216 |
+----------+
1 row in set (24 min 54.06 sec)

For reference:

mysql> select count(*) from Table where blobid is null;
+----------+
| count(*) |
+----------+
| 15262127 |
+----------+
1 row in set (24 min 7.15 sec)

Update: It turns out that the former was doing (not blobid) like '%-%' which turns out to not do anything sensible:

mysql> select not blobid from Table limit 10;
+------------+
| not blobid |
+------------+
|          0 |
|       NULL |
|          1 |
|          0 |
|          0 |
|          0 |
|          1 |
|       NULL |
|          1 |
|          0 |
+------------+
10 rows in set (0.02 sec)

mysql> select blobid from Table limit 10;
+-------------------------------------------------------------------+
| blobid                                                            |
+-------------------------------------------------------------------+
| 5cd1237469cc4b52ca094e215156c582-9ef460ac4134c600a4d2382c4b0acee7 |
| NULL                                                              |
| d20cb4037f8f9ab1de5de264660f005c-2c34209dcfb39251cf7c16bb6754bbd2 |
| 845a8d06719d8bad521455a8dd47745c-095d9a0831433c92cd269e14e717b3a9 |
| 9580ed23f34dd68d35da82f7b2a293d6-bf39df7509d977a1de767340536ebe80 |
| 06c9521472cdac02a2d4b2a18f8bec0f-0a8a28d3b63df54860055f1d1de92969 |
| ed3cd0dd9b55f76db7544eeb64f3cfa0-80a6a3eb6d73c0a58f88b7c332866d5c |
| NULL                                                              |
| b339f6545651fbfa49fa500b7845c4ce-6defb5ffc188b8f72f1aa10bbd5c6bec |
| 642075963d6f69bb11c35a110dd07c2c-8db54ac2d2accae7fa4a22db1d6caae9 |
+-------------------------------------------------------------------+
10 rows in set (0.00 sec)

The documentation says Logical NOT. Evaluates to 1 if the operand is 0, to 0 if the operand is non-zero, and NOT NULL returns NULL. but doesn't describe the behaviour of NOT 'string'. It would appear that a string starting with a number returns 0 and a string starting with a letter returns 1. Either way, neither has a hyphen in.

A while ago I published an article on PostgreSQL user administration. Typically, things have changed since I wrote that article. I thought I'd detail a couple of the differences since I wrote that guide.

The major difference is that you now have roles rather than users and you use the CREATE ROLE command to create them instead of CREATE USER, although the latter command still works. The command line options for the createuser command have changed as a result too. Before superuser and the ability to create new users were the same thing. Now you can give a role permissions to create new roles without giving them superuser powers. The options are now -s for superuser and -S for not superuser, -d to allow them to create databases and -D to disallow database creation and -r to allow the new role to create other roles and -R to prevent them. for a standard user you probably want somethig like:

createuser -S -D -R -P user

The -P makes createuser ask you for a password for the new role.

You can find out more information about the new role system in PostgreSQL in the user management and CREATE ROLE reference sections of the manual.

I'm currently upgrading my blog to PyBlosxom 1.4.3. I apologise for any broken links or entry flooding.

Update: I've finished playing now. I've upgraded to 1.4.3 and I don't think I've broken anything yet.

I've also taken the opportunity to add a couple of plugins to add tagging to entries and added the obligatory tag cloud to the side bar rather than the list of months. I'm going to make some changes to the comment plugin later to add OpenID support. I'd be interested to know of any other pyBlosxom plugins you find useful.

I did manage to make a mistake by using vim to edit entries to add some tags rather than my wrapper script to keep timestamps the same. This is where I'm glad I have a database table with the metadata from all my entries to hand. A quick touch foo.txt -d 2006-06-07 19:02:57+01 later and everything was fixed. Hopefully not too many people got bitten by the few entries that had new dates for a few minutes. Please let me know if you notice anything broken.

Perl doesn't enforce access to modules' namespaces. This would usually be considered a bad thing, but sometimes it allows us to work around problems in modules without changing their code. Here's a perfect example:

I've been writing a script to talk to an XML-RPC endpoint, using Frontier::Client but for one of the requests, the script throws the following error:

wanted a data type, got `ex:i8'

Turning on debugging showed the response type was indeed ex:i8, which isn't one of the types that Frontier::Client supports.

<?xml version="1.0" encoding="UTF-8"?>
<methodResponse xmlns:ex="http://ws.apache.org/xmlrpc/namespaces/extensions">
  <params>
    <param>
      <value>
        <ex:i8>161</ex:i8>
      </value>
    </param>
  </params>
</methodResponse>

Searching through the code shows Frontier::Client is a wrapper around Frontier::RPC2 and the error message happens at the following section:

   } elsif ($scalars{$tag}) {
       $expat->{'rpc_text'} = "";
       push @{ $expat->{'rpc_state'} }, 'cdata';
   } else {
       Frontier::RPC2::die($expat, "wanted a data type, got \`$tag'\n");
   }

So we can see that it's looking up the tag into a hash called %scalars to see if the type is a scalar type, otherwise throws the error we saw. Looking at the top, we can see this hash:

%scalars = (
    'base64' => 1,
    'boolean' => 1,
    'dateTime.iso8601' => 1,
    'double' => 1,
    'int' => 1,
    'i4' => 1,
    'string' => 1,
);

So, if we could add ex:i8 to this scalar, we could fix the problem. We could fix the module, but that would require every user of the script to patch their copy of the module. The alternative is to inject something into that hash across module boundaries, which we can do by just refering to the hash by it's complete name including the package name. We can use:

$Frontier::RPC2::scalars{'ex:i8'} = 1;

Now when we run the script, everything works. It's not nice and it's dependent on Frontier::RPC2 not changing. but it allows us to get on with our script.

I just emailed my MP the following letter:

Dear David Lepper,

I would just like to thank you for signing Auston Mitchell's Early Day Motion 1155 Photography In Public Areas. I have been increasingly concerned with reports of police action against innocent photographers, including most recently a man assaulted by several security guards in Stoke (http://www.flickr.com/photos/happyaslarry/2420960125/). I'm sure you appreciate Brighton's reputation as an artistic city and your support for this motion shows your continued support for the photography community in Brighton.

Yours sincerely, David Pashley

If your MP hasn't signed this EDM, I recommend you contact them to urge them to sign it and if they have, contact them again to thank them.

Archive::Tar is a useful library for working with tar archives from Perl. Unfortunately, one thing it doesn't allow is using data from memory as the archive. From the TODO section:

Allow archives to be passed in as string

Currently, we only allow opened filehandles or filenames, but not strings. The internals would need some reworking to facilitate stringified archives.

Fortunately, it does allow you to use a filehandle. I've previously mentioned about how useful the IO::Handle subsystem in perl is. And we should be able to use it in this case. The module we'll want is IO::String, which is a IO::Handle over a perl scalar. We can use it:

my $tar = new Archive::Tar(new IO::String($data));

Unfortunately when we run this now we get:

Cannot read compressed format in tar-mode at Foo.pm line 41
No data could be read from file at Foo.pm line 41

It turns out that this is because Archive::Tar uses IO::Zlib internally if the file isn't uncompressed, but this doesn't provide the ability to uncompress from a filehandle. The answer is to uncompress the data before passing it to Archive::Tar and the easiest way to do this is to use the IO::Uncompress::Gunzip module, so we can rewrite our code to:

my $tar = new Archive::Tar(new IO::Uncompress::Gunzip(new IO::String($data)));

Now when you run the script, Archive::Tar has an uncompressed tar stream. Yet another situation where IO::Handles comes to the rescue.

Because I always forget when I need to create a new class in perl:

package Foo::Bar;

use strict;
use warnings;

sub new {
   my $this = shift;
   my $class = ref($this) || $this;
   my $self = {};
   bless $self, $class;
   $self->initialize(@_);
   return $self;
}

sub initialize {
   my $self = shift;
}

1;

If you have any useful additions I'd love to know.

Regular viewers will know that I don't think favourably of MySQL. Here is yet another reason. Let's create an InnoDB table:

mysql> CREATE TABLE `User_` (
mysql> ...
mysql> ) ENGINE=InnoDB DEFAULT CHARSET=latin1;

Query OK, 0 rows affected, 1 warning (0.04 sec) 

One warning, but we're running this as part of an import, so we'll fail to spot this and even if we did, we wouldn't be able to get it back out of mysql because SHOW WARNINGS only shows the last command. So let's look at the table we just created:

mysql> show create table User_\G
*************************** 1. row ***************************
       Table: User_
Create Table: CREATE TABLE `User_` (
...
) ENGINE=MyISAM DEFAULT CHARSET=latin1
1 row in set (0.00 sec)

Eh? what's going on? We asked for InnoDB, but have got a MyISAM table. Lets look at the engines available.

mysql> show engines;
+------------+----------+----------------------------------------------------------------+
| Engine     | Support  | Comment                                                        |
+------------+----------+----------------------------------------------------------------+
| MyISAM     | DEFAULT  | Default engine as of MySQL 3.23 with great performance         | 
| MEMORY     | YES      | Hash based, stored in memory, useful for temporary tables      | 
| InnoDB     | DISABLED | Supports transactions, row-level locking, and foreign keys     | 
| BerkeleyDB | NO       | Supports transactions and page-level locking                   | 
| BLACKHOLE  | NO       | /dev/null storage engine (anything you write to it disappears) | 
| EXAMPLE    | NO       | Example storage engine                                         | 
| ARCHIVE    | YES      | Archive storage engine                                         | 
| CSV        | YES      | CSV storage engine                                             | 
| ndbcluster | DISABLED | Clustered, fault-tolerant, memory-based tables                 | 
| FEDERATED  | YES      | Federated MySQL storage engine                                 | 
| MRG_MYISAM | YES      | Collection of identical MyISAM tables                          | 
| ISAM       | NO       | Obsolete storage engine                                        | 
+------------+----------+----------------------------------------------------------------+
12 rows in set (0.00 sec)

Oh, so innodb has been disabled. We can fix that easily by removing skip-innodb from my.cnf.

root@cmsdb01:/var/log# grep skip-innodb /etc/mysql/my.cnf
root@cmsdb01:/var/log#

But hang on a second, that's not in the config file. What's going on? It turns out that the reason InnoDB is disabled is because of the innodb_log_file_size setting not matching the files on disk.

root@cmsdb01:/var/log# grep innodb_log_file_size /etc/mysql/my.cnf
innodb_log_file_size            = 512M
root@cmsdb01:/var/log# ls -lh /var/lib/mysql/ib_logfile*
-rw-rw---- 1 mysql mysql 5.0M 2006-12-19 18:39 /var/lib/mysql/ib_logfile0
-rw-rw---- 1 mysql mysql 5.0M 2006-12-19 18:39 /var/lib/mysql/ib_logfile1

Rumour has it that you can just stop MySQL, delete these log files and start MySQL again. I'm yet to try this as the server in question is in production use. The alternative is to change the innodb_log_file_size setting to match the file.

So in summary the problems with MySQL are:

• Not logging warnings anywhere useful.
• Converting engine types with a warning rather than throwing an error. This can be fixed by setting sql_mode to include NO_ENGINE_SUBSTITUTION.
• Starting up and disabling InnoDB when there is a problem rather than failing to start, giving a false impression that everything is working.

Unfortunately I live in the UK, where 6 months of the year, the time is GMT. Now is the time of year when I discover which of my servers don't have the right timezone configuration and show the wrong time during daylight saving. For future reference, here's how to set the timezone to Europe/London rather than UTC.

root@cms01:/tmp/openssl-0.9.8g# date
Mon Mar 31 08:23:35 GMT 2008
root@cms01:/tmp/openssl-0.9.8g# tzconfig
Your current time zone is set to GMT
Do you want to change that? [n]: y

Please enter the number of the geographic area in which you live:


   1) Africa         7) Australia

   2) America        8) Europe

   3) US time zones     9) Indian Ocean

   4) Canada time zones    10) Pacific Ocean

   5) Asia           11) Use System V style time zones

   6) Atlantic Ocean    12) None of the above


Then you will be shown a list of cities which represent the time zone
in which they are located. You should choose a city in your time zone.

Number: 8

Amsterdam Andorra Athens Belfast Belgrade Berlin Bratislava Brussels
Bucharest Budapest Chisinau Copenhagen Dublin Gibraltar Guernsey Helsinki
Isle_of_Man Istanbul Jersey Kaliningrad Kiev Lisbon Ljubljana London
Luxembourg Madrid Malta Mariehamn Minsk Monaco Moscow Nicosia Oslo Paris
Podgorica Prague Riga Rome Samara San_Marino Sarajevo Simferopol Skopje
Sofia Stockholm Tallinn Tirane Tiraspol Uzhgorod Vaduz Vatican Vienna
Vilnius Volgograd Warsaw Zagreb Zaporozhye Zurich

Please enter the name of one of these cities or zones
You just need to type enough letters to resolve ambiguities
Press Enter to view all of them again
Name: [] London
Your default time zone is set to 'Europe/London'.
Local time is now:      Mon Mar 31 09:23:48 BST 2008.
Universal Time is now:  Mon Mar 31 08:23:48 UTC 2008.

More information is available in the Debian System Administrator Manual.

Recently, we rolled out a Shibboleth Single Sign On service to protect one of our services. However, we started recieving intermittant login failures, both on our automated monitoring and from customers. Curiously these failures tended to happen mostly in the evening, which isn't a peak time for us. Debugging showed that the authentication worked, but the authorisaton was failing. Shibboleth works as an apache module and daemom that protects a service, which communicates with a webservice that does the authenication processing. The log files were showing an occasional SSL error in this communcation link.

INFO shibtarget.SessionCache [43005] sessionGet: trying to get new attributes 
      for session (ID=_d0cd2f93840bb92050b28fa73d19ce4f)
INFO SAML.SAMLSOAPHTTPBinding [43005] sessionGet: sending SOAP message to 
      https://login.example.com/shibboleth/AA
ERROR SAML.SAMLSOAPHTTPBinding [43005] sessionGet: failed while contacting   
      SAML responder: error:1408F06B:SSL routines:SSL3_GET_RECORD:bad 
      decompression
ERROR shibtarget.SessionCache [43005] sessionGet: caught SAML exception 
      during SAML attribute query: SOAPHTTPBindingProvider::send() failed 
      while contacting SAML responder: error:1408F06B:SSL 
      routines:SSL3_GET_RECORD:bad decompression
ERROR shibtarget.SessionCache [43005] sessionGet: no response obtained

We didn't manage to find any suitable solutions on the internet, so we pulled out the trusty wireshark and started looking to see what was going on. We could see that the client was advertising deflate and null compression, and that the server was responding by asking for deflate compression. However the client would then claim that there was a decompression error in the servers response. This opened a few lines of enquiry. I made sure that both ends of the connection were running the same version of OpenSSL and they were both using 0.9.8a from Ubuntu Dapper. Interestingly 0.9.8a is the first version that had compression support. We found a couple of suggestions including forcing connections to be SSL2, which lacked compression or recompiling openssl without zlib support. As the former was easier, we tried that first by putting

SSLCipherSuite SSLv2:-LOW:-EXPORT:RC4+RSA

in /etc/apache2/mods-enabled/ssl.conf as suggested by Debian bug #338008, and this seemed to work for around an hour. Packet sniffing showed that it was still negotiating SSL3 including deflate compression. Clearly we had to try something else.

Rather than recompile OpenSSL without zlib support, I thought I'd try upgrading the version of OpenSSL to something later in case that fixed the decompression bug. the version in Hardy is 0.9.8g, which sadly required recompiling and disabling the Ubuntu change to enable -Bsymbolic-functions during linking. Installing this on the client end didn't fix the problem, however installing it on the server end seemed to fix it. So far it's been running for 24 hours without an error, so fingers crossed that this has fixed it for good.

Installing the Sun Java packages on Debian or Ubuntu require to you accept Sun's license before you can install them. This means that it's not easy to install non-interactively, for example when using pbuilder. Fortunately the license uses debconf to check to see if you have already accepted the license. This means you can use debconf to accept the license before you install the packages. Create a file containing the following lines:

sun-java5-jdk shared/accepted-sun-dlj-v1-1 select true
sun-java5-jre shared/accepted-sun-dlj-v1-1 select true
sun-java6-jdk shared/accepted-sun-dlj-v1-1 select true
sun-java6-jre shared/accepted-sun-dlj-v1-1 select true

Then run /usr/bin/debconf-set-selections <file> and when you install the java packages, you should find it doesn't prompt for the license any more.

By default, warnquota sends out emails with the device name in the message, which probably doesn't make much sense to most non-technical users.

Hi,

We noticed that you are in violation with the quotasystem
used on this system. We have found the following violations:


/dev/mapper/Ubuntu-home

                        Block limits               File limits
Filesystem           used    soft    hard  grace    used  soft  hard  grace
/dev/mapper/Ubuntu-home
               +- 1044404 1000000 1200000  6days    1781     0     0

You can improve this by using /etc/quotatab to assign a more meaningful name to the partition:

/dev/mapper/Ubuntu-home:user directory
/dev/mapper/Ubuntu-shared:shared area

I've been running Hardy on my workstation for a while and had recently noticed that I was failing to type a space after "I". I was doing it far too much for it to just be me failing to press the space bar properly, and it wasn't happening after any other letter. After a little bit of experiementing, I discovered that something was eating shift-space. What was happening was that I was failing to release the shift key quick enough after typing "I" and before I hit the space bar, so it wasn't getting passed on.

Turns out that the problem was a recent update of Hardy installed SCIM, which uses shift-space as a keyboard shortcut. To turn it off, load the SCIM Setup program and go to the FrontEnd Global Setup screen and remove "Shift+Space" from the Trigger hotkey.

Caused confusion for a few minutes. :)

Subject: FELICITATION !!!!!  VOUS VENEZ DE GAGNER (ceci n'est pas un spam)

It's in French; of course it's a spam.

dpkg has a very useful feature where if you delete a conffile (pretty much everything under /etc and a few other files) it isn't replaced when you upgrade the package[0]. This behaviour was confusing me for a while until I realised what was happening. I was attempting to reinstall a package to get the default configuration files back that had been accidentally deleted, but no matter what I tried, the files didn't exist after running dpkg. Once I figured out that dpkg had this behaviour the solution was simple; use the --force-confmiss command line argument.

root@quux:~# dpkg --force-confmiss -i /tmp/foo_2.0.0-build.14_all.deb 
(Reading database ... 33418 files and directories currently installed.)
Preparing to replace foo 2.0.0-build.14 (using .../foo_2.0.0-build.14_all.deb) ...
Unpacking replacement foo ...
Setting up foo (2.0.0-build.14) ...

Configuration file `/etc/foo/foo.xml', does not exist on system.
Installing new config file as you request.
root@quux:~#

[0] If the file didn't exist in the previously installed version, it is installed, so you get new configuration files.

Phishers aren't even trying these days:

The following things stand out:

• The date header is +0900. Suspicion rating: 2/10
• The recorded log in time is in EST. The Halifax and myself are in GMT. Suspicion rating: 6/10
• The recorded log in time hadn't occured by the time I get the email. Suspicion rating: 8/10
• I don't bank with the Halifax. Suspicion rating: 10 million/10

If you've got various indenting and text wrapping options turned on in vim, pasting text into the editor results in screwed up results. You can get around this by turning on paste mode using :set paste and off with :set nopaste. To make things a little easier, you can use the following snippet in your .vimrc to allow you to toggle paste on and off using a single keypress:

nmap <F4> :set invpaste paste?<CR>
imap <F4> <C-O>:set invpaste<CR>
set pastetoggle=<F4>

(Warning: my vim settings have organically grown over the last 10 years, so they may not be the best or modern way of achieving an effect.)

If you're trying to import a dump file created using mysqldump and you get an error like:

ERROR 1005 (HY000): Can't create table './Database/Table.frm' (errno: 150)

Then you've just been bitten by mysqldump being far too stupid. The problem occurs because mysqldump includes foreign key constraints in the initial CREATE TABLE command, so if a table refers to a table that doesn't currently exist, it throws an error. mysqldump does correctly disable the contraints when inserting data into the tables. The correct way for this would be for mysqldump to create all the tables without the constraints, use ALTER TABLE to add the constraints to the tables, and then importing the data into the tables.

The workaround for this problem is to use:

SET FOREIGN_KEY_CHECKS = 0;
source dump.sql
SET FOREIGN_KEY_CHECKS = 1;

Update: Someone has pointed out that it appears that mysql 5 has fixed this problem by including the above statements in the dump.

Breaking news on BBC New24. Just confirmed in the last few minutes. Very few details.... Last night, Tony Blair converted to Catholism

Why is this news? Who cares? Why is the BBC treating this like it's the biggest news item of the year? Why have they rolled out Anne Widdecomme to do a phone interview? His wife is a catholic, his children are catholic, it's been on the cards for a while. He isn't in power any more. It remains to be seen if he has any relevance any more. So why does it matter what denomination he is.

Dear Lazyweb,

I'm possibly asking for the moon on a stick here, but in the office we have VoIP phones, which talk to our Asterisk server. Unfortunately, the ringtone on them are incredibly quiet and I tend to listen to music and don't notice either the ring or the small green flashing light when a call comes in.

The question then is does anyone know of a program which will talk SIP to the asterisk server and notice when a call comes in and turn my music down and display a notification?

In my article on Perl's IO::Handle objects I talked briefly about IO::AtomicFile and IO::Digest. I've just had reason to use these very useful modules to create a script which edits a file in place. These modules allowed me to do the rewrite atomically and optionally make a backup if the contents have changed. The example assumes you have a function called perform_rewrite that takes two file handles as the first two parameters.

use File::Copy;
use IO::File;
use IO::AtomicFile;
use IO::Digest;

sub rewrite_file {
   my $file = shift;
   my $sub = shift;
   my $input = new IO::File($file,'r');
   my $input_md5 = new IO::Digest($input, 'MD5');
   my $output = new IO::AtomicFile($file,'w');
   my $output_md5 = new IO::Digest($output, 'MD5');

   $sub->($input, $output, @_);

   if ($input_md5->hexdigest ne $output_md5->hexdigest) {
           copy ("$file", "$file.bak");
           $output->close();
   } else {
           # we haven't changed so don't bother updating
           $output->delete();
   }
   $input->close();
}

rewrite_file("/foo/bar", \&perform_rewrite, $baz, $quux);

Currently listening to Biffy Clyro's new album, Puzzle, and I have to say it is definitely their best album yet. It's up there with 65daysofstatic for best album of the year so far. It's made a change from all the recent albums where it's taken repeated listenings to like the album. With the first listen I love this album. I am definitely going to have to see them live more often. I've only seen them twice and the gig a fortnight ago was possibly one of my best gigs ever.