If you've not heard of Puppet, it is a configuration management tool. You write descriptions of how you want your systems to look and it checks the current setup and works out what it needs to do to move your system so it matches your description. The idea is to write how it should look, not how to change the system.

Puppet uses a client (puppetd) that talks to the central server (puppetmaster) over HTTPS.The default puppetmaster HTTP server is webbrick, which is a lightweight Ruby HTTP server. While it's simple and allows Puppetmaster to work straight out the box, due to it's pure Ruby structure and Ruby's green thread architecture, it doesn't scale beyond a simple puppet setup. After a while, every medium to large Puppet installation needs to move to the other HTTP server that puppet supports: Mongrel. Mongrel is a faster HTTP library, but supports a lot less features. In particular it doesn't support SSL, which is important with Puppet, as Puppet relies heavily on client certificate verification for authentication. As a result, we need to put another webserver in front that can handle the SSL aspect. As a nice side effect of having to proxy to Puppetmaster is that we can run several puppetmaster processes and improve on the green threads problem that Ruby has. In this blog post, I'm going to describe setting up nginx and mongrel.

The first thing to do is to install the mongrel and nginx packages.

apt-get install mongrel nginx

We need to run nginx on port 8140 and proxy to our mongrel servers on different ports, so lets move puppetmaster off 8140 and configure it to use mongrel while we're at it. Edit /etc/default/puppetmaster and set the following variables:

SERVERTYPE=mongrel
PUPPETMASTERS=4
PORT=18140
DAEMON_OPTS="--ssl_client_header=HTTP_X_SSL_SUBJECT"

This tells the init.d script to use the mongrel server type and to run four of them. The init.d script is clever enough to start up the right number of processes and will set them up to use a sequence of ports for each one, starting at 18140 for the first process, up to 18143 for the last one. The DAEMON_OPTS option tells Puppetmaster how we're going to pass the SSL certificate information from nginx so it can grant or refuse permission.

Now to set up nginx. Put the following in /etc/nginx/conf.d/puppetmaster.conf:

ssl                     on;
ssl_certificate /var/lib/puppet/ssl/certs/puppetmaster.example.com.pem;
ssl_certificate_key /var/lib/puppet/ssl/private_keys/puppetmaster.example.com.pem;
ssl_client_certificate  /var/lib/puppet/ssl/certs/ca.pem;
ssl_ciphers             SSLv2:-LOW:-EXPORT:RC4+RSA;
ssl_session_cache       shared:SSL:8m;
ssl_session_timeout     5m;

upstream puppet-production {
   server 127.0.0.1:18140;
   server 127.0.0.1:18141;
   server 127.0.0.1:18142;
   server 127.0.0.1:18143;
}

In this file we tell nginx where to find the server certificates for your puppetmaster, so your clients can authenticate your server. We also tell nginx the CA certificate to authenticate clients with and set up some SSL details required for Puppet. Finally we create a group of remote servers for our pack of mongrel puppetmasters, so we can refer to them later. If you added more or less servers earlier don't forget to add or remove them here. You also need to replace puppetmaster.example.com with your FQDN. If at a later stage, you find you need ever more performance, you can easily move some of your puppetmaster processes to a separate box and update the upstream list to refer to servers on the remote server.

Finally, we need to set up a couple of HTTP servers. Create /etc/nginx/sites-enabled/puppetmaster with the following contents:

server {
    listen                  8140;
    ssl_verify_client       on;
    root                    /var/empty;
    access_log              /var/log/nginx/access-8140.log;

    # Variables
    # $ssl_cipher returns the line of those utilized it is cipher for established SSL-connection
    # $ssl_client_serial returns the series number of client certificate for established SSL-connection
    # $ssl_client_s_dn returns line subject DN of client certificate for established SSL-connection
    # $ssl_client_i_dn returns line issuer DN of client certificate for established SSL-connection
    # $ssl_protocol returns the protocol of established SSL-connection

    location / {
        proxy_pass          http://puppet-production;
        proxy_redirect      off;
        proxy_set_header    Host             $host;
        proxy_set_header    X-Real-IP        $remote_addr;
        proxy_set_header    X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header    X-Client-Verify  SUCCESS;
        proxy_set_header    X-SSL-Subject    $ssl_client_s_dn;
        proxy_set_header    X-SSL-Issuer     $ssl_client_i_dn;
        proxy_read_timeout  65;
    }
}

server {
    listen                  8141;
    ssl_verify_client       off;
    root                    /var/empty;
    access_log              /var/log/nginx/access-8141.log;

    location / {
        proxy_pass  http://puppet-production;
        proxy_redirect     off;
        proxy_set_header   Host             $host;
        proxy_set_header   X-Real-IP        $remote_addr;
        proxy_set_header   X-Forwarded-For  $proxy_add_x_forwarded_for;
        proxy_set_header   X-Client-Verify  FAILURE;
        proxy_set_header   X-SSL-Subject    $ssl_client_s_dn;
        proxy_set_header   X-SSL-Issuer     $ssl_client_i_dn;
        proxy_read_timeout  65;
    }
}

This creates two servers on port 8140 and 8141 which both proxy all requests to our group of mongrel servers, adding suitable headers to pass on the SSL information. The only difference between them is the X-Client-Verify header. This shows the one problem with using nginx with puppet. Because the client verification success or failure is not available as a variable before nginx 0.8.7, we can't have a single port for both the usual client connection and the initial unauthenticated connection where the client requests a certificate to be signed. As a result, with this setup, you are required to run puppet with --ca-port 8141 the first time you run puppet until the certificate has been signed with puppetca.

Foruntately with versions of nginx later than 0.8.7, you can use a simpler setup shown below. This replaces both files shown above with the single server. Unfortunately, 0.8.7 is not available in any version of Ubuntu yet, not even Karmic.

server {
  listen 8140;

  ssl                     on;
  ssl_session_timeout     5m;
  ssl_certificate         /var/lib/puppet/ssl/certs/puppetmaster.pem;
  ssl_certificate_key     /var/lib/puppet/ssl/private_keys/puppetmaster.pem;
  ssl_client_certificate  /var/lib/puppet/ssl/ca/ca_crt.pem;

  # choose any ciphers
  ssl_ciphers             SSLv2:-LOW:-EXPORT:RC4+RSA;

  # allow authenticated and client without certs
  ssl_verify_client       optional;

  # obey to the Puppet CRL
  ssl_crl /var/lib/puppet/ssl/ca/ca_crl.pem;

  root                    /var/tmp;

  location / {
    proxy_pass              http://puppet-production;
    proxy_redirect         off;
    proxy_set_header    Host             $host;
    proxy_set_header    X-Real-IP        $remote_addr;
    proxy_set_header    X-Forwarded-For  $proxy_add_x_forwarded_for;
    proxy_set_header    X-Client-Verify  $ssl_client_verify;
    proxy_set_header    X-SSL-Subject    $ssl_client_s_dn;
    proxy_set_header    X-SSL-Issuer     $ssl_client_i_dn;
    proxy_read_timeout  65;
  }
}

If you are running another webserver on the server, you may want to delete /etc/nginx/sites-enabled/default which attempts to create a server listening on port 80, which will conflict with your existing HTTP server.

If you follow these instructions, you should find yourself with a better performing puppetmaster and significantly few "connection reset by peer" and other related error messages.

Sometimes you need to review what exactly doing an svn up will do. Fortunately, you can do a couple of things to find out. The first is use svn status -u to find out what files have changed:

/etc/puppet/modules# svn stat -u
       *     1338   exim4_mailserver/files/exim_db.pl
       *     1338   dbplc/files/sort-dbfs.pl
       *     1338   dbplc/files
       *     1338   dbplc/manifests/portal.pp
M            1386   tomcat/files/server.xml
Status against revision:   1386

Here we can see that four files were changed since their current revision of 1338. We can also see that tomcat/files/server.xml is up to date against the repository, but has local modifications.

This is all well and good, but how do we know what the changes are? Well, svn diff is our friend here. By comparing the checkout against the repository, we can see what will be updated.

/etc/puppet/modules# svn diff dbplc/manifests/portal.pp -rBASE:HEAD
Index: dbplc/manifests/portal.pp
===================================================================
--- dbplc/manifests/portal.pp (working copy)
+++ dbplc/manifests/portal.pp (revision 1386)
@@ -22,6 +22,7 @@
       owner => "tomcat55",
       group => "adm",
       mode => 644,
+      require => Package["tomcat5.5"],
    }
 
    apache::config { "portal":

When you're happy, you can run svn up as normal. I've just used this process to sanity check our Puppet config before updating, as it hasn't been updated for a few days.