These were last year’s most successful articles:

1. Generating DDL Scripts from JPA Annotations with Maven
2. Reverting a Commit in Subversion
3. Parsing Command Line Options in Shell Scripts
4. Generating DDL Scripts with Maven
5. Ubuntu Natty vs. nVidia

That’s pretty similar to 2010, the top four are well linked and seem to contain just the right keywords so search engines send a lot of traffic. The Ubuntu posting is a typical problem solving piece from this year that earned me some serious karma points, judging from the comments.

Traffic-wise, rants linked from link aggregators like DZone generate lots of page impressions in a short amount of time. So do problem solving articles until the underlying issue is fixed. But it’s the HOWTO-style articles that generate steady traffic for years.

There was another thing that’s interesting from a SEO point of view. Back in July I switched from a wordpress subdomain to my own domain, wondering what this would do to website traffic. Now I know: Page impressions dropped by 50% and it took almost two months to recover. This would have been really serious if this was an ad-supported website (which it isn’t, the ads you see are from WordPress.com, and I don’t get a share).

Despite this sharp drop, compared to last year, page impressions have increased by more than 40%. While I’m still a very long shot from running a high-traffic blog, I could have made a bit of money that would more than cover typical hosting costs. But I’m still happy with WordPress.com, so I won’t bother.

So, that’s it, enough meta blogging for this year. Thanks for reading and commenting and have a great new year!

Before comparing the alternatives let’s first have a look at my use case. My department runs a medium sized Hudson installation with three build slaves. We have 700 build jobs, most of them using Maven 2 to build software and run rather large test suites. These jobs fall into a small number of categories (on-commit builds, nightly site builds, Sonar builds, release builds, etc.) where jobs in the same category are pretty much identical except for their Subversion URL.

It’s probably hard to understand for CI developers, but to us, our CI server is about as exciting as our Maven repository or our wiki. We don’t care about it most of the time and expect it to run with little to no maintenance. Here’s what we need:

• Subversion support
• On-commit and time-based build triggers
• Running Maven builds
• E-Mail notifications
• Developer self service
• Authentication and authorization

We don’t care much for fancy features. Our main Hudson system is critical to our internal development processes and hasn’t been updated in more than three years. Back then, when you finally managed to find a stable Hudson release for your scenario, there was no way you would ever upgrade. Neither would you install any plugins without a very thorough QA process.

Anyway, three years is a long time in IT and since our build servers are about to be modernized, we’re finally going to upgrade our CI environment with the latest version of either Hudson or Jenkins.

I don’t have any inside knowledge into either project, but to make a decision I’ll sum up my impressions drawn from reading blog posts, looking around websites, mailing list archives and bug trackers. Feel free to comment if I got something wrong.

A Common History

With Jenkins being a fork of Hudson (or maybe the other way round), both projects share a common past. Both projects inherited an old, mostly undocumented code base with lots of open issues in Jira, a huge unmanaged set of dependencies and no test coverage to speak of. From my short glances at the code it looked like a hackish open source project with a few cool ideas but overall little emphasis on quality. In projects like this you need good developers to keep the code base from collapsing under its own weight.

While there were a lot of third-party plugins, many of the more obscure plugins seemed to be buggy or unmaintained, which is a well-known problem in extensible software packages (see Firefox, for example). Still, Hudson core along with its most important plugins was a usable, battle tested CI server.

The Case for Jenkins

Project team: The Jenkins project picked up where Hudson left off, taking many (plugin) developers with it. This seems to be the original development team from before the fork but obviously without Oracle employees.

Activity: There have been regular releases since the fork, the community seems very active and there was even a conference on Jenkins. All in all a healthy open source project.

Releases: One of my biggest complaints about the original Hudson has always been its weird release policy. There was no stable release branch; to get bug fixes, you had to install the latest release which also contained new features. Since this is unacceptable to many companies, the Jenkins project introduced long term support releases (LTS), with "long term" meaning three months. Apparently, you can get 12 months of support from Cloudbees which is better than nothing.

Documentation: User documentation is traditionally sparse on the web, but a book from O’Reilly fills the gap for those who need it. There’s very little on Jenkins’ architecture which is a problem for plugin developers and contributors.

Quality: Looking at Jira, there are currently 515 open bugs in Jenkins core, 244 of them older than a year. Everything else included, there are 4273 open issues, which is (in my opinion) not a healthy amount for a project of this size. This makes me less than confident that obscure bugs plaguing only a minority of users get fixed in a timely manner.

Features: Some features useful for larger installations are only available through Jenkins Enterprise by CloudBees which is annoying. There’s also a large number of third party plugins, but since our requirements are pretty basic, we don’t need any of them.

The Case for Hudson

Project team: Despite many people’s predictions, Hudson is still alive. There seems to be a team at Oracle working on Hudson and there have also been contributions from Sonatype, a well-known player from the Maven universe. I’m not sure how dependent the project is on employees from either companies.

Activity: There have been several releases since the fork, with some rather large changes. Right now, there’s at lot of clean up and refactoring activity going on which is necessary for migrating the project to the Eclipse Foundation. They also spent some time on professionalizing the project, for example defining and documenting processes.

Releases: Release cycles seem to be longer than with Jenkins, with release numbers using a more mainstream numbering scheme. I didn’t find any information on how long releases are supported.

Documentation: There’s a free book on Hudson available from the website which is a work in progress and there’s also serious architectural documentation. Right now, both projects are similar enough to benefit from this documentation, but projects are likely to drift apart over time.

Quality: Currently, Jira shows 2663 open issues with 34 being open bugs in Hudson core (6 of them older than a year). It seems the Hudson team spent some time cleaning up their bug tracker. According to their wiki, an Oracle-based QA team does functional testing of Hudson core along with the most important plugins. From my own experience with good software testers, I know about the huge positive effect this can have on a product.

Features: Release 2.2.0 comes with a job templating mechanism ("cascading") that’s going to be extremely valuable for handling large numbers of similar jobs. I guess if they spiced up the Nested Views plugin, too, this would make the Cloudbees offering pretty much pointless to a lot of users.

Conclusion

The Jenkins project takes pride in frequent releases and its large number of plugins, but honestly, I couldn’t care less. CI is a critical part of our infrastructure. If it doesn’t work, we’re in serious trouble, so I choose quality and stability over features any day. In fact, I would jump at the opportunity to disable features on the job configuration page.

Given its bad track record with Open Source, I don’t trust Oracle, but I haven’t seen enough from Cloudbees to trust them, either. Sonatype has a better reputation though, and so does Hudson’s new home, the Eclipse Foundation. Since Sonatype offers a Hudson-based product, I’m confident that there will be continued development.

To sum things up, I’m leaning towards Hudson. The Hudson project makes a more professional impression, looking like a well-run enterprise IT project rather than your typical open source project. In my opinion, their focus on quality makes it a better match for our use case.

My router also acts as a DNS proxy and provides a few handy DNS records, too. The router itself has the host name fritz.box and every other device on the network is available via HOSTNAME.fritz.box (the router learns the client’s host name via DHCP). This means I can access my desktop machine via amy.fritz.box.

Using my servehttp script or Simon Budig’s similar but more fancy woof script, I can serve files from my desktop machine and download them from my Android device via the browser. Using the browser is nice because I don’t need any additional apps or configuration (a bookmark is useful though).

Services like Ubuntu One or Dropbox work well, but you don’t want to send larger files across the internet. Sharing via Bluetooth might work for small files, but my desktop machine doesn’t support Bluetooth. Using my servehttp script to publish a directory via HTTP works technically, but you have to know the (dynamic) IP address of the desktop machine. While I could assign a static address, it’s pretty inconvenient having to deal with IP addresses (even more so with IPv6).

A perfectly fine set of standards for solving the addressing problem is Zeroconf, modeled after Apple’s Bonjour. Using its Multicast DNS (MDNS) protocol, a device can look up a host’s name by sending a MDNS query to the local network. The device owning the host name responds with its IP address. Since this is just DNS over multicast, it integrates well with your system’s resolver. Also, hosts can announce services to the network. On recent linux distributions you need packages avahi-daemon and the resolver plugin package libnss-mdns. Then you should be able to look up your host on your local network (try ping HOST.local). Unfortunately, Android doesn’t support Zeroconf out of the box.

A widely deployed alternative to Zeroconf is Universal Plug and Play (UPnP). It provides service discovery through announcements like Zeroconf, but has never been properly standardized. Hosts announce services via Simple Service Discovery Protocol (SSDP), a multicast-based UDP protocol. SSDP announcements are basically HTTP extended with a NOTIFY method:

NOTIFY * HTTP/1.1
HOST: 239.255.255.250:1900
CACHE-CONTROL: max-age=1800
LOCATION: http://192.168.2.103:49152/description.xml
NT: urn:microsoft.com:service:X_MS_MediaReceiverRegistrar:1
NTS: ssdp:alive
SERVER: Linux/2.6.38-11-generic, UPnP/1.0, Portable SDK for UPnP devices/1.6.6
X-User-Agent: redsonic
USN: uuid:898f9738-d930-4db4-a3cf-0015c5c15a3f::urn:microsoft.com:service:X_MS_MediaReceiverRegistrar:1

It looks a bit like a redirect; in fact you can download a service description from http://192.168.2.103:49152/description.xml.

There are several media servers that support UPnP. Here’s how you share a file system directory on my desktop machine using ushare:

matthias@amy ~ $ ushare -c /tmp/share/
Warning: can't parse file "/etc/ushare.conf".
Recheck uShare's configuration and try again !
uShare (version 1.1a), a lightweight UPnP A/V and DLNA Media Server.
Benjamin Zores (C) 2005-2007, for GeeXboX Team.
See http://ushare.geexbox.org/ for updates.
Listening on telnet port 1337
Initializing UPnP subsystem ...
UPnP MediaServer listening on 192.168.2.103:49152
Sending UPnP advertisement for device ...
Listening for control point connections ...
Building Metadata List ...
Looking for files in content directory : /tmp/share/
Found 2 files and subdirectories.

This works well for one-off sharing of media files. You can also run ushare as a system service on a fixed set of directories. Obviously, this is only viable for local, trusted networks (have a look at ushare’s web console for even more reasons).

On my Android devices, I use BubbleUPnp to stream music or videos (in supported formats) and optionally download them. Unfortunately, most UPnP apps seem to be media players and ushare doesn’t seem to serve document files (I want my ebooks, dammit).

This means, right now I’m stuck with assigning a static IP address to my desktop machine and serve files via HTTP. Or are there better solutions?

My server doesn’t do a lot: There’s just a couple of deamons, including an Apache instance that servers less than 1000 page impressions per day. Load is pretty much zero most of the time, there’s very little network I/O, and memory use is below limits, too.

After some head scratching, I realized what the problem was: The virtual machine host counts kernel level threads towards the process limit! It’s quite obvious when you think about it; a quick "ps -efL" confirmed the suspicion.

The culprit was Apache with its threaded worker model (apache2-mpm-worker). The default settings were unsuitable for my rather limited virtual machine, they caused Apache to spawn too many threads. Fortunately, after reading the documentation this was rather easy to fix. I edited the configuration (/etc/apache2/apache2.conf on my system), adjusting the default settings:

<IfModule mpm_worker_module>
    StartServers          2
    ServerLimit           4
    MaxClients           64
    ThreadsPerChild      16
    MinSpareThreads       4
    MaxSpareThreads       8
    MaxRequestsPerChild   0
</IfModule>

After an Apache restart, my problems were gone.

This configuration limits the number of active worker threads to 64 (MaxClients), which is well below the virtual machine’s threshold. Those 64 threads run in up to 4 child processes (ServerLimit), with 2 processes created on startup (StartServers). Each server process contains at most 16 threads (ThreadsPerChild), with 4 threads created at minimum and at most 8 idle threads kept in the worker pool (MinSpareThreads and MaxSpareThreads respectively). There’s no limit on the number of requests a process may serve (MaxRequestsPerChild).

An alternative solution would be to switch to the prefork module. It works with processes rather than threads and is a lot easier to configure. Just install the apache2-mpm-prefork and set its MaxClients appropriately.

Keep in mind that browsers typically open several connections to your host, so one client can easily hog half a dozen workers. For my site, that’s no problem, for others it may very well be.

A check in JUnit 4 always looks something like this:

assertThat(actual, predicateName(expected));

The predicateName() factory method is just syntactic sugar, so let’s instantiate the matcher directly to make it more obvious what happens here:

assertThat(actual, new SomeMatcher(expected));

The first argument is of type TypeActual, the expectation’s type (the constructor’s argument) is ExpectedType. This is our implementation of SomeMatcher:

import org.hamcrest.Description;
import org.hamcrest.Factory;
import org.hamcrest.TypeSafeMatcher;

public class SomeMatcher extends TypeSafeMatcher<ActualType> {
    private final ExpectedType expected;

    public SomeMatcher(ExpectedType expected) {
        this.expected = expected;
    }

    @Override
    public boolean matchesSafely(ActualType actual) {
        // return true iff "actual" matches "expected"
    }

    @Override
    public void describeTo(Description descr) {
        // describe expectation
    }

    @Factory
    public static SomeMatcher predicateName(ExpectedType expected) {
        return new SomeMatcher(expected);
    }
}

The @Factory annotation isn’t necessary, but Hamcrest ships with a tool that picks up predicates annotated this way and collects them in a Matchers class for easy importing.

They have pretty decent documentation so setup was simple. You can see that I added a CNAME DNS entry from blog.mafr.de to the blog’s “technical” address at wordpress.com:

matthias@amy ~ $ host -t CNAME blog.mafr.de
blog.mafr.de is an alias for unmaintainable.wordpress.com.
matthias@amy ~ $

The unmaintainable.wordpress.com entry itself is just an alias for lb.wordpress.com which in turn maps to several actual hosts using round-robin DNS. Users who still request URLs from the old site are redirected (I hope this doesn’t affect search engine traffic):

matthias@amy ~ $ curl -D - http://unmaintainable.wordpress.com
HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Sat, 23 Jul 2011 10:12:57 GMT
Content-Type: text/html; charset=utf-8
Transfer-Encoding: chunked
Connection: close
Vary: Cookie
Location: http://blog.mafr.de/

matthias@amy ~ $

Checking my new setup I noticed this:

matthias@amy ~ $ curl -s -D - http://blog.mafr.de | head
HTTP/1.1 200 OK
Server: nginx
Date: Sat, 23 Jul 2011 10:48:54 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: close
Vary: Accept-Encoding
Vary: Cookie
Set-Cookie: hiab=on; path=/; domain=blog.mafr.de
X-hacker: If you're reading this, you should visit automattic.com/jobs and apply to join the fun, mention this header.

Coolest job offer ever :)

Fortunately, there’s a little known but documented mechanism to globally remap all JBoss ports. The file server/default/conf/bindingservice.beans/META-INF/bindings-jboss-beans.xml (for the default profile) contains port bindings that you can activate using the jboss.service.binding.set property from the command line.

Suppose there’s already a JBoss instance running (which uses the default ports-default binding). Using the command line property, we can run a second instance like this:

  bin/run.sh -Djboss.service.binding.set=ports-01

The ports-01 binding adds 100 to each port of the ports-default binding, ports-02 adds 200, and ports-03 adds 300. You can easily add more ports bindings yourself if you need them.

I tested this on JBoss 6.0.0.Final and it worked well. An alternative to port mapping approach is to make your machine listen on multiple IP addresses and have each JBoss instance bind to a different IP address using the jboss.bind.address property. Obviously, this requires no port remapping but changes to your network setup, usually including DNS.

You can configure a lot in Gnome using the gconf configuration system. For all settings that aren’t available through the menus, you can use the graphical gconf-editor tool or the command line utility gconftool-2.

There are two gconf settings: A desktop-wide one that disables cursor blinking for all Gnome applications (/desktop/gnome/interface/cursor_blink). And the other is a setting specific to a Gnome Terminal profile (/apps/gnome-terminal/profiles/Default/cursor_blink_mode for the “Default” profile). While the desktop-wide one is a boolean setting, Gnome Terminal allows to either inherit the desktop-wide setting (“system”) or to set your preference explicitly (“on”, “off”).

This command disables cursor blinking desktop-wide:

  gconftool-2 --set /desktop/gnome/interface/cursor_blink --type bool false

And this command disables blinking just for Gnome Terminal’s Default profile:

  gconftool-2 --set /apps/gnome-terminal/profiles/Default/cursor_blink_mode --type string off

All changes to the gconf database are persistent, so you don’t have to run these commands in every session.

Using my own domain, I can define a personal identifier (let’s say http://example.com/matthias/) that delegates to a real identity provider. All I need is an HTML page on the public internet with some OpenID metadata that refers to an identity provider. The relying party (OpenID speak for the service I want to authenticate to) associates me with http://example.com/matthias/ and I can switch identity providers as I see fit.

Let’s have a look at an example. I want to use my WordPress blog as my identity provider so I can use my existing Google account to log into OpenID-enabled services like Stackoverflow. All I need is a bit of metadata at http://example.com/matthias/:

<html>
<head>
  <link rel="openid.delegate" href="http://unmaintainable.wordpress.com/" />
  <link rel="openid.server" href="http://unmaintainable.wordpress.com/?openidserver=1" />
</head>
<body>
  <!-- ... -->
</body>
</html>

Wikipedia has a list of OpenID identity providers. For more information about the required metadata see the OpenID spec.