Hudson vs. Jenkins Revisited

When the Hudson/Jenkins fork was announced earlier this year, the case seemed pretty clear. There were heroes from the Open Source community, among them Hudson's original author, and there was everybody's favorite villain, Oracle. Back in February, I would have chosen Jenkins without a second thought. Now, almost a year later, things are less clear and I'm not sure which one to pick.

Before comparing the alternatives let's first have a look at my use case. My department runs a medium sized Hudson installation with three build slaves. We have 700 build jobs, most of them using Maven 2 to build software and run rather large test suites. These jobs fall into a small number of categories (on-commit builds, nightly site builds, Sonar builds, release builds, etc.) where jobs in the same category are pretty much identical except for their Subversion URL.

It's probably hard to understand for CI developers, but to us, our CI server is about as exciting as our Maven repository or our wiki. We don't care about it most of the time and expect it to run with little to no maintenance. Here's what we need:

  • Subversion support
  • On-commit and time-based build triggers
  • Running Maven builds
  • E-Mail notifications
  • Developer self service
  • Authentication and authorization

We don't care much for fancy features. Our main Hudson system is critical to our internal development processes and hasn't been updated in more than three years. Back then, when you finally managed to find a stable Hudson release for your scenario, there was no way you would ever upgrade. Neither would you install any plugins without a very thorough QA process.

Anyway, three years is a long time in IT and since our build servers are about to be modernized, we're finally going to upgrade our CI environment with the latest version of either Hudson or Jenkins.

I don't have any inside knowledge into either project, but to make a decision I'll sum up my impressions drawn from reading blog posts, looking around websites, mailing list archives and bug trackers. Feel free to comment if I got something wrong.

A Common History

With Jenkins being a fork of Hudson (or maybe the other way round), both projects share a common past. Both projects inherited an old, mostly undocumented code base with lots of open issues in Jira, a huge unmanaged set of dependencies and no test coverage to speak of. From my short glances at the code it looked like a hackish open source project with a few cool ideas but overall little emphasis on quality. In projects like this you need good developers to keep the code base from collapsing under its own weight.

While there were a lot of third-party plugins, many of the more obscure plugins seemed to be buggy or unmaintained, which is a well-known problem in extensible software packages (see Firefox, for example). Still, Hudson core along with its most important plugins was a usable, battle tested CI server.

The Case for Jenkins

Project team: The Jenkins project picked up where Hudson left off, taking many (plugin) developers with it. This seems to be the original development team from before the fork but obviously without Oracle employees.

Activity: There have been regular releases since the fork, the community seems very active and there was even a conference on Jenkins. All in all a healthy open source project.

Releases: One of my biggest complaints about the original Hudson has always been its weird release policy. There was no stable release branch; to get bug fixes, you had to install the latest release which also contained new features. Since this is unacceptable to many companies, the Jenkins project introduced long term support releases (LTS), with "long term" meaning three months. Apparently, you can get 12 months of support from Cloudbees which is better than nothing.

Documentation: User documentation is traditionally sparse on the web, but a book from O'Reilly fills the gap for those who need it. There's very little on Jenkins' architecture which is a problem for plugin developers and contributors.

Quality: Looking at Jira, there are currently 515 open bugs in Jenkins core, 244 of them older than a year. Everything else included, there are 4273 open issues, which is (in my opinion) not a healthy amount for a project of this size. This makes me less than confident that obscure bugs plaguing only a minority of users get fixed in a timely manner.

Features: Some features useful for larger installations are only available through Jenkins Enterprise by CloudBees which is annoying. There's also a large number of third party plugins, but since our requirements are pretty basic, we don't need any of them.

The Case for Hudson

Project team: Despite many people's predictions, Hudson is still alive. There seems to be a team at Oracle working on Hudson and there have also been contributions from Sonatype, a well-known player from the Maven universe. I'm not sure how dependent the project is on employees from either companies.

Activity: There have been several releases since the fork, with some rather large changes. Right now, there's at lot of clean up and refactoring activity going on which is necessary for migrating the project to the Eclipse Foundation. They also spent some time on professionalizing the project, for example defining and documenting processes.

Releases: Release cycles seem to be longer than with Jenkins, with release numbers using a more mainstream numbering scheme. I didn't find any information on how long releases are supported.

Documentation: There's a free book on Hudson available from the website which is a work in progress and there's also serious architectural documentation. Right now, both projects are similar enough to benefit from this documentation, but projects are likely to drift apart over time.

Quality: Currently, Jira shows 2663 open issues with 34 being open bugs in Hudson core (6 of them older than a year). It seems the Hudson team spent some time cleaning up their bug tracker. According to their wiki, an Oracle-based QA team does functional testing of Hudson core along with the most important plugins. From my own experience with good software testers, I know about the huge positive effect this can have on a product.

Features: Release 2.2.0 comes with a job templating mechanism ("cascading") that's going to be extremely valuable for handling large numbers of similar jobs. I guess if they spiced up the Nested Views plugin, too, this would make the Cloudbees offering pretty much pointless to a lot of users.

Conclusion

The Jenkins project takes pride in frequent releases and its large number of plugins, but honestly, I couldn't care less. CI is a critical part of our infrastructure. If it doesn't work, we're in serious trouble, so I choose quality and stability over features any day. In fact, I would jump at the opportunity to disable features on the job configuration page.

Given its bad track record with Open Source, I don't trust Oracle, but I haven't seen enough from Cloudbees to trust them, either. Sonatype has a better reputation though, and so does Hudson's new home, the Eclipse Foundation. Since Sonatype offers a Hudson-based product, I'm confident that there will be continued development.

To sum things up, I'm leaning towards Hudson. The Hudson project makes a more professional impression, looking like a well-run enterprise IT project rather than your typical open source project. In my opinion, their focus on quality makes it a better match for our use case.

social