Hudson vs. Jenkins Revisited

When the Hudson/Jenkins fork was announced earlier this year, the case seemed pretty clear. There were heroes from the Open Source community, among them Hudson’s original author, and there was everybody’s favorite villain, Oracle. Back in February, I would have chosen Jenkins without a second thought. Now, almost a year later, things are less clear and I’m not sure which one to pick.

Before comparing the alternatives let’s first have a look at my use case. My department runs a medium sized Hudson installation with three build slaves. We have 700 build jobs, most of them using Maven 2 to build software and run rather large test suites. These jobs fall into a small number of categories (on-commit builds, nightly site builds, Sonar builds, release builds, etc.) where jobs in the same category are pretty much identical except for their Subversion URL.

It’s probably hard to understand for CI developers, but to us, our CI server is about as exciting as our Maven repository or our wiki. We don’t care about it most of the time and expect it to run with little to no maintenance. Here’s what we need:

  • Subversion support
  • On-commit and time-based build triggers
  • Running Maven builds
  • E-Mail notifications
  • Developer self service
  • Authentication and authorization

We don’t care much for fancy features. Our main Hudson system is critical to our internal development processes and hasn’t been updated in more than three years. Back then, when you finally managed to find a stable Hudson release for your scenario, there was no way you would ever upgrade. Neither would you install any plugins without a very thorough QA process.

Anyway, three years is a long time in IT and since our build servers are about to be modernized, we’re finally going to upgrade our CI environment with the latest version of either Hudson or Jenkins.

I don’t have any inside knowledge into either project, but to make a decision I’ll sum up my impressions drawn from reading blog posts, looking around websites, mailing list archives and bug trackers. Feel free to comment if I got something wrong.

A Common History

With Jenkins being a fork of Hudson (or maybe the other way round), both projects share a common past. Both projects inherited an old, mostly undocumented code base with lots of open issues in Jira, a huge unmanaged set of dependencies and no test coverage to speak of. From my short glances at the code it looked like a hackish open source project with a few cool ideas but overall little emphasis on quality. In projects like this you need good developers to keep the code base from collapsing under its own weight.

While there were a lot of third-party plugins, many of the more obscure plugins seemed to be buggy or unmaintained, which is a well-known problem in extensible software packages (see Firefox, for example). Still, Hudson core along with its most important plugins was a usable, battle tested CI server.

The Case for Jenkins

Project team: The Jenkins project picked up where Hudson left off, taking many (plugin) developers with it. This seems to be the original development team from before the fork but obviously without Oracle employees.

Activity: There have been regular releases since the fork, the community seems very active and there was even a conference on Jenkins. All in all a healthy open source project.

Releases: One of my biggest complaints about the original Hudson has always been its weird release policy. There was no stable release branch; to get bug fixes, you had to install the latest release which also contained new features. Since this is unacceptable to many companies, the Jenkins project introduced long term support releases (LTS), with "long term" meaning three months. Apparently, you can get 12 months of support from Cloudbees which is better than nothing.

Documentation: User documentation is traditionally sparse on the web, but a book from O’Reilly fills the gap for those who need it. There’s very little on Jenkins’ architecture which is a problem for plugin developers and contributors.

Quality: Looking at Jira, there are currently 515 open bugs in Jenkins core, 244 of them older than a year. Everything else included, there are 4273 open issues, which is (in my opinion) not a healthy amount for a project of this size. This makes me less than confident that obscure bugs plaguing only a minority of users get fixed in a timely manner.

Features: Some features useful for larger installations are only available through Jenkins Enterprise by CloudBees which is annoying. There’s also a large number of third party plugins, but since our requirements are pretty basic, we don’t need any of them.

The Case for Hudson

Project team: Despite many people’s predictions, Hudson is still alive. There seems to be a team at Oracle working on Hudson and there have also been contributions from Sonatype, a well-known player from the Maven universe. I’m not sure how dependent the project is on employees from either companies.

Activity: There have been several releases since the fork, with some rather large changes. Right now, there’s at lot of clean up and refactoring activity going on which is necessary for migrating the project to the Eclipse Foundation. They also spent some time on professionalizing the project, for example defining and documenting processes.

Releases: Release cycles seem to be longer than with Jenkins, with release numbers using a more mainstream numbering scheme. I didn’t find any information on how long releases are supported.

Documentation: There’s a free book on Hudson available from the website which is a work in progress and there’s also serious architectural documentation. Right now, both projects are similar enough to benefit from this documentation, but projects are likely to drift apart over time.

Quality: Currently, Jira shows 2663 open issues with 34 being open bugs in Hudson core (6 of them older than a year). It seems the Hudson team spent some time cleaning up their bug tracker. According to their wiki, an Oracle-based QA team does functional testing of Hudson core along with the most important plugins. From my own experience with good software testers, I know about the huge positive effect this can have on a product.

Features: Release 2.2.0 comes with a job templating mechanism ("cascading") that’s going to be extremely valuable for handling large numbers of similar jobs. I guess if they spiced up the Nested Views plugin, too, this would make the Cloudbees offering pretty much pointless to a lot of users.

Conclusion

The Jenkins project takes pride in frequent releases and its large number of plugins, but honestly, I couldn’t care less. CI is a critical part of our infrastructure. If it doesn’t work, we’re in serious trouble, so I choose quality and stability over features any day. In fact, I would jump at the opportunity to disable features on the job configuration page.

Given its bad track record with Open Source, I don’t trust Oracle, but I haven’t seen enough from Cloudbees to trust them, either. Sonatype has a better reputation though, and so does Hudson’s new home, the Eclipse Foundation. Since Sonatype offers a Hudson-based product, I’m confident that there will be continued development.

To sum things up, I’m leaning towards Hudson. The Hudson project makes a more professional impression, looking like a well-run enterprise IT project rather than your typical open source project. In my opinion, their focus on quality makes it a better match for our use case.

Advertisements
This entry was posted in tools and tagged , , , , . Bookmark the permalink.

9 Responses to Hudson vs. Jenkins Revisited

    • mafr says:

      Hmm. The article’s main argument seems to be community activity back in March (9 months ago). I try to make a decision on more factors than just this one.

      Also, I don’t like the fact that the author works for Cloudbees and doesn’t disclose it in the article. In my opinion, that’s pretty bad style and it somewhat reduces my trust in the numbers presented (and also in Cloudbees).

  1. gorlok says:

    I had used Hudson in the past. But now we use Jenkins and we feel happy with it. We have ~200 jobs. Mostly Maven 2/3. We use Jenkins for deploy, testing, build, publish to artifactory, update sonar, etc. I don’t think we will change it any time soon. CI is critical for us too.

    What do you means with “rather than your typical open source project”? Every project (even open source projects) is unique.

    Anyway, both options are good. It depends on your priorites.

    • mafr says:

      With typical I mean that usually, open source projects are pretty low on processes and quality assurance. That’s where they differ from corporate projects and that’s why they can be so much fun :)

      Other departments in my company run Jenkins instances, too, with some setups being a lot larger than ours. On the other hand, they have dedicated resources to evaluate new Jenkins releases and write patches.

  2. Dirk Olmes says:

    I don’t agree with your statement on the bad QA status of Jenkins. We have roughly the same setup you describe but use the release plugin in addition. Most jobs are free form, though.

    I introduced Hudson around 2007 in the company. Updates were not done regularly but around once per month. When the Jenkins fork occurred we followed the open source version.

    Ever since we started to use Hudson/Jenkins, I only installed a bad version once. I’d say that’s not a bad ratio, though. Comparable to any commercial release, IMHO.

    • mafr says:

      Maybe we hit a bad streak of releases, but we were not the only ones. Colleagues of mine tell similar stories.

      Our main problem today is that sometimes Hudson loses contact to one of its slaves and we have to kill the slave and reconnect the node. This happens about once a month so we can live with it. I’ve seen distributed applications (really expensive ones) that were a lot less stable :)

    • Brian says:

      @Dirk says:
      “When the Jenkins fork occurred we followed the open source version”.

      Are you implying that Jenkins is “the open source version” ? Both Jenkins and Hudson are open source and both are under the same license.

  3. jwong says:

    thanks for this post. it sums up the issues pretty well; we’ve been pondering the same dilemma, and decided for the time being to stick with hudson, primarily in hopes if optimizing maven integration and improvement in the job cascading feature, which is exciting but doesn’t work quite well enough yet to make it useful. Now that it’s 6 mos later, are you still sticking with hudson?

    • mafr says:

      We’ve set up a Hudson test instance for evaluation that works pretty well, but we haven’t switched it to production yet. I’ll post a follow-up as soon as we’ve really taken the plunge.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s