Saturday, 3 April 2010

Continuous Intergration

It's about time I started blogging about CI again, given that I have developed and worked with some of the best CI systems I have had the pleasure to encounter.

When I first got involved with CI it was with Java development. At the start we simply used Ant to carry out our builds and tests on remote servers. As time progressed, we started to look at some of the CI build tools that were available to us. We came across Cruisecontrol, which I think was the tool de jour when it came to build servers. I was naturally impressed with the features offered by Cruisecontrol at the time. We were able to put together a simple CI environment to carry out our builds and/or churns. As time progressed, we looked for ways to integrate Cruisecontrol into our main repository - the company I was working with at the time used Perforce as it's SCM, we also had a tool called the repository that held all of our application meta data (we made mobile phone apps, so the meta data was a collection of definitions per handset and carrier). By the time I left that company, we had almost a complete integration of our build environment with our repository and SCM.

From here, I moved to a .Net software house. I brought with me the concept of Cruisecontrol and CI. My new employers could be best described as a group of energetic younger people who worked on the bleeding edge. This time round, it wasn't mobile phone apps, instead it was the ultra serious realm of financial software. When I joined this company, the guys had just put the finishing touches to a migration to .Net, easy to implement but unforseen issues cropped up - that is a story for another day ;). These guys were up for anything when it came to CI and automated testing. I introduced them to Cruisecontrol and Ant for creating a CI build environment, the head of development (my boss) at the time was very interested in implementing a unit testing regime for the newly migrated .Net code. I remember working with an intern who had been tasked with creating a set of guidelines for unit testing within the integration code, this was about the time I started looking at Cruisecontrol.Net and Nant.

Given that my new place was a .Net house, it made little sense to me to implement a suite of Java based tools when there were viable .Net versions available. With this said and done, the company branched out into the new realms of .Net CI! It was a success from the get go, the upper management liked the idea of seeing nice graphs and reports on build status, the developers like the idea of being told when a specific build passed testing - it allowed pretty much anyone in the developer and analyst pool to see what was happening with their build and others.

I feel that good CI is often based on some simple foundations. First off would have to be good code. It doesn't matter if it doesn't pass the build at a lower level, those guys should be used to new build to go through some teething problems with production code, this is something that CI would come to reduce further down the line. Next, we needed a decent environment in which to build our code. At the start, we used Crusiecontrol.Net on one builds server, but we had another three servers available to us should we need them. As unit testing was rolled out for all of development, this allowed everyone to see problems before a build made it out onto a test environment. Naturally, this annoyed some developers who didn't see the point in unit testing their code, being in the build development team I could see the importance of unit testing and so could my colleagues. The next important thing that makes up a good CI environment is Software Configuration Management. I have worked with some excellent SCM's and some truly awful (read Visual Source Safe). At this point the company was using a tool called Surround (made by Seapine). It was a fairly good SCM, a little clunky and had some performance issues, we needed to move away from it however as we needed to get to a point where we could run multiple projects from a common library of assemblies. Surround wasn't really up to this. Being a .Net house, we had access to all of the latest tools from MS, naturally Visual Studio was in there. At around about this point in time, MS was starting to promote a tool called Team Foundation Server, this was essentially MS's take on CI. It came with all all the bells and whistles that a good CI system should offer, there was built in SCM, built in reporting, built in data collection and the ability to carry out Team Builds (MS's take on a Cruisecontrol style too). It also came with a tool called MSBUILD, which I like very much. MSBUILD was the replacement for NMake, I am going to assume that most people reading this article will be familiar with Apache Ant and/or Nant. MSBUILD borrows a lot from these tools. Essentially, it is a XML based script that allowed you to automate a great deal of pre-build, build and post-build activities. At it's simplest, it allows you to carry out a build on the command line.

Now, when it came to the CI system that I built, I didn't want to use MSBUILD, at the time it seemed to constricted for me, instead I turned to what is probably MSBUILD's biggest rival, Nant. At the present time, both these tools are pretty much identical in what they can do and how they are built and maintained, back in the day though Nant offered slightly more flexibility over MSBUILD.

Now it came down to the system that I built. One of the strong points of Nant (and MSBUILD) is that you can write your own tasks for your build/test configuration. This is great when you need to add some small things into the config - this could be something as simple as plugging in a third party tool etc. Both of these tools offer a lot of functionality from the get go, all I had to write for the first CI system there was integration with our SCM, I needed to create some Nant tasks that would work with our SCM.

When we first went live with this system, there were some complaints. These mainly came from the developers working on new build, they would now be informed of breaks very early on in the build process automatically, Cruisecontrol.Net allowed you to do this. There was some resistance to unit testing also at this time, some developers simply did not want to see their code fail at such an early stage. Both the head of development and myself held the same view - unit tests are a must have as they find issues way in advance of a build making it out into test, where an analyst may or may not find the issue. Also, the build development team was also subject to unit tests, so across the entire company unit testing became a must be done (especially when managerial start looking at lovely reports that showed them test coverage and pass rates. A lot of the developers that complained were of the "just get it to run" frame of mind. Thankfully, these developers were vastly in the majority.

As time progressed, we found that Cruisecontrol.Net was beginning to be a little constrictive for us. We were only using it for build and test for the entire company (read multiple projects) on one build server. By this time, upper management was extremely impressed with what had been achieved with the setup, it has sped up the build and test time across the board. When companies are successful, they often start branching out to get more clients to use their systems, this was no different to the one I was currently working for. It was financial software, so the money to be made from efficient development was astronomic. As the shape of the overall development changed, so did our attitude to building it changed also. With the current setup, we couldn't handle these sorts of builds on a daily basis, at this point we were running CI for one workstream only and the system would not be able to cater for different builds for different projects. We were under using the build environment we had, we only used one machine out of the four available to us. We knew we needed to change our build tools and management was concerned that we would loose development time having to migrate from one system to another. We quickly put that fear down by explaining that the build scripts we were using were not dependant on the CI system, in other words, the Nant scripts we used for all of the builds would work with any CI tool. This brought a sigh of relief to many a manager a team lead. Right up until this point, we were running a complete build and test for the new build for just one customer automatically. This would take place all the way through the day at scheduled times, this meant that developers could work away to their hearts content pretty much 24/7 (and some did come close to this).

This made moving to a new CI system fairly painless, it also allowed us to see what we could do for the build, not just a test and build but hopefully a test, build, deployment and further testing. A complete lights off system that would roll back to a working build should the current code fail.


  1. You might also want to look at Parabuild for Continuous Integration. It is a bit more stable than CruiseControl and it is free for small teams.

  2. Agreed, have just downloaded it to compare it to Team City and CC.Net. This is definitely a refreshing alternative to CC.Net.

    I also took a look at DevEnv, which is completely free and something I could very well end using myself some time soon.