Testing culture
May 2012
++
Never patch release your code. Instead, remove any external dependencies, keep your master branch militantly clean, move your QA process to before you commit your changes, and always release HEAD.
Yesteryear
I spent a long thinking patch releases were the done thing, but now I realise they just mean you've got a broken development process.
Yesteryear we worked like this.
A team of people write some code and then commit it to a code repository. And lets say it then gets deployed to an integration environment, and once everyone is happy with that then they deploy it to a staging environment and on to production. And you tag that release version 0.1.0.
Lets say you then decide to add a feature to that code or fix a bug, so you start writing code on top of version 0.1.0. That's fine, and you release that code in the same way as above, and you call that version 0.2.0, and then you set off with a plan to build version 0.3.0.
But wait! Someone has spotted a serious problem with version 0.2.0.
So we've got a choice to make if we want to fix that problem:-
v0.1.0 - version last known to have worked. v0.2.0 - version in production now. v0.3.0 - version on integration environment.
Do we, a) rollback to the last known good version 0.1.0, b) patch the code in the production release, and called it version 0.2.1, or c) put the fix in version 0.3.0 and release that ASAP?
The old me would pick either option a) or b).
Probably a) if it were a serious problem spotted in the minutes after release or b) if it was a problem that could wait a few hours to resolve, but looking back at I realise now these options highlighted serious deficiencies with our development process.
The best option is always c).
Not picking c) meant past projects fell in to one or all of these traps.
Problem #1 - Clean Trunk
We couldn't release immediately because we had code in our master branch (or trunk) we weren't confident of putting in to production, so any release was blocked until this was resolved. In other words we merged code in to master that nobody had taken a second look at or that we didn't have sufficient tests around it to prove it worked.
This is easy to fix. Firstly by writing more tests at various levels (unit, integration etc.). And secondly adopting various development approaches (pairing, peer reviewing etc.) that socialise the code before it's merged in to trunk. Being militant about what's merged in to the master branch is important here. Have high standards.
Problem #2 - Dependencies
Sometimes not being able to release was because we had dependencies that needed to be released first.
At the very beginning of a project, say, release v0.1.0, this is often impossible to avoid, especially where new systems are being created, but after that point a dependency you can't quickly turn off will slow your release process down to the slowest denominator, blocking any changes that need to be made more quickly.
So, it's not always possible to plan for dependencies not being delivered but having a means to stub things out (feature toggles etc.) is the idea here.
Problem #3 - QA
Having non-automated process after the code is merged in to master prevented us from releasing quickly. This is most apparent in the QA process.
Most projects employ a tester. It's very easy to see this person as a backstop - something that happens after the programming stops and before the code gets released. But this is lazy and another blocker to a seamless release process.
Developers should care enough about the quality of what they are producing to check it works perfectly before they dump it in to the source code management system and in front of users - both by writing integration tests or pairing with someone on the team to walk through the change (ie. HCI, testing the change with different use cases).
Sometimes we found it was hard to test something locally - Eg, you can't connect to a production database, or emulate a state of some dependent system, or physically connect a test device to the developers virtual machine - and committing the changes to the integration environment was the only reasonable way to test them. But this breaks out clean trunk rule (above), so spending time removing these problems yields an environment where nobody has excuses for not testing something.
Rather than employing an army of testers to plough through the work the team has done at regular intervals, use them to impart their expertise to developers to help them understand context in which the code with be used. Help them understand how to test.
And as for people who like writing code but think it's some one else's job to check it for them. Well, try to avoid that.
Master
The advantage of the above three principals is that you avoid the hassle of patching, which isn't so much a hassle in practical terms (although typically around our way is does cause more effort that I'd like), more a bad process smell. We've tried to keep to these rules while building <a href="http://m.bbc.co.uk/news%mdquot;>m.bbc.co.uk/news.