Anti-pattern: The release vehicle.

At my current client site you cannot get a piece of compiled code into production unless you can find an appropriate ‘release vehicle’, i.e. a planned high ceremony release of the component which has been officially prioritised, scheduled and funded. (Note: The same does not apply to non-compiled code such as JSPs or XML templates containing complex XPath expressions).

Somebody very clever, who probably had a beard (Grady Booch?), once said that "Regular releases into production are the lifeblood of the software development process.". I agree. My current client also seems to be in agreement but cannot extract themselves from the constraints their existing processes.

The client in question has a successful agile adoption. Walking round the development teams you see task boards, burn downs and SCRUM meetings. Go to a management meeting and you’ll hear them talk about two week iterations and the importance of continuous integration. At a strategic level, the organisation (which is very large) is still waterfall orientated. This has implications for the way in which work is financed. Funds for the development, testing and deployment of a certain application are released on waterfall inspired milestones. This, in conjunction with a legacy of long development cycles has led the this ‘release vehicle’ anti-pattern.

The organisation has an unwillingness to make a deployment of a component into production unless there is named and funded change request which covers its release. Activities within development, possibly funded internally as ‘business as usual’ do not have such CRs. Therefore, a development activity such as refactoring for technical debt reduction or improving performance might get engineering buy in but will not get released into production until some CR happens to touch the same application.

It is common to see refactorings made which then sit in source control for literally months as they wait for an excuse to go live. Medium to low priority defects or useful CRs which lack very high prioritisation from marketing never get executed because the programme manager does not have a release identified for the change.

The application suite can appear inert to external parties as it takes a considerable period for changes to make it through the full release cycle. This erodes confidence. If I was a product owner and saw that a team was taking six months to execute my minor change I am not going to be inclined to believe that the same team can turn around my big important changes quickly. I am going to be looking for other mechanisms to get my changes into production and earning money quickly. Once I find a route that works I am going to keep using it.

Why do people like the release vehicle?

  • It is the way the whole software lifecycle as exposed to the rest of the organisation works. The QA team don’t test a component unless they have funding from marketing. Marketing won’t be paying for something that has no role in a prioritised proposition. The Operations team won’t support the deployment actives for our component if they don’t have the cash from the same marketing team.
  • It looks like it is easier to manage for PMs. Releases (because they are infrequent) are a big deal, involve lots of noise, planning, disruption to everyday working pattern.
  • It reduces the infrastructure costs. It costs resource to make a release unless every aspect including testing and operational deployment is fully automated (and even then there is potential cost, dealing with failures etc.). It costs resource to automate a manual build process. Engineers appreciate that fully automated build processes are a priority because in the end they reduce costs and increase agility. It is that age old problem of trying to convince not just the build team, but the build team’s manager and the build team’s manager’s manager that it is worth diverting resource in the short term to fix a problem in order to make a saving in the long term.

** This is a symptom of our strategic failure to get agile adopted beyond the development group. Until we do so, we will continue to hit these sort of issues.

What we should do instead:

We should schedule frequent (bi-weekly, ideally more frequent) updates in production from the trunk of source control for every component. We should not need an excuse for a release. The release process should be as cheap as possible, i.e. automated build, regression test, deployment and smoke test. The code in the trunk is supposed to always be production ready and the automated tests should keep it that way.

If we achieve this we should:

  • Reduce complexity in branch management (no merging changes made months ago).
  • Avoid a massive delay between development and deployment which is not cost effective and makes support very hard.
  • Increase our perceived agility and responsiveness.
  • Enable refactoring to improve non-functionals (stability, latency, dependency versions, capacity).
  • Prevent a release from being a ‘special occasion’ which requires significant service management ceremony.

If you release all the time everybody knows how to release. If you release twice a year every release involves re-education of the teams involved on deployment, load testing, merging etc.. etc. This increases the cost and risk that it fails.

Note: Having frequent, regular, low ceremony releases is greatly eased by having a fully automated build and deploy process but you can have one without the other. As stated above, having such a build process makes regular deployments to production cost effective but is an enabler rather than the justification for this change to working practice.

One Response to “Anti-pattern: The release vehicle.”

  1. Dean Warren

    I agree with you general thrust. However, how do we address the organisation issues? I can see ways around the budgeting problem, but the waterfall/ milestone based attitude, coupled with a general risk aversion – especially to things we don’t understand, like “change” and “technology” – is a difficult nut to crack.
    I think that waterfall makes people feel like they are in control of the process: they have time to understand what’s going on. I, personally, find this approach distracting, because it brings focus on what is being built rather than the benefits it will deliver (outputs verses outcomes – management should be interested in the outcomes). Agile, of course, can be used to deliver measurable benefits in an incremental manner, therefore you think it would be embraced by programme managers and business leaders alike.
    The move to a more agile approach and your vision of almost continuous release requires a more “empowered” environment – a threat to some managers. I would argue for a more goal/ objective based approach with devolved budgets based on longer term goals rather than discrete projects. Based on these goals, the product backlog can be fed based on individual departmental objectives – hopefully getting around the problems of convincing a marketing executive that your “invisible” technical change is actually important to the organisation.
    While we are cracking this nut, there might be some pain – someone has to have the vision to lead an organisation through that, and if the people at the top can only see as far as the next quarterly and annual results that might be a difficult vision to sell.
    Thanks – your article made me think about this… perhaps you or one of your colleagues has come comment to make on the broader business culture change aspect of this – I’d be interested to read about Valtech’s experience with this.

Leave a Reply

  • (will not be published)

XHTML: You can use these tags: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>