Introducing agile to large organizations is a subtractive process, not an additive one

Working in technology in government, the word “agile” comes up a lot. Usually it’s in the context of agile software development, but it gets applied pretty broadly to other parts of digital government work. I get emails on a semi-regular basis from companies pitching agile training certifications or agile coaching.

“Agile” gets mentioned enough that can sometimes seem like it applies to everything: is anything not agile? In that sense, it’s almost used as a synonym for “good”, which isn’t particularly useful. But there’s a deeper meaning behind it that’s easy to miss, that’s particularly relevant to organizations that have a lot of long-established processes.

What is agile?

In the software development world, “agile” as an idea originated from the Agile Manifesto, written in 2001 by a group of software developers. They were disillusioned with how IT worked in large companies at the time. The Agile Manifesto laid out a simple set of values to do software development differently:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

In the words of the authors (on a beautifully retro website), “while there is value in the items on the right, we value the items on the left more.” It’s a call to software developers – and those they work with – to focus their time on things that matter.

Agile is usually described as the opposite of “waterfall” software development, where the planning and eventual development of a piece of software follows a linear path from concept, to business planning, to requirements and specifications documents, to implementation. Waterfall software development has been abandoned by most of the technology industry, but is still widely used in government and other large organizations.

Don’t just do more things

Working in government technology today might feel similar to how the authors of the Agile Manifesto felt back in 2001, where their companies spent a lot of time on detailed waterfall project planning requirements and reports at the expense of building good software.

Since that time, “agile” has become the byline of a much larger industry – of project management consultants, training programs, software tools, and other efforts to do software development differently. When you hear terms like “scrum”, “extreme programming”, “kanban”, “lean software development”, and others, these are all techniques and approaches that evolved out of similar origins as the Agile Manifesto.

In many government teams, there’s a lot of ongoing interest in “being more agile” or “becoming agile”. This often translates into doing some of the activities that feel like staples of agile software development: daily standups, tracking tickets in JIRA, putting a lot of post-it notes on walls. There’s a whole industry peripherally around government that encourages this, offering training programs and certifications in various agile-themed approaches.

Doing these activities can be motivating, especially for teams that want to modernize how they work – they’re not bad as a first step. But adding these things (daily standups and JIRA tickets, for example) without removing established processes that slow a team down is a recipe for frustration. Being agile means choosing one approach (say, working software) over another (comprehensive documentation, following the Manifesto values above). It’s about deliberately prioritizing what you spend your time on. Just adding more things to do undoes the benefits of agile work.

What this looks like in practice

The IT Strategy Team at ESDC wrote a phenomenal blog post a month ago titled, “We are inadvertently promoting risks in IT”. It describes some of the project management policies and processes in use in their department:

In the PMLC [project lifecycle framework], an IT project is typically expected to produce between 21 and 39 project management artefacts, would require between 48 and 60 project management activities, and would involve 16 stakeholders. Note that in the 16 stakeholders only 5 of them are from the IT Branch.

This heavy project governance process, though it is intended as a means to reduce risks, ends up promoting them instead. Given that the effort of making small IT changes do not warrant the effort of going through this burden, we end up waiting. We wait until we have enough small changes piled up to justify the effort in seeking permission to make them. We inadvertently end up promoting feature bloat.

In organizational environments like this one, adding agile practices without removing some of the traditional waterfall activities isn’t likely to lead to significant change. Doing agile training courses, getting certified, and doing daily standups – but then still having to produce almost 40 traditional project management artefacts, each with dozens or hundreds of pages – isn’t a recipe for success.

In some ways, this is a symptom of how people implementing a process in government often don’t have the authority to change it. It’s easy to start doing daily standups, but to stop doing burdensome project management exercises depends on someone else (often several layers up a hierarchy) giving you permission to work differently.

For people in those leadership positions, several layers up, one of the simplest and most valuable things they can do is to remove established organizational policies and procedures that impose or reinforce waterfall-style ways of working. In most government institutions, there’s no shortage of those.

Why it matters

Adding agile activities but not getting rid of waterfall practices ultimately leads to a lose-lose situation. People get cynical about the agile activities (all the post-it notes, for example) that feel showoff-ish when you’re still only able to ship your software once a year. If you really do have to do all the waterfall practices anyway, then the agile activities can feel like a waste of time. It’s easy to revert to the mean, or to what feels familiar.

In Digital Transformation at Scale (the seminal book on digital government today), the authors touch on this:

As agile has grown in popularity, it has spawned a wide variety of imitators, many of them combining bits of agile and bits of waterfall. ‘Wagile’ doesn’t offer the best of both worlds though; you won’t get the thing you planned on the date you planned it, and you won’t get something that meets user needs either.

Ultimately, they write:

Waterfall works much less well in a landscape where people’s needs and the underlying technology are constantly changing. Agile is a rejection of applying false certainty to delivering policy with technology.

Technology changes very quickly. People’s behaviour, more importantly, is impossible to predict in the way that waterfall software requirements expect. It can only be observed and learned from, and your software iterated on in turn. To do that, you need to ship frequently, test with real users, learn from their feedback, and keep shipping continued improvements. And to do that, you need to stop doing the waterfall things that hold you back.

If you’re interested in learning more, be sure to read “Detecting Agile BS” – a shockingly excellent government publication – from the US Defence Innovation Board, or their other concept papers here.