“Charbonneau Loops” and government IT contracting
Dan Hon’s newsletter (which is excellent) had a great anecdote last week about the New York City Sanitation Department hiring McKinsey to study and design a procurement for trash containerization. Which is exciting (urban planning!) but also, as Dan says, alarming:
This kicked off my usual consultant-consulting-on-RFP threat-response mechanism, this time focussing on: under what circumstances and in what environment is an external consultant needed to design that organization’s RFP for them? …
In general – and in my experience – this kind of thing happens, where external consultants are needed, because a bunch of hollowing-out of experience has happened in favour of outsourcing and contract management.
This reminded me of something I’ve been meaning to write about (since May 2020, thanks Evernote) that’s somewhat prevalent in Canadian government IT contracting. That something is “Charbonneau Loops”, which, yes, is a term I made up.
What are Charbonneau Loops?
Let’s start with a brief throwback to the Charbonneau Commission, a judicial inquiry into corruption in the Québec construction industry that published its final report in 2015 (officially the Commission d’enquête sur l’octroi et la gestion des contrats publics dans l’industrie de la construction).
There were a lot of corrupt-y things that the commission uncovered (envelopes full of cash to municipal politicians, for example) but the part that stood out, to me, was the commission’s recommendation (on page 1,336 of a 1,741 page report) that the Québec provincial government rebuild its in-house capacity to oversee transportation engineering projects (see English translation):
En l’absence d’expertise interne, les donneurs d’ouvrage publics dépendent de consultants externes pour la planification, la réalisation et le contrôle des travaux à réaliser. L’impartition de certaines tâches à des firmes externes ne pose pas problème, lorsqu’elle répond à des besoins d’expertise spécifique ou en raison d’une hausse temporaire du volume de travail à effectuer. Toutefois, le manque d’expertise interne empêche les donneurs d’ouvrage d’apprécier à leur juste valeur les solutions proposées par leurs fournisseurs et d’évaluer le travail accompli par ces derniers. Lorsque la perte d’expertise interne est trop prononcée, ils sont alors complètement dépendants des firmes externes. (p. 1,243-1,244)
Ce rapport met entre autres en lumière d’importantes hausses de coûts et déplore que les ingénieurs du [Ministère des Transports du Québec] soient devenus «des gestionnaires de projet cantonnés dans des tâches administratives » alors que les firmes de génie-conseil préparent « 100 % des estimations relatives aux contrats d’infrastructure routière à Montréal et 95 % dans les autres régions ».
Il peut être avantageux de recourir aux firmes de génie-conseil lorsque les besoins d’un donneur d’ouvrage requièrent une expertise très pointue ou se manifestent durant une période de pointe. La Commission juge toutefois qu’il est primordial de rééquilibrer le recours aux firmes de génie-conseil et de redonner aux donneurs d’ouvrage les coudées franches lorsque vient le temps de déterminer si des travaux doivent être réalisés à l’interne ou octroyés en sous-traitance. (p. 1,336, Recommandation 23)
Up until the 1990s, the Ministère des Transports du Québec had in-house engineers who oversaw construction and engineering work for highways, overpasses, bridges, and so on. From then on, to save costs, the provincial government instead largely contracted out oversight of these projects to other engineering firms.
You can see where this is going. The total number of firms who were capable of either a) doing highway engineering work or b) overseeing highway engineering work in Québec was not that big. On one project, company A would be doing the work, and company B would be overseeing their work on behalf of the government. On the next project, company B would be doing the work and company A would be overseeing it, and so on.
They all talked to each other, and (to paraphrase a lot) found creative ways to cut corners, keep the extra money, and not call each other on it or report it to the provincial government. For a couple of decades.
Overall – minus the envelopes of cash – this is a pattern that you see in other areas of public sector procurement. When the same set of vendors is sometimes doing the work, and sometimes overseeing it, you get a repeating pattern that I call a Charbonneau Loop.
How do Charbonneau Loops happen?
In most cases, I should be really clear, they don’t involve actual corruption. Charbonneau Loops happen whenever you have:
- Insufficient internal capacity to provide oversight directly
- A small pool of established vendors
And, it almost goes without saying, you’re doing work with a sufficiently high level of complexity that it requires active oversight. This includes infrastructure construction projects (like highways), and it also includes IT implementation, client service delivery, and a whole range of other fields.
It doesn’t include everything; landscaping and gardening, for example (with apologies to gardeners!) doesn’t necessarily need the same level of oversight. If done poorly, it likely won’t affect people’s life and livelihood the way a collapsed bridge or failed benefit program IT system would.
In some areas – public sector IT, for example – “oversight” can refer to a range of things. In this context, it could include planning procurement activities, and developing documentation for these; it could include project management and coordinating the work of other vendors; it could include reviewing and approving other vendors’ work; and it could include more traditional oversight roles like security compliance or privacy assessments.
You can see #1 above show up when you see government organizations issue RFPs for “procurement support” (like Dan’s New York City example, above), for different kinds of “design authority” work, to write requirements, or to support project management offices.
#2 is harder to determine, because it depends on a longer pattern that shows up over several years’ of procurement activities. Are there companies that are sometimes implementers, and sometimes oversight providers? How frequently do the same companies show up in different aspects of the same projects?
Charbonneau Loops ultimately happen when the “pool” of companies (receiving public sector contracts for a given type of work) is small enough that the same companies are sometimes overseeing, and sometimes overseen, by their peers in that same pool. Even if they never actually coordinate with each other – even if they don’t have any conversations whatsoever – they’re all incentivized to be a little bit less critical of each other as a result.
Ultimately, each of the companies in these kinds of environments recognizes that – in the next iteration of the Loop – that if they carefully scrutinize and call out poor-quality work when they’re in an oversight role, their peers will take it out against them when their situations are reversed. It’s “better” for everyone (every company, at least) if they’re gentler on each other, and report back that things are pretty fine to the public sector organizations that hired them.
All of this is compounded by the very blurry line between public servants and vendors that tends to occur in these environments. Imagine the boundaries of public service so porous that you couldn’t tell if you are meeting with colleagues or contractors? It’s not only because the contractor has been around at least a year with no sign of leaving, but also because contractors get government email addresses and have access to all internal systems. Some may even hold senior advisor roles to Assistant Deputy Ministers and Deputy Ministers via Interchange programs.
It’s not corruption, but it’s a set of incentives that aren’t aligned in a way that’s favourable to good public sector outcomes.
How do you fix it?
In short, just like the two criteria above, there are two overall solutions:
- Rebuild or increase your in-house capacity (as a public sector organization) to provide oversight directly
- Increase the size of the “pool” of vendors who are regularly participating in procurements
Beyond having sufficient in-house capacity to provide effective oversight, more in-house capacity gives you the option of implementing in-house, reducing or eliminating the need for vendors at all. For government IT work, there are some great pieces from Mark Lerner and colleagues on increasing in-house tech capacity: Should this role be in-house, or outsourced? and The Government Technology Silver Bullet: Hiring In-House Technical Talent. The takeaway from a lot of recent research (including these pieces) is that governments’ heavy dependency on outside companies makes it hard for them to hold these companies to account.
When it comes to increasing the number of vendors participating in government IT procurements, this has been an acknowledged problem for a while. In Canada, burdensome procurement processes make it hard for small companies to participate. Departments are locked-in to existing proprietary software systems, making it difficult to switch vendors despite high costs.
Other countries have undertaken a lot of interesting and innovative approaches to tackle this, particularly the UK government’s Digital Marketplace which had an explicit goal of bringing in small, geographically-diverse vendors that hadn’t previously worked with the government. Being able to cut ties with underperforming vendors (without jeopardizing your project!) is also a critical but very under-exercised set of skills.
When Charbonneau Loops happen in the government IT space, they’re often related to the participation of large management consulting firms. These firms have the expertise to provide project management and oversight activities, and to help design procurements, and they have large IT professional services wings. They also tend to have arrangements with specific large software vendors (for example, a go-to case management suite, a go-to proprietary database vendor, and so on). It’s not surprising that – as departments have accelerated their digital transformation efforts – spending on IT consulting services has increased noticeably.
In data that’s publicly available, it isn’t possible to track the participation of specific firms to specific IT projects at a scale that would illustrate Charbonneau Loop patterns. There’s no smoking gun here (and this blog isn’t the place for it!). Outside of occasional news articles (usually based on access to information requests), it’s rare to know which companies worked on which projects.
Charbonneau Loops are far from the only reason government IT software projects fail. Building custom software is hard. Building custom software that you didn’t know was custom software is harder. That’s partly why being able to hold companies to account – ensuring that their work is functional, reliable, and high-quality – is so critical.
And as departments attempt to modernize both how they work, and the IT systems their work depends on, they tend to rely on the same small set of vendors. Given this, it’s worth keeping a close eye out for Charbonneau Loop situations. And it’s worth keeping in mind that, regardless of any other good advice they might provide, established vendors are not going to come out and say: established vendors are a key structural problem.
These days I have A Lot Of Thoughts about IT vendors! For slightly more academically-rigorous thoughts, don’t miss Prof. Clarke and I’s “Ask Me Anything” session at FWD50 in November!