Rethinking Public Procurement
The Hidden Drivers of High Costs and Strategies for Reform
When you’re reading about government projects, the final cost estimates always seem to have a few more zeros than you anticipate. For instance, a recent report about failures in payroll HR software for California schools includes the following details:
$40m to build a payroll system for the San Francisco Unified School District, and maybe $43 more on contractors to clean it up
$95m for payroll software for the LA school system
Marin county stopping two payroll contracts; one for $90m and one for $30m
The article focuses the in failures of these systems to actually work and the knock-on fees for expensive consultants to fix the bugs as well as resulting litigation. But even if the software worked perfectly; the charges for software contracting here just seem astronomical, especially when you consider that these kinds of payments are presumably duplicated in every school district around the country — just to achieve a simple software task (payroll) which every private firm is able to manage fairly cheaply.
Another annoying education-related cost bloat problem came from the American Rescue Plan, which allocated $122 billion for schools to deal with the Covid pandemic in 2021. However, few schools used the money to upgrade ventilation, or AC systems — and those that did often bought questionable devices of limited use. Despite that cash infusion, many schools lack basic heating, cooling, and ventilation systems which have demonstrable impacts on learning. Meanwhile, just to build one school building these days can easily cost over $50m.
So why are local governments being taken for such a ride here?
I think the basic answer comes from a few quotes in the piece above:
Mahmood stressed that he has great empathy for government entities changing payroll systems, as software must be customized to account for all their intricacies.
“The legacy companies—the SAPs, the Oracles—they know how to navigate the government procurement process.”
The Problems with Procurement
The following sequence seems to be repeated in government all over:
Rather than buying off-the-shelf tools, government agencies put into place exhaustive requirements to comply with bespoke needs and requirements, many of which are “nice to have” rather than essential, or even completely unnecessary.
Governments then do minimal bidder outreach, limiting the number of firms able to meet these requirements.
Therefore only a handful of companies bother to enter the government procurement market, and deal with the hassles of catering to government specification demands, lawsuit risk, insurance requirements, and so forth.
Those firms then earn monopoly rents, stay as contractors in the system forever, and face minimal monitoring. They botch up job after job, charging enormous fees. Little of what they produce passes the “export test” in being used for non-government purposes.
The lack of state capacity and ownership of the task by government agencies compounds the problem. Important jobs get farmed out to costly consultants, while efforts to fix earlier problems compound in the form of delays and changed program specifications which just add additional cost and complexity to the problem.
Breakdowns in governance and accountability, particularly at very local levels, which understanding of and capacity to fix any of these issues.
So there is basically a structural problem of government procurement, and it shows up all over the place.
Infrastructure Cost Bloat
One place these procurement problems have been fleshed out the most in infrastructure, thanks to the work of the Transit Cost Project and others. The basic fact you’ve probably heard is that it costs the United States much more to build infrastructure compared to the rest of the world; the Anglosphere in general tends to have high costs; whereas higher state capacity regions in Continental Europe and East Asia tend to offer the cheapest and highest quality infrastructure.
Now, many of the problems here are not procurement policy specific, but the basic political structure of high-cost countries does aggravate the situation. As a general rule, the US system relies on “adversarial legalism” as Robert Kagan put it to limit state power. ie, the US uses the judiciary, lawsuits, and citizen voice more broadly to limit government, which elsewhere are constrained through bureaucracy and regulation. As Ed Glaeser and others found in their survey, more effective procurement countries actually have less rules in place. Going back to the Progressive Era in the US, we’ve placed more straightjackets on public officials to reduce corruption and do the right things, but the long-term consequence of this kind of micromanagement limit the discretion and oversight really necessary to reduce costs.
So in infrastructure you get to years and decades of delays as understaffed government agencies hollow out in capacity, rely on overpriced contractors to make key decisions, and defer to community activists and local opposition in any project all of which winds up adding costs. Opponents of any project have many recourses through blocking lawsuits and environmental protections which result in proliferating veto points. A simple example here would be the residents in Old Lyme successfully blocking plans to shorten the New York City-Boston Amtrak line.
Procurement adds to all these problems. As Zach Liscow, William Nober, and Cailin Slattery argue; state Departments of Transportation have been steadily losing employees, a sign of poor state capacity, often substituting for high cost consultants. Project scopes for highway resurfacing change all the time, resulting in change orders and additional delays. There’s limited competition for procurement contracts, driven by an annoying process to enter in the system and limited bidder outreach to get other firms involved. It all adds up to escalating costs, even in relatively straightforward areas like highway resurfacing, and these issues seem to be compounded in more complicated projects like mass transit.
Unemployment Insurance Systems
Another software-related failure in government procurement comes in the form of state unemployment insurance (UI) systems. This one has really material consequences for policy: we can’t “right-size” checks for recipients, and so wind up with fairly arbitrary pandemic UI policies dictated by software limitations.
For instance, Oregon’s UI system alone received $86 million to upgrade in 2009; but this isn’t even scheduled to be completed until 2025. These systems are often built on poorly maintained COBOL code, and as of 2019 only 28% of unemployed people received any unemployment benefits at all.
Folks at the Niskanken Center and the Manhattan Institute have various suggestions on reforming the UI system for better coverage and modernization: but a necessary condition in all of this is to reform the basic computational and administrative capacity of the UI system. For instance, the Department of Labor’s UI staff has actually fallen from 145 in the 1970s to 66 today. As a result, the federal UI office is so small it can’t effectively work with state UI agencies, which are left to procure entirely distinct systems coded in byzantine languishing computer languages at enormous cost and poor effectiveness.
Fixing the Problem: Lessons from Space
Things are not all doom-and-gloom though; some government agencies have managed to fight procurement cost bloat. Eli Dourado also has a nice piece on how NASA drastically lowered costs in the ISS resupply program. Federal procurement requires agencies to generate bespoke program requirements that they own, for which the firm charges development costs plus fees. But through the COTS program, NASA set up a structure by which 1) firms were expected to pay part of their own development costs, to encourage them to commercialize the end product; 2) NASA used fixed-price milestone based payments to avoid paying for stuff that didn’t work; and 3) NASA had goal-based evaluation based on the end product (ISS cargo supply) rather than pre-specifying the means. The end product was successful both in achieving the goals of space shuttle supply, as well as boostrapping new firms like SpaceX.
SpaceX in general illustrates how drastic cost reductions are possible, and how problems of procurement, though concentrated in government, also extend to private firms and non-profits. The basic ways that SpaceX drastically cut costs are covered in Eric Berger’s Liftoff, in this post by Trung Phan, and in Walter Isaacson’s biography. There’s no one secret here, but instead relentless questioning of whether requirements and constraints are necessary or extraneously added in the design process; and pursuing cost minimization on each item that’s actually required, shifting to in-person design when necessary and buying off-the-shelf whenever possible rather than relying on bespoke contracts and cost-plus accounting.
The Department of Defense has managed to do similar procurement reform as well (and, maybe relatedly, has similar legal transaction authority as NASA to sidestep the typical procurement process). I haven’t been able to find great sources on it, but at some point in the 1980s the DoD realized that the market for semiconductors was growing so rapidly that their bespoke requirements were starting to get in the way. So they shifted to buying chips off-the-shelf. This then transformed the government from being a specialized customer, to instead just another incremental source of demand, helping to broaden the overall semiconductor ecosystem.
There are other problems with federal procurement as well: expiring budgets lead to use-it-or-lose it dynamics; delays in payment can limit the ability of smaller firms to deal with the government; and insurance requirements for federal contractors often come up as a source of annoyances. Liscow Nober and Slattery also emphasize Disadvantaged Business Enterprise Requirements, which may produce rents for a small class of connected subcontractors without meaningfully improving costs or advancing gender and racial equity overall.
So to recap, there are probably a few straightforward (but politically challenging) ways for governments to revamp procurement and save a lot of money:
Limit extraneous requirements and try to buy existing commercially available products whenever possible.
To the extent you need specialized development, try to use the opportunity to seed a new industry rather than create a separate walled-off procurement niche, and solicit new bidders with transparent criteria.
Monitor existing contracts by adherence to pre-specified goals, don’t micromanage how contractors achieve the end objective, don’t change the objectives half-way through, and oversee the whole enterprise through well-staffed and compensated public servants.
Other #Content
Together with Sabrina Howell, I have a piece looking at tax policies for private equity.
Testing Romer and Romer narrative shocks using LLMs
Joe Gagnon has a nice podcast with David Beckworth; he emphasizes the inflation shock resulting from sectoral reallocation. ie, there was a huge services → goods reallocation over the pandemic, which pushes up prices a lot in goods sectors where demand is really high and industry-level Philipps Curves can be nonlinear. This lines up with some other work thinking about this sectoral reallocation problem and macro shocks.
A neat Odd Lots episode which talks about the impact of rising rates and insurance costs on home builders. I liked the discussion of real estate common ownership in both directions: for good (internalize amenity spillovers on neighboring parcels) and ill (anti competitive price markups)
John Coglianese, Maria Olsson, and Christina Patterson have a nice paper using micro data to look at the impact of monetary tightening in Sweden. If you squint, you’ll see the prices particularly producer prices, move fairly quickly before unemployment adjusts. You see more unemployment in sectors with more rigid contracts, consistent with nominal rigidities playing a role.