The Cognitive Failures of Covid Response

A typology of behavioral mistakes in our pandemic thinking

Why has the Covid pandemic been so bad across many developing countries, the United States included? There have been some important successes — the rapid development of vaccines boosted by Operation Warp Speed, for instance. But mostly our public health response has consistently lagged, especially compared with many countries in East Asia. A lot of this blame is probably best laid at the hands of politicians, who have often avoided accountability as a “rallying around the flag” effect boosted incumbent ratings even in high death toll countries.

But failures in expert advice by some public health experts also played a role. I don’t think the problem is the public health community at large, which overall has done a phenomenal job at generating a lot of knowledge, much of which has directly translated into specific guidelines. But given the scale of our problem, I think it is productive to revisit the decision-making process through the pandemic and try to identify the systematic biases and failures of thought which have made our lives a little worse. There is an inherent Monday morning Quarterbacking aspect to this type of exercise. Still, I think we have learned a lot about weaknesses in our state capacity through this pandemic and we should try to learn and do better. Briefly, these are:

1. Over-emphasis on Peltzman Risk

2. Blank Slate Frequentism

3. Jawboning the Public

4. Forgetting the reasons behind your guidelines

5. Individual v. Population Health

6. Not looking at successful responses in other countries

Let me break these down.

A Typology of Cognitive Failures During Covid

1) Over-emphasis on Peltzman Risk. Peltzman risk is the concept of risk compensation; that people do more risky behavior as their perceived risk level falls. The concern is that if we test for Covid more, for example, people might actually respond by going more lax on other social distancing measures such that we wind up in a less safe situation. This may sound odd, but it’s the rationale offered at UNC for their decision to not go for a mass-testing regime:

UNC health experts said they have the capacity to do [mass testing], but it wouldn’t be productive and has drawbacks.

The virus has a 4-day incubation period, so a person could test negative today, but then test positive tomorrow and that could make things worse, said Dr. David Weber, medical director of UNC Hospitals’ Departments of hospital epidemiology.

“Sometimes it gives people a false assurance of ‘I’m negative so don’t have to follow physical distancing or masking or other protective mechanisms’ ” Weber said.

Dr. Erica Pettigrew, medical director of the Orange County Health Department and the medical director of occupational Health at UNC Health Care, said the “testing everyone” strategy has given rise to some issues in hospitals.

“As we do more and more testing of patients, we see that people may get a little bit more lax in their PPE, in their masking or symptom monitoring,” Pettigrew said.”

We know now this is nuts — UNC’s decision not to go for mass testing wound up resulting in an early outbreak and end to in-person classes, while the mass testing regime places elsewhere did just fine. But this has been a repeated concern about every risk mitigation strategy we have tried during Covid: we can’t wear masks because they lead to a false sense of security; we can’t use tests of the asymptomatic or rapid tests because of the false sense of security.

As it turns out, people have measured the presence of Peltzman risk with respect to mask wearing, and found little evidence that people who are masked up actually do substitute risk and engage in less social distancing. If anything, they appear to distance more — perhaps the very presence of masks makes Covid risk salient to people. Now that particular result may not hold up; and sometimes the Peltzman risk increase may be high. But point is that second-order bankshot effects like this are, well, generally second order. Now, there are going to be some cases where the compensatory risk increase actually happens to be large enough that it outweighs the initial risk reduction. But as a general rule I think it’s a better way to bet that some generic risk reduction strategy will have an effect in the intended direction, with part of the benefit offset by the Peltzman effect.

What’s a little odd is that the medical community generally recognizes this in other cases — people don’t worry that much that encouraging PrEP for instance will result in a countervailing increase in risky sexual behavior — but for some reason the dominant response in our Covid response is a hand-waving assumption that actually good will make us worse off so let’s not have any nice things.

2) Blank Slate Frequentism. Suppose you are trying to figure out the effect of wearing masks. There are two ways to think about this problem. A Bayesian puts some prior probability on the idea that masks are helpful against respiratory disease, based for instance on how effective fabric is at stopping particle flow. Then you update based on whatever data you can find; but if the data is low-quality you basically stick with the prior to form your posterior.

Then, to figure out if you should wear a mask, you also do what Jim Savage refers to as ABIYLFOYP: Always Be Integrating Your Loss Function Over Your Posterior Estimate. Meaning: you look at how costly it is to do the action. Even if you think masks aren’t that helpful; if it’s not that expensive to mask up you do it anyway.

This formalizes some common sense intuition, but is not really how the competing statistical dogma of frequentism works. Under that framework, you instead decide on your null hypothesis — you start by presuming that masks don’t work — and only change that belief if you receive a sufficiently strong signal (that some data realization would be pretty likely under that null). Ideally, you rely on some sort of Randomized Control Trial to generate a “gold standard” evidence base. The downside with this approach is what to do if you have strong, but not statistically significant, evidence that masks are helpful. Under a dogmatic frequentist view, you can’t use any of this evidence: you just failed to reject the null. But you still have to take some action, so the frequentist recommends not masking up — in opposition to your prior beliefs, the weak signal from the data, and the relative benefits and costs of masking.

This is basically the problem with our evidence base on masks. Even at this stage in the pandemic, we still don’t have any rock solid evidence — either from observational data or RCTs — for how effective masks are. We just have lots of good scientific reasons to think masks work; lots of weak evidence from observational data they seem to work; and a cost-benefit tradeoff that says we might as well wear them. So we have gone with masks eventually — but we really wasted months sticking with a dogmatic frequentist prior that we had no “gold standard” evidence masks were effective (here is a trial which would potentially answer the question).

We also saw blank slate frequentism with vaccine effectiveness. For a long time, people were really sticking to a party line that vaccines lower the incidence of severe illness, but we don’t yet have evidence that they actually lower transmission to others. Lower transmission seems to be a the general pattern with most other diseases, and there was some evidence (from both the AstraZeneca trial and the Moderna trial) that people with the vaccine genuinely got infected less often; but still you had people arguing we can’t assume anything since “we don’t know” about transmission, and so need to mask up and distance exactly as much as before.

I wouldn’t take it too far, but there is a sense here in which the availability of knowledge from RCTs — which should just be some unambiguously good thing — somehow results in a systematic devaluation of all other forms of knowledge such that we are somehow worse off.

3) Jawboning the Public. This concept of shading the truth is pretty well illustrated by Fauci’s description of how he described herd immunity to the public:

When polls said only about half of all Americans would take a vaccine, I was saying herd immunity would take 70 to 75 percent,” Dr. Fauci said. “Then, when newer surveys said 60 percent or more would take it, I thought, ‘I can nudge this up a bit,’ so I went to 80, 85.”

We need to have some humility here,” he added. “We really don’t know what the real number is. I think the real range is somewhere between 70 to 90 percent. But, I’m not going to say 90 percent.

The optimistic interpretation here is that Fauci has a range of plausible estimates, and skews to one side of the range to try to achieve a desirable outcome from the public. I think even this aspect of skewing the truth is dangerous and risks losing the public’s trust if they feel experts are engaging in a noble lie to manufacture a desired outcome.

Another example here would be Trump admitted to Woodward he was minimizing the pandemic to avoid causing fear.

4) Forgetting the reasons behind your guidelines. In 1994, a case of the bubonic plague hit Surat, India and there was some economic fallout from the resulting loss of travel and trade. This response was a little excessive, since the consequence of localized plague outbreaks is not that bad. And it threatened to encourage countries to cover up disease outbreaks, rather than report them to the WHO.

This led to the WHO recommending against travel restrictions during Covid, though we now recognize they have wound up being really helpful, especially for countries like New Zealand, Australia, Taiwan, and South Korea that pair travel restrictions with quarantine on the ground. Matt Yglesias has a great discussion of this here — some very specific and contextualized recommendations with respect to a local plague epidemic morphed into a blanket “no travel ban” advice that wound up being counterproductive when applied to a totally different situation.

For the US specifically, another problem was that the reference mental model so many people had in their minds was HIV — a public health crisis that required a very different set of interventions than the respiratory crisis that we actually faced. For many Asian countries, by contrast, more recent experiences in combating SARS, etc. were more top of mind, and local public health departments converged more quickly on a set of targeted interventions — Japan with their 3C’s strategy for instance.

5) Individual v. Population Health. Michael Mina raises this point well:

The standard for treating patients is radically different from treating populations. When treating an individual, you really want to know what’s going on, and only want to recommend the absolute best treatment. But for populations, it’s often good enough to have something that works okay but is better than nothing.

A good example is with rapid tests — many experts have been very reluctant to advocate their use, as they are not quite as good for diagnostic purposes as the full PCR. “The whole idea is to use the right test for the right patient at the right time,” as one expert says. This ignores the whole benefit of rapid tests — they are better at catching people who are currently contagious relative to doing nothing.

Another example here would be California’s decision to temporarily ban vaccinations from one of the vaccine batches due to allergic reactions. These allergic reactions seem to have posed relatively little harm relative to the enormous gains from mass vaccination. But a framework of maximizing individual health leads to an overly precautious framework in balancing risks against benefits. You can also imagine health systems here being overly concerned with the costs resulting from their actions — What if we knowingly send out a vaccine with allergic side effects? — relative to the costs of inaction — What about all of the deaths that would result from a slower vaccine rollout?

These examples points to a broader failure in not adopting the population health framework and thinking about what’s necessary to combat a society-wide pandemic rather than an individual illness. Obviously, public health experts themselves have been at the forefront of developing and spreading the population health framework. But this has been lost a little bit in some of the communication through the crisis.

6) Not looking at successful responses in other countries. You can also see Fauci here, on March 8, scoffing the mass wearing in Asia and saying there is no reason people should be walking around in the US wearing masks and it doesn’t provide “perfect protection.” By contrast, in New Zealand Michael Baker explains that "We looked to other countries... Asia got it right, the Western world got it wrong..."

There seems to be a real strain of American exceptionalism across many of our institutions. To the extent we are willing to look overseas at all, it is to a sample of European countries that are fun to vacation in. But the age of Western exceptionalism in state capacity is long gone. East Asia has become a global leader in many areas of public administration — and it’s not just China, there are plenty of democratic countries to look at.

These countries innovated a really successful series of interventions against Covid — including travel restrictions, enforced quarantines, central quarantine for infected individuals, universal masking, and in-depth contract tracing. We have slowly, over time, adopted aspects of this response in piecemeal. But we have never really bothered to get centralized quarantines, for instance, even though that may cut down the rate of household transmission a lot. We never really bothered to figure out how to get contact tracing to work either.

We have just become fundamentally unserious as a society in learning from international best practices, and complacent if not arrogant in our status quo. This is dooming not only our Covid response, but also our creaking healthcare, ramshackle infrastructure, and poor quality of public administration in general. It’s time for a reverse Meiji in which Americans start to look abroad and take best practices, instead of coming up with endless excuses for why we just have to suffer the status quo.

Value Capture and the 2nd Ave Subway

In other content — I had the great pleasure of speaking with Greg Shill, Jeff Linn, and Chris Severen about a paper on the 2nd Avenue Subway with Stijn Van Nieuwerburgh and Constatine Kontokosta. It was a great wide-ranging discussion that brought up issues about how firms and people can relocate with changing transit improvements, how to think about the role of transportation as having persistent effects on urban development (contrasting with LA), and question of public finance and infrastructure investment.

Part of why you are seeing this boost in newsletter, podcasts, videos, etc. is due to the lockdown — we aren’t able to interact in person, and so are switching to digital tools. I think undoubtedly part of this is temporary and we will appreciate talking to people in person again rather than dealing with a lot of Zoom fatigue. But I also expect we will pick and choose the specific digital tools that make more sense, and offer a way to drastically scale up our effects to disseminate research and information. So I hope this podcast continues — I think it’s a great tool to sample research at the frontier, and the back and forth discussion recalls the best of the seminar environment but in a format open to everyone.