Lockdowns Did Not Save Lives, Concludes Meta-analysis

The Covid-19 pandemic measures were a milestone in how modern Western societies restricted freedoms in the face of a new pathogen. It’s fair to say that we panicked in those fateful spring months of 2020. Ever since, the heated conversations, angered populations, lost friendships, and moralistic battles have split societies down the middle.

Back then politicians, partly influenced by poor epidemiological modeling, opted for a set of policies we’ve gotten used to call “lockdowns.” They usually involved various degrees of mandating the closure of public places, that schoolchildren be sent home from school, that employers vacate their premises such that employees wouldn’t physically interact, or strict government edicts that you mustn’t leave your home. 

Two years into this experiment, it’s about time to assemble the evidence. Did lockdowns live up to their trotted potential? Did they “save lives” and “stop the spread” and all the other slogans we painfully heard talking heads sputter?

Many have tried. There are plenty of studies that show no virus-mitigating effects of lockdowns (but much secondary harm). The thing with compiling such lists of studies is that they’re assembled ad hoc, selecting on the result rather than the study itself. Stacking more such potentially cherry-picked studies on top of each other, isn’t really advancing the scientific claim that lockdowns don’t prevent death. It’s amassing confirming evidence for a certain hypothesis rather than comprehensively investigating how the full range of studies measure up. 

To gauge a large and sprawling field, scientists use meta-studies – a kind of methodological studies that systematically search for studies and incorporate their result into a combined whole. Jonas Herby of the Center for Political Studies in Copenhagen, Denmark, Lars Jonung of Lund University, and Steve Hanke of Johns Hopkins have done precisely that for the early period before July 1st, 2020. In ‘A Literature Review and Meta-Analysis of the Effects of Lockdowns on Covid-19 Mortality’, just published as a working paper with Johns Hopkins’ Studies in Applied Economics series, they assemble the evidence that lockdowns averted deaths from Covid-19. 

Since there’s a lot of scope for fiddling with studies that make up a meta-analysis, here’s the full selection strategy that the authors used: 

1. They screened over 18,000 studies, most of which weren’t related to the narrow lockdown efficacy question. 
2. 1,048 studies remained, where most were excluded for not answering the two core eligibility questions:
   1. Does the study measure the effect of lockdowns on mortality?
   2. Does the study use an empirical diff-in-diff approach?
3. Of the 117 studies that remain, the authors exclude 83 that were duplicates, used modeling, or synthetic controls. Structural-break studies weren’t enough, the authors argue, “as the effect of lockdowns in these studies might contain time-dependent shifts, such as seasonality.”

34 studies thus make it into their analysis, and they are divided into three segments: mortality impacts associated with the stringency of Covid policies (following the much-publicized Oxford metric); Shelter-in-Place studies; and studies that target specific non-pharmaceutical interventions. 

Studies like Flaxman et al. in the scientific magazine Nature, which professed millions of lives saved through lockdown measures, are excluded because of their forcing study design: 

“the only interpretation possible for the empirical results is that lockdowns are the only thing that matters, even if other factors like season, behavior etc. caused the observed change in the reproduction rate […]
Flaxman et al. illustrate how problematic it is to force data to fit a certain model if you want to infer the effect of lockdowns on COVID-19 mortality.”

You can’t assume the conclusion you wish to prove. 

Similarly, they follow Christian Bjørnskov at Aarhus University in excluding synthetic-control studies.  Bjørnskov shows that in many such studies, the country characteristics they synthetically create looked nothing like the real-world countries they imitated, and so greatly questioned the empirical numbers derived from such exercises. 

Browsing through the result summary of the 34 final studies is dire reading for the believer in lockdowns (the authors publish a table with a brief description of all). A few show measures that correspond positively with Covid mortality. Of the ones that do find statistically significant results of the right sign (with lockdowns having a negative effect on mortality) the impacts are remarkably small: often single-digit percentages, with several studies reporting results around zero.  

The combined estimates in the stringency studies (averted deaths as a proportion of total Covid deaths) band around zero, with only a single study (Fuller et al. 2021) finding a large impact of lockdowns on Covid-19 mortality. When adjusting the combined estimate for that study’s highly imprecise estimate, Herby, Jonung and Hanke find that the precision-weighted average effect of lockdowns on Covid-19 mortality is -0.2%: 

“based on the stringency index studies, we find little to no evidence that mandated lockdowns in Europe and the United States had a noticeable effect on COVID-19 mortality rates.”

The more precise the estimate and the more clean and comprehensive the study, the closer to zero are lockdowns’ effect on Covid-19. Read that again. When we run the numbers carefully, any initial protective effect from lockdowns on Covid deaths go away. 

The Shelter-in-Place studies don’t fare much better. While the bottom-line figure is a little better (-2.9%), again, most studies show effects that cluster around zero (or low negative single-digit percentages): 

We find no clear evidence that SIPOs had a noticeable impact on COVID-19 mortality. Some studies find a large negative relationship between lockdowns and COVID-19 mortality, but this seems to be caused by short data series which does not cover a full COVID-19 ‘wave’. Several studies find a small positive relationship between lockdowns and COVID-19 mortality. Although this appears to be counterintuitive, it could be the result of an (asymptomatic) infected person being isolated at home under a SIPO can infect family members with a higher viral load causing more severe illness. 

Finally, in the NPI segment we can discern a sliver of vindication for the lockdown argument. The set of studies are a little more scattered as they assess different interventions (schools, border closures, gatherings, mask etc) and thus harder to compare. Still, writes Herby, Jonung, and Hanke: 

“there is no evidence of a noticeable relationship between the most-used NPIs and COVID-19. Overall, lockdowns and limiting gatherings seem to increase COVID-19 mortality, although the effect is modest (0.6% and 1.6%, respectively) and border closures has little to no effect on COVID-19 mortality”

The biggest effect that comes out of this meta-analysis is the effect of closing non-essential businesses, particularly bars, that was associated with 10.6% fewer covid deaths. 

The authors are pretty severe in their final conclusions. Lockdowns didn’t meaningfully reduce Covid-19 mortalities: “the effect is little to none.”

The best case we can make for lockdowns is that the minor impact they may have had in temporarily averting deaths, are not worth the hassle, the pain, the societal upheaval, the misery and human suffering that accompanied them. 

Is anyone responsible ever going to admit that policy error?