A closer look at the Johns Hopkins lockdown study

On Thursday, February 3, American Experiment reported on a new study by economists at Johns Hopkins University that finds the body of empirical evidence shows lockdowns had “little to no effect on COVID-19 mortality.” That’s obviously a big deal and a big story. It runs directly counter to the leading narratives used to justify the burdensome school closures, business shutdowns, and shelter-in-place orders.  

No doubt this study will be met with criticism. No one is going to be comfortable admitting the devastation wrought by these orders might have been unnecessary.

If it’s not met with criticism, it will likely be met with silence, which is exactly what Fox News is currently reporting. Their analysis of Grabien transcripts shows “a full-on media blackout of the new study.” Television networks CNN, MSNBC, ABC, CBS, and NBC all ignored the study, as well as major print and online news outlets, including The New York Times, The Washington Post, The Associated Press, Reuters, USA Today, Axios, and Politico.

Yet, this study cannot be ignored. Mounting the best response to COVID-19 and any new epidemics in the future depends on taking an honest look at the last two years. So, it’s well worth taking a closer look at what the Johns Hopkins economists found.

A solid methodology

The study rests on a solid methodology that follows a traditional meta-analysis framework. It begins with a systematic search that identified 18,590 studies, then screened those studies based on the strength of how the study established a relationship between lockdowns and mortality. 

The “difference-in-difference” criteria they use focuses on studies that measure the difference in how mortality changed after the fact compared to a control group (i.e., a geographical region that did not lockdown in the same way). This offers the best measure of what actually happened. Thus, they excluded studies that involved before-and-after comparisons, simulations, and counterfactuals based on modeled forecasts. Ultimately, this process winnowed the number of studies that qualified to be included in the meta-analysis to 24.

Each of the studies was then placed into one of three groups. The first group assessed effectiveness based on a lockdown stringency index of Oxford University that measures the strength of the lockdown across nine parameters, such as closing schools, stay-at-home requirements, and travel bans. The second group included studies on the effectiveness of shelter-in-place orders, and the third included studies of specific orders.

When evaluating the studies, the authors considered four quality dimensions. First, peer-reviewed studies were considered higher quality than working papers. Second, studies assessing longer periods that ended after May 31, 2020, were considered higher quality because they covered the first wave more fully. Third, because about three weeks passes between infection and death, studies that identified an early effect on mortality sooner than 14 days after the lockdown policies began were considered lower quality. Fourth, social science studies were considered higher quality than natural science studies because socialist scientists have more expertise in evaluating policy interventions.  

Results show little to no effect on mortality

What did the Johns Hopkins authors specifically find in these 24 studies?

The lockdown stringency index studies offer the best assessment of whether a stronger, more comprehensive lockdown reduced mortality. Among the seven studies in this group, only one finds a substantial effect. Considered together, the meta-analysis of the studies finds that lockdowns across Europe and the U.S. reduced COVID-19 mortality by just 0.2 percent.

There were 13 shelter-in-place order studies; the Johns Hopkins analysis finds that these orders were also ineffective, and reduced mortality by only 2.9 percent. Notably, five of these studies actually find shelter-in-place orders increased mortality by a small amount. They posit this could be due to asymptomatic people with COVID-19 who infect other family members when forced to isolate at home.

The remaining studies of specific interventions found no evidence that complete or partial lockdowns, school closures, border closures, and limiting gatherings had any “noticeable effect on COVID-19 mortality.” They did find some evidence that business closures reduce mortality, but this is likely related to bar closures. Two studies suggest a possible effect from mask mandates, but, as the authors note, there were only two of these studies and one was limited to employer mask mandates.

Why no effect?

The authors close with a discussion on possible reasons why lockdowns had little to no effect on mortality. They first note how people will social distance themselves when a “pandemic rages” regardless of what the government mandates. They reference one study showing voluntary behavior is 10 times as important as mandated behavior. Second, mandates impact only a fraction of the contacts that can spread COVID-19. Third, people might also respond by taking more risks of exposure when the initial high-risk wave subsides. Finally, they note how “unintended consequences may play a larger role than recognized.” This includes forcing asymptomatic contagious people to isolate with family, as noted earlier. They also point out that lockdowns can restrict people from meeting in safer, outdoor spaces. 

Quick criticism from likely sources

Three scientists already posted negative “expert reactions” on the Science Media Centre website, alongside one positive reaction. The critics include Imperial College of London Professor Neil Ferguson, whose simulations helped drive lockdown decisions. Recall, his simulation projected 2.2 million Americans would die if no action was taken. Another of these critics includes University of Oxford Professor Seth Flaxman, who published a widely cited paper in Nature that estimated 3.1 million deaths had been averted in Europe due to lockdowns and other interventions.

The Johns Hopkins economists referenced the work of both professors. They noted that the Ferguson “predictions were questioned by many scholars” in their discussion on what motivated their research. Also, they highlighted why the Flaxman study did not meet their criteria for inclusion.

Clearly, these critics have a vested interest in opposing the Johns Hopkins findings and so their immediate response must be read with that in mind. Overall, the points they make do little to undermine Johns Hopkins.

They complain that the definition of “lockdown” as mandatory non-pharmaceutical interventions is problematic and overly broad. Ferguson notes the U.K. has been on lockdown since March 2021 because people diagnosed with COVID-19 must self-isolate for five days. Ferguson’s Imperial College of London colleague Samir Bhatt notes this definition would make mask mandates a lockdown. They are correct that it’s a broad definition. But that’s exactly why they split the studies into three groups with the third group focused on specific lockdown policies like mask mandates. That makes sure the studies are indeed comparable.

These critics made some other fair points about the nature of studying lockdowns, but they also point that the Johns Hopkins economists address in their study. They note the importance of timing and how lockdowns immediately focus on reducing transmission, while impacts on hospitalization and mortality are delayed. True; that’s why the authors considered studies assessing a longer period higher quality. The critics also note the importance of the combined impact of the lockdown measures. Again, fair point, but that’s why the first group of studies the authors analyze focus on the stringency index.

Any study will suffer from certain weaknesses.  It’s the imperfect nature of empirical research, which is why studies often come to competing conclusions. That’s why a meta-analysis like the Johns Hopkins study is so important. It collects the best research on a particular question and weighs their findings on statistical significance to determine an overall answer from the body evidence. 

Since lockdowns were first imposed in the spring of 2020, people across the world protested. At the time, people warned of the high and devastating cost of school closures to kids’ development, business closures to jobs and the economy, and shelter-at-home orders to mental health. But these widely recognized tradeoffs were ignored in favor of doing something. Well, this Johns Hopkins study now adds to the evidence that doing something may have accomplished nothing at a devastating cost.