Royal Society Ignores High-Quality Evidence and Embraces Politically Acceptable Conclusions

This week saw the publication of a suite of systematic reviews by the Royal Society (RS) on the effect of non-pharmaceutical interventions in the pandemic. 

Politico headlined with ‘Top review says Covid lockdowns and masks worked, period.’ The Guardian led with ‘Lockdowns and face masks “unequivocally” cut the spread of Covid, report finds,’ and the i newspaper stated: ‘Masks and social distancing did reduce Covid infections, new report shows, proving lockdown sceptics wrong.’

So there you have it, a slam dunk, sceptics, you were all wrong. You should have masked up and stayed in lockdown.

Even more so when you listen to the Chair of the report’s group, Mark Walport, who said: “There is sufficient evidence to conclude that early, stringent implementation of packages of complementary NPIs was unequivocally effective in limiting SARS-CoV-2 infections.”

Four systematic reviews informed the effectiveness of non-pharmaceutical interventions in the Covid pandemic. However, here is some of what these reviews report.

A systematic review on environmental control measures:

Many of these studies were assessed to have critical risk of bias in at least one domain, largely due to confounding factors that could have affected the measured outcomes. As a result, there is low confidence in the findings.

Testing, contact tracing and isolation interventions among the general population on reducing transmission of SARS-CoV-2:

One study, an RCT, showed that daily testing of contacts could be a viable strategy to replace lengthy quarantine of contacts. Based on the scarcity of robust empirical evidence, we were not able to draw any firm quantitative conclusions about the quantitative impact of TTI interventions in different epidemic contexts.

Effectiveness of face masks for reducing transmission of SARS-CoV-2:

We analysed 35 studies in community settings (three RCTs and 32 observational) and 40 in healthcare settings (one RCT and 39 observational). Ninety-one percent of observational studies were at ‘critical’ risk of bias (ROB) in at least one domain, often failing to separate the effects of masks from concurrent interventions.

Effectiveness of international border control measures during the COVID-19 pandemic:

There is little evidence that most travel restrictions, including border closure and those implemented to stop the introduction of new variants of concern, were particularly effective.

The report makes the same errors that the UKHSA and Public Health England did. They ignored the critical biases and the confounders when drawing conclusions. Some of the comments misunderstand the evidence required for making healthcare decisions.

Chris Dye, Professor of Epidemiology at the University of Oxford, who led the review on masks for the Royal Society, said if they had only looked at randomised controlled trials, they would have come to the same conclusion as the Cochrane review. However, the researchers behind the paper released Thursday chose to analyse a larger body of studies and found strong evidence that masks work.

So, if we ignore high-quality evidence, we arrive at the conclusion we want – they fully understand the politics. Low-quality evidence means the estimated effect will differ substantially from the actual effect – we’ve known this for quite some time, and it is fundamental to the delivery of evidence-based interventions. An approach that uses low-quality evidence shouldn’t inform healthcare, and it doesn’t. That’s why we have NICE, which uses the best available evidence to develop recommendations that guide health, public health, and social care decisions. 

Did the reviewers, for instance, ask if there was a protocol for any of these studies – something we have previously pointed out? There were none, despite protocols being essential for robust research.

There is something we do agree with in the report, that the “future assessments should also consider the costs as well as the benefits of NPIs, in terms of their impacts on livelihoods, economies, education, social cohesion, physical and mental well-being, and potentially other aspects.” However this report looked at none of that.  The single focus on one outcome, ignoring harms, further hinders informed decision-making.

The RS report wants us to believe that RCTs are impossible during a pandemic: “While RCTs should not be discounted, it is highly likely that most information in a future pandemic will continue to be observational.”

Yet the pandemic has reemphasised the importance of high-quality randomised clinical trials and highlighted the need for preparation, coordination and collaboration. 

The Royal Society review shows that some academics are losing their ability to think critically. Instead of retrofitting evidence to preconceived conclusions, it would be much better to report the uncertainties and set out those questions that need addressing. Refusal to acknowledge uncertainties does a disservice to society and undermines public trust in research.

Staying at home decreases your risk of all sorts of hazards – in the short term, you won’t get run over and you’ll reduce the risk of an infection or an accident. But what matters is the costs of what happens when you reemerge.

A report has found social distancing and wearing face masks “unequivocally” reduced the spread of infections.

Professor Carl Heneghan: “There’s a mismatch between the conclusion and spin of it… they’re not being critical anymore, it’s a disservice to science.” @JuliaHB1 pic.twitter.com/EfET0E4241— TalkTV (@TalkTV) August 24, 2023

Reprinted from the DailySceptic