Cochrane Fact-Checked with Absurd Results

On February 17th fact-checker Iria Carballo-Carbajal, a neuroscientist by training, but apparently without any education in epidemiology, published a “fact-check” article on the Health Feedback website. In her headline, Carballo-Carbajal makes the following statement: “Multiple studies show that face masks reduce the spread of COVID-19; a Cochrane review doesn’t demonstrate otherwise.”

This article is now being used by social media companies to suppress all references to the Cochrane study. I became aware of this on March 10th when I received a notification that a post by a member of a Facebook group I manage contained “false information.”

The post referred to an opinion piece on the Cochrane review in the New York Times, published on February 10th. The “independent fact-checker” resource referred to was the aforementioned article by Carballo-Carbajal. Getting a fact-check stamp can be a serious issue for a newspaper, no less so for a scientific institution. Therefore it didn‘t come as a surprise that already on March 10th Cochrane editor Karla Soares-Weiser published a statement trying to downplay the study results, incorrectly claiming the study only aimed at assessing the effectiveness of interventions to promote mask wearing, while the clearly stated objective of the study is to assess the effectiveness of the pysical interventions themselves, not only the effectiveness of their promotion.

The same day the New York Times published a piece claiming in the headline that masks certainly work, but for the most part dedicated to smearing Cochrane study author Dr. Tom Jefferson. For example, the article claims Jefferson stated in an interview that there is no evidence that the SARS-CoV-2 virus is airborne, while what he actually says is that there are many transmission routes and further evidence is needed to ascertain precisely how transmission occurs.

This chain of events is a shockingly clear example of how the censorship industry works. It is all the more shocking considering how seriously flawed Carballo-Carbajal‘s “fact-check” article is, factually, logically and ethically.

1.     The strawman

Carballo-Carbajal begins by creating a strawman, in this case a claim attributed to Dr. Robert Malone, referring to a recent post on his blog. Under the heading “Claim,” the purported claim is stated thus: “Face masks are ineffective at reducing the spread of COVID-19 and other respiratory diseases, a Cochrane review demonstrates.” This claim, quoted besides a picture of Dr. Malone, is nowhere to be found in his blog post.

As if this were not enough, Carballo-Carbajal continues, presenting what she calls a “full claim:” Review “failed to find even a ‘modest effect’ on infection or illness rate:” “the CDC Grossly Exaggerated the Evidence Supporting Mask Mandates.”

The problem with this is that while Dr. Malone is correctly quoted in the first part of the paragraph, the second is something he simply does not say in his blog post.

2.     The ad hominem

Carballo-Carbajal then takes it upon herself to attack Dr. Malone, claiming he has spread “misinformation about Covid-19 vaccines”, referring to another article, also published by Health Feedback. Now, in what does the purported misinformation consist, according to that article? The article is a “fact-check” of a Washington Times opinion piece by Dr. Malone and Dr. Peter Navarro published in 2021, where they argue against the US government‘s universal vaccination policy, arguing that it is based on four flawed assumptions. First, that universal vaccination can eradicate the virus, second that the vaccines are highly effective, third that they are safe, and fourth that vaccine-mediated immunity is durable.

Carballo-Carbajal could hardly have been less lucky with her reference. It is now abundantly clear that universal vaccination cannot eradicate the virus, that vaccine-mediated immunity wanes very quickly, even to the point of becoming negative, as infection studies and reinfection studies have already shown. The fact that the vaccines are not “(near) perfectly effective,” quoting Malone and Navarro’s article, is long since obvious; it is in fact the reason why they cannot eradicate the virus.

As for the third point, this is what Malone and Navarro say in their article: “The third assumption is that the vaccines are safe.  Yet scientists, physicians, and public health officials now recognize risks that are rare but by no means trivial.  Known side effects include serious cardiac and thrombotic conditions, menstrual cycle disruptions, Bell’s palsy, Guillain-Barre syndrome, and anaphylaxis.” In other words, they aren‘t safe, they have many known rare side-effects, and this actually becomes clearer as time passes.

In short, Carballo-Carbajal tries to disqualify Dr. Malone by accusing him of “misinformation” about something else than the subject of her article. This is the classic ad-hominem tactic almost universal in “fact-check” pieces. Her failure is spectacular, as all the purported pieces of “misinformation” are now already verified facts.

3.     The argumentation

Carballo-Carbajal‘s main summary (including “Details” and “Key take away”) is the following:

Claims that face masks are ineffective at reducing the spread of COVID-19 based on a Cochrane review didn’t take into account the limitations of the review. While many users presented this review as the highest-quality evidence, the individual studies it evaluated varied greatly in terms of quality, study design, populations studied, and outcomes observed, which prevented the authors from drawing any definitive conclusions. 

Randomized controlled trials are considered the gold standard when assessing the effectiveness of an intervention. However, this type of study can vary greatly in quality, particularly in complex interventions such as face masks, affecting the reliability of the results. In this context, many scientists consider that randomized controlled trials should be seen as a part of broader evidence including other study designs. When taking those studies into account, evidence suggests that widespread mask usage can reduce community transmission of SARS-CoV-2, especially when combined with other interventions like frequent handwashing and physical distancing.

I shall now break up this statement into parts and then verify the validity of each part. We must keep in mind that the source quoted is Dr. Malone‘s blog post, thus any reference to “claims” must be to Malone‘s blog post, which is the only source quoted. References to unidentified sources, such as “many websites and social media posts” must be disregarded for the obvious reason that no references are provided:

1.     Statement: Dr. Malone claims the Cochrane review shows masks are ineffective at reducing the spread of Covid-19.

Discussion: As shown above, Dr. Malone does not make this claim. Instead he claims the study “failed to find even a ‘modest effect’ on infection or illness rate.” There is a critical distinction between claiming A doesn‘t work and claiming A has not been proven to work. The two do not have the same meaning.

Verdict: Carballo-Carbajal‘s statement is false.

2.     Statement: Dr. Malone does not take into account the limitations of the review when making this claim.

Discussion: To begin with, Dr. Malone never makes the claim referred to, but a different claim. This notwithstanding, in his blog post he clearly cites the study authors‘ disclaimer on the uncertainty about the effects of face masks: “The low‐moderate certainty of the evidence means our confidence in the effect estimate is limited, and that the true effect may be different from the observed estimate of the effect.“… “[t]he high risk of bias in the trials, variation in outcome measurement, and relatively low compliance with the interventions during the studies hamper drawing firm conclusions and generalising the findings to the current COVID‐19 pandemic.  It is therefore untrue that Dr. Malone “didn’t take into account the limitations of the review.“

Verdict: Carballo-Carbajal‘s statement is false.

3.     Statement: “[T]he individual studies […] evaluated [in the review] varied greatly in terms of quality, study design, populations studied, and outcomes observed, […] [preventing] the authors from drawing any definitive conclusions.”

Discussion: The study results are clear: “There is low certainty evidence from nine trials (3,507 participants) that wearing a mask may make little or no difference to the outcome of influenza‐like illness (ILI) compared to not wearing a mask (risk ratio (RR) 0.99, 95 percent confidence interval (CI) 0.82 to 1.18. There is moderate certainty evidence that wearing a mask probably makes little or no difference to the outcome of laboratory‐confirmed influenza compared to not wearing a mask (RR 0.91, 95 percent CI 0.66 to 1.26; 6 trials; 3,005 participants). … The use of a N95/P2 respirator compared to a medical/surgical mask probably makes little or no difference for the objective and more precise outcome of laboratory‐confirmed influenza infection (RR 1.10, 95 percent CI 0.90 to 1.34; moderate‐certainty evidence; 5 trials; 8,407 participants).”

Those results are repeated in the Authors‘ conclusions, adding the disclaimer that “[t]he high risk of bias in the trials, variation in outcome measurement, and relatively low compliance with the interventions during the studies hamper drawing firm conclusions and generalising the findings to the current COVID‐19 pandemic.”

This disclaimer is the straw onto which Carballo-Carbajal clings with all her might. But as the lead author of the study has explained, this does not change the results of the study, it only states that the results may be affected by uncertainties arising from limitations of the studies used. In his own words:

“It’s called caution, and it’s called being honest with the evidence that we have found. This is the best evidence that we have” (see reference below).

It looks as if Carballo-Carbajal does not understand the meaning of a disclaimer in a scientific paper; instead she tries to use this to invalidate the study results and back up her claim that masks work, despite the evidence. A disclaimer in a study does not invalidate its results.

Verdict: Carballo-Carbajal‘s statement is misleading.

4.     Statement: Randomized controlled trials are considered the gold standard when assessing the effectiveness of an intervention.

Discussion: The reference on which this statement is based is Dr. Malone‘s blog. While this statement may well be true, inferring that something is generally “considered the gold standard” based on the opinion of one scientist, is a grave logical error.

Verdict: Carballo-Carbajal‘s statement logically invalid.

5.     Statement: Gold standard studies vary greatly in quality.

Discussion: This claim is not backed up by any evidence. It may be true, or it may not.

Verdict: Carballo-Carbajal‘s statement is not supported by evidence.

6.     Statement: Many scientists consider that randomized controlled trials should be seen as a part of a broader evidence.

Discussion: The source for this is an article in The Conversation by three epidemiologists and one expert in primary healthcare. The authors certainly make this claim, but without quoting any reference. Thus, the statement that it is based on the opinion of “many epidemiologists” is simply false. This is a claim made by three epidemiologists and considering the vast number of people with that training, the word “many” is most certainly not warranted. It must be added that appeal to numbers (argumentum ad populum) is a logical error.

Verdict: Carballo-Carbajal‘s statement is not supported by evidence. Its purported relevance is based on argumentum ad populum, a logical error.

7.     Statement: When studies that do not fulfill the requirements of gold standard research are taken into account, they show widespread mask usage can reduce community transmission.

Discussion: It is of course true that by lowering the standard you may get different results, but this statement is problematic, for Carballo-Carbajal seems to draw from it the conclusion that despite the result of the Cochrane review, masks do in fact prevent transmission. This is evident from this passage, toward the end of the article: “A growing body of evidence from RCTs and observational studies suggests that consistent mask-wearing can effectively reduce the spread of respiratory viruses like SARS-CoV-2 in both healthcare and community settings. … For the time being, face masks are another layer of protection in addition to vaccination, frequent handwashing, and physical distancing when the circulation of respiratory viruses is high.”

This means Carballo-Carbajal‘s claim is not only that low quality studies suggest something; the final statement shows that she clearly claims that what they suggest is actually true. This claim is even clearer in her headline: “Multiple studies show that face masks reduce the spread of COVID-19.” A subtle difference on the surface, but an all-important one. It means it is justified to rephrase the original statement as: “When studies that do not fulfill the requirements of gold standard research are taken into account, they show widespread mask usage can reduce community transmission and this is a valid conclusion.”

This brings us to the question of why the low quality studies Carballo-Carbajal quotes were not included in the Cochrane review. Luckily we have a transcript of a detailed interview between the lead author of the study, Dr. Tom Jefferson (JF) and Dr. Carl Heneghan (CH), where this is discussed in detail:

CH. Now look, I’m going to take you to task here. In the author conclusions people are going to read this review and start to look at this and say, look, we’ve got the high quality evidence, we’ve got randomised controlled trials and particularly at the mask level they’re going to say, look, you’re showing in the community this lack of effect, but you start with the high risk of bias in the trial, variation in outcome measurement, and relatively low adherence with the intervention during the studies, which hampers us drawing firm conclusions. Now I push that point because the obvious answer then is to go to all the observational studies where people have done systematic reviews and certainly drawn firm conclusions about what to do. So could you just elaborate on what that means in the context of 78 trials – that’s a lot of randomised control trial evidence – can you elaborate on what that means?

TJ. It’s called caution, and it’s called being honest with the evidence that we have found. This is the best evidence that we have, but unlike some of the ideologists pushing the idea that non-randomised studies, observational studies could give answers, some of them come up with sweeping answers, sweeping statements, certainties, which simply do not belong to science. Science is not about certainty, science is about uncertainty, it’s about trying to move on the agenda, and accumulate knowledge. The use of non-randomised studies in respiratory virus assessment of interventions with respiratory viruses means that people do not understand, those who did those studies do not understand the play of several factors. For instance the seasonality, for instance the capricious comings and goings of these agents, they’re here one day, and gone the next. If you look at the SARS-CoV-2 behaviour in the UK surveillance for the last 12 months its up and down, and it’s just completely independent of any intervention, and also it’s very quickly up and very quickly down. Observational studies cannot account for that. Also, a very large proportion of observational studies are retrospective, and so they are subjected to merciless recall bias; researchers draw conclusions from data that they got from asking questions such as “Can you remember a month ago how many times you wore a mask” or “What you did on this or what you did on the other day” without keeping a diary. This is just simply not science. Inferring meterage, distancing, when the original studies did no such thing. So this is just an endless list of bias which cannot be taken into account by observational studies. And the only way that we have to answer questions is to run large prospective randomised control trials to answer a specific question in a specific population.”

As Jefferson explains here, the limitations of observational studies make it virtually impossible to draw from them the conclusion that Carballo-Carbajal does. Carballo-Carbajal quotes a number of observational studies to support her claim. I will not go through all those here, but looking at a some examples should be sufficient to provide evidence for some of the problems Jefferson discusses, as well as refuting some of Carballo-Carbajal‘s unsubstantiated conclusions.

For example one of the studies quoted, Wang et al, concludes that face mask use by the primary case and family contacts before the primary case developed symptoms was 79 percent effective in reducing transmission. This is a retrospective observational study where the evidence for mask usage is purely based on participant‘s after-the-fact own self-reporting. 

Another one, Mello et al. shows how viral particles accumulate in masks, but Carballo-Carbajal takes this as evidence that “[a]vailable data indicates that mask-wearing is more effective when combined with other control measures, such as physical distancing and frequent handwashing.”

To summarise, Carballo-Carbajal claims that since high quality studies do not prove the effectiveness of masks against transmission, then unreliable observational studies, which are excluded from the “gold standard” meta-review, precisely because of their unreliability, prove what the high quality studies fail to prove.

Verdict: Carballo-Carbajal‘s (rephrased) statement is false. Without rephrasing it is irrelevant.

8.     Statement: The effect of mask usage is greater when combined with other interventions.

Discussion: This statement is problematic. It is already clear from the high quality evidence provided by the Cochrane review that the claim that masks reduce transmission is unproven. This means claiming they add to the protection provided by other interventions must be incorrect.

Verdict: Carballo-Carbajal‘s statement is false.

Carballo-Carbajal begins by falsely attributing to Dr. Robert Malone two claims which he has never made. Those false claims become the basis of her “fact-check.”

She then wrongly accuses Dr. Malone of making false statements regarding a different matter, an ad hominem argument irrelevant to the subject of the article.

Out of the eight claims made by Carballo-Carbajal in her summary, backed up by her main text, four are plainly false, one is logically invalid, one is misleading and two are not supported by any evidence, out of which one is based on a logical error also.

Considering how this seriously flawed article is now apparently used to suppress the dissemination of an important scientific paper, to press the Cochrane editor-in-chief into making false claims about the objective of the paper and downplay its results, and to censor a review of the findings by an important mainstream newspaper, there is clearly an urgent need to act strongly against the so called “fact-checking” industry. The level to which this censorship has escalated is a clear and present threat to scientific research and development.