Not long ago I thought I had discovered the worst misinterpretation of a study on Covid vaccines, but I have just encountered another contender for the title. It was a study about vaccination and traffic crashes.
The authors explain that “[They] tested whether COVID vaccination was associated with the risks of a traffic crash,” and they concluded that “COVID vaccine hesitancy is associated with significant increased risks of a traffic crash.”
They meant much more than “associated.” They meant that vaccine hesitancy increases the risk of a traffic crash, a causal claim, just like the claim that Covid vaccines reduce the risk of death.
How do I know that that was their claim?
They remarked that the study was not a randomized trial, and they used statistical methods to support inference on cause-and-effect relationship from an observational study.
Their inference was false. These are the true conclusions:
1. Their study shows another example of the healthy vaccinee bias.
2. Their study shows failure to remove the bias by the most stringent statistical method.
Let me start with a crash course on association versus causation.
Association is a statistical phenomenon. Causation is reality. During the Covid era, many have heard the statement “Association is not (necessarily) causation,” which is true. But the two ideas are connected. How?
The connection is best explained by using a simple causal diagram, where an arrow means causation.
Two mechanisms can create an association between A (e.g., vaccination) and B (e.g., a traffic crash).
1) A affects B (causation)
2) A and B share a cause, C (confounding)
If A does not affect B, yet they share a cause, A and B will still be associated. That’s one reason why an association is not necessarily causation. A randomized trial eliminates any cause of the treatments we assign (e.g., one drug versus another) except the mechanism of randomization. That’s why we need randomized trials to make strong causal claims. Confounding is gone.
One more point: any arrow may be considered a summary of a causal chain. For example, C → B could represent C → X → Y → Z → B.
End of course. There is an ocean of complicated material, but that’s all we need to know.
The authors of the article know about causal diagrams. They show an elaborated “directed acyclic graph” (left diagram), which is a fancy name for a “causal diagram.”
Amazingly, the variable “vaccination status” does not show up in their diagram, only “vaccine hesitancy,” the name they gave to the actual variable they analyzed: vaccinated or not.
Science is not judgmental, so I replaced “vaccine hesitancy” with “decision” (to be vaccinated or not), whatever its causes might be (right diagram). Then, I added “vaccination status” (A), which is the effect of “decision.” The two variables are nearly perfectly correlated. If I decide to be vaccinated, most likely I will be vaccinated. Likewise, if I decide to not be vaccinated. I am ignoring instances when a person is cognitively unable to decide or there is no access to the vaccine or a physically enforced injection…
As you can see in my diagram, no causal arrow connects either “decision” or “vaccination status” with a traffic crash. No A → B. The only possible causal connection, unclearly alluded to in the authors’ diagram, is through Covid: unvaccinated → infection → fatigue → crash. We may ignore this chain since we know that vaccination does not reduce the risk of an infection, possibly the opposite.
So, why might vaccination and crashes be associated?
By now, you know the answer. They share many causes — C in my diagram — some of which were measured in the study, and many that were not. Based on the diagram, a randomized trial would have found no association between vaccination and a traffic crash, no evidence of any effect.
As expected from their diagram and mine, the authors indeed found an association between vaccination and a crash. The unvaccinated appeared to be at higher risk of a crash than the vaccinated or vice versa: vaccination appeared to protect against a traffic crash. Some of the shared causes operated to create the appearance of a reduced risk, whereas others operated in the opposite direction. The net effect of all shared causes produced pseudo-effectiveness against a serious traffic crash.
That’s another example of the healthy vaccinee bias, of which the authors were aware. More “favorable characteristics” of the vaccinated made them less likely to be engaged in a serious crash, which could have resulted in traffic death, one of many non-Covid causes of death. Those characteristics, which also made them more likely to be vaccinated, reduced their risk of a crash — not a decision to take a Covid vaccine or taking it.
Ironically, the authors checked for the healthy vaccinee bias by using a method called “negative controls.” They studied the association of vaccination with other endpoints on which no effect of vaccination was expected. Yet they failed to realize that the endpoint in their study is exactly that kind of endpoint. A priori, vaccination is not expected to affect the risk of a crash, according to their own diagram and common sense. The “effect” they found was confounding bias.
Even more ironically, hospitalization for injury or trauma was considered a “negative control” endpoint for studies of the flu vaccine by no other than a co-author of a key publication on the effectiveness of Covid vaccines. (I don’t know why he did not apply this method to studies of Covid vaccines. I wasn’t allowed to ask.)
The risk of a crash in the unvaccinated was 1.72 times the risk in the vaccinated, or vice versa: the pseudo-effect of vaccination was a risk ratio of 0.58, or pseudo-vaccine effectiveness of 42%.
Having causality in mind, the authors tried to adjust the estimate by several methods and showed various results. They described the most rigorous attempt as follows:
The purpose of the second propensity score analysis was to be stringent when matching an unvaccinated individual 1-to-1 with a vaccinated individual and excluding cases where any person had a medical diagnosis.
You don’t need to know statistics to recognize intuitively that this is indeed a rigorous method.
Did they get a risk ratio of 1, the true null effect, from their most stringent attempt to remove the healthy vaccinee bias? No, they got 1.63 (adjusted) instead of 1.72 (unadjusted). That’s all that rigorous adjustment has achieved. (Both numbers are technically odds ratios.)
So, when you read reviews of the sparse literature on methods to remove the healthy vaccinee bias, remember this article on vaccination and traffic crashes. Relying on measured variables can fail to remove the bias, and that’s all that we need to know.
The best we can do at this time is explained elsewhere. It is not sophisticated at all, although there is more to explore. The real problem we face is not scientific: the data we need on non-Covid death is usually hidden.
P.S. I was an associate editor of The American Journal of Epidemiology, and my record contains about 200 publications, some of them in the so-called top medical journals. Shall I reformat this post or others on the topic and submit them to a journal to get the stamp “peer-reviewed?”
I gave up long ago.
Republished from the author’s Medium
Published under a Creative Commons Attribution 4.0 International License
For reprints, please set the canonical link back to the original Brownstone Institute Article and Author.
