Flu Vaccine Studies: Pseudo-Randomized or Untrustworthy Results?

Preamble

The validity of a study, or a series of studies, may be questioned by several methods, one of which is identifying unexpected results. Consider the following example, which I used to teach in an introductory epidemiology course.

Having yellow-stained fingers is associated with lung cancer, but we know that the association does not reflect causality. Yellow-stained fingers do not cause lung cancer. Rather, the association is explained by smoking — a confounder — a cause of both yellow fingers and lung cancer.

Suppose a group of researchers reported that the association in question was not removed after adjustment for smoking status. Obviously, something is wrong, either in the data or in the way it was analyzed. Regardless of the reason, we will not trust any inference from that study, including the estimated effect of smoking. It is equivalent to rejecting a witness’s entire testimony after detecting a single relevant lie. That’s an analogy, of course, not a claim about intentional lying by the researchers.

The VISION Network

Each year the CDC reports estimates of the effectiveness of the flu vaccine in the previous season. These estimates are derived from a version of the case-control study that is called the test-negative design. Both cases and controls are selected from people who sought care for respiratory symptoms (outpatient setting) or were hospitalized (inpatient setting). Cases tested positive for the flu; controls tested negative.

Since the Covid pandemic, the largest source of data has been a CDC-associated network called VISION. It is described on the CDC website:

Observational Studies Versus Randomized Trials

Reports from the VISION network are observational studies; they are not randomized trials. People do not flip a coin to decide whether to get the annual flu shot. Those who get the shot differ from those who do not on many attributes, some of which are measured and others are not. Some of these attributes also affect the clinical outcome (i.e., confounders).

It is well known that vaccinated people are healthier on average than their unvaccinated counterparts. That’s the healthy vaccinee effect, which leads to confounding bias. At least part of the association between vaccination and a clinical outcome is not causality. Since the estimated effectiveness is exaggerated, it is called positive confounding.

Some of the confounding, though not all, may be removed by matching or by statistical adjustment. For example, if the unadjusted association shows an odds ratio of 0.5 (vaccine effectiveness of 50%), the adjusted odds ratio might be only 0.75 (25% effectiveness).

That’s not the case in randomized trials. In large enough trials, randomization results in a similar distribution of every baseline variable in the two (or more) arms of the trial. Therefore, the two groups — say, vaccine recipients and placebo recipients — are called “exchangeable.” They are similar in every respect except the randomized treatment. There is no need for any adjustment, and adjustment should not change the estimated effect.

The take-home message is this. In observational research of vaccine effectiveness, we expect smaller effectiveness after adjustment. If the association is largely unchanged, something must have been wrong, or a miracle happened (pseudo-randomization to vaccination status). Miracles don’t happen often, however.

Drowning in Numbers

A study of vaccine effectiveness typically contains hundreds of numbers, and the VISION reports are no exception. Although more information is preferred, it also makes it more difficult to identify questionable results. The difficulty is compounded if we try to examine a series of studies, say, over the years and search for patterns in the data. Referring to the witness analogy, it is equivalent to identifying a lie in a long testimony that contains many details.

Unlike a testimony, however, there is a solution, which I used to teach: reduction. Focus on a narrow set of results about which you have clear expectations; that is, you have a general idea of what the results should show. If you detect questionable results, your task will be completed. You may question the validity of any estimate from the study.

And that’s what I did. I examined the data from VISION for the elderly (age 65+) between 2021 and 2026. I focused on three aspects of the data: 1) outpatient setting and hospitalization; 2) effectiveness against any strain of the flu; and 3) adjusted estimates versus unadjusted estimates.

To simplify, I display the data in three tables, each showing results from two consecutive seasons or a pair of results from the same season (interim and final). The VISION reports have been traced from the CDC website and PubMed.

The First Two Seasons

The VISION network began to report on the flu vaccine right after the pandemic (2021–2022). For that season, they reported near-zero effectiveness in the elderly against an outpatient encounter, and about 30% effectiveness against hospitalization (Table 1). Such a dichotomy is unexpected, and I doubt it was observed before. The effect of the flu vaccine might differ in magnitude between two clinical settings — perhaps being stronger for the more severe outcome — but it should not be around zero in just one setting. Symptomatic infection is a continuum.

Moreover, adjustment for the variables they modeled has made no material difference (a change from 31% to 29%). Did that sample of hospitalized patients happen to resemble two arms of a randomized trial (pseudo-randomization to vaccination status)? Unlikely. Was something inherently wrong? That’s the likely explanation.

In the following season (2022–2023), the estimated effectiveness was similar in the two settings: about 40% (Table 1 above). Again, however, adjustment has made no material difference (only minor attenuation, 46% to 41%, for hospitalization). How come? These are data from two observational studies, one in an outpatient and another in an inpatient setting, yet both results largely match data that was obtained from a randomized trial. No meaningful confounding is detected. Pseudo-randomization to vaccination status again? Two more miracles?

Two Estimates from One Season

Effectiveness in 2023–2024 was reported in two papers: interim results and final results (Table 2). The adjusted estimates are constrained in a narrow range around 40%, but we get there differently.

In the final report, we observe miraculous pseudo-randomization again. No confounding needed to be removed or, more likely, no confounding was removed. In the interim report we observe an unusual phenomenon — negative confounding. Instead of attenuated associations after adjustment, as expected from the healthy vaccinee effect, the opposite is observed: stronger associations. Somehow the baseline health of the vaccinated in the two samples for the interim report (outpatient, hospital) was worse, not better, than that of the unvaccinated.

Interestingly, however, the vaccinated were not sicker in the two samples that were used for the final report (adjustment has made no difference). It is difficult to think of a more peculiar behavior of data from two reports by the same network for the same season. That the estimates end up being similar is irrelevant.

The Last Two Seasons

We have an interim report for both 2024–2025 and 2025–2026. The phenomenon of pseudo-randomization is gone, but negative confounding is evident again (Table 3). And this time it is extremely strong. Adjustment has increased the estimated effectiveness by over 20 percentage points in the hospital setting (both seasons), by 19 percentage points in one outpatient setting (2024–2025), and by 12 percentage points in another (2025–2026). That’s the opposite of the healthy vaccinee bias.

What Have We Learned?

Of 12 estimates over five flu seasons, half are compatible with no meaningful confounding (miraculous pseudo-randomization) or miraculous cancellation of biases in opposite directions. The other half shows negative confounding, an unexpected result. None shows positive confounding — the consequence of the healthy vaccinee effect, which was consistently documented in the US and elsewhere during the pandemic. (It was evident in the elderly as well for both the Covid vaccine and the flu vaccine.) Either VISION data for the elderly substituted for a randomized trial half of the times, or we have gathered enough evidence to question the validity of VISION’s estimates of effectiveness. Which would you choose?

Perhaps we will learn more from the final reports for the last two seasons. Or perhaps it is time for the NIH to solicit requests for applications for cohort studies of the annual flu vaccine. The test-negative design is loaded with methodological problems on top of a faulty analysis. How about showing us — for the next flu season — the results from several cohorts versus test-negative case-control studies? It will be money well spent, if awarded wisely (and asymptomatic flu is a required endpoint).

One final thought:

I often wondered why the effectiveness of the flu vaccine has been consistently mediocre, usually below 50%. The usual explanation is mismatching with the circulating strains, but is it possible that much of what is being estimated is a compilation of biases?

Republished from Medium