We Cannot Trust the CDC Estimates of Flu Vaccine Effectiveness

Each year the CDC publishes an estimate of the effectiveness of the flu vaccine in the previous flu season. Recently, the NIH director criticized the test-negative design from which the estimates are derived. He was right. The basic premise of the design is a two-edged sword: on the one hand, restricting the sample to people who sought medical care might reduce confounding by healthcare-seeking behavior; on the other hand, that restriction might add another type of bias — colliding bias — which is not as widely appreciated. The net bias remains unknown.

This, however, is not the only shortcoming of test-negative case-control studies of the flu vaccine. In this post, I will expose the shaky results of a large study of the flu vaccine in 2022–2023, when the vaccine was well-matched to the dominant strain. The study was based on the VISION Vaccine Effectiveness Network, one of several networks that collaborate with the CDC. Below are the published results.

Confounding by the Background Risk of Infection

The risk of infection always varies during the flu season. It was high in October through December 2022 and low in January through March 2023 (Figure).

Given a changing risk of infection, a valid comparison of the vaccinated and the unvaccinated requires similar distributions of the two populations over time. This is not the case because vaccination is associated with calendar time (rollout).

[The authors show the vaccination status at the time of seeking care, but most people got vaccinated by the end of December, and the percentage of vaccinated people stabilized in January at about 45% of the encounters.]

As shown below (Table), the share of the vaccinated population in October through December (47%), a period of high background risk, was lower than the comparable share in the unvaccinated population (61%). Of course, the complementary shares in January through March, a period of low risk, were reversed: 53% versus 39%.

In technical terms, vaccinated people accumulated more exposure time when the background risk of infection was low (53%), and unvaccinated people accumulated more exposure time when the background risk was high (61%). Moreover, since the authors excluded events that happened within two weeks of vaccination, those who were vaccinated in the second half of December 2022 contributed events only in January 2023, a time of lower risk. I will return to this analytical decision in the next section.

It is simple to grasp the bias (left table below) if we consider an extreme example where no one was injected in the first period and everyone was injected a saline solution at the beginning of the second period (right table). If we compare the rate of infection in the “vaccinated” to the rate in the “unvaccinated,” the saline injection would appear effective…

This bias was explained in the context of the Covid vaccines during the pandemic and was demonstrated in a study from Ontario, Canada. As far as I know, it was not appreciated in the context of the flu vaccine, where the rollout typically follows the rising wave and is completed around the winter peak.

Confounding by time trends in the background risk can be avoided in a cohort design with matching an unvaccinated person to a vaccinated person on the vaccination date (and terminating the observation when the former is vaccinated, if they are).

Immortal Time Bias

As I mentioned above, the authors excluded some events. They write:

“Events among patients with documented vaccination <14 days before the index date were excluded. Index date was defined as the earlier of the associated influenza test or the ED/UC visit or admission date.”

The exclusion of early events in the vaccinated is a well-known source of bias, leading to an inverse association with vaccination and adding a bias component to an estimated effect. Both the name — immortal time bias — and the mechanism are too technical to explain here.

Recently, I showed how immortal time bias operated in a study of a Covid vaccine in Qatar. Removal of the bias, by including those early events, has drastically changed estimates of effectiveness, sometimes cutting the numbers by half. If this bias is removed in the study of the flu vaccine, estimates of effectiveness in the range of 30% to 40% might change to 20% or lower.

How many early events were excluded? Probably many, but the number is hidden. According to a flowchart, almost 2,000 outpatient encounters were excluded because vaccination happened 1–13 days before the index date or vaccination status was unknown. No breakdown.

How much of the estimated effectiveness in the VISION network is due to the combination of immortal time bias and confounding by time trends in the background risk? I cannot offer a quantitative answer, but it is certainly a lot, if not all of the association. Elsewhere, I showed zero effectiveness of the flu vaccine in that season by re-analyzing data from a cohort study that reported effectiveness against both symptomatic infection and asymptomatic infection.

There are other questionable findings in the study, which will be discussed in the rest of the post.

Lower effectiveness against hospitalization?

The table below shows unadjusted and adjusted estimates of vaccine effectiveness against an outpatient encounter (left) and hospitalization (right), overall and in various strata.

The adjusted estimates for hospitalization (column D) were almost always smaller than the adjusted estimates for an outpatient encounter (column B). That’s unusual. We expect similar or stronger effects with increased severity (outpatient to inpatient) because each step adds another risk ratio multiplier (≤1) from two sequences of conditional probabilities. What is the explanation? What do the authors have to say on the topic?

First, they write that they “found similar VE within the same health systems across ambulatory and inpatient settings.”

Similar? Is consistently lower effectiveness, sometimes substantially lower, accurately described as “similar?”

Second, they acknowledge that something is unexpected and struggle to provide (unconvincing) explanations.

The Healthy Vaccinee Bias

In the majority of the analyses of outpatient encounters and in all analyses of hospitalizations, the estimated effectiveness was lower after adjustment (columns A vs. B; columns C vs. D). The explanation is confounding bias. The vaccinated were healthier than the unvaccinated, and therefore, at least part of the unadjusted association reflects the better health status of the vaccinated, which offered some protection.

The healthy vaccinee bias is well known. (I devoted many posts to this topic in the context of the Covid vaccines.) Unfortunately, it cannot be completely removed by regression models, no matter how sophisticated they are. Some aspects of health status are not captured by measured variables. The so-called adjusted estimates are still biased.

What kind of models?

The authors write:

“Models were adjusted for prespecified confounders including age, study site, and calendar time, as well as any covariate with an SMD >0.20. Age and calendar time were modeled as natural cubic spline variables. Also, inverse-propensity-to-be-vaccinated weights (IPVWs) were estimated via generalized boosted regression trees and used in logistic regression models to account for additional imbalances between vaccinated and unvaccinated groups.”

The authors used unusually complex models that included classical covariates (some in a non-linear form) plus inverse probability of treatment weighting (IPTW) to account for “additional imbalance.” It is unclear how “additional imbalance” was detected and which variables were used to compute the weights. No one would have been able replicate their analysis based on this description, even if they were given the dataset.

Higher Effectiveness in the Elderly?

Another set of questionable results is shown below. It was stated in the abstract.

We typically expect lower effectiveness in the elderly because of attenuated immune response with aging. Indeed, we observe a somewhat smaller VE against outpatient encounters in the elderly (41% vs. 45%). Unexpectedly, however, effectiveness against hospitalization is much stronger in the elderly (41% vs. 23%). A trustworthy result?

Apparently, the authors noticed the peculiar results, and they argue that “VE by age group cannot be directly compared as most young adults received standard-dose inactivated vaccines and most older adults received enhanced products such as high-dose inactivated or adjuvanted vaccines.”

Well, this explanation does not explain why the remarkable benefit of “enhanced products” was only observed for hospitalization. Indeed, the authors concede: “Despite most vaccinated older adults receiving enhanced vaccine products, VE [for outpatient encounters] was similar compared to younger adults who mostly received standard-dose inactivated vaccines.”

In short, these findings remain unexplained. They cannot be trusted.

The Outcome of Hospitalized Flu Patients

Although not explicitly stated, the authors show a set of results from a nested cohort design.

They write:

“As a secondary objective, to explore whether patient characteristics and in-hospital outcomes were different between vaccinated and unvaccinated influenza-positive cases, we compared proportions of patients with more severe clinical outcomes by influenza vaccination status stratified by age (18–64, ≥65 years)…”

Stated differently, they compared the outcome, including death, of hospitalized flu patients according to their vaccination status. Unfortunately, they did not try to adjust for baseline characteristics, so the inference is limited. I will focus on the case fatality in the elderly (almost 70% of all deaths).

In the population of hospitalized elderly flu patients, the vaccinated were older and sicker than the unvaccinated, but the differences were generally small. The fatality of the former was 50% higher: risk ratio = 4.0/2.7 = 1.5. Although no formal adjustment is possible, we can try some hypothetical examples.

Suppose vaccination was helpful and the true effect ranged from a risk ratio of 0.5 to 0.75 (50% to 25% effectiveness against death) if an elderly person is hospitalized because of the flu. Then, confounding should have changed a risk ratio of 0.75 (true) or 0.5 (true) to 1.5 (biased), which is a two-fold (1.5/0.75) or a three-fold (1.5/0.5) shift on the ratio scale. If true, that’s extreme confounding, perhaps stronger than what might be expected from the reported differences in age and some baseline characteristics. Did the flu vaccine offer any protection in those patients?

It is interesting to read the authors’ account of these data. They write:

“Baseline demographic characteristics and underlying medical conditions were similar across vaccinated and unvaccinated groups within this age strata…The percentage experiencing severe in-hospital clinical outcomes including ICU admission, receipt of IMV, or death, was similar across vaccination groups.”

They don’t even claim any hidden benefit. They assume no meaningful confounding but consider the different fatality (4.0% vs. 2.7%) as “similar.” If we follow their reasoning, we may wonder whether vaccination increased the risk of death in these patients.

Epilogue

I chose to examine one paper closely rather than criticize the generic methodology because this paper demonstrated a series of problems, some of which are shared by other CDC-based studies of the flu vaccine.

Since randomized trials will never be conducted, other designs should be sought. I mentioned two possibilities in previous posts: 1) A cohort study with two outcomes: symptomatic infection and asymptomatic infection; 2) Regression discontinuity design. So far, neither showed promising effects of the annual flu shot.

Republished from Medium