Scientific Meta-Analysis is Broken

All people of good faith should be troubled by the information I lay out below.

It took me eight years to figure out how to describe this problem and it’s massive — it threatens the very existence of humanity.

I. How systematic review and meta-analysis are supposed to work

Imagine a world where scientists, doctors, government regulators, and the general public actually cared about high quality data. We all know that such a world does not exist. But just as a thought experiment, let’s imagine that everyone involved in science and medicine really wanted high quality data to make appropriate medical decisions to improve health.

Even with the best of intentions, it’s pretty difficult to draw accurate conclusions about any medical intervention. Every human is different so there is enormous variability in any group of people that is studied. Any chemical or biological intervention will have a range of effects. Other factors — the weather, what someone had for breakfast, existing chemicals in one’s system, hormonal cycles, etc. can influence research outcomes. And there are always errors in administration, note taking, reporting, etc. in any clinical trial.

To overcome some of these problems, large data sets are sought. The larger the data set the better because that enables various small natural variations to cancel each other out so that one can look for the larger signals about safety and efficacy.

Because humans are always biased, double-blind randomized controlled trials are used so that the investigators themselves do not know who has received the medical intervention.

However, large, well-conducted, double-blind randomized controlled trials are very expensive. One way to overcome the problem of small sample sizes is to group lots of different clinical trials together in a systematic review or meta-analysis.

In a systematic review one formulates a research question, identifies the keywords or phrases associated with that question, and then engages in a search of high quality medical databases (Google Scholar, PubMed, etc.) to identify every study related to that question over a certain time period (e.g. the last ten years or the last twenty years). Then one reads through the studies to identify any that are contaminated by bias or low quality methods and those are thrown out. Then one summarizes the remaining high quality studies to see if the results agree with each other and to identify areas of disagreement.

In a meta-analysis one follows the same initial steps from systematic review (formulate a research question, identify keywords, find every relevant study, read them, and throw out any low quality studies). Then one obtains the data from all of these studies (it’s often available online or one can write to the researchers to request it) and pools that data to create one massive data set. Finally, one runs the analysis (about safety or efficacy) on that new larger sample.

This problem of how to obtain reliable, reproducible, medical results is so enormous that an entire field has developed called Evidence-Based Medicine to help clinicians sort through the available data. They create “evidence hierarchies” to help clinicians grade the reliability of various forms of evidence. Inevitably at the top of most evidence hierarchies are systematic review and meta-analysis — because they have the largest data sets.

That’s how the system is supposed to work.

II. The reality

How does one decide which studies to include and which studies to exclude?

It is standard practice to exclude from a systematic review or meta-analysis any study that is not a double-blind randomized controlled trial.

However, we know that pharmaceutical companies using Contract Research Organizations have figured out how to game RCTs to create their preferred results. (Chapter 5 of my thesis is very good on this point.) Contract Research Organizations select healthy young people with no underlying conditions to decrease the possibility of side effects, use a long ramp-up period and kick people out of the study who show any initial side effects, kick people out of the study for going to the emergency room (for violating the trial protocol), and other gimmicks to give pharmaceutical companies favorable results every time.

Common sense would suggest to exclude any study contaminated by a conflict of interest. Indeed there is a whole field of research into the “funding effect.” As the name suggests, these studies focus on how financial conflicts of interest influence research outcomes. These studies universally find that ANY financial conflict of interest no matter how small — literally a single slice of pizza — changes research outcomes in the direction of the funder. So the presence of any financial conflict of interest means that the results will not accurately reflect the underlying reality. Of all the findings in modern science, the funding effect is the largest, most reliable, and most reproducible.

By the way, there is no such thing as a small amount of bias nor a manageable amount of bias. Once a financial conflict of interest is introduced the results shift in the direction of the funder — but the size of the effect is unpredictable.

So a proper systematic review or meta-analysis will only use RCTs, but even RCTs are often manipulated by Pharma, and studies with any financial conflict of interest no matter how small should be excluded.

III. It is impossible to do a proper systematic review or meta-analysis of the mainstream vaccine literature.

A. The censorship is extreme.

Professors self-censor in universities so that they don’t lose their jobs. Literally they will not conduct a study that could possibly show harms from vaccines because they do not want to be blacklisted.

Scientific journals block publication of any study that shows harms from vaccines. It won’t even make it to peer review. It’ll just be rejected outright.

Databases including Google and Google Scholar block access to critical studies. So it’s as if those studies never existed.

B. There are no mainstream RCTs.

The vaccine industry claims that because vaccines are “Safe & Effective™️” it would be unethical to have an unvaccinated control group. This is an example of the “circular reasoning fallacy” — an argument that assumes the very thing it is trying to prove is true. But thus far they’ve gotten away with it and they simply do not conduct RCTs of vaccines (as proved by ICAN).

C. All mainstream studies of vaccine safety and efficacy are contaminated by financial conflicts of interest.

The studies are either conducted directly by pharmaceutical companies (via Contract Research Organizations) or scholars who are taking money from pharmaceutical companies.

So under the standard rules of Evidence-Based Medicine, one must exclude from any systematic review or meta-analysis all of the studies that are generally relied upon by the NIH, FDA, CDC, and the mainstream media to claim that vaccines are safe and effective. If one wants to engage in proper evidence-based medicine one has to admit that the current evidence generally relied upon to make vaccination decisions is scientifically meaningless.

IV. What would a proper systematic review or meta-analysis of vaccine safety and efficacy actually look like?

Remember in our thought experiment we are imagining a world where “everyone involved in science and medicine really wants high quality data to make appropriate medical decisions to improve health.” Okay so what would that look like?

I’ve already shown that one has to exclude all mainstream vaccine studies because none of them are RCTs and all of them are contaminated by financial conflicts of interest. That’s not me talking, those are just the rules of evidence-based medicine.

So in order to find the truth — the best and most reliable evidence — one would need to go outside the mainstream and conduct a new sort of systematic review or meta-analysis.

One would need to formulate a research question (e.g. “are vaccines associated with diabetes”), identify the related keywords, and then conduct a complete search of alternative sources and what’s called “the grey literature” (that is usually censored and suppressed) including:

• The Defender/Children’s Health Defense,
• The Highwire/ICAN,
• The Vaccine Reaction/NVIC,
• Mercola,
• Trial Site News,
• Substack (where, for example, A Midwestern Doctor does better work than the NIH, FDA, and CDC combined),
• Brownstone Institute,
• International research papers (the best vaccine research is happening outside the U.S.),
• Working papers on SSRN,
• Retracted papers (Pharma demands retraction of any paper that is directly over the target so one has to evaluate the retraction notice and the underlying paper for oneself),
• books (that bypass the censors),
• first person accounts on blogs and testimony from warrior moms and dads (using search engines other than Google),
• court proceedings including depositions and affidavits,
• legislative witness testimony, and
• documents turned over via FOIA requests,
• in addition to Google Scholar, PubMed, and other mainstream databases.

This is a much more difficult process (it takes more effort and exposes one to censorship and blacklisting). That’s why people don’t do it. But if one really wants to do a proper systematic review or meta-analysis of the vaccine evidence, that’s how one would do it.

And yes, we do have over 100 vaxxed vs. unvaccinated studies that happened via natural experiment and other clever study designs. But given the problems identified with RCTs in Section II of this essay I believe that a proper systematic review or meta-analysis of vaccines should also include other ways of knowing beyond RCTs.

Systematic review and meta-analysis should not just be a reflection of power — who can fund the most RCTs, who can perform the most statistical wizardry. They should be about gathering the most high quality information from wherever we can find it to create the most comprehensive picture of any problem and any proposed remedy.

V. What about “the funding effect” in skeptical studies?

Let’s talk for a moment about possible conflicts of interest among the researchers in exile.

Team Vaccine (e.g. Offit, Hotez, and Reiss who are all making millions of dollars from the iatrogenocide) often point out that there is some money to be made from fighting to expose the harms from vaccines. It’s nowhere close to the hundreds of billions of dollars a year that Pharma brings in from vaccines and from selling treatments for vaccine injury — but it’s not zero. CHD, ICAN, Mercola, etc. do bring in a few million dollars a year in revenue that they use to build a movement to stop the iatrogenocide. Independent scholars including Jeremy Hammond, Jennifer Margulis, and I can make a bit better than a subsistence living from reporting the truth.

(I should also point out that Team Vaccine seems unaware of the fact that they have indicted themselves and their studies in the process.)

But I want to make a case that any reasonable definition of a financial conflict of interest does not apply to the alternative and grey literature (and vaccine skeptics in general) for the following reasons:

1. We would absolutely love to be wrong. To take just my own experience as an example — I’ve gone to bed every night and woken up every day for eight years hoping and praying that I am wrong about the harms of vaccines. I once spent three days reading about every possible type of logical fallacy and applying them one by one to my work to see if I was wrong. (I wasn’t. However, the various lists of logical fallacies actually fit the actions of the mainstream vaccine pushers quite well.) If it turns out that I’m wrong I will be one of the happiest people in the world. And that’s true throughout the movement — we would all love to be wrong. Yet the evidence grows every day that vaccines cause tremendous harms.

2. When we make a methodological mistake we get torn to shreds by Pharma and their well-paid functionaries. Pharma watches and reads everything we do looking for the slightest misstep so that they can pounce and then claim that all contrary evidence is incorrect. Said differently, vaccine skeptics undergo the most rigorous peer review of any scholars on Earth and if mistakes are made they will be found out quickly, announced publicly, and widely distributed.

Of course Pharma dislikes any study that shows harms. But in the process, a large secondary and tertiary literature builds up as Pharma attacks and scholars defend their work. One has to read it all but over time it becomes pretty clear which scholars have proven their mettle and which studies have withstood rigorous scrutiny and should be included in a systematic review or meta-analysis. Skeptical scholars tend to be smarter and faster than Team Vaccine because we welcome criticism and questioning and that improves our skills year after year.

3. Yes there is some money to be made working in the resistance but everyone working to understand the harms from vaccines is making far less money than they would be making if they accepted the standard narrative. It seems like a stretch to claim that someone who is losing hundreds of thousands of dollars or even millions of dollars in lifetime earnings in order to tell the truth somehow has a financial conflict of interest. Indeed, one could argue that the work we do is the opposite of a financial conflict of interest.

4. Pharma’s abuse of the legislative process and capture of government and academia created the need for a freedom movement to fund alternative research in the first place. In the absence of liability protection, plaintiffs’ lawyers would have funded hundreds of millions of dollars of vaccine safety studies (just like they do with other toxic products), the vaccine schedule would not exist, and there would be no epidemic of vaccine injury in this country.

If government weren’t captured by Pharma, it would have funded independent research into vaccine safety. Pharma’s capture of the political system, regulatory agencies, and the knowledge production process in science and medicine — thus corrupting it all — created the need for a freedom movement, otherwise there would be no honest research whatsoever. Independent citizens stepping up to fill the void created by the corruption of industry, the cowardice of academia, and the complete failure of government is not a conflict of interest, it’s patriotism, democracy, survival, actual science, and just plain decency.

VI. Conclusion

Proper science is hard. There’s just a lot of variability in humans and the vast majority of proposed medical interventions do not produce benefits. One way to improve scientific research is to get really large samples. But those studies are expensive. One can improve scientific results through systematic review and meta-analysis that combine the results from all previous high quality studies on a topic. Evidence-Based Medicine requires that studies that are not RCTs and those characterized by financial conflicts of interest should be excluded from systematic reviews and meta-analyses.

There are no mainstream RCTs in connection with vaccines and all mainstream vaccine studies are contaminated by financial conflicts of interest. Therefore it is impossible to do a proper systematic review or meta-analysis of vaccines using mainstream studies. Furthermore, Pharma’s ability to manipulate RCTs suggests that one should include additional types of high quality data to improve research results.

The best way to do a systematic review or meta-analysis of vaccine benefits and harms is to turn to alternative sources and the “grey literature” by searching the results from the leading skeptical groups (CHD, ICAN, Mercola, and others). There are over 100 vaxxed vs. unvaccinated studies that show harms from vaccines. But we should also include a wider range of studies to build the most comprehensive possible understanding of vaccine risks.

Some conflicted parties raise concerns that the rebel researchers are conflicted too. But a skeptical orientation toward one’s own results (“we’d love to be wrong”), close scrutiny from hostile outsiders, and enormous financial losses involved in telling the truth provide reasons why these results are correctly understood as the most reliable evidence we have. Furthermore it should be understood that telling the truth in the face of widespread lies by industry, government, and academia is not a conflict of interest; it’s courageous, honorable, and represents the best of the human spirit.

Republished from the author’s Substack