The Real-World Data Dilemma

On 22 May 2020, the Lancet published a study that changed the course of the Covid-19 pandemic.

It purported to analyse the medical records of 96,032 patients hospitalised with Covid-19 across 671 hospitals and six continents, delivering a damning conclusion about an old antimalarial drug — hydroxychloroquine.

Patients treated with hydroxychloroquine alone or with a macrolide antibiotic appeared more likely to develop dangerous heart rhythm disturbances and more likely to die.

The study’s findings dealt a major blow to hydroxychloroquine — a cheap, off-patent drug that was already at the centre of a fierce political battle.

President Donald Trump had publicly championed hydroxychloroquine as a potential Covid treatment and called for research to be accelerated — even revealing that he was taking the drug himself.

Within days of the study appearing in the Lancet, the World Health Organisation suspended the hydroxychloroquine arm of its multinational Solidarity trial. Other trials were interrupted, and treatment policies changed around the world.

The study carried enormous authority because of its size. Nearly 100,000 patient records appeared to provide the kind of real-world evidence clinicians desperately wanted while conventional randomised trials were still underway.

But there was a problem — investigators quickly concluded the database was fake.

The company behind it was Surgisphere, an obscure Chicago operation. What followed was one of the most extraordinary scientific collapses in modern history, exposing how easily global health policy could be swayed by unverified data.

Six years later, Surgisphere is worth revisiting — not as a one-off pandemic anomaly, but because the weakness it revealed in modern healthcare data has only become more consequential.

The Sleuths

Among those scrutinising the Lancet paper was physician James Todaro, who had been closely following the emerging hydroxychloroquine literature.

Todaro wasn’t surprised that hydroxychloroquine showed little benefit in severely ill patients. What caught his attention was the substantially higher mortality — an extraordinary safety signal for a cheap, decades-old drug with a well-established safety profile.

Clinicians, statisticians, epidemiologists, and independent researchers began pulling apart the paper.

One of the first major discrepancies came from Australia. The study claimed to include 600 Australian patients from five hospitals, with 73 deaths recorded by 21 April 2020. But only 67 Covid deaths had been recorded in the entire country by that date.

Something wasn’t adding up. The data contained more Australian deaths than Australia itself had recorded.

The Guardian contacted hospitals in Melbourne and Sydney whose participation would have been necessary to reach the numbers reported in the paper. All denied contributing to the database and said they had never heard of Surgisphere.

The company said one Asian hospital had accidentally been assigned to Australia. The figures were corrected, and the authors maintained that the mistake did not alter the conclusions.

But researchers kept looking.

The study also claimed to contain detailed clinical data from hospitals across Africa, including information derived from continuous cardiac monitoring.

Collecting and transmitting that level of granular, near-real-time information requires expensive and sophisticated electronic hospital infrastructure — technology that many African hospitals simply did not have.

Researchers questioned how Surgisphere could possibly have obtained such comprehensive electronic records from facilities where the systems needed to generate them were unlikely to exist.

Then there was North America. The study claimed to have records for approximately 63,000 hospitalised Covid patients from the region — roughly equivalent to the total number of Covid hospitalisations reported across the entire United States at the time.

If the numbers were genuine, Surgisphere had somehow captured detailed clinical records for virtually every hospitalised Covid patient in America, across hundreds of independent hospitals using different electronic medical-record systems.

It was an extraordinary claim. If true, Surgisphere had accomplished something that major health systems and technology companies had struggled to achieve for years: connecting hundreds of hospitals using different electronic medical-record systems and extracting detailed, standardised clinical information on tens of thousands of patients across multiple countries, almost in real time.

Todaro later described a database with that capability as the “holy grail” of medical research. The idea that an obscure company had quietly built it without anyone in the field knowing about it was increasingly difficult to believe.

The Company Behind the Data

A global healthcare database spanning hundreds of hospitals would require substantial technical infrastructure — software engineers, database architects, statisticians, data-security specialists, and clinical informatics teams.

But investigators found nothing resembling such an operation.

Surgisphere’s website was heavy with references to artificial intelligence, machine learning, and advanced analytics, but provided little evidence of how its vast international data network actually operated.

Its public footprint showed only a handful of employees and no visible army of data engineers capable of building and maintaining the system described in the papers.

The company’s founder and lead author, Sapan Desai, was a vascular surgeon who had also written science-fiction novels.

Media investigations uncovered even stranger details about the small group of people associated with the company. The Guardian revealed that one employee, listed as a marketing executive, was an adult model and events hostess.

As researchers and journalists dug deeper, pages disappeared from Surgisphere’s website, and claims about academic relationships were altered or removed.

The company had supposedly built a research asset of extraordinary value — a database spanning hundreds of hospitals and tens of thousands of patients that would ordinarily generate years of publications across numerous diseases.

If only it were true.

Who Had Actually Checked the Data?

The Lancet paper contained an important declaration: all four authors stated that they had full access to the data and accepted responsibility for its integrity.

But that assurance soon unravelled.

On 2 June 2020, more than 200 scientists, clinicians, and statisticians signed an open letter detailing serious concerns about the study’s methods and data, and demanding independent validation.

An audit could have resolved the dispute.

But when asked to provide the data, the study’s lead author, Harvard cardiologist Professor Mandeep Mehra, reportedly replied: “Our data sharing agreements with the various governments, countries and hospitals do not allow us to share data unfortunately.”

Three of the four authors then said they could no longer vouch for the primary data because they had been unable to complete an independent review.

By 4 June 2020, it was all over. The authors abandoned the study, and the Lancet retracted it. On the same day, the New England Journal of Medicine retracted another Covid study built on the same database.

A Failure Beyond Surgisphere

The scandal was widely treated as an extraordinary case of scientific fraud during a chaotic time, but its true significance lies in what it exposed.

Neither the peer-reviewers nor the journal editors had verified the underlying data, and outside researchers were barred from inspecting it. None of that stopped the study from rewriting global health policy overnight.

Recognising the failure, the Lancet published a statement in September 2020 titled “Learning from a retraction,” introducing new safeguards for research reliant on large datasets.

The journal mandated that multiple authors directly access and verify raw data, required data-sharing statements, and introduced specialised data-science peer review.

These reforms were a clear admission of systemic failure, but they left a critical flaw unresolved.

Requiring authors to verify the dataset delivered to them is not the same as requiring them — or independent researchers — to verify those data against the original source medical records.

Six years later, that distinction has become critical. Modern medicine is embracing real-world evidence at an unprecedented scale, relying on massive aggregations of electronic health records, insurance claims, and disease registries to evaluate major drugs and vaccines.

Massive datasets containing the health records of millions of patients now underpin thousands of studies on drug safety, vaccines, and treatment effectiveness.

Regulators are increasingly relying on them too.

Last December, then-FDA Commissioner Marty Makary announced plans to expand the agency’s use of “real-world evidence” in regulatory decisions.

“We’re removing unnecessary barriers that have prevented us from using powerful real-world evidence to get life-changing treatments to patients faster,” Makary said.

There are valid reasons for the shift. Electronic health records covering millions of patients can reveal rare side effects and long-term outcomes that clinical trials may be too small or too short to detect.

But there is a problem. Much of this research relies on enormous, heavily processed datasets that researchers and independent reviewers cannot readily check against the original hospital records.

The danger became clear during the Surgisphere scandal in 2020, when the database underpinning a major Covid-19 study that influenced health policy turned out to be fake.

Six years later, the same fundamental problem remains.

If neither researchers nor independent reviewers can trace the data back to the original medical records, how do we know they are reliable?

Who Controls the Data?

Patient records are scattered across thousands of hospitals, stored in different computer systems, and protected by strict privacy laws.

Before researchers can analyse them, those records must be extracted, stripped of identifying information, converted into standardised codes, and assembled into research-ready databases.

The use of these datasets has exploded. A PubMed search for “real world evidence” or “real world data” shows how rapidly the field has grown over the past two decades (see graph).

Between the original medical record and the scientist who eventually analyses it sits something of a black box.

This has created a powerful role for middlemen — companies and research networks that extract, process, and package medical records into datasets for researchers to analyse.

Commercial data companies, such as IQVIA, TriNetX and Merative, along with research networks such as OHDSI and PEDSnet, now occupy this space.

They largely draw on real patient data from electronic health records, Medicare, insurance claims, and other healthcare databases.

But some, such as OHDSI, can also work with synthetic data — artificially generated patient records that mimic the characteristics of real healthcare data but do not represent actual patients.

By the time researchers receive the data, the records may already have passed through multiple stages of extraction, coding, filtering and software transformation.

Several organisations involved in this process also have financial ties to the pharmaceutical industry.

TriNetX, for example, received venture backing from Merck’s Global Health Innovation Fund. OHDSI was launched by Janssen, and receives financial support from pharmaceutical companies including Takeda, Janssen, AstraZeneca, Merck and Pfizer.

Those relationships do not mean the databases are inherently corrupted or biased, but when organisations involved in processing the data also have commercial interests, independent scrutiny of how the data are handled becomes even more important.

The Secrecy of PEDSnet

PEDSnet provides a window into how this data pipeline works.

This large US research network turns electronic health records from major children’s hospitals into datasets for medical research. During the pandemic, it was used for influential studies examining the safety and effectiveness of Covid-19 vaccines in children.

In 2024, researchers published a study using PEDSnet data to evaluate Pfizer’s Covid vaccine in adolescents. The result was striking.

The authors reported 98.4% effectiveness against documented infection in adolescents during the Delta wave, with no statistically significant waning after the first dose — effectively claiming near-perfect protection.

Another high-profile PEDSnet study later reported that Pfizer’s vaccine was 95.4% effective against long Covid in adolescents during the Delta period, based on records from 20 US health systems.

To some researchers, those extraordinary figures seemed “highly implausible,” prompting them to look more closely at the underlying data. But when they tried to re-examine them, they hit a wall.

The paper stated that the data were “not publicly available due to privacy concerns,” adding that they “may be available through submitting PEDSnet Study Approval applications.”

In a public exchange on X, one of the study’s senior authors, biostatistician Professor Jeffrey Morris, stated, “I don’t have access to those data.”

Morris later explained that access was governed by strict consortium protocols.

“The PEDSNet data is a consortium repository of complete medical records for millions of children so…they are not ‘public’ by law,” he said, adding that his colleagues “are not at liberty to share the data.”

One of the papers, however, stated that eight authors had “accessed and verified the data.” Corresponding author Christopher Forrest was among them.

MD Reports contacted the authors, including Forrest, asking about data access, but there was no response. PEDSnet also did not respond.

It therefore remains unclear what was actually verified — and whether anyone checked the research dataset against the original hospital records.

So when scientists say they have “verified” a dataset, what exactly have they verified?

Who Gets to See the Underlying Data?

A 2021 report prepared for the US Department of Health and Human Services provides a rare look inside these research networks.

It examined 15 major networks, including PEDSnet, PCORnet, FDA Sentinel, and OHDSI.

Access to patient-level information is tightly controlled. The report noted that “only authorized individuals at each data provider organization have access to patient-level information.”

Outside researchers can submit instructions specifying what they want analysed. Depending on the network and project, they may receive deidentified or aggregated results — sometimes only summaries — rather than the underlying medical records.

These restrictions are intended to protect patient privacy. But deidentifying, summarising, enhancing, and imputing data requires additional processing, and each step creates another opportunity for errors or distortions to creep in.

The HHS report itself warned that automated quality checks cannot catch every error. Data that pass validation can later be “found erroneous in unexpected ways, even in environments with long-established data workflows.”

That goes to the heart of the problem.

Researchers are often unable to check whether the data they receive faithfully reflect what was originally recorded at the hospital. If an error was introduced before the dataset reached them, it may simply pass through unnoticed.

That is precisely what Surgisphere exposed — researchers analysed a database they believed contained genuine hospital records, without being able to verify those records at their source.

Compounding the problem, independent researchers trying to scrutinise or replicate published findings often struggle to obtain the dataset at all.

The result is a black box at both ends — researchers cannot verify the data against the original medical records, and outsiders cannot independently check the data after publication.

When Researchers Get to Check

A recent dispute over HPV vaccination in Scotland shows what can happen when researchers are given access to the underlying data.

The original Scottish study used national health registry records and made international headlines after reporting “zero cases” of invasive cervical cancer among women vaccinated against HPV at ages 12 or 13.

Two Australian researchers, Associate Professors Samir Saidi and Mark Jones, requested the underlying dataset from the Scottish researchers.

They received it in full, deidentified and without financial cost or legal restrictions.

When Saidi and Jones reanalysed the anonymised data, the picture changed dramatically.

They found key differences between the vaccinated and unvaccinated groups, questioned the population figures used in the analysis, and argued that the celebrated “zero cancer” finding was largely explained by the women’s young age and short follow-up.

The merits of the statistical debate aside, the important point is that the data could be checked. There was no black box, and without access, the problems Saidi and Jones identified would probably never have surfaced.

The Reproducibility Problem

That kind of scrutiny is fundamental to science. Important findings should be open to independent checking and replication.

NIH Director Jay Bhattacharya has made replication and reproducibility a major agency-wide priority.

The NIH describes replication and reproducibility as “foundational to the conduct of gold standard science,” with its new initiative focused on data sharing, standardisation, and independent replication.

That push now extends to the provenance of real-world data. The NIH’s new Data COUNTS initiative aims to preserve the “full data provenance and lineage” of electronic health records — allowing researchers to trace where the data came from and what happened to them along the way.

Longstanding NIH publishing principles state that, where ethically appropriate, “all datasets on which the conclusions of the paper rely must be made available” to reviewers during peer review and upon reasonable request after publication.

In practice, researchers seeking access to data can face lengthy application processes, outright refusals, or fees that make independent scrutiny difficult or impossible.

Patient privacy is essential, but it need not preclude independent scrutiny.

Researchers can be given controlled access through secure systems, while independent audits can verify whether research datasets accurately reflect the original records without exposing patients’ identities.

At the very least, there should be a clear audit trail showing how the original records were turned into the dataset used in the study.

Medicine’s New Black Box

The problem exists at both ends.

Before researchers see the data, patient records have already been extracted, enhanced, coded, and transformed. After publication, independent researchers may be unable to obtain the dataset to check the findings for themselves.

Between the two sits an increasingly powerful layer of commercial data companies and research networks controlling how medical information is processed and who gets to see it.

Surgisphere was not simply an isolated pandemic scandal. It exposed a weakness in a research system increasingly dependent on enormous datasets that few people evaluating the resulting science will ever see.

Six years later, the datasets are vastly larger, the software more complex, and their influence over healthcare greater than ever.

The NIH now says research data should have traceable “provenance and lineage.” That standard should extend through the entire research chain.

Researchers must be able to examine the underlying data themselves, or there must be an independent way to verify where the data came from and how they were processed.

Otherwise, “accessed and verified” risks becoming little more than a box to tick, while medicine is still being asked to trust the black box.

Republished from the author’s Substack