Our pathbreaking research

Health research that fails the reproducibility standard wastes time and money, erodes the public’s trust in science, and negatively impacts patient care. 

A bedrock principle of science—and of OHDSI—is that research findings must be reliable and reproducible. If a study’s result is accurate, others should be able to repeat that study and reach the same conclusion. 

But studies are rarely held to the reproducibility standard. A few, like studies of the drug Vioxx, developed by Merck, are so flawed that they lead to thousands of patient deaths, their flaws surfacing only in hindsight. But most research isn’t subjected to such intense scrutiny, even under peer review. A 2016 survey by the journal Nature of 1,500 scientists reported that 70% had failed to reproduce at least one other scientist’s experiment, while 50% had failed to reproduce one of their own. 

One reason, according to OHDSI Director of Clinical Research Asieh Golozar, is practical. Ensuring a study is reproducible would involve reviewing not only the results, she says, but also the study’s design and statistical methodology, plus the code underlying whatever analytic approach was employed. 

Journals don’t publish all that information, Golozar says, noting that data is often proprietary and print space is limited. If journals did link to it, other investigators could better understand a study’s conclusions and its analytical approach, then generate additional evidence from other databases of the same type. 

To help remedy what science’s watchdogs have called a “reproducibility crisis,” the OHDSI Center recently unveiled a new service led by Golozar. The service will pressure-test a study’s validity by attempting to replicate it with not only the original dataset, but also a network of similar databases. 

The service is at once a public good, a training tool, and a potential revenue generator for the OHDSI Center, Golozar says. Postdoctoral fellows will do the work, guided by senior faculty. Organizations most likely to benefit include pharmaceutical companies, nongovernmental policymaking organizations, and federal regulatory agencies, such as the FDA and the European Medicine Agency. 

According to Golozar, health research that fails the reproducibility standard wastes time and money, erodes the public’s trust in science, and negatively impacts patient care. The Reproducibility Service alone can’t remedy that, but it can elevate reproducibility as a mark of study quality and dependability. 

The imprimatur could prove valuable, particularly for companies and federal agencies, which invest huge sums in health research and its oversight. “Our goal long term,” says Golozar, “is to work with organizations to improve research quality, and by extension, patients’ health.” 

Our goal long term … is to work with organizations to improve research quality, and by extension, patients’ health.” 

Asieh Golozar
Director of Clinical Research, OHDSI Center at Northeastern’s Roux Institute