Our paper arose from conversations about reproducibility that began all the way back in June 2016. One of us (David Singerman) is a historian of science curious about science’s present all around him at MIT. Another (Aaron Stupple) is a practicing physician with an interest in the philosophical and historical underpinnings of science and medicine. And the third (Leo Anthony Celi), in addition to his work as an intensive-care doctor, leads several MIT projects at the forefront of digital medicine. All of us were paying attention to the “reproducibility crisis” that was just then attracting huge public interest thanks in large part to the work of Brian Nosek and the Center for Open Science.
Reproducibility is most often honored as a principle rather than a practice in science. We’re all familiar with the oft-stated reasons more scientific work isn’t actually re-done: it’s expensive, for one thing, and there’s little joy and even less prestige in confirming what someone else already claims to have found. What’s important, we are always told, is that someone could confirm your experiment’s results if they wanted to. The most important part of reproducibility is -ibility.
As our conversations progressed, we realized that even the vibrant contemporary discussion of reproducibility could benefit from the interdisciplinary combination of our perspectives. A historically-informed approach seemed especially important to the dynamic and rapidly evolving field of digital medicine, in which the costs of actually reproducing someone else’s work may be lower than in any other area of research.
To see why that might matter, we considered a few historical examples of when new technologies were reshaping science. For instance, according to the historians Steven Shapin and Simon Schaffer, in the 1660s Robert Boyle (of chemistry’s Boyle’s Law) and his colleagues at the newly-formed Royal Society advocated a novel experimental method for learning things about nature, which they contrasted against both the deductive reasoning of philosophers or the secretive community of alchemists. Instead, Boyle proposed to induce in public. A trusted experimenter—meaning, in the 1660s, a gentleman like Boyle—would conduct a sophisticated experiment or “trial” of some natural phenomenon, often using an instrument of his own devising. He would then describe the parameters of that trial in exhaustive detail in writing and often before an audience of peers too. The detail would let readers and listeners accept the truth of the finding as though they themselves had been participants. Yet as Shapin and Schaffer discovered, nobody was able to actually reproduce Boyle’s most iconic instruments or experiments from the written descriptions alone. We found other examples—James Joule and Louis Pasteur—where justly famous and important scientific work was decidedly non-reproducible in reality.
Those Royal Society presentations might look like the beginnings of our familiar system of peer review. But as our synthesis of current scholarship demonstrated, peer review wasn’t designed to judge truth, but rather evolved to disburse public research funds. Reviewers assess the ibility part, and don’t reproduce anything, not least because such reproductions are expensive in time, money, and effort. And all the incentives in peer review, and the publication ecosystem in general, encourage work that is attention-grabbing and novel while not fundamentally questioning the presuppositions of potentially senior reviewers.
So where does that leave digital medicine? In our paper, we discuss implications of combining the low cost of digital medicine analyses and the immense interest in the field. On the one hand, there will inevitably be hyperbolic reports of discoveries from analyzing data from huge numbers of people or cases. On the other, if such data is publicly available and so is the code used to analyze it, it may be less costly than ever to actually re-run a digital medicine experiment.
In the 1660s, people traveled to Oxford to examine Boyle’s vacuum chamber of wood, metal, and glass in order to try to build one at home. In 2020, thousands of researchers might be able to re-run digital medicine experiments on their laptops. Rather than wait to see whether hype or transparency becomes the dominant force in digital medicine, we conclude our paper by calling for the field to address the problem of reproducibility now, not after a crisis arrives.