This site uses cookies. By continuing to use this site you agree to our use of cookies. To find out more, see our Privacy and Cookies policy.
Skip to the content

Share this

Free weekly newswire

Sign up to receive all our latest news direct to your inbox.

Physics on film

100 Second Science Your scientific questions answered simply by specialists in less than 100 seconds.

Watch now

Bright Recruits

At all stages of your career – whether you're an undergraduate, graduate, researcher or industry professional – brightrecruits.com can help find the job for you.

Find your perfect job

Physics connect

Are you looking for a supplier? Physics Connect lists thousands of scientific companies, businesses, non-profit organizations, institutions and experts worldwide.

Start your search today

Blog

On the frontline of the ‘reproducibility crisis’

Significant. (Click to view full cartoon. Courtesy: xkcd/Randall Munroe)

Significant. (Click to view full cartoon. Courtesy: xkcd/Randall Munroe)

By Margaret Harris

The “reproducibility crisis” in science has become big news lately, with more and more seemingly trustworthy findings proving difficult or impossible to reproduce. Indeed, a recent Nature survey found that two-thirds of respondents think current levels of reproducibility constitute a “major problem” for science. So far, physics hasn’t been affected much; the crisis has been most severe in fields such as psychology and clinical research, which, not coincidentally, involve messy human beings rather than nice clean atomic systems. However, that doesn’t mean it’s irrelevant to physicists. Last month, I had the pleasure of speaking to three physics graduates who have become personally involved in addressing the reproducibility crisis within their chosen profession: medicine.

Henry Drysdale, Ioan Milosevic and Eirion Slade are third-year medical students at the University of Oxford. All three earned their undergraduate degrees in physics, and they now make up one-third of COMPare – an initiative by Oxford’s Centre for Evidence-Based Medicine (CEBM) that tracks “outcome switching” in clinical trials. As Drysdale explained to me over coffee in an Oxford café, researchers who want to perform clinical trials have to state beforehand which “outcomes” they intend to measure. For example, if they are trialling a new drug to treat high blood pressure, then “blood pressure after one year” might be their main outcome. But researchers generally keep track of other variables as well, and often their final report focuses on a positive result in one of these other parameters (a dip in the number of heart attacks, say), while downplaying or ignoring the drug’s effect on the main outcome.

“In its purest form, outcome switching is when you don’t report the thing that you said you were going to report, and instead you report something that is either more favourable or less bad,” Drysdale explains. He and his colleagues believe this is misleading, but more than that, he says, switched outcomes may be a sign that the reported result is just a statistical fluke. The reason for this is a little bit subtle, but essentially, if you measure enough things, it’s very likely that at least one of them will turn up a statistically significant correlation completely at random – a point made nicely by the xkcd cartoon above.

What Drysdale and his colleagues are doing with COMPare (the initials stand for CEBM Outcome Monitoring Project) is to flag up cases where outcome switching has happened and write to the scientific journal that published the offending study. The results so far have been “kind of shocking”, Drysdale says. Of the 67 trials published in the top five medical journals from the end of October 2015 to December 2015, the team found that 58 (87%) displayed some level of outcome switching. Most of these cases were probably not malicious or dangerous; Milosevic notes that sometimes there are good reasons for researchers to shift their attention away from pre-specified outcomes. But if the study doesn’t explain why such a shift was made – or even mention it at all – it’s impossible for readers to judge whether the decision was valid.

The thing that really surprised the team, though, was the medical community’s blasé attitude. One journal they wrote to claimed that since trial protocols and registries (which contain information about pre-specified outcomes) are public, interested readers could simply look them up, compare them to the published study and see which (if any) outcomes were switched. In theory, that’s true, but “some of [the studies] have been all over the place – the outcomes are impossible to decipher and we can’t figure out what they’re trying to measure”, Milosevic says. “Our argument is that it’s the journal’s responsibility to police this. They’re presenting this work in their journal, and an interested reader shouldn’t have to spend two hours to decipher whether a report is accurate.”

In Slade’s view, resistance to the team’s criticisms reveals a degree of “wilful ignorance” about flaws in the way science is being conducted. “I think in the future there are going to be issues other than outcome switching that are going to come out of the woodwork,” he says. “We’re really relying on the authors of trials, the publishers, the editors of journals, the peer reviewers, to get on board when these are discovered and say ‘We need to change the culture in which we work.’ ”

Towards the end of the conversation, I asked the trio whether their background in physics had any bearing on their involvement with COMPare. Drysdale replied that it helped to have an understanding of statistics, while Milosevic joked that it required “a certain type of personality” to trawl through hundreds of entries in registries and meticulously record all the data in a spreadsheet. Slade, though, thought it went deeper than that. “Most physicists strive for internal consistency, especially when coming up with physical theories,” he observed. “The idea that you’re only comfortable if things are consistent with themselves is definitely of benefit in this project, where we are trying to establish whether medical research is consistent with itself. And the conclusion we’ve come to is that in terms of outcome switching, unfortunately, it’s not.”

If you’d like to learn more about the three COMPare physicists, including how they got into medicine after starting out in physics, look out for the careers section of the July issue of Physics World. For more information on COMPare, you can visit the website.

This entry was posted in General and tagged , , , , . Bookmark the permalink.
View all posts by this author  | View this author's profile

3 comments

  1. R Wilson

    You say medicine and psychology “involve messy human beings rather than nice clean atomic systems”. But medicine also involves big-pharma where millions can be made by “proving” a drug is effective. A cure for acne would be very lucrative, even if it had only been shown to work one in 20 times, so long as the marketing ignores the 19 failures..

  2. M. Shah

    I agree with R. Wilson, it is not a matter of what entities are under question, it is a matter of big money involved in the field of medical science, which is becoming increasingly commercial, political and dishonest. This is based on my own experience with this lucrative industry which is no more about saving mankind or serving humanity.

    • Jim

      True, when one reads about companies upping the price of medicinal drugs a hundred fold, this is profiteering at the huge expense of the NHS and can thus directly or indirectly be costing peoples health or lives. These profiteers are no doubt stashing their millions offshore as well, avoiding tax and social responsibility. How is it that these practices are legal?

Guidelines

  • Comments should be relevant to the article and not be used to promote your own work, products or services.
  • Please keep your comments brief (we recommend a maximum of 250 words).
  • We reserve the right to remove excessively long, inappropriate or offensive entries.

Show/hide formatting guidelines

Tag Description Example Output
<a> Hyperlink <a href="http://www.google.com">google</a> google
<abbr> Abbreviation <abbr title="World Health Organisation" >WHO</abbr> WHO
<acronym> Acronym <acronym title="as soon as possible">ASAP</acronym> ASAP
<b> Bold <b>Some text</b> Some text
<blockquote> Quoted from another source <blockquote cite="http://iop.org/">IOP</blockquote>
IOP
<cite> Cite <cite>Diagram 1</cite> Diagram 1
<del> Deleted text From this line<del datetime="2012-12-17"> this text was deleted</del> From this line this text was deleted
<em> Emphasized text In this line<em> this text was emphasised</em> In this line this text was emphasised
<i> Italic <i>Some text</i> Some text
<q> Quotation WWF goal is to build a future <q cite="http://www.worldwildlife.org/who/index.html">
where people live in harmony with nature and animals</q>
WWF goal is to build a future
where people live in harmony with nature and animals
<strike> Strike text <strike>Some text</strike> Some text
<strong> Stronger emphasis of text <strong>Some text</strong> Some text