Taking the pulse of biomedical science: reproducibility, racing and speaking out loud

Last night, my college (Clare Hall) hosted the termly fellow-student interaction dinner, and I happened to sit next to a renowned scientist and group leader at the MRC Laboratory of Molecular Biology (LMB) in Cambridge. He did not seem interested in discussing molecular biology; instead, we entered a brief, yet insightful, conversation on a worrying development in scientific culture. My thoughts on this development have been brewing for a long time, and yesterday’s conversation has now triggered their release.

Google “reproducibility crisis” and you are struck by the billions of articles addressing one of the greatest issues in modern science. Over the last few years, this hot topic has given rise to multiple editorials and comments in prestigious journals such as Nature and Science, reflecting the pervasiveness of the problem. Earlier this week, Science notified its readers about the IAP’s (InterAcademy Partnership) new guide: Doing Global Science: A Guide to Responsible Conduct in the Global Research Enterprise. Yet another attempt at promoting best research practices.

One would hope that the intensified focus on research conduct will ultimately instigate a change in the current status quo. A status characterised by a stream of retracted research papers and an inability of industry to replicate seminal findings. The prestigious Science and Nature are two of my favourite journals, and many scientists consider it a career victory if their paper is accepted by one of the two. However, if the seminal papers published in these and similar journals may later have to be retracted, we are left with the uncomfortable feeling that something is not quite right. Somehow, following the reproducibility discussion is like following current EU attempts at solving the refugee crisis: high-profile politicians are emphasising moral values and humanitarian help, yet the problem continues to escalate and people remain stuck at Greek borders.

Biomedical research, in particular, is strongly affected by the reproducibility crisis. The natural variation inherent to biological systems makes it impossible to conduct the perfectly controlled experiment. This, in turn, necessitates a certain study size  to ensure that the effects under study are not missed or falsely estimated. Hence, a reasonable statistical power (i.e. the likelihood of detecting an effect if it is truly present) is a key step towards better reproducibility. This power has to be increased further if the size of the effect is small; we all know that finding a needle in a haystack is harder when the size of the former decreases. Moreover, the nature of probability teaches us that the higher the number of relationships being tested, the greater the risk of obtaining false-positive findings. Thus, without rigorous independent validation, creative –omics studies should at best be interpreted with healthy scepticism given the likelihood of them having an extremely low so-called positive predictive value (for more on this topic, I would highly recommend the following paper: Why Most Published Research Findings Are False).

With all this in mind, one might wonder what is preventing scientists from following the right path of research conduct, i.e. making sure that their studies are well-powered and mechanisms validated in great depth?

Science and politicians become corrupted for the very same reason: money. The highly competitive scientific funding race makes hot, but poorly validated, scientific findings commonplace. As stated in Why Most Published Research Findings Are False Corollary 5:  “The greater the financial and other interests and prejudices in a scientific field, the less likely the research findings are to be true.” Followed by Corollary 6: “The hotter the scientific field […], the less likely the research findings are to be true.” If I could add a Corollary 7, it would be: “The hotter the scientific field and the greater the financial and other interests, the less likely the research findings are to be shared with the wider scientific community to foster collaboration.” This bias towards high-impact findings and fear of data sharing deteriorates reproducibility, stifles scientific progress and causes biomedical scientists to fail our duty to produce results with potential benefits for human health.

This money-driven business is also behind the recent development of labs into mini-factories of more than a dozen group members, driven by the false belief that more people produce more findings, thereby leading to more papers and money. My neighbour at yesterday’s dinner confirmed the flaws of this approach, and used it as an example why LMB groups seek the exact opposite structure: the smaller the groups, the better the science. When the group is smaller, the principal investigator is in a much better position to scrutinise the raw data together with the Postdoc or PhD student behind it. This scrutiny, in turn, ensures that the final output is more likely to be sound, as opposed to data that has only been looked at by the researcher generating it (who is likely to be biased), and that has only been presented to the group leader in a processed format. Today, it is the norm – not the exception – that a principal investigator is detached from the raw data-generating process, leaving it up to the moral and training of each individual group member to ensure scientific integrity.

Scientific training, however, is another important aspect that suffers when academia turns into a flawed business enterprise. To fund their factories, research institutes take in more PhD students than they have space for, and each individual PhD student has to accept less individual mentoring. Students are then faced with one of two detrimental options: 1) To fight for the attention of their supervisor, leading some students to manipulate data in order to present exciting results; 2) To navigate their 3-year journey on their own, suffering from multiple set-backs due to lack of guidance. Not to mention the money that is wasted in this process. It is sad to see that in places like Cambridge, this environment is far more common than the University would like to admit.

It is also sad that many junior researchers feel unable to voice some of their concerns. Partly because there is an established hierarchical culture in the scientific community, and as a young PhD student, you often want to avoid being noticed for causing “problems”. Hence, internal knowledge about dodgy science by some groups may continue to go unnoticed in the wider community simply because the knowledge of misconduct survives among the student population who are afraid of damaging their own future career by criticising established, senior researchers. Moreover, it is not always clear who you can express your concerns to in order for them to be dealt with in an objective manner.

Even more benign aspects of academic work, such as determining the order of authorship on a paper, may be blown out of proportion in the absence of transparency. In the biomedical community, in particular, authorship order does not always correspond to the actual contribution of each individual. It is common for internal politics to govern the final list of authors, and many junior researchers find it hard to stand their ground if they disagree with the decision of their supervisor. Once again, it is not immediately clear who you can consult for objective advice if this happens to be the case.

If any senior people happen to read this piece, they may frown and think that I am too junior and inexperienced to express an opinion on the current state of biomedical science. If they do, I would not be surprised. Similarly, I have experienced a colleague tell me off for joining scientific discussions involving senior researchers, just because I belong to the lower end of the hierarchy. To such people, I have only one thing to say: You are wrong.

I would argue that young researchers are in the perfect position to enter the discussion on the future of biomedical science because they still retain many of the idealistic values which may long be gone in some of their older colleagues, a point that is also made by Daniele Fanelli, a senior scientist at the Meta-Research Innovation Center at Stanford (METRICS) in Palo Alto, California. If we only discuss the concerns among ourselves, without opening up to the wider scientific community, our hopes for improvement will remain just that – hopes.

I hope that this blog piece will reach out to other junior researchers who either share or oppose my view, so that we can have a healthy debate about the future of biomedical science.



2 thoughts on “Taking the pulse of biomedical science: reproducibility, racing and speaking out loud

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s