Tensions in Scientific Culture Contribute to Reproducibility Challenges

“Once a system is running a certain way it’s hard to change course.”—Richard Harris, award-winning science correspondent for National Public Radio

Scientists, publishers, journalists and the public talk of the problem of reproducibility in experimental science. There are committees, symposia, peer-reviewed articles, blogs and opinion pieces documenting the issue and exploring remedies to the challenge of reproducibility. NPR science correspondent Richard Harris takes this societal challenge one step further in his book with a title that in no uncertain terms calls attention to the root cause of the reproducibility challenge—Rigor Mortis: How Sloppy Science Creates Worthless Cures, Crushes Hope and Wastes Billions.

PLOS interviewed Harris about the lack of rigor in experimental research that he describes in his book, and from a discussion that covered materials standards, publishing fewer papers with greater confidence, text and data mining and the cultural shift required to improve the rigor in science, one unifying theme emerged—tension. Tension between evaluation and culture, hands-on training and coursework, between making work public and ensuring its reliability, and between responsible reporting and satisfying the needs of a news organization.

These lines of tension contribute to the current structural rigidity in the culture of science that, according to Harris, make scientific rigor a difficult challenge to address. As he delved into biomedical sciences in 2014 after nearly a decade reporting on climate change and the environment, Harris took a broad look at the state of the research enterprise and asked, what are the consequences of limited funds on the structural influences supporting the current state of academic biomedical research?

Evaluation and Culture

The biggest obstacles influencing rigor, according to Harris, are the “underlying cultural issues” that confront science. He cites the financial crunch, career pressures and the hyper competitiveness of science, particularly in biomedical sciences, as examples. These are, he thinks, also the hardest problems to solve. “Even if you poured huge amounts of new money into biomedical research this problem would not go away quickly. It’s changing a mindset.” Fundamentally the “incentives are misaligned,” says Harris, to reward numbers of papers and how many of them are published in high profile journals, rather than “careful work where one is highly confident in results.”

Hands-on Training and Coursework

On the point of experimental responsibility, Harris’ book follows a path of practical recommendations to improve scientific rigor, including:

• Improve experimental design and provide methods training
• Validate cell lines, antibodies, gene constructs
• Apply appropriate statistical analysis
• Disclose experimental and analytical methods

During his research for the book, Harris was quite surprised to find out how little formal training there is in experimental methodology, particularly in biomedical research. He did note, however, that NIH is now funding attempts to develop curriculum, following an unsuccessful search to find the best training courses in the country to replicate. There is “huge room for improvement” in this area, he says, but how to integrate this into a training program? Changing a system that’s already in place to accommodate that is difficult, especially without reward and recognition for faculty teaching those courses. “Should it start more robustly at the end of the undergrad career?” One potential solution discussed during the interview is for post-docs to teach these courses to incoming graduate students, as part of a summer or first-quarter orientation program.

Making Work Public and Ensuring Reliability

Steps can be taken to improve the situation. For example, researchers with grants from the National Institutes of Health are required to authenticate the cell lines used in their work. “There’s no simple solution to be imposed from top down, it needs to also work bottom up,” says Harris when considering what key additional recommendations the community might consider. “Young scientists are more open to sharing and that leads to transparency, that helps solve some of the issues.” It’s not a silver bullet but it improves things if people can put their data out there, he says. Increased sharing and transparency addresses a number of these issues, and “to the extent that the culture of the young scientist is open to that, that’s great. Although it’s hard for them to change the culture, but over time this can help.”

A “more nimble publication system might encourage scientists to publish confirmatory or negative results,” Harris states. When asked specifically about the role of preprints and alternative forms of science communication, he acknowledges that experiments in openness and transparency are interesting although it’s unclear how successful they will be. Will that additional literature be less reliable, he wonders? “It’s the job of the entire community, not simply the scientist who makes a claim, to figure out what’s right and what’s wrong,” he says, when discussing the viable options of prepublication sharing of work and use of blogs and commenting as forums for peer review. However, as a science journalist, he ponders: “Do we want more literature out there or do we want more strict checks and less literature? Personally, I would like to see less literature, have people spend more time thinking about what they’re doing and being sure they’re right.”

Responsible Reporting and News Flow

Science journalists can help the issue of scientific rigor, acknowledges Harris. “They should step back from doing the story of the day. It takes more time to think about things from a broader perspective, but that’s more important than ever.” Science journalists must find a balance between that and satisfying the needs of their news organizations. Even an award-winning science correspondent like Harris admits that he needs to think differently about how he does his job on a day to day basis. “Look at what’s published and fits into the broader trend and context of similar results, and what it means elsewhere,” he recommneds. These are results that should be the focus of science journalists.

Easing the Tug of War

The good news is that many institutions now evaluate candidates for jobs and promotions based on a set limit of publications, chosen by the author to best represent their contributions to science. Additional efforts to broaden evaluation within the constraints of the existing scientific culture include recognition and credit for reagent validation, peer review activities and training others on experimental responsibility. Dedicating more time to thinking and less time to drafting and revising manuscripts may not the working philosophy of many labs or the culture of science, but it’s an issue getting attention and was expressed as a concern by leaders of the organization Rescuing Biomedical Research.

Everyone has a stake in the current structure of the scientific enterprise, says Harris, from journals caring about impact factors [PLOS de-emphasizes journal impact factors] to deans making sure scientists draw in funding and overhead for their institutions.

“The biomedical research enterprise is driven by economics; economic systems are much harder to change and that should be something the overall enterprise should be thinking about—how to rethink that.”

Harris believes clinical medicine in the 1990s experienced similar cultural stresses, but ClinicalTrials.gov, a web-based resource from the National Library of Medicine that provides public access to information about clinical studies and the availability of experimental drugs, got people “past some of these issues without making them rethink their place in the scientific universe.” Hopefully academic biomedical research can do the same.

****************

Richard Harris, science correspondent for National Public Radio (NPR) received the 2010 Kavli Science Journalism Award from the American Association for the Advancement of Science (AAAS) for his reporting on BP’s Deepwater Horizon oil spill in the Gulf of Mexico. He also received the award in 1988 and 1995. He currently covers biomedical sciences with a focus on investigative stories and in 2014 completed an eight-part series examining the stresses on biomedical research in the US caused by fluctuating funding levels.

Tug of War Image Credit: falco; pixabay.com