What science is worth shouting about?

A few weeks ago I wrote about the different ways results of a clinical trial can be reported (What are the right numbers for JUPITER). Inspired by blog posts by Eva Amsen (Failure) and Sally Church (Over hyped cancer drugs or sensational journalism?), I thought more about what makes scientific findings worth talking about outside of your immediate research community. I will look at cancer research, and it is obvious that some research findings are more exiciting than others. But it less obvious that it is as important who is communicating the research and what is the intended audience. This is especially true for translational research1, where research findings can change the way we treat patients and drug companies (and others) can potentially earn a lot of money.

The academic researcher
Academic researchers involved in basic or clinical cancer research are interested in publications and grant money. As Eva said: the unit of success is the publication record. Basic cancer research can be very exciting and touches many related fields from signal transduction to epigenetics. But what confuses me is when basic research findings are related to the treatment of human cancer, which is a completely different story. Cell lines are not living organisms, mice are not men and very few basic research findings make it into a clinical trial.
Unfortunately less than one in five studies in cancer (they all have to be registered with clinicaltrials.gov since 2005) have been published in peer-reviewed journals2. Negative results will usually not be published, and this so-called publication bias creates many problems. Just imagine 5 clinical trials with a new drug where one trial will show an advantage for the new treatment. This trial will be published in a nice journal, probably one negative trial will be published in a small journal and the remaining 3 trials will remain unpublished. Just looking at the published literature will of course give the wrong impression, but these are the only data that are available to most people.

The drug company
A drug company is interested in marketing approval of a drug. This approval is obtained from the Food and Drug Administration (FDA) in the United States and from the European Medicines Agency EMEA for most of Europe. Approval of a cancer drug usually requires one or more large randomized trials (so-called phase III trials) that shows that the drug improves survival. And up to 50% of these phase III trials fail to show the desired effect, even when earlier (so-called phase II) trials were positive3. Satraplatin is one recent example of a drug that showed promising results in the treatment of prostate cancer but failed to prolong survival – and therefore was not approved by the FDA.

The insurance company (or whoever pays for medical care)
The drug erlotinib was approved for the treatment of pancreatic cancer, but the median benefit in overall survival was less than 2 weeks4. Insurance companies are interested in treatments that are not only effective, but also cost-effective. For cancer treatments one can calculate the cost per quality-adjusted life year (QALY). Adding erlotinib to gemcitabine in the treatment of pancreatic cancer would cost about $410,000 per year of life saved5. Costs over $100.000 per year of life saved are considered high and the high cost is the reason that the British National Institute for Health and Clinical Excellence (NICE) decided to not recommend 4 new drugs for the treatment of renal cancer in August 2008. NICE doesn't question the effectiveness of these drugs, but wants these drugs to become cheaper.

The media
Cancer research sells. Cancer is very common and many people can tell sad stories of friends or relatives that died because of cancer. And we want to hear encouraging stories. But the media often confuse promising findings in basic research or early clinical trials with a new cure for cancer. One of the more famous examples is the statement Judah is going to cure cancer in two years by James Watson in a 1998 New York Times story on angiogenesis research (A Cautious Awe Greets Drugs That Eradicate Tumors in Mice). Another example is this recent article (cited by Sally Church in her blog post mentioned above) in the Times (Cancer drug could save the lives of 10,000 a year). Ben Goldacre has written more than one critical blog post about how traditional media report cancer research (e.g. Still no cure for cancer hysteria).

The patient
A patient with cancer needs a drug treatment that either cures his cancer or prolongs survival. The new treatment should either be better or have fewer side effects than the current standard of care. But many cancer patients can't wait until a new cancer drug is approved and available for prescription. So whenever possible, patients should participate in a clinical trial. Clinical trial participation by adult cancer patients is unfortunately low (less than 5% compared to more than 50% for children with cancer6), but this is the best way to receive a promising new drug as early as possible (and to stop receiving it when the drug has been shown to be ineffective or toxic).

We should be careful how we report research findings outside of specialist journals or research meetings, especially if these findings could have important consequences outside our immediate research community. And as a critical reader we should assume that most published research findings are false7.

fn1. Translational Research: Getting the message across. Nature 2008 doi:10.1038/453839a

fn2. Practicing on the Tip of an Information Iceberg? Evidence of Underpublication of Registered Clinical Trials in Oncology. Oncologist 2008 doi:10.1634/theoncologist.2008-0133

fn3. Analysis of Phase II Studies on Targeted Agents and Subsequent Phase III Trials: What Are the Predictors for Success? J Clin Oncol 2008 doi:10.1200/JCO.2007.14.8874

fn4. Erlotinib plus gemcitabine compared with gemcitabine alone in patients with advanced pancreatic cancer: a phase III trial of the National Cancer Institute of Canada Clinical Trials Group. J Clin Oncol 2007 doi:10.1200/JCO.2006.07.9525

fn5. Does a Statistically Significant Survival Benefit of Erlotinib Plus Gemcitabine for Advanced Pancreatic Cancer Translate Into Clinical Significance and Value? J Clin Oncol doi:10.1200/JCO.2007.13.0401

fn6. How Sociodemographics, Presence of Oncology Specialists, and Hospital Cancer Programs Affect Accrual to Cancer Treatment Trials J Clin Oncol 2002 doi:10.1200/JCO.2002.08.056

fn7. Why Most Published Research Findings Are False. PLoS Med 2005 doi:10.1371/journal.pmed.0020124

This entry was posted in Snippets. Bookmark the permalink.

4 Responses to What science is worth shouting about?

  1. Heather Etchevers says:

    Great post, Martin. I of course agree with everything you wrote and applaud what appears to my biased mind to be a very fair appraisal of the situation. Last year I attended a European forum sponsored by the “Drug Information Association”:http://www.diahome.org/DIAHome/FlagshipMeetings/Home.aspx?meetingid=16920 (this year’s is in Berlin), as a all-fees-paid representative of a patient association. It was an eye-opener for me, to see with my own eyes the multiple processes of how and which drugs get all the way into a patient’s body, and the different interests of all the stakeholders you’ve outlined above, of whom the patients were by far the smallest minority in this forum.
    Lest you attract the ire of Maxine, may I suggest you change _as a critical reader we should assume that most published research findings are false_ to _”… should be taken with a healthy dose of skepticism”_?

  2. Martin Fenner says:

    Heather, I agree that the paper title _most published research findings are false_ is a little bit harsh. In 2008 the same author wrote “Why Current Publication Practices May Distort Science”:http://dx.doi.org/10.1371/journal.pmed.0050201. Both papers have been discussed by Maxine and others on Nature Network before, e.g. “here”:http://network.nature.com/groups/precedings/forum/topics/3222.

  3. Maxine Clarke says:

    I think this is a very well-argued and informative post, but I think that the last sentence (in regard to ref 7) is a non-sequitur. Naturally, as Heather intimates, I do not agree with it as a general statement; I think it is a pity that it has been stated as such, so baldly, in this otherwise excellent post.

  4. Martin Fenner says:

    Maxine, let me rephrase the last sentence so that the point I want to make becomes clearer:
    _And as a critical reader we should assume that only a small proportion of basic and clinical research translates into something that will directly help a patient._