Last week the New England Journal of Medicine (NEJM) published a paper on selective outcome reporting in clinical trials (Vedula et al. 2009). The primary and secondary outcome(s) of a clinical trial could for example be survival in cancer patients or rate of heart attacks and other cardiovascular events in patients taking cholesterol-lowering drugs. These outcomes are defined in the study protocol before the first patient is treated, and whether or not the primary outcome is reached (using statistical testing) defines whether a trial was positive or negative. The study protocol is approved by an institutional review board (IRB) and can only be changed later (and that includes changes in the protocol-defined outcomes) if again approved by the IRB.
The study in the NEJM examined the outcomes of 20 clinical trials for the drug gabapentin using internal documents of the drug companies Pfizer and Parke-Davis, and compared them to the outcomes reported for those trials that were published as peer-reviewed papers. The internal company documents were obtained as the result of litigation in which the company admitted guilt for off-label marketing (i.e. marketing for uses that were not approved by the Food and Drug Administration) of gabapentin.
12 of the 20 trials were published as peer-reviewed papers. In 8 of these 12 papers the primary outcome differed from the primary outcome defined in the protocol. New primary outcomes were introduced in 6 papers, and protocol-defined primary outcomes were not reported in 5 papers. Trials with primary outcomes that were not significant were either not reported as full paper or were reported with a changed primary outcome.
Selective outcome reporting has also been reported by other authors and is not limited to trials funded by drug companies.1 Selective outcome reporting is a major problem because it a) distorts our scientific knowledge and b) is unethical as it involves research on human subjects. This could for example lead to repeated studies of a clearly ineffective or harmful drug or intervention. The distortion of our scientific knowledge by selective outcome reporting in scientific journals is also a concern for research in other areas. Confirmation of important research findings by independent groups is important, but a lot of research is probably repeated simply because negative results were not published. Some of these reporting biases could be avoided by making more unpublished research data available, either by Open Notebook Science or by publishing “unexciting negative” findings in peer-review journals or preprint archives such as Nature Precedings.
Clinical medicine tries to solve the problem of publication bias by making public registration of clinical trials mandatory before patient enrollment. This registration is required by most major medical journals since 20042. U.S. legislation also requires clinical trial registration and since 2008 (FDAAA: Push to open data in clinical medicine) this includes outcome reporting within 12 months after data for the last subject were received in the publicly available Clinicaltrials.gov database.3 These efforts should ensure ethical standards of clinical research, and help to avoid the kinds of biases reported in the paper by Vedula et al.
Clinical trial registries obviously also serve other purposes, as they help interested patients and their relatives and treating physicians to find clinical trials they want to participate in, and they help researchers and clinicians to do a systematic overview of the ongoing research in a particular field. Just as there are centralized databases (e.g. PubMed Central and the Current Controlled Trials metaRegister of controlled trials) and institutional repositories for fulltext papers, there are also both central (e.g. Clinicaltrials.gov)4 and institutional clinical trial registries. But in contrast to institutional repositories, there are no standard software tools available that an institution can use. I am involved in building a clinical trial registry for our institution, and the effort is both technically demanding, and socially challenging. The information in a clinical trial registry is constantly changing and we need to keep the efforts required by the individual reseachers at a mininum. We also have to walk a fine line of what information can be made publicly available, and here we follow the standards provided by both Clinicaltrials.gov and the WHO.
Vedula, S., Bero, L., Scherer, R., & Dickersin, K. (2009). Outcome Reporting in Industry-Sponsored Trials of Gabapentin for Off-Label Use New England Journal of Medicine, 361 (20), 1963-1971 DOI: 10.1056/NEJMsa0906126
fn1. Chan AW et al. CMAJ 2004 doi:10.1503/cmaj.1041086
fn2. De Angelis C et al. NEJM 2004 doi:10.1056/NEJMe048225
fn3. EudraCT, the mandatory European clinical trials registry is regretably not available to the public.
fn4. Both PubMed Central and Clinicaltrials.gov are efforts by the U.S. National Institutes of Health.