Dear Newspapers: Individual Studies Do Not Exist In A Vacuum

Newspapers B&W (1)

All too frequently, newspapers portray individual studies as the definitive answer on a given topic.  This is a problem because most studies are not the definitive answer on anything. That is why researchers are constantly trying to replicate each  others’ work.

Just because one study finds a relationship between A and B, does not mean that other studies will be able to replicate that finding, or that it will extend to other situations. On the face of it, this seems like an incredibly obvious statement.  And yet it’s something that newspapers often forget, and which I think could have some very negative consqeuences.

As an example, I’d like to bring your attention to the media attention surrounding a recent article on sedentary behaviour.

The story

Earlier this year my friend and colleague Valerie Carson published an interesting paper examining the health impact of various types of sedentary behaviour in a sample of 2500 children and adolescents.  They created a clustered risk score (CRS) which took into account a child’s waist circumference, blood pressure, cholesterol, and inflammation, and then examined whether it was associated with 3 different measures of sedentary behaviour – accelerometry (an objective measure of movement), self-reported TV watching, and self-reported computer use.

Here is what they found (emphasis mine):

Volume and patterns of sedentary behavior were not predictors of high CRS after adjusting for MVPA and other confounders (P > 0.1). For types of sedentary behavior, high TV use, but not high computer use, was a predictor of high CRS after adjustment for MVPA and other confounders. Children and adolescents who watched ≥4 hours per day of TV were 2.53 (95% confidence interval: 1.45-4.42) times more likely to have high CRS than those who watched <1 hour per day.

The study is an interesting one and has some important implications.  Not surprisingly, it has received some well-deserved media attention in the past few weeks.  Here is what the Daily Mail had to say:

Watching TV most damaging pastime for inactive children, increasing risk of heart disease

Watching television is the most damaging activity an inactive child can indulge in, a study has warned.

Exploring the health impact of different types of sedentary behaviour, scientists discovered that high levels of TV viewing were associated with an increased risk of heart disease, compared with other pursuits such as computer use.

It is now hoped that the findings will encourage parents to be more aware of the damaging effects certain activities can have.

And here is a sampling of other headlines from around the web:

Net Doctor: Watching TV ‘worse for heart than computer use’

The Hartford Courant: Sitting At Computer Healthier Than Sitting At TV

My Health News Daily: TV Harms Kids’ Health More than Computer Time

The Kingston Whig Standard: Television loses battle of sedentary giants

Reading those articles and headlines, a person could quite reasonably conclude that TV watching is worse for a kid’s health than using a computer. Case closed, right?  Not exactly.

The above study looked at one specific group of children, and a specific group of health outcomes. Last month, our group in Ottawa published another paper (led by Dr Gary Goldfield) looking at different types of sedentary behaviour and heart disease risk factors in a cohort of overweight and obese teens (in contrast, the earlier study was on a sample of nationally representative youth). Interestingly, we found that neither TV time nor computer time was associated with increased risk in this group in our dataset it was video games that were by far the most  important sedentary behaviour.

From our abstract:

Seated video gaming was the only sedentary behaviour associated with elevated BP and lipids before and after adjustment for age, sex, pubertal stage, parental education, body mass index (BMI), caloric intake, percent intake in dietary fat, physical activity (PA) duration, and PA intensity. Specifically, video gaming remained positively associated with systolic BP (adjusted r = 0.13, β = 1.1, p<0.05) and total cholesterol/HDL ratio (adjusted r = 0.12, β = 0.14, p<0.05).

Given this new information, if we only care about the results of individual studies, the Daily Mail may want to re-write their headline to read:

Watching TV Playing video games most damaging pastime for inactive children, increasing risk of heart disease

But of course that would be ridiculous.  We expect studies to disagree, that’s the way science works. You look at slightly different populations, different measures, etc, and suddenly things change.  Everyone knows that, and yet it’s not the way that science is typically portrayed by newspapers and other news agencies.

Why is this a problem?

Put yourself in the shoes of someone who just read the Daily Mail article, and who now believes that TV viewing is the single most damaging sedentary behaviour for kids to engage in.  What reaction are you going to have when you read a similar article about our new study, suggesting that TV viewing and computer use aren’t important at all, but that video games are actually “the most damaging activity an inactive child can indulge in”?

You would probably be confused – if television was so important last week, how is it so completely unimportant this week?!?  You might begin to question why these researchers can’t get their act together and figure out what’s actually going on, rather than making one claim and then following it up with a contradictory one.  And then you may tune out from any articles on the topic in the future, since you can’t really trust those researchers to stick with a finding for more than a few weeks anyway.  In more controversial areas of research (e.g. cell phones and cancer), this approach of sensationalizing every new study can have a very negative impact on the public discourse.

What is the solution?

The way to solve this problem is not to write a bunch of articles saying that TV viewing is less important than computer use.  To be honest, all journalists really need to do is dial back the enthusiasm a bit, rather than painting every study as a GROUNDBREAKING NEW FINDING.

Journalists may also want to shift away from writing about individual studies, and look instead to systematic reviews. This is what researchers and policy-makers are doing already.  We know that many published findings turn out to be false (some have argued that most findings are false) and so when we want to know the definitive answer to a question, we look at systematic reviews rather than individual studies.

Trying to understand the health impact of any given behaviour (e.g. sedentary behaviour, physical activity, smoking, etc) is a bit like trying to make a map of a city by taking thousands of independent pictures using different angles, distances, and resolution, without knowing how all the pictures link together. Any one picture (or study) tells you relatively little about the city, and some pictures may seem to contradict (e.g. one picture may suggest the city is grassland, while another picture may suggest it is incredibly urban). But if you take enough pictures from enough angles, you start to get a pretty good sense of what the city looks like.

Systematic reviews are an attempt to bring order to that chaos by organizing the pictures, grouping types of pictures together, and placing more weight on the high quality pictures, while reducing the emphasis of low quality pictures, or simply throwing them out entirely.

If journalists focused more on systematic reviews rather than individual studies (and there are plenty of systematic reviews coming out these days), they’d be less likely to steer people in the wrong direction, and more likely to be spreading a message that will hold up over the long term.

On the bright side…

Did I mention that my friend Val and I have new papers published?  They are both open access so you can read and share them for free (Val’s paper here, mine here). Despite the differing findings, they are both good papers that will hopefully stand the test of time.

And if you have any other suggestions for ways to present data from individual studies to the general public, I’d love to hear them!


ResearchBlogging.orgGoldfield, G., Kenny, G., Hadjiyannakis, S., Phillips, P., Alberga, A., Saunders, T., Tremblay, M., Malcolm, J., Prud’homme, D., Gougeon, R., & Sigal, R. (2011). Video Game Playing Is Independently Associated with Blood Pressure and Lipids in Overweight and Obese Adolescents PLoS ONE, 6 (11) DOI: 10.1371/journal.pone.0026643

Carson, V., & Janssen, I. (2011). Volume, patterns, and types of sedentary behavior and cardio-metabolic health in children and adolescents: a cross-sectional study BMC Public Health, 11 (1) DOI: 10.1186/1471-2458-11-274

Related Posts Plugin for WordPress, Blogger...
This entry was posted in News, Sedentary Behaviour and tagged , , . Bookmark the permalink.

11 Responses to Dear Newspapers: Individual Studies Do Not Exist In A Vacuum

  1. Yannis Guerra says:

    Great post. Sadly it wont apply to all the journalism that is done to sell, instead of inform. And with most news org cutting down their science correspondents, the situation is likely to become worse and worse

  2. Excellent points. I agree that writing about specific studies rather than broader trends is a problem, because most(though not all) studies represent an incremental advance rather than a “groundbreaking” one; and as you point out, covering individual health-related studies can sometimes lead to conflicting and contradictory coverage. Given that science news is heavily driven by press releases, however, I would be surprised if this were to change any time soon. Reminds me of a PhD Comics cartoon about “The Science News Cycle”, see here at this link:

  3. Julia Belluz says:

    Very happy to see you’re tackling this very real issue. It’s one of the problems that inspired the Science-ish blog with McMaster, the Medical Post and Maclean’s. I use (mostly) systematic reviews and other high-quality evidence (instead of single studies) to evaluate health claims in the media. Glad to see we may be on the right track, bridging the “research to reporting gap”.

    • Travis says:

      Thanks for the comment! Actually, while writing this post I was thinking about the columns that do it right and yours was one that came to mind. I originally titled the post “Dear Journalists” and the retitled it to “Dear Newspapers”, since it really seems that newspapers and newswires are the ones who do this the most, while magazine columns take a broader and more nuanced view.

      As always, keep up the great work with Science-Ish.

  4. Audrey Silk says:

    Mutant Dragon beat me to it. While I couldn’t agree more that journalists and whole news organizations have this bad habit of sensationalizing a study and love that you’ve pointed a well-deserved public finger at them, some blame is to be shared by SOME party associated with the study. But for self-aggrandizing press releases ABOUT the study, the media wouldn’t know that they exist to write about it. Not only that, press releases are many times released prior to the publication of the paper in a journal! Otherwise known as a form of activism called Science By Press Release. No one can even review the study to check it against what the press release says which in turn is a spoon-feeding to the journalists. So maybe another solution is to a) not issue press releases until the study has been published and; b) include a very clear instruction to the media that summarizes your warning. Though I fear there’s little hope that the activist scientists that walk among you will go along with either a or b.

    • Travis says:

      Absolutely. Scientists and/or PR folks share some of the blame here.

      I do find though that often interviews are setup in such a way as to force your answers to fit into the “what is the big news?!?” mindset. I was interviewed on a national news channel earlier this week about sedentary behaviour, and the first question was “What has this new study shown?”. Which was odd, since there was no new study to discuss, and no new study had been discussed in the pre-interview. And that’s not the first time that has happened to me. So I think that we scientists need to do our part, but the impact will be limited unless journalists are on board.

  5. Pingback: Glenn Beck, Newt Gingrich Spar About Individual Mandate, Climate Change (VIDEO) | Yes You Can Sugars

  6. I second what others are saying here: keep doing what you’re doing–you’re helping to demystify what goes on behind the scenes in the scientific media.

    The general media, however, is what it is. It reduces, simplifies, exaggerates and dramatizes. Regular people need to know that the way scientific studies are rendered in the media is a unique form of inaccuracy. Thanks for writing this and many other of the posts here on this subject.


  7. Pingback: Sensationalism in Science News « Science Picks

  8. Pingback: It's Not Just Penn State | PA Laws

  9. Pingback: Nutrition Newsmakers of 2011 | No Baloney