Correcting the myths about missing drug trials

Not all drug trials get published. This is a problem because doctors, journalists, and others look to published data for the fullest picture of whether and how a drug works, and in whom.

The international AllTrials campaign launched a US branch this week. We need better reporting of clinical trials: Although we have legislation supposedly mandating trials to be published, many still aren’t. A 2012 study found that only 22% of trials complied with the law. According to, the FDA hasn’t ever fined anyone for violating the law.

The last time I wrote about publication bias in drug trials—just a quick pointer to this excellent Salon piece by Rob Waters—some readers just didn’t get it. Here are a few of the comments that show why we need to communicate better about why publication bias is a problem:

“There’s really no use for the data if it shows that [a drug] doesn’t work.” (here)

“People generally aren’t interested in failure. Failure isn’t progress.” (here)

“So drug co’s don’t waste their time publishing tests of products that don’t work? #efficient #Scary #journalism #WellDuh” (here)

These readers assume the missing trials are unimportant ones. If a handful of trials show that a drug works, and only those get published, don’t we have the information we need?

No, we don’t. Say there’s an antidepressant, and half the trials show that it works and half show that it doesn’t. (Why the difference? Maybe they were done on different patient populations, or maybe some were done with better methodology than others.)

Trials aren’t just for the FDA to read at approval time.

  • Researchers build their future projects on what has and hasn’t worked in the past—which they know about from published studies and data sets.
  • Writers like me look up studies to understand the evidence behind how and whether a drug works.
  • Doctors read summaries of the latest studies to find out what drugs are more effective than others, and how best to use them. (I know because I write a lot of these summaries.)
  • Reviewers, like those that write the Cochrane reviews, collect and compare trials to figure out the big picture of a treatment’s effectiveness. (Not surprisingly, the Cochrane group is a major sponsor of Alltrials.)

In addition to answering the question “Does this drug work?” published data also helps answer questions like “Does this drug work better than these older ones?” and “How do the benefits stack up against harms?” (data that theNNT presents very clearly, by the way—mostly based on Cochrane reviews.)

If a large fraction of the trials for an antidepressant are missing, we could end up with a skewed view of how well it works. That’s exactly what happened with antidepressants as a class, according to this study led by Erick Turner. Trials published in journals painted a much rosier picture of the drugs’ effectiveness than the data submitted to the FDA. And we don’t know if the FDA had complete data, either; Alltrials suggests that regulators often don’t.

Evidence-based medicine is only as good as the evidence it’s based on. Hiding data skews the understanding that doctors and researchers rely on; it isn’t harmless.

Category: Uncategorized | Comments Off on Correcting the myths about missing drug trials

Using math to make Guinness

If you ever read public health research, you’ve probably encountered the term “Student’s t-test,” or just “t-test.” The experimenters will do this magical test, and suddenly conclude that everything is awesome. But even when you’re familiar with the t-test and what it does, very little thought goes into where this came from, or who came up with it. Well, today I’m going to tell you the origins of this staple of public health research.

William Sealy Gosset, statistician and rebel | Picture from Wikimedia Commons

Let me tell you a story about William Sealy Gosset. William was a Chemistry and Math grad from Oxford University in the class of 1899. After graduating, he took a job with the brewery of Arthur Guinness and Son, where he worked as a mathematician, trying to find the best yields of barley.

But this is where he ran into problems.

One of the most important assumptions in (most) statistical tests is that you have a large enough sample size to create inferences about your data. You can’t make many comments if you only have 1 data point. 3? Maybe. 5? Possibly. Ideally, we want at least 20-30 observations, if not more. It’s why when a goalie in hockey, or a batter in baseball, has a great game, you chalk it up to being a fluke, rather than indicative of their skill. Small sample sizes are much more likely to be affected by chance and thus may not be accurate of the underlying phenomena you’re trying to measure. Gosset, on the other hand, couldn’t create 30+ batches of Guinness in order to do the statistics on them. He had a much smaller sample size, and thus “normal” statistical methods wouldn’t work.

Gosset wouldn’t take this for an answer. He started writing up his thoughts, and examining the error associated with his estimates. However, he ran into problems. His mentor, Karl Pearson, of Pearson Product Moment Correlation Coefficient fame, while supportive, didn’t really appreciate how important the findings were. In addition, Guiness had very strict policies on what their employees could publish, as they were worried about their competitors discovering their trade secrets. So Gosset did what any normal mathematician would.

He published under a pseudonym. In a startlingly rebellious gesture, Gosset published his work in Biometrika titled “The Probable Error of a Mean.” (See, statisticians can be badasses too). The name he used? Student. His paper for the Guinness company became one of the most important statistical discoveries of the day, and the Student’s T-distribution is now an essential part of any introductory statistics course.

So why am I telling you this? Well, I’ve talked before about the importance of storytelling as a way to frame scientific discovery, and I’ve also talked about the importance of mathematical literacy in a modern society. This piece forms the next part of that spiritual trilogy. Math is typically taught in a very dry, very didactic format – I recite Latin to you, you remember it, I eventually give you a series of questions to answer, and that dictates your grade in the class. Often, you’re only actually in the class because it’s a mandatory credit you need for high school or your degree program. There’s very little “discovery” occurring in the math classroom.

Capturing interest thus becomes of paramount importance to instructors, especially in math which faces a societal stigma of being “dull,” “boring” and “just for nerds.” A quick search for “I hate math” on Twitter yields a new tweet almost every minute from someone expressing those sentiments, sometimes using more “colourful” language (at least they’re expanding their vocabulary?).

There are lots of examples of these sorts of interesting anecdotes about math. The “Scottish book” was a book named after the Scottish Café in Lviv, Ukraine, where mathematicians would leave a potentially unsolvable problem for their colleagues to tackle. Successfully completing these problems would result in you receiving a prize ranging from a bottle of brandy to, I kid you not, a live goose (thanks Mariana for that story!) The Chudnovsky Brothers built a machine in their apartment that calculated Pi to two billion decimal places. I asked for stories on Twitter and @physicsjackson responded with:

Amalie (Emmy) Noether is probably the most famous mathematician you’ve never heard of | Photo courtesy Wikimedia Commons

There’s also the story of Amalie Noether, the architect behind Noether’s theorem, which basically underpins all modern physics. Dr Noether came to prominence at a time when women were largely excluded from academic positions, yet rose through the ranks to become one of the most influential figures of that time, often considered at the same level of brilliance as Marie Curie. Her mathematical/physics contemporaries included David Hilbert, Felix Klein and Albert Einstein, who took up her cause to help her get a permanent position, and often sought out her opinion and thoughts. Indeed, after Einstein stated his theory of general relativity, it was Noether who then took this to the next level and linked time and energy. But don’t take my word for it – Einstein himself said:

In the judgment of the most competent living mathematicians, Fräulein Noether was the most significant creative mathematical genius thus far produced since the higher education of women began.

While stories highlight the importance of these discoveries, they also highlight the diversity that exists within the scientific community. Knowing that the pantheon of science and math heroes includes people who aren’t all “math geniuses” can make math much more engaging and interesting. Finally, telling stories of the people behind math can demystify the science, and engage youth who may not consider math as a career path.



This post originally appeared on PLOS Sci-Ed

Category: Epidemiology, History of Public Health | Comments Off on Using math to make Guinness

The ‘Dad Bod’ Explained: A Study of Weight Gain during Fatherhood



The ‘Dad Bod’ | Image source: Flickr

Does a woman’s pregnancy affect the weight of her partner? And is weight gain sustained while the kids grow up? Anecdotally, many people would undoubtedly say yes from their own experiences. Lack of sleep, less time for exercise, and having a cheeky bite of junk food with the kids all add up in a serious way. We poke fun at the ‘dad bod’, but is it a real phenomenon among fathers?

This is actually a difficult question to answer. The transition to parenthood occurs over time. As we also inevitably age over time, our metabolisms sadly slow down and we usually gain weight. So, how to disentangle the causal effect on weight gain of becoming a parent versus simply getting older?

A study published last week in the American Journal of Men’s Health set out to answer this exact question. They investigated the effect of the transition to fatherhood on young men’s body mass index (BMI) at the population level in America.

Data were from the National Longitudinal Study of Adolescent to Adult Health, a nationally representative study of 10,263 young men aged 12-21 years old when the study began in 1994-1995 (1). They followed the men up for over 20 years, until they were aged 25-34 years old in 2007-2008. The authors assessed the men’s BMI, whether and when they had had children, and many other important demographic and lifestyle factors.

Men were grouped into three categories: resident fathers, non-resident fathers, and non-fathers. The trajectory of standardized BMI over time is shown below.


Time is shown in “Fatherhood years”, where negative years are pre-birth of the child, 0 years is at the birth of the child, and positive years are post-birth of the child.

The ‘Fatherhood effect’ was an average weight gain of 4.4 pounds (2.6% increase in BMI) for a 6-foot tall man becoming a first-time dad and residing with his children.

First-time dads who did not live with their children had a 2% increase in BMI, on average, from the time they became fathers. For a 6-foot tall man, this translates to 3.3 pounds gained.

During the time period when the fathers gained weight post-birth, the childless men actually declined by about 1% in BMI until the end of the study. For a 6-foot tall man, this translates to 1.4 pounds lost. Weight change in the non-father group was attributable to age (there was a small overall increase in BMI in this group, despite the slight decline in the latter half of the study), while additional weight change in fathers was attributed to the life transition of having children (1).

In a press release, the lead author of the study, Dr Craig Garfield, Associate Professor of Pediatrics and Medical Social Sciences at Northwestern University Feinberg School of Medicine and attending pediatrician at Ann & Robert H. Lurie Children’s Hospital of Chicago, said,

Fatherhood can affect the health of young men, above the already known effect of marriage. The more weight the fathers’ gain and the higher their BMI, the greater the risk they have for developing heart disease as well as diabetes and cancer.

You have new responsibilities when you have your kids and may not have time to take care of yourself the way you once did in terms of exercise. Your family becomes the priority.

We now realise the transition to fatherhood is an important developmental life stage for men’s health. It’s a magical moment where so many things change in a man’s life. Now the medical field needs to think about how can we help these men of child-rearing age who often don’t come to the doctor’s office for themselves.

There you have it. The ‘Dad Bod’ is a real phenomenon, and not just a joke on social media. Weight gain associated with life transitions such as parenthood is a real concern, and public health, social, and clinical supports should be available for men during this time.


1. Garfield CF, Duncan G, Gutina A, Rutsohn J, McDade TW, Adam EK, Levine Coley R, Chase-Lansdale PL. Longitudinal study of body mass index in young males and the transition to fatherhood. Am J Mens Health 2015; Epub ahead of print DOI: 10.1177/1557988315596224

Category: Determinants of health, Epidemiology, Nutrition, Time trends | Tagged , , , , , | 3 Comments

The Case for Unlimited Tablet Time for Toddlers

This sounds extreme, but first let me ask: how many parents do you think actually keep track of their kids’ screen time? If the TV is on but one of the children wanders out of the room, does that count? What if they’re following along to a yoga video? What if the kid borrows Mom’s phone at dinner to ask Google what snails eat?

Guidelines abound that encourage limiting “screen time.” The American Academy of Pediatrics, for example, recommends two hours or less per day, and says screens “should be avoided” for kids under 2. While I hate to see kids vegging out in front of the TV, I think these limits are based more on knee-jerk reactions (kids these days and their screens!) than on anything that’s actually meaningful to kids’ development.

Most of the research on screen time comes from studies of kids who watch TV compared to kids who don’t, as Emily Oster explains at Fivethirtyeight—and the effects typically disappear when demographic differences like income, race, and education are taken into account.

Television probably isn’t inherently bad for kids, and other tablet activities like video games shouldn’t be lumped in with TV viewing anyway. I surveyed the research on this when I wrote about tablet time for Lifehacker. To summarize the important differences:

Television is passive: stuff happens, and you get to watch. The stuff that happens isn’t personalized to you, either: somebody’s grandma might be on TV, but it’s not your grandma.

One of the pediatricians who authored the AAP’s recommendations wrote in JAMA Pediatrics (careful to note that he was speaking for himself and not the AAP) that play on an iPad is far more similar to playing with blocks, or reading a book with a caregiver, than it is to passive TV watching.

The evidence-based recommendations from nonprofit Zero to Three focus on finding appropriate content rather than setting blanket limitations on time, although their evidence for the importance of “appropriate” content is weak in places.

Maybe it just sounds better to say kids should be playing educational video games with Dad instead of flinging birds at asteroids over and over again. But I don’t know about that. My toddler has no problem with his letters and numbers, having figured them out through some combination of real life and, ok, maybe TV—but he understands orbital mechanics better from Angry Birds Space than I ever did from high school physics class.

(When I mentioned Angry Birds in my Lifehacker piece, I got commenters insisting that I was mistaken and my toddler couldn’t possibly be doing more than aimless swiping. Here’s news: if you don’t think a one-year-old can clear levels on Angry Birds, then you haven’t spent much time with one-year-olds lately.)

My kids each have a tablet (a Nook HD+, bought on a very good sale, and rooted to run cyanogenmod’s version of Android), and they decide their own age-appropriate uses for it. The five-year-old uses his to build elaborate things in Minecraft (a sort of lego-block world) and to research whatever is on his mind (“OK Google, show me pictures of narwhal skeletons.”) The two-year-old explores the physics of Angry Birds and Monument Valley. Both children play problem-solving games like Cut the Rope and Bad Piggies, sometimes for hours. And, to be fair, they also watch a lot of Youtube.

Imagine, for the moment, that you wanted a kid to spend hours on a tablet. What problems do you foresee? They’d get bored with it. They’d find new and different things to do with it than whatever you proposed. They’d leave it at the bottom of the toybox for days if some newer, more interesting toy came into their life. Guess what? That’s exactly what happens when you allow unlimited screen time and the novelty wears off.

I don’t think a forbidden fruit policy is the best. If my kids only got an hour of screen time a day, I promise you they would only use it to watch cartoons. But sometimes they send texts to their dad or their grandparents, or they sit down with me to puzzle out an adventure game, or the older boy will teach his little brother how to install and play something new.

Today, Google and touchscreens and online communication are just part of the background of everyday life. My kids know how to skip ads on videos and how to navigate a website even if they can’t read the words on it. They know what information you can get from a phone, and will pipe up to recommend I text Daddy or ask Google or punch something into the GPS. Keeping kids away from screens is as nonsensical as if parents of the past kept their kids away from the radio, or the telephone, or pencils and paper.

For the Lifehacker piece, I asked Clare Smith, a language development researcher who has written about screen time, whether she agrees with my view on this. Here’s what she said:

We now live in a world where this technology opens up opportunities for extended social engagement, learning, work and leisure. In fact, it is becoming the norm, and our children will be expected to be proficient in this technology. It is just another form of media that can and should be used in whichever way an individual chooses. Choosing devices and apps is just the same as choosing toys or books, and each choice should be made on its own merits. My own children are embracing social networking and gaming and we are trying our best to guide them through the associated risks and benefits. Just as a conscientious parent teaches a child road safety, stranger danger, eating a healthy balanced diet and a disciplined approach to learning, so this may extend to learning about modern technology and the online world.

Screen time isn’t something to protect kids from; it’s just part of our world. Let’s stop pretending that it’s some kind of tragedy to hand a toddler an iPad.

Category: Uncategorized | Tagged , , , , | 1 Comment

Questioning the link between citrus fruit and skin cancer



Does consumption of citrus fruits lead to increased melanoma risk? Image source: NIH, Public Domain

A research article titled ‘Citrus Consumption and Risk of Cutaneous Malignant Melanoma’ was just published online in the Journal of Clinical Oncology (1). You may have seen the several news headlines associated with the research. It caught my eye, as this relationship was news to me. Given that the results of the research would add a caveat to current cancer prevention recommendations, which encourage 5 or more servings per day of fruits and vegetables, it is worthwhile to break the study down.

The biological rationale for citrus fruits and malignant melanoma


Psoralen derivatives.

Citrus fruits are high in psoralens, which are a group of naturally occurring chemical compounds called furanocoumarins. Furanocoumarins are the compounds that cause grapefruit to adversely interact with several medications in the gut (this is why grapefruit is not served in hospitals). Furanocoumarins increase skin photosensitivity when applied topically to the skin and also when taken orally. For example, they are used in combination with UVA light to treat psoriasis. Psoralens also used to be used in tanning creams to enhance tanning.

So, although the idea may not be readily apparent to many, the authors of the study didn’t pull the idea of out thin air. The biological rationale is legitimate and worthy of investigation. They had done previous research looking at antioxidant nutrients and melanoma risk, and had found that women with a higher intake of orange juice and dietary vitamin C unexpectedly had an increased risk for melanoma, while this effect was not apparent in women who only took supplementary vitamin C (1). The authors then hypothesised that some other compound present in foods containing vitamin C (i.e. citrus fruits) must be responsible for the increased melanoma risk they observed (1). Psoralens were a compound that made biological sense, and this study was born.

What did they do?

Prospective data came from 63,810 women in the U.S. Nurses’ Health Study and 49,617 men in the Health Professionals Follow-Up Study. The study began in 1984 for the women and 1986 for the men, where both groups filled out a ‘baseline’ questionnaire about their diet, along with providing other data such as medical history and other lifestyle risk factors. The study participants were followed forward to track any new diagnosis of malignant melanoma, and they filled out ‘follow-up’ questionnaires every two to four years. Each person was followed until one of three events occurred: first diagnosis of any new cancer, death, or end of the study in January 2010 (men) or June 2010 (women).

In the food frequency questionnaires, the study participants were asked how often, on average, they had consumed grapefruit (half), oranges (one), and grapefruit and orange juices (on small glass) in the past year. The total individual citrus fruit servings defined ‘overall citrus consumption’, the main exposure variable in the study. This variable was updated every two to four years to better reflect long-term dietary intake.

The researchers used Cox proportional hazards models to estimate hazards ratios for malignant melanoma according to frequency of consumption of:

a) overall citrus fruit

b) grapefruit

c) grapefruit juice

d) oranges

e) orange juice

The authors accounted for potential confounding variables in the analysis, meaning that the results are independent of these factors: age, family history of melanoma, natural hair colour, number of arm moles, sunburn susceptibility as child or adolescent, number lifetime blistering sunburns, cumulative ultraviolet flux since baseline, average time spent in direct sunlight since high school, body mass index, physical activity, smoking status, and intake of total energy, alcohol, coffee, and vitamin C from supplements. Analyses for women also accounted for menopausal status and postmenopausal hormone therapy.

What did they find?

The risk of malignant melanoma increased with increasing overall citrus fruit consumption, in a ‘dose-response’ fashion (1).

Men and women who reported eating citrus fruits more than 1.5 times per day had a 36% increase in risk for malignant melanoma compared with people who ate citrus fruits less than twice per week.

Curiously, when the different types of citrus fruits were looked at individually, grapefruit and orange juice consumption were associated with increased melanoma risk, but grapefruit juice and oranges were not. The study authors have been met with some criticism about the consistency of the results across different types of citrus fruits and the small number of cases of malignant melanoma that the conclusions are based on (to be fair, it is a rare cancer).

What does it mean?

The lead author of the study, Shaowei Wu, a postdoctoral research fellow at Brown University, said in the press release,

“While our findings suggest that people who consume large amounts of whole grapefruit or orange juice may be at increased risk for melanoma, we need much more research before any concrete recommendations can be made.”

The authors conclude that the lack of associations they observed with oranges and grapefruit juice was because people ate less of them than they did of orange juice and grapefruits. Dubious? It’s hard to say, given that errors in the measurement of people’s actual consumption could also explain the results. People may make errors in recalling how often they ate citrus fruits in the past year, adding noise to the data. It has also been pointed out that the study participants were all health professionals, and so they may be better at detecting skin lesions indicative of melanoma than the general population (2).

My question is, how many people actually eat citrus fruits more than 1.5 times per day? Also, if the absolute lifetime risk of developing malignant melanoma is about 2.1% (2), then crudely calculated, a 36% relative increase in risk due to eating citrus fruits more than 1.5 times per day (assuming causality) would result in a lifetime risk of 2.9% for malignant melanoma. Because people’s baseline risk of malignant melanoma is relatively low, a 36% increase in relative risk does not mean as much as it would for a more common cancer, such as breast cancer.

Where do we go from here?

This study is an example of the kind of research that can lead to a lot of public confusion. People say that ‘everything causes cancer’, and it’s partly from reading headlines like the ones accompanying this study. It is difficult because there might be a public health relevance to the findings – if eating citrus fruits increases cancer risk to a significant degree, then the public should know. But, it’s hard to know what to do when we also know that citrus fruits are a key source of vitamin C and count towards daily servings of fruit and vegetables. As Marion Nestle said, ‘I’d worry much more about alcohol and cigarettes‘ (3). It will be interesting to see if the results of this research are replicated in other studies.


1) Wu S, Han J, Feskanich D, Cho E, Stampfer MJ, Willett WC, Qureshi AA. Citrus consumption and risk of cutaneous malignant melanoma. J Clin Oncol 2015; 33 doi: 10.1200/JCO.2014.57.4111

2) Surveillance Research Program, National Cancer Institute. SEER Stat Fact Sheets: Melanoma of the Skin. (accessed 11 July 2015).

3) Eunjung Cha A. Citrus consumption and skin cancer: how real is the link? Washington Post. 29 June 2015. (accessed 11 July 2015).





Category: Cancer, Epidemiology, Nutrition, Science Outreach | Tagged , , , , | 6 Comments

Can Lightning Strike an Indoor Pool?

Two swimming pool weather policies have surprised me in recent years. One was when I showed up to swim laps at an outdoor pool as it was beginning to drizzle. “Come on in,” I was told; as long as there was no lightning, the pool was still open. So I had one of my weirdest swims ever: you turn your head out of the water to breathe, but your face is still wet. Thanks, rain.

Here’s the other: an indoor pool I know closes during thunderstorms. Surely they’ve bought into some bizarre urban legend, right?

If it’s a myth, it’s a popular one. As one YMCA explains,

The YMCA of the USA, FEMA, the American Red Cross, NOAA, and the National Lightning Safety Institute all advise to not use plumbing of any kind during thunder and lightning and to stay clear of windows.

Meanwhile, the Director of Aquatics at a university athletic department agrees with my first impression. He writes that closing indoor pools for lightning is “THE urban myth” and adds:

At Penn State University we keep our four indoor pools open during electrical storms with the approval of our Environmental Health and Safety Department, the Risk Management Department and our High Voltage Experts on campus.

From there, the debate gets weird, and is mostly conducted on early-2000s-era websites. (I saw more blinking text and Comic Sans researching this topic than I ever thought I would see in an afternoon in 2015.) Several of the anti-pool-closing websites mentioned a Dr. Vicki Weiss, quoted in 2008 in Aquatics International as saying that “a pool closure policy is in violation of the National Electric Code section 250.4(A)(1) and you will be subject to regulatory enforcement.”

I looked up the National Electric Code section 250.4(A)(1). Here’s what it says:

250. Grounding and bonding

(A) Grounded Systems.

(1) Electrical System Grounding. Electrical systems that are grounded shall be connected to earth in a manner than will limit the voltage imposed by lightning, line surges, or unintentional contact with higher-voltage lines and that will stabilize the voltage to eart during normal operation.

Informational note: An important consideration for limiting the imposed voltage is the routing of bonding and grounding electrode conductors so that they are not any longer than necessary to complete the connection without disturbing the permanent parts of the installation and so that unnecessary bends and loops are avoided.

There’s nothing in there about pool closing policies. It does say that electrical systems should be bonded and grounded to minimize possible damage from lightning. Does that mean that you are totally perfectly safe when in a pool?

Not necessarily. The Redwoods Group, which provides insurance to some YMCAs, notes that some pool buildings were built before this regulation was in effect, some have metal bleachers or lifeguard chairs that aren’t connected to ground, and that the bonding in a well-built system can still deteriorate over time. That means some pools are probably safe in a lightning storm, some definitely aren’t, and there are plenty whose status isn’t clear.

Now the inconsistent policies make more sense. Aquatics directors who are confident that their building is protected from lightning may choose to keep the pool open during storms; others may know that theirs is not safe or may choose to close out of caution. Lightning has been observed in indoor pools, causing injuries and electrical damage.

One last point: a detail both sides repeat is that nobody has ever been killed by lightning while swimming indoors. But NOAA’s data on lightning fatalities don’t distinguish between indoor and outdoor swimming. This factoid is based on an absence of evidence, not evidence of absence.

How do most lightning fatalities occur? Fishing and camping, according to NOAA, with many occurring as somebody seeks shelter too late. Men make up 81% of total fatalities, and 90% in the fishing and sports categories. Better to get inside (or out of a pool, if asked) than to end up like one of these unfortunate but comically animated pixel people.

Category: Uncategorized | 3 Comments

Using Video Games to Model Real Life Outbreaks

Those of you who know me know that I’m a video game nerd. And comic book nerd. And just nerdy nerd in general. So when I read an article that used World of Warcraft to model disease outbreaks, I jumped on it.

World of Warcraft is a MMORPG (Massively Multiplayer Online Role Playing Game) and forms the butt of many jokes in shows like Community, The Simpsons and others. I’ve never played it myself, but I lived with a guy who did so picked up a few things. Basically, you pick a player class (barbarian, wizard etc) and then join a “guild” and do quests together. These vary from the mundane to the epic (“kill this dragon”). It is, allegedly, a lot of fun. And a lot of that fun comes from being in a group of 50-60 like minded people, all playing out their fantasies as an elf, warlock, goblin etc.

World of Warcraft (WoW) has a very intricate world that has grown up around it. Gold provides an in-game economy, and treasures you gain from slaying foes give people items to trade. And since it is based around the actions of people, each quest can be very different from the last. Sometimes this can result in inadvertently hilarious consequences; the video below shows a guild meticulously planning their attack. However, when a player decides that he’s had enough, he runs in screaming his name (“LEEEEROOOOYYYYY JENNNKINNNSSSSS”). This results in his team panicking, and all their planning going to waste. To quote Robert Burns: “The best-laid schemes o’ mice an’ men / Gang aft agley.” I’m pretty sure he was talking about WoW when he wrote that.

So you have this society with thousands of players all logging on regularly, heavily invested in their characters, spending anywhere upwards of 40-60 hours a week in the game. What happens when a “virus” is introduced into the game?

Video games have a long and storied history of teaching people and modelling how they react to external stimuli. For example, the game Pandemic (link here) taught people that the safest place in the case of a zombie outbreak is Madagascar, because they close their port IMMEDIATELY. You might be able to infect every other country in the world, but if you don’t get to Madagascar early, you lose. (Ed note: I’ve never successfully infected Madagascar. It’s infuriating).

Pandemic 2. Good luck infecting Madagascar!

Pandemic 2. Good luck infecting Madagascar!

In the World of Warcraft … world … the developers occasionally introduce updates to add functionality, balance characters and add quests. An update issued in September of 2005 added a new boss character called “Hakkar the Soulflayer” who would cast a spell named “Corrupted Blood” that caused you to lose health points every second. After you were infected, you could then pass the disease onto your teammates and other players who were around you. It was limited to one area though, and only the one boss character could cast the spell. But then players found out that if you teleported to a nearby town, the spell would continue to infect others who had nothing to do with your quest, and weren’t fighting that boss, but just relaxing in “Ye Olde Tavern.”

The “Corrupted Blood” spell being spread between players

This resulted in two main effects on the in-game world, and paralleled how people respond to outbreaks in the real world.

First, the good. Some players (generally higher level “healing” characters) would provide their services to try and either remove the plague or heal those afflicted. Lower level characters would stand further away, and direct people away from the infected areas, trying to stop the spread of the disease. Blizzard (the company that makes WoW) also introduced a voluntary quarantine of those infected.

On the other hand, there were those who wanted to spread the disease. These people worked to bypass safety measures that Blizzard Entertainment implemented to try and control the spread of the disease, and infected non-player characters, as well as summoned animals. Every time Blizzard introduced a new security measure, they tried to bypass it, as well as spread the disease in ways that the developers could not have predicted.

What does this mean for researchers?

Well, we have a thriving online community that is infinitely better than a mathematical model or simulation. While we can predict how we *think* people will respond to a disease outbreak and model that using software and code, we can’t be sure. The “Corrupted Blood Incident” adds in the human factor and an element of randomness that improves our model. In addition, we can also use it to model experiments that would be otherwise unethical. Introducing a disease that “kills” players in a game is a minor annoyance (you have to wait to come back to life, you lose some gold, no big deal). Introducing this in the real world would be impossible. In addition, you can see how “Public Health” announcements are viewed – do players take them seriously? How do they respond when you tell them to avoid an area? Finally, for those interested in bioterrorism, you could model the spread of the disease between terrorist cells, and observe how these cells interact and spread the disease.

That being said, the use of video games isn’t necessarily the way forward for research as there are too many unknown factors. Death is nothing more than a minor annoyance in the WoW universe, and so people are much less cautious as a result. As a result, the stakes (and consequences) for both parties are considerably lowered. In addition, for those interested in biosecurity, these individuals are not committed to sowing anarchy or ruining the world, they’re just having fun. And so their actions may not mimic those of bioterrorists.

In closing, I think video games have a huge potential for both educating those who are not familiar with epidemiology, and could be very useful as a starting point for researchers. They can’t replace our existing methods and techniques – nor are they meant to. They can, however, open a whole other door though, and increase the awareness of Epidemiology in the community and provide us with opportunities, insights, and a rich testing environment.
Lofgren, E., & Fefferman, N. (2007). The untapped potential of virtual game worlds to shed light on real world epidemics The Lancet Infectious Diseases, 7 (9), 625-629 DOI: 10.1016/S1473-3099(07)70212-8
Balicer, R. (2007). Modeling Infectious Diseases Dissemination Through Online Role-Playing Games Epidemiology, 18 (2), 260-261 DOI: 10.1097/01.ede.0000254692.80550.60


Ed note: A version of this originally appeared on

Category: Determinants of health, Epidemiology, Infectious disease, Preventable Deaths, Science Outreach, Social Media | Tagged , , , , , | 5 Comments

The problem with P values: defining clinical vs. statistical significance



The problem with P values

Today we warmly welcome guest writer Sean Sinden to PLOS Public Health Perspectives. His biography is at the end of the post.

The practice of null hypothesis testing has traditionally been used to interpret the results of studies in a wide variety of scientific fields. Briefly, significance testing involves the calculation of an outcome statistic, known as the P value. The P value represents the probability of finding a difference, by chance, between two sets of values larger than that which was observed, assuming no difference between the two sets of values. Conventionally, if that probability is less than 0.05 the outcome is deemed “statistically significant”. If this sounds confusing, it’s because it is!

P values are commonly misinterpreted and misused to answer research questions, but in actuality they fail to provide much information to the reader (1). This method of statistical analysis has been met with criticism throughout its history and increasingly so in the past few decades. The shortcomings of significance testing have been identified in both the academic and non-academic literature. I encourage anyone interested in the mechanistic limitations of significance testing to seek out such resources.

The P value doesn’t begin to explain the importance of a study’s outcome or the amount of the effect observed, though many researchers mistakenly believe it to.

Null hypothesis significance testing does not explain how much better – or worse – a group of individuals did compared with another group, just that there was a difference between the two groups (2). In short, significance testing only gives us statistical significance and says nothing about a study’s practical significance or clinical applicability. There are numerous examples of a statistically significance result having no practical significance and vice-versa, two of which I have included below:

 A primary HIV prevention medication known as Truvada – a combination of tenofovir and emtricitabine – was approved by the US Food and Drug Administration (FDA) in 2012. The two major side effects of this drug, taken as a daily oral dose, are a mild, non-progressive decrease in kidney function and a small decrease in bone mineral density (3). The effect on kidney function was found to be statistically significant (P = 0.02), but was considered to be “sub-clinical” by the authors (4). In a separate study, the same drug caused a small, but statistically significant decrease in bone mineral density from baseline, the clinical significance of which was unknown but was not associated with an increased occurrence of bone fracture during the study (5). Hence, Truvada does not appear to have any significant clinical side effects (see the Table below). However, in a cohort of 1603 at-risk individuals in the US, 185 people – roughly 12% of respondents – cited concern about side effects as their reason for not taking the medication (6). This is an alarming misconception for a disease that infects approximately 50,000 Americans and 2 million people globally per year (12,13), highlighting the need for better knowledge translation to improve public understanding of research outcomes.

Table. Statistical and clinical significance of Truvada side effects
Side effect Statistically significant? Clinically significant?
Reduced kidney function (4) Yes: P = 0.02 No: the ‘sub-clinical’ effects on kidney function were reversible and manageable
Decreased bone mineral density (5) Yes: P = 0.02 for a 3% decrease in bone mineral density Unclear: no increase in fractures associated with loss of bone mineral density in the study

On the other hand, research results that are not found to be statistically significant may have clinical applications. In the field of exercise physiology, a 1% change in performance might be considered clinically applicable.

To put this in perspective, a difference of 1% in the 100m sprint is the difference between Donovan Bailey’s 1996 Olympic record of 9.84 seconds, and Asafa Powell’s 2007 world record of 9.74 seconds.

Such a difference might not be deemed statistically significant in a typical research study, due to the difficulty of obtaining precision around such a small difference as a result of limited sample size or large variability in measurement (7). Another good example is from a study where researchers found a 2.9-minute improvement in a cycling time trial lasting 160 minutes with a supplement intervention. This difference was not statistically significant (8). However, if the researchers were to look at the results from an application standpoint, their conclusion may have been different: the 2.9 minute time improvement translated into a 1.8% increase in performance, suggesting that competitive athletes would probably benefit from the supplement intervention and that further investigation is warranted (7).

In recent years there has been a move away from using null hypothesis significance testing alone, or at all, in many scientific fields. Making inferences using magnitude-based measures such as confidence intervals, which is becoming increasingly popular, allows researchers to estimate the size of an effect in relation to clinical and practical importance (1,9,10). P-values may still have a place in our statistical assessment of study outcomes, but not should be the defining value for accepting or rejecting an outcome (11). Researchers, scientific writers, and the public need to look at the outcomes of research from a practical standpoint, and not only use the outdated and dichotomous view of statistical significance. Do not make conclusions about research outcomes solely based on significance testing without considering the practical significance and applicability of the observed effect.


Sean Sinden is an MSc student in the School of Kinesiology at the University of British Columbia, specialising in exercise physiology. Sean Sinden’s research is focusing on the physiology of doping and the impact of environmental conditions on asthmatic athletes. He is also interested in knowledge translation, science writing, and public understanding of scientific findings.

Twitter: @seanmsinden




  1. Batterham AM, Hopkins WG. Making Meaningful Inferences About Magnitudes. Int J Sports Physiol Perform. 2006;1(1):50–7.
  2. Wilhelmus KR. Beyond the P. I: Problems with probability. J Cataract Refract Surg. 2004;30(9):2005–6.
  3. Krakower DS, Jain S, Mayer KH. Antiretrovirals for Primary HIV Prevention: the Current Status of Pre- and Post-exposure Prophylaxis. Curr HIV/AIDS Rep. 2015;12(1):127–38.
  4. Solomon MM, Lama JR, Glidden DV, Mulligan K, McMahan V, Liu AY, et al. Changes in renal function associated with oral emtricitabine/tenofovir disoproxil fumarate use for HIV pre-exposure prophylaxis. AIDS. 2014;28(6):851–9.
  5. Liu AY, Vittinghoff E, Sellmeyer DE, Irvin R, Mulligan K, Mayer K, et al. Bone Mineral Density in HIV-Negative Men Participating in a Tenofovir Pre-Exposure Prophylaxis Randomized Clinical Trial in San Francisco. PLoS ONE. 2011;6(8):e23688–11.
  6. Grant RM, Anderson PL, McMahan V, Liu A, Amico KR, Mehrotra M, et al. Uptake of pre-exposure prophylaxis, sexual practices, and HIV incidence in men and transgender women who have sex with men: a cohort study. Lancet Infect Dis. 2014;14(9):820–9.
  7. Hopkins WG, Hawley JA, Burke LM. Design and analysis of research on sport performance enhancement. Med Sci Sports Exerc. 1999;31(3):472–85.
  8. Madsen K, MacLean DA, Kiens B, Christensen D. Effects of glucose, glucose plus branched-chain amino acids, or placebo on bike performance over 100 km. J Appl Physiol. 1996;81(6):2644–50.
  9. Wilkinson M, Winter EM. Clinical and practical importance vs statistical significance: Limitations of conventional statistical inference. Int J Ther Rehabil. 2014;21(10):488-95.
  10. Cumming G. The new statistics: why and how. Psychol Sci. 2014;25(1):7–29.
  11. Greenland S, Poole C. Living with p values: resurrecting a Bayesian perspective on frequentist statistics. Epidemiology. 2013;24(1):62–8.
  12. Joint United Nations Programme on HIV/AIDS. UNAIDS Fact Sheet: Global Statistics. (accessed June 11, 2015)
  13. Centre for Disease Control and Prevention. HIV in the United States: at a glance. (accessed June 11, 2015)
Category: Epidemiology, Guest Posts, Science Outreach | Tagged , , , | 2 Comments

Does Big Grape Juice Control Nutrition Research? An Interview with Michele Simon

The “Sustaining Sponsors” thanked at the 2015 ASN meeting

Corporations cozying up to researchers create massive conflicts of interest. It’s an old story when the villain is a pharmaceutical company. But food companies need to make money too, and what better way than funding and publicizing research on their products?

I had plenty of time to think about this issue while I was at the American Society for Nutrition conference this spring. Here’s what I wrote on Lifehacker:

As I write this, I’m at the American Society of Nutrition conference, listening to scientific presentations on nutrition—and the number of corporate logos is astounding. Not just the biggies like Pepsi and Kellogg, but specialized groups too: if there’s a study on cranberry juice, for example, you can bet there will be an Ocean Spray logo on the acknowledgements slide.

Corporate funding doesn’t mean the study is bogus, but topics with good funding are more likely to be investigated (or in other words, funding doesn’t always bias the answer, but does buy the question.) To use another example from this conference, take the news that adding eggs to your salad makes the vitamins in the vegetables more available to your body. Amazing, right? It’s actually not news: fat-soluble vitamins are more available to your body when you eat them with fat. The same researcher who did this work has shown the same effect with other foods, including oils. So why are eggs getting the press? Take a look at the funder (which most of the news articles aren’t reporting): the American Egg Board.

Public health lawyer Michele Simon has been thinking about it too. Last week she released a report (written with the Alliance for Natural Health) detailing the American Society for Nutrition’s reliance on corporate sponsors. I caught up with her by phone:

In this report, why are you focusing on ASN specifically? How do they compare to other groups?

My previous report was a deeper analysis of the Academy of Nutrition and Dietetics [a trade group for nutritionists in the US] and that organization is even more influenced by the food industry. That was a good example of what happens when you form these kinds of cozy relationships with the junk food industry. It’s not just dietitians, but also nutrition researchers that have conflicts. The ASN and its members’ research form the foundations of the nutrition advice that comes from other professions, like dietitians, and even the federal government.

There have been exposes here and there of individuals in the ASN, such as David Allison, who was called out on ABC News for his connections to Coca-Cola and PepsiCo, but I wanted to look at how this membership trade organization sets a tone. When you go to your professional annual meeting and you see sponsorships by the likes of Coca-Cola, what does that tell you as a researcher? That it’s perfectly acceptable to say yes when, the next day, the Coca-Cola scientific officer says “let’s team up on some research.”

Of course not all ASN members have conflicts, far from it, but ASN is normalizing the idea that it’s OK to have science influenced by corporations that have a direct economic stake in the outcome of that science.

In the medical field, it seems like researchers are very up-front about their funding: there’s always a disclosure statement. When I went to the ASN conference, I didn’t always see disclosures attached to research. Do nutrition scientists need to get better about disclosures?

The disclosure trend originally came out of the deep conflicts of the pharmaceutical industry and medical science, and that has somewhat filtered into the food realm. But I’ve had conversations with academics about the role of disclosures and it’s really become this feel-good thing. It’s an expected thing to do. ‘Yes, we disclose our conflicts.’ But that doesn’t make it OK.

Obviously it’s better than nothing, knowing that Coca-Cola funded that study that says soda doesn’t cause obesity. OK, now I can completely discount that study. But you [the consumer] may not have seen this disclosure. There are many examples of industry funded research that gets widely reported in the media without disclosures.

I’ve noticed the same thing too: things that were presented at the ASN conference, you see them in the news the next day without any mention of sponsors. I got to thinking, and I’m curious about your thoughts on this—and not even Coca-Cola, but something like Ocean Spray was sponsoring a ton, and the Egg Board–

Yes, and the National Dairy Council. It all flies under the radar of most media outlets.

So lots of stories make the news about the antioxidants in cranberries being good for you and I’m thinking, you know, it’s probably not just cranberries, but Ocean Spray is the one paying for it, so we hear a lot about cranberries. It’s almost like a selection publication bias, even aside from whether the research is right or not.

Those companies that have a stake in positioning themselves as healthy are going to be the ones funding the research to prove their point, and they’re ones that have the money. Obviously a company like Ocean Spray or Welch’s is really well positioned to fund research versus, say the peach council or the cherry board.

You might get the exact same or even better nutritional value by consuming whole fruit, but those sectors don’t have the same funding sources, and they’re not going to make as much money as Welch’s from slapping a claim on a 100% sugar vehicle. It’s a joke to me to even be talking about juice as a source of antioxidants given the sugar content, which likely negates any alleged health benefit.

So we’re talking about corporate ties and funding and so on. Now, there’s funding and there’s influence, and they’re related but not necessarily the same thing. Can you explain a little about how these companies are influencing the ASN or how we know the scope of their influence?

I would agree that giving money isn’t necessarily bad, but it does raise serious concerns, for example, around buying access. One of the types sponsorships that companies can purchase, is $35,000 for the hospitality suites. So that is a perfect opportunity to socialize with researchers and we can assume that if a company is sponsoring it, they’re sending their representatives, and there are probably other corporate representatives showing up at the hospitality suite because they know that’s where they can cozy up to researchers. Relationship building is politics 101. When you form relationships with people, then they don’t think of you as evil, and then it’s normal to engage in all kinds of dealings with them.

But the more direct influence comes from the two examples of policy positions I explained in the report. One, against the FDA’s proposal to require added sugar to be listed on the Nutrition Facts panel, completely flies in the face of every other health organization that has supported that proposal, and just common sense: why shouldn’t consumers have additional information about the sugar that’s in their food?

And then the example, even worse to me, was a complete defense of processed food, again flying in the face of nutrition science and common sense, and, even consumer trends. So it can’t be an accident that the sugar association and the processed food industry are major funders of ASN when they come out with positions like that that are just antithetical to basic science.

Since we’re talking about conflicts of interest here, can you tell me about yourself? How do the findings in this report relate to the kind of work you do for clients?

I’m an independent consultant, and I’ve done this work for almost 20 years in various ways, collaborating with different organizations. For this report, it was a collaboration with the Alliance for Natural Health, and they said we want to expose this group’s industry ties. I said that sounds like a good idea, I wanted to do that too, so we teamed up to do it together.

Everything I’ve done is a complete open book at Every single report I’ve done, everything I’ve written is either as a freelance writer (sometimes for no pay) or in in collaboration with nonprofits or law firms or just from funders who support my work. I do a variety of things, including public speaking. I’m kind of unusual in that no one else really does what I do, so I’ve been creative in how I do this work, but it’s all there.

What inspired you to do this work?

I got inspired by Marion Nestle‘s work, a few years before her seminal book Food Politics came out. She was writing about the influence of the meat and dairy industries on the dietary guidelines and I thought that’s exciting because I had recently adopted a vegetarian diet and I hadn’t seen anybody else writing about that. So I decided to follow in her footsteps and then I broadened my horizons from there.

To me, public health law is really about pushing for changes in law and policy that make healthy eating easier. Unfortunately the general public doesn’t understand some basic public health principles. Many people think, “it’s just up to individuals and parents to make better choices” without an understanding of how much our environment influences our behavior. It’s hard to educate people about how to eat right when they leave their house and all they find is McDonald’s and the corner liquor store.

Category: Uncategorized | Tagged , , , , | 3 Comments

Society for Epidemiologic Research 48th Annual Meeting

Epidemiology is a powerful tool that, when used carefully and creatively, can help us make sense of the world we live in.

Epidemiology is a powerful tool that, when used carefully and creatively, can help us make sense of the world we live in.

Epidemiology is the science of public health, hence it is fitting that we cover the Society for Epidemiologic Research (SER) 48th Annual Meeting here on PLOS Public Health Perspectives. The meeting runs this week, from the 16th to 19th of June in Denver, Colorado. Over these four days, the scientific sessions will cover diverse topics ranging from ‘nature’ and ‘nurture’ in social disparities in obesity, persistent problems and resilience 10 years after Hurricane Katrina, and novel risk factors and methods in the epidemiology of the ageing brain. Other interesting topics (there are too many to list!) include mediation analysis for health disparities research, the etiologic heterogeneity of breast cancer, and the widening educational disparities in life expectancy among women in the United States. Over 200 talks will be given on these topics and many more.

A major methodological theme running across the sessions is that of causal inference. Epidemiology is a field that attempts to measure cause-effect relationships, despite the many minefields we wade through in doing so at the edge of our scientific capability. Our decisions at every step of the research process are crucial to this goal, including formulation of the research question, design of the study, participant recruitment, measurement techniques, and statistical analysis. And once our individual studies are complete, how do we put it all together to infer causality from our collective results? I am particularly looking forward to the Keynote Cassel Lecture on the Scientific Approach to Causal Inference by Judea Pearl.

In addition to the cutting-edge scientific content of the meeting, there will be several workshops and career development events such as breakfast with the experts and lunchtime roundtables. These will be great places for any savvy students and post-docs to network and build their careers. Career development and networking is an important part of these conferences that should never be underestimated by early career researchers.

On Twitter, the event will be covered with the hashtag #SERMTG2015 (a tad long, but it’s certainly specific!). I will be live-tweeting as many sessions as I can get to, so follow me: @1lindsayk. I will also be at the student and post-doc committee social events, and lacing up for the 5km Fun Run on Wednesday morning. I will be presenting some of my PhD research in the ‘Predestination of the Modifiable: Lifecourse Approaches to Behavioural Epidemiology’ session at 4:15 pm on Thursday, 18th June in the Columbia room, so come say hi!

Image source: Odra Noel, Royal Society Summer Science Exhibition 2013

Related Posts Plugin for WordPress, Blogger...
Category: Epidemiology, Social Media | Tagged , , | 1 Comment