STAT’s headline was particularly boosterish:
Play on! In a first, brain training cuts risk of dementia years later.
It’s just a few months since the US Federal Trade commission fined a company $2 million for false advertising based on brain training claims like this. And in October 2014, an international scientific consensus statement tried to stem this tide. Yet here we are again. Sigh!
This time, the results aren’t even just getting the usual claim of being “promising”: in the STAT article, they’re “highly, highly promising”! And that’s only on the basis of a conference presentation – not access to a report of the results in question.
That’s an unreliable source of information. Details that come out about trials later are often quite different to what was presented at conferences. A study of cardiology articles post-conferences from 1999-2002 found around 40% had discrepancies, and a sports medicine study found 63% had at least 1 major issue. And the day before this ACTIVE trial hype landed, I had tweeted about a study that probably has the record on these discrepancies: 96% – plastic surgery FTW!
Several of the media stories make a further mistake, though: they say we need to wait for the article to be sure. But 1 study isn’t enough to be sure.
This is what the scientific consensus statement says:
A single study, conducted by researchers with financial interests in the product, or one quote from a scientist advocating the product, is not enough to assume that a game has been rigorously examined. Findings need to be replicated at multiple sites, based on studies conducted by independent researchers who are funded by independent sources. Moreover, participants of training programs should show evidence of significant advantage over a comparison group that does not receive the treatment but is otherwise treated exactly the same as the trained group.
It would be great if there were a simple and accessible way to reduce the risk of dementia – even just a little. But no, there is no good reason in this latest report of the ACTIVE trial to invest the time or money for the products behind this new round of publicity.
Let’s go back to the beginning to make sense of what’s happening. The trial’s acronym stands for Advanced Cognitive Training for Independent and Vital Elderly (ACTIVE). There’s a big slew of publications on it – you can see them here at the bottom of its trial registration entry.
Although some of the reporting refers to a new trial, it isn’t. Even reporting on 10-year results isn’t new: those results came back in 2014 – months before the scientific consensus statement that there was no good evidence for claims of dementia prevention.
I wrote a critique of that paper here (originally on PubMed Commons). The people who developed this product had the goal of ultimately marketing it. As the trial progressed, with little impact being found, the trial was modified – trying to boost the impact. Still, after all that, here was the bottom line:
The current study showed weak to absent effects of cognitive training on performance-based measures of daily function.
And that’s key. Cognitive training can teach people to perform better on the specific function of that training. But that doesn’t necessarily translate into anything else at all. The performance-based measures did not rely on that learned skill only.
So what about these new “halved the risk of dementia” statements? Measuring dementia after 10 years wasn’t one of the pre-specified primary or secondary outcomes of the trial listed when the trial was registered – or in the article detailing the trial’s methods back in 2001. Media reports refer to a “secondary analysis”…
This means that the trial was not set up with methods designed to be able to answer the question about an effect on dementia – such as having enough people for the less common outcome of brain disease. More about the risks of subsequently adding measures and new hypotheses after you start here.
Here’s what Julie Steenhuysen from Reuters reports from one of the authors, Jerri Edwards:
Edwards did a secondary analysis of the 10-year data, looking at the time it took individuals to develop dementia.
She found that the group that did speed training showed 33 percent less risk of dementia relative to the control group, while the memory and the reasoning interventions offered no such benefit.
People who completed 11 or more speed training sessions were at 48 percent less risk for developing dementia over the 10 years of the study, Edwards said.
By the time you have multiple groups of people, each with progressively smaller subgroups (different numbers of sessions and so on), and multiple measures at multiple times, you increase your chances of pulling very misleading rabbits out of the hat.
Statistical significance testing isn’t enough to rely on here. (More on that here.) There needs to be a good reason to be confident this is cause and effect: how plausible is it that some cognitive training prevents a brain disease years later? And you need to see the results confirmed in different studies.
The product’s website bills it as being the most scientific brain training of all, supported by “100+ published research studies”: “Real science to believe in”.
The consensus statement again:
It is customary for advertising to highlight the benefits and overstate potential advantages of their products. In the brain-game market, however, advertisements also assure consumers that claims and promises are based on solid scientific evidence, as the games are “designed by neuroscientists” at top universities and research centers. These claims are reinforced through paid advertising and distributed by trusted news sources.
Neuroskeptic warns about all the “neurobunk” out there:
[M]ost “brain training” has little to do with neuroscience as such. Yes, you do use your brain when you’re practicing and training on these software titles, but that’s nothing special, since you use your brain to do everything.
Steenhuysen reports that Edwards said:
I’m sick of our studies being ignored.
I don’t think they are being ignored. They just fail to convince. Perhaps the next paper will break new ground.
But perhaps it will just be déjà vu all over again.
Related Absolutely Maybe posts:
* The thoughts Hilda Bastian expresses here at Absolutely Maybe are personal, and do not necessarily reflect the views of the National Institutes of Health or the U.S. Department of Health and Human Services.