“Understanding Psychosis and Schizophrenia” and mental health service users

understand coverDoes Understanding Psychosis and Schizophrenia exploit, disrespect, and marginalize service users?

Genre confusion.

The 180-page Understanding Psychosis and Schizophrenia produced by the British Psychological Society Division of Clinical Psychology is a puzzling document. We need to know its genre to decide what standards we apply in evaluating it. The authors tell us:

The report is intended as a resource for people who work in mental health services, people who use them and their friends and relatives, to help ensure that their conversations are as well informed and as useful as possible. It also contains vital information for those responsible for commissioning and designing both services and professional training, as well as for journalists and policy-makers. We hope that it will help to change the way that we as a society think about not only psychosis but also the other kinds of distress that are sometimes called mental illness.

“Well-informed” by what or whom? How is the information “vital”? Does “vital” assume “trustworthy” and “credible”?

As I will cover in a later blog issue, the document strikingly lacks the transparency that it would need to be taken seriously.  Understanding Psychosis conforms to none of the well-defined processes and standards – checks and balances – expected to be met by professional organizations producing a report aimed at policy-makers and the general public.

mental ellf1For now, note these psychologists did not engage other professionals with complementary viewpoints and expertise. And the writing  was closed to anyone not already expressing strongly held particular opinions. When critics nonetheless provided a detailed analysis of some crucial points at the popular blog, Mental Elf, the authors of Understanding Psychosis retweeted and favorited a denunciation of them as a “circle jerk,” i.e., mutually masturbating.

circle jerk

Please click to enlarge

how vulgar

Key stakeholders were simply excluded – primary care physicians, social workers, psychiatrists, police and corrections personnel who must make decisions about how to deal with disturbed behavior, and –most importantly- the family members of persons with severe disturbance. There was no check on the psychologists simply slanting the document to conform to their own narrow professional self-interests, which we are asked to accept as “expertise.”

Is Understanding Psychosis evidence-based?

Understanding Psychosis occasionally cites some empirical findings, but can’t be seen as evidenced-based. That would require transparent, systematic strategies for gathering, interpreting, and integrating evidence that are simply not there.

Indeed, I think it is an excellent document for PhD students and trainees to practice debunking the creation of false authority by selective citation and miscitation and ignoring of contradictory studies. I suggest that they arm themselves with Google Scholar and tools provided in

Greenberg, S. A. (2009). How citation distortions create unfounded authority: analysis of a citation network. BMJ, 339.

Then start checking the citations provided for seemingly evidence-based statements in Understanding Psychosis. Ask questions like “What relevant studies are not cited? What studies are misinterpreted or simply cited for findings they did not contain?” Go to Google Scholar or Web of Science and find out.

For instance, take the opinion

In view of the problems with diagnoses, many researchers and clinicians are moving away from using them, and recent high-profile reports have recommended this. 55 56.

Check the references and see that the authors of Understanding Psychosis are the “many researchers and clinicians.” They are praising their own opinion pieces as “high-profile.”

55. British Psychological Society (2013). Division of Clinical Psychology position statement on the classification of behaviour and experience in relation to functional psychiatric diagnoses: time for a paradigm shift. Leicester: British Psychological Society.

56. Division of Clinical Psychology (2011). Good practice guidelines on the use of psychological formulation. Leicester: British Psychological Society.

The authors of Understanding Psychosis would have embarrassed themselves if they stated outright “It is our opinion that…and we consider our opinion high-profile and you should be duly impressed.” They depend on readers not checking references.

Argument from cherry picked quotes.cherrypicking

Understanding Psychosis is a collection of quotes. We might be inclined to interpret this as a strength, a sign of collaborative  participatory research.

Or maybe this represents qualitative research allowing  people to speak for themselves, rather than requiring that their experiences be processed through others’ filters and concepts.  But bona fide, credible qualitative research requires that biases of  investigators not intrude upon what they report.  Some controls must be visibly present preventing the investigators from doing so.

Quotes are carefully selected to support by the psychologists opinions expressed before the document was prepared – like 15 years ago in their Recent Advances in Understanding Mental Illness and Psychotic Experiences.

Many quotes are not from people suffering from schizophrenia. In most instances, we are not given sufficient information to determine this.  The authors systematically withhold information that would allow readers to determine who is and who is not a service user.

In this issue of Mind the Brain, I examine implications of this heavy dependence on these particular quotes. I will question whether Understanding Psychosis involves using and even exploiting service users, pitting more highly functioning ones against those who are functioning less well and their families who have to deal with them when they cannot take care of themselves.

Where do the quotes in Understanding Psychosis come from?

Some quotes were simply pasted in from the 2000 Recent Advances in Understanding Mental Illness and Psychotic Experiences.

Presumably people had relevant experience in the interim for our grasping the relevance to what it like living with schizophrenia and other psychoses – if that was actually their circumstances. Unfortunately, no follow-up is provided. The authors did not respond to repeated inquiries to asking whether they even obtained permission to use these quotes.

The quotes also have been trimmed of most details about their context that are available in original sources. Going to the original sources, we find the sources deliberately sampled people who were not service users.

Yup, people stripped of their identities are paraded out without the benefit of information that would render their experiences meaningful. Readers can’t independently assessment the uses to which the psychologist authors of Understanding Psychosis put these quotes.

What is not at issue is whether people with unusual experiences can get our attention when they talk about them. What is at issue is that a group of professionals take these quotes out of context and insist that they be accepted as the primary basis for – as their title states – our understanding of psychosis and schizophrenia.

Some of the quotes come from sources like

Jackson, L., Hayward, M. & Cooke, A. (2011). Developing positive relationships withvoices: A preliminary grounded theory. International Journal of Social Psychiatry, 57(5), 487–495.

Freeman, D., Garety, P.A., Bebbington, P.E., Smith, B., Rollinson, R., Fowler, D. et al. (2005). Psychological investigation of the structure of paranoia in a non-clinical population. British Journal of Psychiatry, 186, 427–435.

Heriot-Maitland, C., Knight, M. & Peters, E. (2012). A qualitative comparison of psychotic-like phenomena in clinical and non-clinical populations. British Journal of Clinical Psychology, 51(1), 37–53.

Jackson et al report

Five men and seven women were recruited through local NHS services, community advertisement and the local branch of the Hearing Voices Network.

Freeman  et al report

An anonymous internet survey [was]… e-mailed the address of a website where they could take part in a survey of ‘everyday worries about others’.

Heriot-Maitland et report interviewing 12 participants, who reported “psychotic-like ‘out-of-the-ordinary’ experience (OOE) in the past five years.”

The quotes come from persons who are lucid enough to be recruited for small studies of , highly selected articulate persons. They certainly don’t display the distorted thought and behavior disorder and simple incoherence of many people with acute and chronic schizophrenia.

I agree with the Understanding Psychosis authors that few people who have ‘psychotic-like’ experiences meet criteria for a diagnosis of schizophrenia. But should we accept a carefully cherry-picked and edited group of quotes as the basis for revising our understanding of people who do meet criteria?

A number of quotes sound like people who are high functioning and showing an unusual degree of  fantasy-proneness:

P 29 I work four days a week in a professional job; I own my own house and live happily with my partner and pets. Occasionally I hear voices – for example when I have been particularly stressed or tired, or I have seen visions after a bereavement. Knowing that many people hear voices and live well, and that some cultures see these experiences as a gift, helps me to never catastrophise or to worry that it may be the start of a breakdown. Although I am lucky that the experiences have never been as upsetting as some people’s, if someone had told me it was madness I could have got into a vicious cycle and struggled to get out.

Some of the quotes seem to represent clinically significant distress, but probably not psychosis or schizophrenia.

p 53 One thing that you might hear a lot about is that anxiety is a trigger of suspicious thoughts. I have never been that good at recognising my own anxiety. Quite a high level of anxiety is pretty normal for me. So normal that I wouldn’t normally do anything about it, but I now recognise that it sets the background for the expected potential threats in any situation, and so the suspicious thoughts and ideas of reference can pop right in there. I find people as having the most potential as a source of threat and because of that I am prone to suspicious thoughts about others. So now what I do is try to address the level of anxiety I feel in these situations. Adam

We’re not provided any information suggesting this suspiciousness is the psychotic symptom, paranoia.

P 43 After being almost killed by my ex-boyfriend when I was 16 I have had OCD. I have also developed paranoia about someone trying to kill me. If I have conflict with someone over anything I worry they are going to kill me or have someone come and kill me. I wake up worried someone is in my bedroom. I think about trying to be ready to protect myself if someone comes at me. I don’t think I would have this if I had not been traumatised half my life ago.  Josephine

Yale Professor Joan Cook and other colleagues and I recently published  a mixed method study of a national sample of psychotherapists providing residential treatment to veterans for posttraumatic stress disorder.  A number reported difficulties deciding whether the “voices” that some veterans describe represented schizophrenia or vivid re-experiencing symptoms consistent with posttraumatic stress syndrome, for which exposure therapy is indicated.

The authors of Understanding Psychosis express a clear disdain for making diagnostic distinctions. But,  it is important for clinicians to decide about the nature of clients’ distress in order to decide how to treat it. They best do so by formulating a hypothesis based on evidence tied to diagnoses, and then sympathetically probing.  Gradual exposure to past trauma would likely tame the distress of someone meeting criteria for PTSD. But this could prove absolutely terrifying and decompensating for someone whom additional information suggested a diagnosis of psychosis. So clinicians have to have some evidence-based ideas to probe and make decisions or proceed blindly.

Some quotes probably refer to brief psychotic reactions. Responding to Understanding Psychosis, Allan Frances noted

Brief psychosis is considered a mental disorder, but it is just a transient one with excellent prognosis and no reason to expect long-term impairment. The symptoms emerge suddenly in response to stress and usually disappear just as suddenly (especially if the stress is removed), often never to reappear. This is common in many cultures, and I have seen it fairly often in college students away from home for the first time, in travelers in strange lands, and in people who have had something terrible happen to them. Antipsychotic medicine is needed only briefly, if at all.

Quotes were selected to fit the authors’ conviction that what other professionals call psychosis or schizophrenia is an understandable reaction to life events. But if we go to the larger literature, the associations between adverse experiences and psychosis, even in a meta-analysis of one of the authors of Understanding Psychosis, are not large enough that would suggest such strong causality.  Adverse experiences are linked to lots of negative outcomes, but generally do not lead to psychosis or schizophrenia, even if there is a significant, but not overwhelming correlation.

Understanding Psychosis is not a transparent, systematic review of available evidence. Authors are mustering quotes to fit their preconceived notions. And leaving out quotes and details that don’t fit.

American psychiatrist Bernard “Barney” Carroll slammed the arrogant response of President of the American Psychiatric Association President  Jeffrey Lieberman to media coverage of Understanding Psychosis. Barney called it over-the-top” and a “disservice to psychiatry.” Yet, this was not before he nailed the report for its “domesticating psychosis”:

Hallucinations become the experience of hearing voices; delusions become the experience of unusual beliefs; paranoid thinking becomes the experience of anxiety – never mind that the great majority of patients with clinical anxiety disorders are not at all paranoid in the way that psychotic patients are. They also make much of the fact that milder forms of these “experiences” are common in the general population – as are milder forms of many clearly medical symptoms. In short, they fail to acknowledge the state transition that demarcates mild or prodromal symptoms from outright psychotic illness.

… The BPS document fails adequately to convey the range of symptoms and associated behaviors in psychosis/schizophrenia. Even when these are mentioned, they are not addressed in a way that matches their clinical salience. Thus, decompensating psychotic crises are discussed unhelpfully in the framework of poor sleep habits. Acute inpatient psychiatric units are discussed in a patronizing way and are faulted as being unhelpful for some patients – never mind their rescue function. Catatonia as a common feature is not acknowledged. Psychotic terror and panic are not acknowledged. Formal thought disorder with truly crazy speech is not acknowledged.

A disclosure of my past.

I’m struck by the huge gap between the clear, articulate statements in the quotes provided in Understanding Psychosis and the incoherent mumbling and sometimes raging of people who are acutely psychotic.  I wonder how many of the authors have ever tried to conduct an interview with someone in that state.

cowboy entering belgium-1-page-001My clinical training involved six years of live supervision at the Mental Health Institute (MRI)provided by professionals widely recognized for their innovative work in analyzing the communication of persons considered as having schizophrenia –  Paul Watzlawick, John Weakland, and Richard Fisch – although they would have objected to that diagnostic label.

At the time, I probably was more anti-diagnosis than many of the authors of Understanding Psychosis are today. But then as Director of Research at Mental Research Institute, I witnessed the disaster of its Soteria Project. I’ll leave that for another time, but Wikipedia states

The Soteria project was admired by many professionals around the world who aspired to create mental health services based on a social, as opposed to a medical, model. It was also heavily criticized as irresponsible or ineffective. The US Soteria Project closed as a clinical program in 1983 due to lack of financial support, although it became the subject of research evaluation with competing claims and analysis. Second generation US successors to the original Soteria house called Crossing Place is still active, although more focused on medication management.

While Paul, John, and Dick were widely recognized for their work analyzing communication with severely disturbed persons, they operated with a sense that at some point the disturbance of thought and behavior could became too much to carry on a discussion. And talking to highly disturbed persons, they knew not to take what was being said literally.

Who was selected for inclusion in Understanding Psychosis and who was excluded and left silent?

Many patients with acute and chronic psychosis are essentially nonverbal and cannot communicate their distress. Sure, they can’t provide coherent quotes for the psychologists who assembled Understanding Psychosis, but it is irresponsible for those psychologists to pretend these people don’t exist or that the quotes they assembled represent their best interest.

Many patients who meet criteria for schizophrenia will times be unable to take care of themselves or to make basic decisions.  The burden of caring and decision-making will fall on family members if they are available. The alternative for persons with schizophrenia is to become homeless or go to jail or prisons because more appropriate beds and hospitals are not available. Nowhere in Understanding Psychosis are we reminded that persons with schizophrenia sometimes need sanctuary in hospitals.

Nowhere are we reminded that 10% of persons with schizophrenia will die by suicide. There is recent evidence that psychotic people may account for nearly 1/3 of suicide attempts with intent to die.

If I were a family member of someone with schizophrenia, I would be damn angry at the gap between the quotes in Understanding Psychosis what I knew about the person for whom I had to provide care. I’d also be angry that no one in my situation had been invited to participate as a stakeholder.

Psychologists in search of opportunities to work with YAVIS clients

purchase of friendshipThe carefully selected quotes suggest people who would be more satisfying to work with than many persons with psychosis and schizophrenia. Reading them, I was immediately reminded of William Schofield[‘s  50-year-old book Psychotherapy: The Purchase of Friendship in which he lamented the strong tendency of mental health professionals wanting to work with the YAVIS: clients who are young, attractive, verbal, intelligent, and successful.  One of the authors of Understanding Psychosis also co-authored the widely misrepresented Lancet study of cognitive behavioral therapy for psychosis and could tell us how difficult and ineffective  it was doing  therapy in  that study with the older patients who had more psychotic  episodes.

Despite the authors of the Lancet study having distanced themselves from earlier claims showed cognitive therapy had effects equivalent to antipsychotic medication, authors of Understanding Psychosis persist in making the claim to service users:

It would also appear that CBT can bring comparable benefits even when people choose not to take medication.

As we would expect from recommendations produced by tightly knit groups representing single professions, Understanding Psychosis is a bid for more resources for its authors to work with clients with whom they want to work.  But like any policy recommendations, we need to examine the evidence and look at where those resources would come.

Please click to enlarge

Please click to enlarge

I’ll leave that discussion to another blog post, but take a look at the graph on the left. It represents the dramatic shift in resources from inpatient beds to outpatient treatment settings. The profoundly disturbed persons who need those beds would undoubtedly be less suitable for the conversations that the Understanding Psychosis psychologists want to be having. The long term reduction in inpatent services represents not so much deinstitutionalization as transinstutionalization.  A lack of those beds means that persons in need of them are being relegated to jails and prisons. In the United States, the Los Angeles jails represent the largest mental health treatment facility in the United States and the conditions for the severely disturbed are abominable. Similar situations hold in the UK.

An inpatient psychiatrist recently wrote in the New York Times:

We also need to rethink how we care for another group of vulnerable patients who have been just as disastrously disserved by policies meant to empower and protect them: the severely mentally disabled.

He went on:

We have worked to minimize the use of restraint and seclusion on my unit, but have seen the frequency of both skyrocket. Nearly every week staff members are struck or scratched by largely nonverbal patients who have no other way to communicate their distress. Attempting to soothe these patients monopolizes the efforts of a staff whose mission is to treat acute psychiatric emergencies, not chronic neurological conditions. Everyone loses.

Professor-Simon-Wessely-007Somebody in the UK should be speaking up for the inarticulate vulnerable persons with schizophrenia needing inpatient beds who are silenced and marginalized by the authors of Understanding Psychosis.  Where the hell is Simon Wessely when they need him?

Promoting an unrealistic view of schizophrenia?

If the authors of Understanding Psychosis were truly interested in providing authoritative information for persons with schizophrenia or psychosis, their family members, and professionals who come into contact with them, they would’ve provided the latest evidence about long-term course and outcome.

For instance, a key English study provides a 10 year follow-up individuals with a first episode of psychosis initially identified in either southeast London or Nottingham.

Morgan, C., Lappin, J., Heslin, M., Donoghue, K., Lomas, B., Reininghaus, U., … & Dazzan, P. (2014). Reappraising the long-term course and outcome of psychotic disorders: the AESOP-10 study. Psychological medicine, 44(13), 2713-2726.

At follow-up, of 532 incident cases identified, at baseline 37 (7%) had died, 29 (6%) had emigrated and eight (2%) were excluded. Of the remaining 458, 412 (90%) were traced and some information on follow-up was collated for 387 (85%). Most cases (265, 77%) experienced at least one period of sustained remission; at follow-up, 141 (46%) had been symptom free for at least 2 years. A majority (208, 72%) of cases had been employed for less than 25% of the follow-up period. The median number of hospital admissions, including at first presentation, was 2 [interquartile range (IQR) 1–4]; a majority (299, 88%) were admitted a least once and a minority (21, 6%) had 10 or more admissions. Overall, outcomes were worse for those with a non-affective diagnosis, for men and for those from South East London.Conclusions Sustained periods of symptom remission are usual following first presentation to mental health services for psychosis, including for those with a non-affective disorder; almost half recover.

Put differently, overall

12% (9% for non-affective) of our sample recovered within 6 months of contact with services and did not have a further episode, 20% (14% for non-affective) never had an episode lasting more than 6 months, and around 50% (40% for non-affective) had not experienced symptoms in the 2 years prior to follow-up.

And then there is the most recent comprehensive systematic review and meta-analysis.

Jääskeläinen, E., Juola, P., Hirvonen, N., McGrath, J. J., Saha, S., Isohanni, M., … & Miettunen, J. (2012). A systematic review and meta-analysis of recovery in schizophrenia. Schizophrenia Bulletin, sbs130.

We identified 50 studies with data suitable for inclusion. The median proportion (25%–75% quantiles) who met our recovery criteria was 13.5% (8.1%–20.0%). Studies from sites in countries with poorer economic status had higher recovery proportions. However, there were no statistically significant differences when the estimates were stratified according to sex, midpoint of intake period, strictness of the diagnostic criteria, duration of follow-up, or other design features. Conclusions: Based on the best available data, approximately, 1 in 7 individuals with schizophrenia met our criteria for recovery. Despite major changes in treatment options in recent decades, the proportion of recovered cases has not increased.

One in 7 people with schizophrenia meet criteria for recovery, and the portion has not increased in recent decades. Compare that with the unrealistically cheery assessment offered in Understanding Psychosis:

Even if people continue to hear voices or hold unusual beliefs, they may nevertheless lead very happy and successful lives. Sometimes a tendency to ‘psychosis’ can be associated with particular talents or abilities.

And

p 30 People who continue to have severe and distressing experiences may lead happy and successful lives in all other respects, such as work and relationships.

Sure, the authors of Understanding Psychosis keep reminding us of the cliché that everybody is different. But they are asking us to make clinical and policy decisions that are life altering for some people and could be life-ending for others. We can’t afford to ignore a larger body of relevant data.

In light of the data from long term follow-up studies, Understanding Psychosis should be seen as a cruel hoax perpetrated against more typical severely disturbed mental health service users, their family, and policymakers

Category: evidence-supported, mental health care, professional organizations, psychosis, psychotherapy, schizophrenia | Tagged , , , , | 7 Comments

Biomarker Porn: From Bad Science to Press Release to Praise by NIMH Director

Concluding installment of NIMH biomarker porn: Depression, daughters, and telomeres

Pioneer HPA-axis researcher Bernard “Barney” Carroll’s comment left no doubt about what he thought of the Molecular Psychiatry article I discussed in my last issue of Mind the Brain:

Where is the HPA axis dysregulation? It is mainly in the minds0@PubSubMain@NIHMS2@s@0@44595.html of the authors, in service of their desired narrative. Were basal cortisol levels increased? No. Were peak cortisol levels increased? They didn’t say. Was the cortisol increment increased? Only if we accept a p value of 0.042 with no correction for multiple comparisons. Most importantly, was the termination of the stress cortisol response impaired? No, it wasn’t (Table 3). That variable is a feature of allostasis, about which co-author Wolkowitz is well informed. Termination of the stress response is a crucial component of HPA axis regulation (see PubMed #18282566), and it was no different between the two groups. So, where’s the beef? The weakness of this report tells us not only about the authors’ standards but also about the level of editorial tradecraft on display in Molecular Psychiatry. [Hyperlink added]

You also can see my response to Professor Carroll in the comments.

I transferred  another  comment  to the blog from my Facebook wall. It gave me an opportunity to elaborate on why

we shouldn’t depend on small convenience samples to attempt to understand phenomena that must be examined in larger samples followed prospectively.

I explained

There are lots of unanswered questions about the authors’ sampling of adolescents. We don’t know what they are like when their mothers are not depressed. The young girls could also simply be reacting to environmental conditions contributing to their mother’s depression, not to their mother’s depression per se. We don’t know how representative this convenience sample is of other daughters of depressed mothers. Is it unusual or common that daughters of this age are not depressed concurrent with their mothers’ depression? What factors about the daughters, the mothers, or their circumstances determine that the mother and daughter depression does not occur at the same time? What about differences with him him dthe daughters of mothers who are prone to depression, but are not currently depressed?  We need to keep in mind that most biomarkers associated with depression are state dependent, not trait dependent. And these daughters were chosen because they are not depressed…

But with no differences in cortisol response, what are we explaining anyway?

The Molecular Psychiatry article provides an excellent opportunity to learn to spot bad

From  http://www.compoundchem.com/2014/04/02/a-rough-guide-to-spotting-bad-science/

From http://www.compoundchem.com/2014/04/02/a-rough-guide-to-spotting-bad-science/

science. I encourage interested readers to map what is said in that into the chart at the right.

This second installment of my two-part blog examines how the exaggerations and distortions of the article reverberate through a press release and then coverage in NIMH Director Thomas Insel’s personal blog.

The Stanford University press release headline is worthy of the trashy newspapers we find at supermarket checkouts:

Girls under stress age more rapidly, new Stanford study reveals

The press release says things that didn’t appear in the article, but echoes the distorted literature review of the article’s introduction in claiming well-established links between shortened telomeres, frequent infections in chronic disease and death that just are not there.

The girls also had telomeres that were shorter by the equivalent of six years in adults. Telomeres are caps on the ends of chromosomes. Every time a cell divides the telomeres get a little shorter. Telomere length is like a biological clock corresponding to age. Telomeres also shorten as a result of exposure to stress. Scientists have uncovered links in adults between shorter telomeres and premature death, more frequent infections and chronic diseases.

From http://news.stanford.edu/news/2014/october/telomeres-depression-girls-10-28-2014.html

From http://news.stanford.edu/news/2014/october/telomeres-depression-girls-10-28-2014.html

And the claim of “the equivalent of six years” comes from direct quote from obtained from senior author Professor Ian Gotlib.

“It’s the equivalent in adults of six years of biological aging,” Gotlib said, but “it’s not at all clear that that makes them 18, because no one has done this measurement in children.”

Dr. Gotlib  seems confused himself about what he mean by the 10 to 14-year-old girls having aged an additional six years. Does he really think that they are now 18? If so in what way? What could he possibly mean – do they look six years older than age matched controls? That would be really strange if they did.

I hope he lets us know when he figures out what he were saying, but he shouldn’t have given the statement to the Stanford press officer unless he was clear what he meant.

The press release noted that Dr. Gotlib had already moved on to intervention studies designed to prevent telomere shortening these girls.

In other studies, Gotlib and his team are examining the effectiveness of stress reduction techniques for girls. Neurofeedback and attention bias training (redirecting attention toward the positive) seem promising. Other investigators are studying techniques based on mindfulness training.

That’s a move based on speculation, if not outright science-fiction. Neurofeedback has some very preliminary evidence for effectiveness in treating current depression, but I would like to see evidence that it has any benefit for preventing depression in young persons who have never been depressed

neurofeedbackGotlib’s claims play right into popular fantasies about rigging people up with some sort of apparatus that changes their brain. But everything changes the brain, even reading this blog post. I don’t think that reading this blog post has any less evidence for preventing later depression than neurofeedback. Nonetheless, I’m hoping  that my blogging implants a healthy dose of skepticism in readers’ brains so that they are immunized against further confusion from exposure to such press releases. For an intelligent, consumer oriented discussion of neurofeedback, see Christian Jarrett’s

Read this before paying $100s for neurofeedback therapy

Attention bias training is a curious choice. It is almost as trendy as neurofeedback, but would it work?  We have the benefit of a systematic review and recent meta-analysis that suggests a lack of evidence for attention bias training in  treating depression and no evidence for preventing it. If it’s ineffectual in treating depression, how could we possibly expect it to prevent depression? Evidence please!

Let’s speculate about the implications if the authors found the cortisol differences between the daughters of the depressed mothers and daughters of controls that they had hypothesized but did not find. What then could have been done for these young girls? Note that the daughters of depressed mothers were chosen because they were functioning well, not currently depressed themselves. Just because they were different from the control girls would not necessarily indicate that any cortisol variables were in the abnormal range. Cortisol levels are not like blood pressure – we cannot specify a level below which cortisol levels have to be brought down for better health and functioning.

Note also that these daughters were selected on the basis of their mothers being depressed and that could mean the daughters themselves were facing a difficult situation. We can’t make the mother-bashing assumption that their mother’s depression was inflicting stress on them. Maybe any psychobiological stress response that was evident was due to the circumstances that led to the depression of their mother. We don’t know enough to specify what levels of cortisol variables would be optimal and consistent with good coping with the situation – we let even specify what is normal. And we don’t know how the daughters would recover from any abnormalities without formal treatment when their circumstances changed.

Bottom line is that these investigators did not get the results they hypothesized. Even if they had, results would not necessarily to lead to clinical applications.

Nonetheless, the director of NIMH saw fit to single this paper out or maybe he was just picking up on the press release.

my blogThomas Insel’s Personal Blog: Depression, Daughters, and Telomeres.

Thomas Insel’s Director’s Blog starts by acknowledging that there are no genetic or imaging markers predicting risk for depression, but research by Stanford Psychology Professor Ian Gotlib and colleagues in Molecular Psychiatry is “worth watching.”

Insel describes Gotlib’s “longitudinal” research as following depressed mothers’ early adolescent daughters.

The young girls have not yet developed depression, but 60 percent will become depressed by the age of 18.

I can find no basis in the article for Insel’s claim that Gotlib has found 60 per cent of these girls will be depressed by age 18. The estimate seems exaggerated, particularly given the case mix of mothers of these girls. It appears that some or most of the mothers were drawn from the community. We cannot expect severe course and biological correlates of depression that we would expect from a more inpatient sample.

Searching the papers coming out of this lab, I could only find one study involving a 30 month follow-up of 22 daughters of depressed mothers in the same age range as the sample in the Molecular Psychiatry article. That’s hardly a basis for the strong claim of 60% becoming depressed by 18.

Insel embellishes the importance of differences in telomere length. He perpetuates the illusion that we can be confident that differences in telomere length suggest these girls were experiencing accelerated aging and what have high risk for disease when the girls reached middle and late age. Without the backing of data from the paper or the existing literature, Insel zeros in on

Troubling early sign of risk for premature biological aging and possibly age-related chronic diseases, such as cardiovascular disease. Investigating the cause and timing of decreased telomere length—to what extent it may result from abnormalities in stress responses or is genetically influenced, for example—will be important for understanding the relationship between cellular aging, depression, and other medical conditions.

Insel ponders how such young, healthy girls could possibly show signs of aging. According to him the answer is not clear, but it might be tied to the increased stress reactivity these girls show in performing laboratory tasks.

But as Professor Caroll noted, the study just does not much evidence of “increased stress reactivity.”

neurofeedback2jpgNonetheless, Insel indicates that Gotlib’s next step is

Using neurofeedback to help these girls retrain their brain circuits and hopefully their stress responses. It will be a few years before we will know how much this intervention reduces risk for depression, but anything that prevents or slows the telomere shortening may be an early indication of success.

It’s interesting that Insel sidestepped the claim in the press release that Gotlib was trying out a cognitive behavioral intervention to affect stress reactivity. Instead he presents a fanciful notion that neural feedback will somehow retrain these girls’ brain circuits and reduce their stress response throughout their time at home and prevent them getting depressed by their mother’s depression.

Oh, if that were only so: Insel would be vindicated in his requiring for funding that researchers get down to basic mechanisms and simply bypass existing diagnoses with limited reliability, but at least some ties to patients’ verbal reports of why they are seeking treatment. In his world of science fiction, patients, or at least these young girls, which come in to have their brains retrained to forestall the telomere shortening that is threatening them not only with becoming depressed later, but with chronic diseases and middle and late life and early death.

So, let’s retrace what was said in the original Molecular Psychiatry article to what was claimed in the Stanford University press release and what was disseminated in the social media of Dr. Insel’s personal blog. Authors’ spin bad science in a peer-reviewed article. They collaborate with their university’s press relations department by providing even more exaggerated claims. And Dr. Insel’s purpose is served by simply passing them on and social media.

There’s a lot in Dr. Insel’s Personal Blog to disappoint and even outrage

  • Researchers  seeking guidance for funding priorites.
  • Clinicians in the trenches needing to do something now to deal with the symptoms and simple misery that are being presented to them.
  • Consumers looking for guidance from the Director of NIMH as to whether they should be concerned about their daughters and what they should do about it.

A lot of bad science and science fiction is being served to back up false promises about anything likely to occur in our lifetimes, if ever.

promising treatmentTaxpayers need to appreciate where Dr. Insel is taking funding of mental health with research. He will no longer fund grants that will explore different psychotherapeutic strategies for common mental health problems as they are currently understood – you know, diagnoses tied to what patients complain about. Instead he is offering a futuristic vision in which we no longer have to pay for primary care physicians or mental health clinicians spending time talking to patients about the problems in their lives. Rather, patients can bring in a saliva sample to assess the telomere length. They then can be rigged up to a videogame providing a social stress challenge. They will then be given neurofeedback and asked to provide another saliva sample. If the cortisol levels aren’t where they are supposed to be, they will come back and get some more neurofeedback and videogames.

But wait! We don’t even need to wait until people develop problems in their lives. We can start collecting spit samples when they are preteens and head off any problems developing in their life with neural feedback.

Presumably all this could be done by technicians who don’t need to be taught communication skills. And if the technicians are having problems, we can collect spit samples from them and maybe give them some neurofeedback.

Sure, mild to moderate depression in the community is a large and mixed grouping. The diagnostic category major depression loses some of its already limited reliability and validity when applied to this level of severity. But I still have a lot more confidence in this diagnosis than relying on some unproven notions about treating telomere length and cortisol parameters in people who do not currently complain about mental health or their circumstances. And the lamer notion that this can be done without any empathy or understanding.

It’s instructive to compare what Insel says in this blog post to what he recently said in another post.

He acknowledged some of the serious barriers to the development of valid, clinically useful biomarkers:

Patients with mental disorders show many biological abnormalities which distinguish them from normal volunteers; however, few of these have led to tests with clinical utility. Several reasons contribute to this delay: lack of a biological ‘gold standard’ definition of psychiatric illnesses; a profusion of statistically significant, but minimally differentiating, biological findings;‘approximate replications’ of these findings in a way that neither confirms nor refutes them; and a focus on comparing prototypical patients to healthy controls which generates differentiations with limited clinical applicability. Overcoming these hurdles will require a new approach. Rather than seek biomedical tests that can ‘diagnose’ DSM-defined disorders, the field should focus on identifying biologically homogenous subtypes that cut across phenotypic diagnosis—thereby sidestepping the issue of a gold standard.

All but the last sentence could have been part of a negative review of the Molecular Psychiatry article or the grant that provided funding for it. But the last sentence is the kind of nonsense that a director of NIMH can lay on the and research community and expect it to be reflected in their grant applications.

But just what was the theme of this other blog post from Dr. Insel? P-hacking and the crisis concerning results of biomedical research not being consistently reproducible.

The relentless quest for a significant “P” value is only one of the many problems with data analysis that could contribute to the reproducibility problem. Many mistakenly believe that “P” values convey information about the size of the difference between two groups. P values are actually only a way of estimating the likelihood that the difference you observe could have occurred by chance. In science, “significance” usually means a P value of less than 0.05 or 1 in 20, but this does not mean that the difference observed between two groups is functionally important. Perhaps the biggest problem is the tendency for scientists to report data that have been heavily processed rather than showing or explaining the details. This suggests one of the solutions for P-hacking and other problems in data analysis: provide the details, including what comparisons were planned prior to running the experiment.

Maybe because Insel is Director of NIMH, he doesn’t expect anybody to call him on the contradictions in what he is requesting. In the p-hacking blog post, he endorsed a call to action to address the problem of a lot of federal money being wasted on research that can’t lead to improvements in the health and well-being of the population because the research is simply unreliable and depends on “heavily processed” data for which investigators don’t provide the details. Yet in the Depression, Daughters, and Telomeres post he grabs an outrageous example of this being done and tells the research community he wants to see more of it.

 

porn

Category: biomarkers, HPA Axis, hype, maternal depression, mental health care, stress | Tagged , , , , | 3 Comments

NIMH Biomarker Porn: Depression, Daughters, and Telomeres Part 1

Does having to cope with their mother’s depression REALLY inflict irreversible damage on daughters’ psychobiology and shorten their lives?

telomerejpg

Telomere

A recent BMJ article revived discussion of responsibility for hyped and distorted coverage of scientific work in the media. The usual suspects, self-promoting researchers, are passed over and their University press releases are implicated instead.

But university press releases are not distributed without authors’ approval.  Exaggerated statements in press releases are often direct quotes from authors. And don’t forget the churnaling journalists and bloggers who uncritically pass on press releases without getting second opinions.  Gary Schwitzer remarked:

Don’t let news-release-copying journalists off the hook so easily. It’s journalism, not stenography.

In this two-part blog post, I’ll document this process of amplification of the distortion of science from article to press release to subsequent coverage. In the first installment, I’ll provide a walkthrough commentary and critique of a flawed small study of telomere length among daughters of depressed women published in the prestigious Nature Publishing Group journal, Molecular Psychiatry. In the second, I will compare the article and press release to media coverage, specifically the personal blog of NIMH Director Thomas Insel.

whackI warn the squeamish that I will whack some bad science and outrageous assumptions with demands for evidence and pelt the study, its press release, and Insel’s interpretation with contradictory evidence.

I’m devoting a two-part blog to this effort. Bad science with misogynist, mother bashing assumptions is being touted by the  Director of NIMH as an example to be followed. When he speaks, others pay attention because he sets funding priorities. Okay, Dr. Insel, we will listen up, but we will do so skeptically.

A paper that shares an author with the Molecular Psychiatry paper was criticized by Daniel Engber for delivering

A mishmash of suspect stats and overbroad conclusions, marshaled to advance a theory that’s both unsupported by the data and somewhat at odds with existing research in the field.

The criticism applies to this paper as well.

But first, we need to understand some things about telomere length…

What is a Telomere?

Telomeres are caps on the ends of every chromosome. They protect the chromosome from losing important genes or sticking to other chromosomes. They become shorter every time the cell divides.

I have assembled some resources in an issue of Science-Based Medicine:

Skeptic’s Guide to Debunking Claims about Telomeres in the Scientific and Pseudoscientific Literature

As I say in that blog, there are many exaggerated and outright pseudoscientific claims about telomere length as a measure of “cellular aging” and therefore how long we’re going to live.

I explain the concepts of biomarker and surrogate endpoint, which are needed to understand the current fuss about telomeres. I show why the evidence is against routinely accepting telomere length as a biomarker or surrogate endpoint for accelerated aging and other health outcomes.

I note

  • A recent article in American Journal of Public Health claimed that drinking 20soda kills ounces of carbonated (but not noncarbonated) sugar-sweetened drinks was associated with shortened telomere length “equivalent to an approximately 4.6 additional years of aging.” So, effects of drinking soda on life expectancy is equivalent to what we know about smoking’s effect.
  • Rubbish. Just ignore the telomere length data and directly compare the effects of drinking 20 ounces soda to the effects of smoking on life expectancy. There is no equivalence. The authors confused differences in what they thought was a biomarker with differences in health outcomes and relied on some dubious statistics. The American Journal of Public Health soda study was appropriately skewered in a wonderful Slate article, which I strongly recommend.
  • Claims are made for telomere length as a marker for effects of chronic stress and risk of chronic disease. Telomere length has a large genetic component and is correlated with age. When appropriate controls are introduced, correlation among telomere length, stress, and health outcomes tend to disappear or get sharply reduced.
  • A 30-year birth cohort study did not find an association between exposure to stress and telomere length.
  • Articles from a small group of investigators claim findings about telomere lengths that do not typically get reproduced in larger, more transparently reported studies by independent groups. This group of investigators tends to have or have had conflicts of interest in marketing of telomere diagnostic services, as well as promotion of herbal products to slow or reverse the shortening of telomere length.
  • Generally speaking, reproducible findings concerning telomere length require large samples with well-defined phenotypes, i.e., individuals having well-defined clinical presentations of particular characteristics, and we can expect associations to be small.

Based on what I have learned about the literature concerning telomere length, I would suggest

  • Beware of small studies claiming strong associations between telomere length and characteristics other than age, race, and gender.
  • Beware of studies claiming differences in telomere length arising in cross-sectional research or in the short term if they are not reproduced in longitudinal, prospective studies.

A walk-through commentary and critique of the actual article

Gotlib, I. H., LeMoult, J., Colich, N. L., Foland-Ross, L. C., Hallmayer, J., Joormann, J., … & Wolkowitz, O. M. (2014). Telomere length and cortisol reactivity in children of depressed mothers. Molecular Psychiatry.

Molecular Psychiatry is a pay-walled journal, but a downloadable version of the article is available here.

Conflict of Interest Statement

The authors report no conflict of interest. However, in the soda article published December 2014, one of the authors of the present paper, Jun Lin disclosed being a shareholder in Telomere Diagnostics, Inc., a telomere measurement company. Links at my previous blog post take you to “Telomeres and Your Health: Get the Facts” at the website of that company. You find claims that herbal products based on traditional Chinese medicine can reduce the shortening of telomeres.

Jun Lin has a record of outrageous claims. For instance, in another article, that normal women whose minds wander may be losing four years of life, based on the association between self-reported mind wandering and telomere length. So, if we pit this claim against what is known about the effects of smoking on life expectancy, women can extend their lives almost as much by better paying attention as from quitting smoking.

Hmm, I don’t know if we have undeclared conflict of interest here, but we certainly have a credibility problem.

The Abstract

Past research shows distorted and exaggerated media portrayals of studies are often already evident in abstracts of journal articles. Authors engage in a lot of cherry picking and spin results to strengthen the case their work is innovative and significant.

The opening sentence of the abstract to this article is a mashup of wild claims about telomere length in depression and risk for physical illnesses. But I will leave commenting until we reach the introduction, where the identical statement appears with elaboration and a single reference to one of the author’s work.

The abstract goes on to state

Both MDD and telomere length have been associated independently with high levels of stress, implicating dysregulation of the hypothalamic-pituitary-adrenal (HPA) axis and anomalous levels of cortisol secretion in this relation.

hpa useWhen I showed this to a pioneer in the study of the HPA axis, he remarked:

If you can find coherence in this from the Abstract you are smarter than I am…The phrase dysregulation of the HPA axis has been used to support more hand waving than substance.

The abstract ends with

This study is the first to demonstrate that children at familial risk of developing MDD are characterized by accelerated biological aging, operationalized as shortened telomere length, before they had experienced an onset of depression; this may predispose them to develop not only MDD but also other age-related medical illnesses. It is critical, therefore, that we attempt to identify and distinguish genetic and environmental mechanisms that contribute to telomere shortening.

This breathless editorializing about the urgency of pursuing this line of research is not tied to the actual methods and results of the study. “Accelerated biological aging” and “predispose to develop… other age-related medical illnesses” is not a summary of the findings of the study, but only dubious assumptions.

Actually, the evidence for telomere length as a biomarker for aging is equivocal and does not meet American Federation of Aging Research criteria.  A large scale prospective study did not find that telomere length predicted onset of diabetes or cardiovascular disease.

And wait to when we examine whether the study had reproducible results concerning either shorter telomeres and depression or telomeres being related to cortisol reactivity.

The introduction

The 6-paragraph introduction packs in a lot of questionable assumptions backed by a highly selective citation of the literature.

A growing body of research demonstrates that individuals diagnosed with major depressive disorder (MDD) are characterized by shortened telomere length, which has been posited to underlie the association between depression and increased rates of medical illness, including cardiovascular disease, diabetes, metabolic syndrome, osteoporosis and dementia (see Wolkowitz et al.1 for a review).

Really? A study co-authored by Wolkowitz and cited later in the introduction actually concluded

telomere shortening does not antedate depression and is not an intrinsic feature. Rather, telomere shortening may progress in proportion to lifetime depression exposure.

“Exposure” = personal experience being depressed. This would seem to undercut the rationale for examining telomere shortening in young girls who have not yet become depressed.

But more importantly, nether the Molecular Psychiatry article nor the Wolkowitz review acknowledge the weakness of evidence for

  • Depression being characterized by shortened telomere length.
  • The association of depression and medical illness in older persons representing a causal role for depression that can be modified by or prevention or treatment of depression in young people.
  • Telomere length observed in the young underlying any association between depression and medical illnesses when they get old.

Wolkowitz’s “review” is a narrative, nonsystematic review. The article assumes at the outset that depression represents “accelerated aging” and offers a highly selective consideration of the available literature.

In neither it nor the Molecular Psychiatry article we told

  • Some large scale studies with well-defined phenotypes fail to find associations between telomeres and depressive disorder or depressive symptoms. One large-scale study co-authored by Wolkowitz found weak associations between depression and telomere length too small to be detected in the present small sample. Any apparent association may well spurious.
  • The American Heart Association does not consider depression as a (causal) risk factor for cardiovascular disease, but as a risk marker because of a lack of the evidence needed to meet formal criteria for causality. Depression after a heart attack predicts another heart attack. However, our JAMA systematic review revealed a lack of evidence that screening cardiac patients for depression and offering treatment reduces their likelihood of having another heart attack or improves their survival. An updated review confirmed our conclusions.
  • The association between recent depressive symptoms and subsequent dementia is evident with very low level of symptoms, suggesting that it reflects residual confounding and reverse causation  of depressive symptoms with other risk factors, including poor health and functioning. I published a commentary in British Medical Journal  that criticized  claim that we should begin intervening for even low symptoms of depression in order to prevent dementia. I suggested that we would be treating a confound and it would be unlikely to make a difference in outcomes.

I could go on. Depression causally linked to diabetes via differences in telomere length? Causing osteoarthritis? You gotta be kidding. I demand quality evidence. The burden of evidence is on anyone who makes such wild claims.

Sure, there is lots of evidence that if people have been depressed in the past, they are more likely to get depressed again when they have a chronic illness. And their episodes of depression will last longer.

In general, there are associations between depression and onset and outcome of chronic illness. But the simple, unadjusted association is typically seen at low levels of symptoms, increases with age and accumulation of other risk factors and other physical co-morbidities. People who are older, already showing signs of illness, or who have poor health-related behaviors tend to get sicker and die. Statistical control for these factors reduces or eliminates the apparent association of depressive symptoms with illness outcomes. So, we are probably not dealing with depression per se.  If you are interested in further discussion of this see my slide presentation, see

Negative emotion and health: why do we keep stalking bears, when we only find scat in the woods?

I explain risk factors (like bears) versus risk markers (like scat) and why shooting scat does not eliminate the health risk posed by bears,.

I doubt few people familiar with the literature believe that associations among telomeres and depression, depression and the onset of chronic illness, and telomeres and chronic illness are such that a case could be made for telomere length in young girls being importantly related to physical disease in their mid and late life. This is science fiction being falsely presented as evidence-based.

The authors of the Molecular Psychiatry paper are similarly unreliable when discussing “dysregulation of the hypothalamic-pituitary-adrenal (HPA) axis and anomalous levels of cortisol secretion.” You would think that they are referring to established biomarkers for risk of depression. Actually, most biological correlates of depression are modest, nonspecific to depression, and state, not trait-related – limited to when people are actually depressed.

MDD and ND [nondepressed] individuals exhibited similar baseline and stress cortisol levels, but MDD patients had much higher cortisol levels during the recovery period than their ND counterparts.

We did not find the expected main effects of maternal depression on children’s cortisol  reactivity.

  • They misrepresent a directly relevant study that examined cortisol secretion in the saliva of adolescents as a predictor of subsequent development of depression.  It actually found no baseline measure of cortisol measures predicted development of depression except cortisol awakening response.

In general, cortisol secretion is more related to stress than to clinical depression. One study concluded

The hypothalamic—pituitary—adrenal axis is sensitive to social stress but does not mediate vulnerability to depression.

depressed girlWhat is most outrageous about the introduction, however, is the specification of the pathway between having a depressed mother and shortened telomere length:

The chronic exposure of these children to this stress as a function of living with mothers who have experienced recurrent episodes of depression could represent a mechanism of accelerated biologic aging, operationalized as having shorter telomere length.

Recognize the argument that is being set up: having to deal with the mothers’ depression is a chronic stressor for the daughters, which sets up irreversible processes before the daughters even become depressed themselves, leading to accelerated aging, chronic illness, and early death. We can ignore all the characteristics, including common social factors, that the daughter share with their mothers, that might be the source of any daughters’ problems.

This article is a dream paper for the lawyers for men seeking custody of their children in a divorce: “Your honor, sole custody for my client is the children’s only hope, if it is not already too late. His wife’s depression is irreversibly damaging the children, causing later sickness and early death. I introduced as evidence of an article by Ian Gotlib that was endorsed by the Director of the National Institute of Mental Health…

Geraldine Downey and I warned about this trap in a classic review, children of depressed parents, cited 2300 times according to Google Scholar and still going strong. We noted that depressed mothers and their children share a lot of uncharted biological, psychological, and environmental factors. But we also found that among the strongest risk factors for maternal depression are marital conflict, other life events generated by the marriage and husband, and a lack of marital support. These same factors could contribute to any problems in the children. Actually, the husband could be a source of child problems. Ignoring these possibilities constitutes a “consistent, if unintentional, ‘mother-bashing’ in the literature.”

The authors have asked readers to buy into a reductionist delusion. They assume some biological factors in depression are so clearly established that they can serve as biomarkers.  The transmission of any risk for depression associated with having a depressed mother is by way of irreversible damage to telomeres. We can forget about any other complex social and psychological processes going on, except that the mothers’ depression is stressing the daughters and we can single out a couple of biological variables to examine this.

Methods

The Methods lacks basic details necessary to evaluate the appropriateness of what was done and the conclusions drawn from any results. Nonetheless, there is good reason to believe that we are dealing with a poorly selected sample of daughters from poorly selected mothers.

We’re not told much about the mothers except that they have experienced recurrent depression during the childhood of the daughters. We have to look to other papers coming out of this research group to discover how these mothers were probably identified. What we see is that they are a mixed group, in part drawn from outpatient settings and in part from advertisements in the community.

Recall that identification of biological factors associated with depression requires well-defined phenotypes. The optimal group to study would be patients with severe depression. We know that depression is highly heterogeneous and that “depressed” people in the community who are not in specialty treatment are likely to just barely meet criteria. We are dealing with milder disorder that is less likely to be characterized by any of the biological features of more severe disorder. Social factors likely play more of a role in their misery. In many countries, medication would not be the first line of treatment.

Depression is a chronic, remitting, recurrent disorder with varying degrees of severity of overall course and in particular episodes. It has its onset in adolescence or early adulthood. By the time women have daughters who are 10 to 14 years old, they are likely to have had multiple episodes. But in a sample selected from the community, these episodes may have been mild and not necessarily treated, nor even noticeable by the daughters. The bottom line is we should not be too impressed with the label “recurrent depression” without better documentation of the length, severity, and associated impairment of functioning.

Presumably the depressed mothers in the study were selected because they were currently depressed. That makes it difficult to separate out enduring factors in the mothers and their social context versus those that are tied to the women currently being depressed. And because we know that most biological factors associated with depression are state dependent, we may be getting a skewed picture of the biology of these women – and their daughters, for that matter – then at other times.

Basically, we are dealing with a poorly selected sample of daughters from a poorly selected sample of mothers with depression. The authors are not telling us crucial details that we need to understand any results they get. Apparently they are not measuring relevant variables and have too a small sample to apply statistical controls anyway.As I said about another small study making claims for a blood test for depression, these authors are

Looking for love biomarkers in all the wrong places.

Recall that I also said that results from small samples like this one often conflict with results from larger, epidemiologic studies with larger samples and better defined phenotypes. I think we can see the reasons why developing here. The small sample consist only of daughters who have a depressed mother, but who have not yet become depressed themselves and have low scores on a child depression checklist. Just how representative is the sample? What proportion of daughters this age of depressed women would meet these criteria? How are they similar or different from daughters who have already become depressed? Do the differences lie in their mothers or in the daughters or both? We can’t address any of these questions, but they are highly relevant. That’s why we need more larger clinical epidemiologic studies and fewer small studies of poorly defined samples. Who knows what selection biases are operating?

Searching the literature for what this lab group was doing in other studies in terms of mother and daughter recruitment, I came across a number of small studies of various psychological and psychobiological characteristics of the daughters. We have no idea whether the samples are overlapping or distinct. We have no idea about how the results of these other modest studies confirm or contradict results of the present one. But integrating their results with the results of the present study could have been a start in better understanding it.

As noted in my post at Science Based Medicine, we get a sense of the methods section of the Molecular Psychiatry article of unreliability in single assessments of telomeres. Read the description of the assay of telomere length in the article to get a sense of the authors having to rely on multiple measurements, as well as the unreliability of any single assessment. Look at the paragraph beginning

To control for interassay variability…

This description reflects the more general problems in the comparability of assessment of telomeres across individuals, samples, and laboratories problems that, that preclude recommending telomere length as a biomarker or surrogate outcome with any precision.

Results and Interpretation

As in the methods, the authors fail to supply basic details of the results and leave us having to trust them. There is a striking lack of simple descriptive statistics and bivariate relations, i.e., simple correlations. But we can see signs of unruly, difficult to tame data and spun statistics. And in the end, there are real doubts that there is any connection in these data between telomeres and cortisol.

The authors report a significant difference in telomere length between the daughters of depressed women versus daughters in the control group. Given how the data had to be preprocessed, I would really like to see a scatter plot and examine the effects of outliers before I came to a firm conclusion. With only 50 daughters of depressed mothers and 40 controls, differences could have arose from the influence of one or two outliers.

We are told that the two groups of young girls did not differ in Tanner scores, i.e., self-reported signs of puberty. If the daughters of depressed women had indeed endured “accelerated aging,” would it be reflected in Tanner scores? The authors and for that matter, Insel, seem to take quite literally this accelerated aging thing.

I think we have another seemingly large difference coming from a small sample that is statistically improbable to yield such a difference, given past findings. I could be convinced by these data of group differences in telomere length, but only if findings were replicated in an independent, adequately sized sample. And I still would not know what to make of them.

The authors fuss about  anticipating a “dysregulation of the hypothalamic-pituitary-adrenal (HPA) axis and anomalous levels of cortisol secretion.” They indicate that the cortisol data was highly skewed and had to be tamed by winsorizing, i.e., substituting arbitrary values for outliers. We are not told for how many subjects this was done or from which group they came. The authors then engaged in some fancy multivariate statistics, “a piecewise linear growth model to fit the quadratic nature of the [winsorized] data.”  We need to keep in mind that multilevel modeling is not a magic wand to transform messy data. Rather, it involves some assumptions that need to be tested and not assumed. We get no evidence of the assumptions being tested and the small sample sizes is such that they could not be reliably tested.

The authors found no differences in baseline cortisol secretion. Moreover, they found no differences in distress recovery for telomere length, group (depressed versus nondepressed mother), or group by telomere interaction. They found no effect for group or group by telomere interaction, but they did find a just significant (p< .042) main effect for telomere length on cortisol reactivity. This would not to seem to offer much support for a dysregulation of the HPA axis or anomalous levels of cortisol secretion associated with group membership (having a depressed versus nondepressed mother). If we are guided by the meta-analysis of depression and cortisol secretion, the authors should have obtained a group difference in recovery, which they didn’t. I really doubt this is reproducible in a larger, independent sample, with transparently reported statistics.

Recognize what we have here: prestigious journals like Molecular Psychiatry have a strong publication bias in requiring statistical significance. Authors therefore must chase and obtain statistical significance. There is miniscule difference from p<.042 and p<.06 – or p<.07, for that matter – particularly in the context of multivariate statistics being applied to skewed and winsorized data. The difference is well within the error of messy measurements. Yet if the authors had obtained p<.06 or p<.07, we probably wouldn’t get to read their story, at least in Molecular Psychiatry.*

Stay tuned for my next installment in which I compare results of this study to the press release and coverage in Insel’s personal blog.  I particularly welcome feedback before then.

*For a discussion of whether “The number of p-values in the psychology literaturethat barely meet the criterion for statistical significance (i.e., that fall just below .05) is unusually large,” see Masicampo and LaLande (2012)  and Lakens (2015).

Category: biomarkers, HPA Axis, hype, maternal depression, Mind-body, stress | Tagged , , , , | 2 Comments

Will following positive psychology advice make you happier and healthier?

smile or dieSmile or Die the European retitling of Barbara Ehrenreich’s realist, anti-positive-psychology book Bright Sided:How Positive Thinking Is Undermining Americacaptures the threat of some positive psychology marketers’ advice: if you do not buy what we sell, you will face serious consequences to your health.

Barbara Fredrickson, along with co-authors including Steven Cole, make the threat that if we simply pursue pleasure in our lives rather than meaning, there will be dire consequences for our immune system by way of the effects on genomic expression.

People who are happy but have little-to-no sense of meaning in their lives have the same gene expression patterns as people who are enduring chronic adversity.

A group consisting of Nick Brown, Doug McDonald, Manoj Samanta, Harris Friedman and myself obtained and reanalyzed the data on which Fredrickson et al based their claim. We concluded:

Not only is Fredrickson et al.’s article conceptually deficient, but more crucially statistical analyses are fatally flawed, to the point that their claimed results are in fact essentially meaningless.

objecrtive approach to moralIn workshops, books, and lucrative talks to corporate gatherings, Fredrickson promises that practicing the loving-kindness meditation that she markets will send you on an upward spiral of physical and mental health that ends who knows where.

My co-authors – this time, Nick Brown, Harris Friedman and James Heathers– and I examined her paper and obtained her data. Re-analyses found no evidence that loving-kindness meditation improved physical health. The proxy measure for physical health in this study – cardiac vagal tone – is not actually reliably related to objective measures of physical health and probably wouldn’t be accepted in other contexts. And it was not affected by loving-kindness meditation anyway.

Katrinaloverimages http://www.fanpop.com/clubs/katerinalover/images/30154750/title/dont-worry-happy-photo

Katrinaloverimages
http://www.fanpop.com/clubs/katerinalover/images/30154750/title/dont-worry-happy-photo

The simplest interpretation of Fredrickson’s interrelated and perhaps overlapping studies of loving-kindness meditation is that lots of people drop out from follow-up and any apparent effect of the meditation is actually due to unexplained deterioration in the control group. And though data concerning the participants’ practice of mediation were collected, none were presented concerning whether participants assigned to mediation actually practiced it or how it affected physical and mental health outcomes. Why were the data collected if they were not going to be reported? They could be used to address the crucial question of whether actually practicing meditation affects health and well-being.

Another queen of positive psychology advice, Sonia Lyubomirsky, proclaims in a highly cited paper:

The field of positive psychology is young, yet much has already been accomplished that practitioners can effectively integrate into their daily practices. As our metaanalysis confirms, positive psychology interventions can materially improve the wellbeing of many.

I showed these claims are based on a faulty meta-analysis of methodologically-poor studies. In addition to Lyubomirsky’s highly-cited meta-analysis, I examined a more recent and better meta-analysis by Bolier and colleagues. It showed that the smaller and poorer-quality a study of positive psychology interventions is, the stronger the effect size. With the more recent studies included in Bolier’s meta-analysis, I concluded:

The existing literature does not provide robust support for the efficacy of positive psychology interventions for depressive symptoms. The absence of evidence is not necessarily evidence of an absence of an effect. However, more definitive conclusions await better quality studies with adequate sample sizes and suitable control of possible risk of bias. Widespread dissemination of positive psychology interventions, particularly with glowing endorsements and strong claims of changing lives, is premature in the absence of evidence they are effective.

I’m quite confident that this conclusion holds for effects on positive affect and general well-being as well.

Actually, when Lyubomirsky attempted to demonstrate the efficacy she claims for positive psychology interventions, she obtained null results but relegated her findings to a book chapter that was not peer reviewed. Yet, her marketing of the claim that positive psychology interventions improve well-being continues undaunted and gets echoed in the most recent papers coming out of the positive psychology community, such as:

Robust evidence exists that positive psychology interventions are effective in enhancing well-being and ameliorating depression.

Advice gurus claim that practicing positive psychology interventions will lead to health and well-being without a good scientific basis. But another literature attempts to identify small changes in everyday and laboratory behavior that can have lasting benefits. These studies are not explicitly evaluating interventions, but the claim is that they identify small behaviors with potentially big implications for well-being and happiness.

Let’s start with an example from the Wall Street Journal (WSJ):

Walk this way: Acting happy can make it so

Research shows people can improve their mood with small changes in behavior

Elizabeth Dunn, Associate Professor of Psychology at the University of British Columbia, provides an orientation:

There are these little doses of social interactions that are available in our day” that can brighten our mood and create a sense of belonging. “I don’t think people recognize this.”

depressed postureThe article starts with a discussion of work by Johannes Michalak from the Department of Psychology and Psychotherapy at Witten Herdecke University, Germany. In one study, 30 depressed psychiatric inpatients were randomized to instructions to sit in either a slumped (n =15) or an upright (n =15) position and then completed a memory test. The idea is that an emotion like depression is embodied. Adopting a slumped posture should increase a depressive negative bias in recall. The abstract of the original article reports:

Upright-sitting patients showing unbiased recall of positive and negative words but slumped patients showing recall biased towards more negative words.

Michalak conducted another study in which the gait of 39 college students was manipulated with biofeedback so as to simulate either being depressed or nondepressed as they walked on a treadmill. During the period on the treadmill, the experimenter read 40 words to them and they were tested for recall. The abstract of the original study reports:

The difference between recalled positive and recalled negative words was much lower in participants who adopted a depressed walking style as compared to participants who walked as if they were happy.

As would be with expected with such small sample sizes, results were weak. Analyses were unnecessarily complicated. It’s not clear that effects would persist if more basic statistics were  presented. For instance, did patients assigned to the “depressive” slump condition in the first study recall fewer positive and more negative words, or both, or neither? Certainly in the second study with college students, there were no differences in recall of positive words and only small differences in recall of negative words. Claims in the abstract were based on the construction of a more complicated composite positive variable.

Michalak is following a familiar strategy in the positive psychology literature – indeed, one that is more widely followed in psychology: If you cannot obtain positive findings in straightforward, simple analyses, then (1) adopt flexible rules of analyses, such as selective introduction of covariates and making up new composite variables; (2) don’t report the simple statistics and analyses in tables where readers could check them; and (3) spin your results in the abstract because that is what most readers will rely on in deciding what your study found

Michalak claims that these studies point to manipulation of the embodiment of depression as a means of treating depression:

There is a mutual influence between mood and body and movement…There might be specific types of movements that are specific characteristics of depression and this feeds the lower mood. So it’s a vicious cycle.

Presumably, with this as a premise, depressed patients could obtain a clinically-significant improvement in mood if they sat up straight and walked faster. Maybe, but this is at best speculative and premature. Michalak does not directly test the take-away message the author of the WSJ article wants to give: Even if you are not depressed, you can improve you mood by sitting up straight and adopt what Michalak calls a “happy walking style.”

To be fair to Michalak, he may be pumping up the strength and significance of his findings and promoting himself a bit. But unlike the rest of the authors discussed in the WSJ article, he is not yet prematurely turning some scientific papers of modest significance and strength of findings into press releases, TED talks and positive psychology products like books and workshops.

But let’s turn to the work of Nicholas Epley that is next described in the article. Epley is a Professor of Behavioral Science, University of Chicago Booth School of Business and author of Mindwise: How We Understand What Others Think, Believe, Feel, and Want. Epley does not have a TED talk, but got mentioned in the Business Blog of the Financial Times as not needing one. And he’s available through the Washington Speakers Bureau, whose website proclaims it is “connecting you with the world’s greatest minds”.

According to the WSJ article:

“I used to sit in quiet solitude on the train,” Dr. Epley said. “I don’t anymore. I know now from our data that learning something interesting about the person sitting next to me would be more fun than pretty much anything else I’d be doing then,” he said.

This is a reference to his article

Epley, N., & Schroeder, J. (2014). Mistakenly seeking solitude. Journal of Experimental Psychology: General, 143(5), 1980.

talking to strangersThe study actually involves trains, buses, and taxis and giving different participants instructions to either connect with strangers, remain disconnected, or commute as normal.

In the train experiment, a composite measure was substituted for a simpler measure of whether these various strategies made participants happier:

To obtain an overall measure of positivity, we first calculated positive mood (happy minus sad), then standardized positive mood and pleasantness, and then averaged those two measures into a single index.

Epley study one

Error bars represent the standard error around the mean of each condition.

A one-way analysis of variance indicated a significant difference among the three conditions (p< .05) explaining a modest 6% of participants’ variation in the composite measure of mood. However, consulting Figure 1 in the article suggests the effect was in the difference between instructions to remain disconnected versus the other two conditions, not between connecting with strangers versus commuting as normal. See an excerpt of Figure 1 to the left.

Results were somewhat stronger when the experiment was replicated on buses, (p = .02), with 10% of the variance in participant mood explained by the condition to which participants were assigned. See figure to the right.Epley bus experiment 2

When the experiment involved talking to a taxi driver, significant results were obtained (p <.01). But this time, pairwise differences between conditions were tested and there was no significant difference between the connecting and the control condition, only between the control condition and the condition in which participants were instructed not to talk to the taxi driver.

This may not be rocket science, but it is apparently worthy of press releases, media coverage, and positive psychology products. The results are overall weak and may even disappear in straightforward analyses with simple measures of happiness. The most robust interpretation I could construct was if someone asks you to refrain from talking to others on the train or bus or even a taxi driver, you probably should ignore them. I offer this advice for free, and have no intention of presenting it in a TED talk with unattributed anecdotes.

Re-enter Elizabeth Dunn, Associate Professor of Psychology at University of British Columbia. Dr. Dunn is the author of Happy Money: The Science of Smarter Spending, has presented a  TED talk. She is available as a speaker through the Lavin Agency, which according to its website, is “making the world a smarter place.” The WSJ article reports on:

Sandstrom, G. M., & Dunn, E. W. (2013). Is efficiency overrated? Minimal social interactions lead to belonging and positive affect. Social Psychological and Personality Science,

Participants were instructed to either avoid any unnecessary conversation with a barista at Starbuck’s and simply be efficient in getting their coffee or:

“have a genuine interaction with the cashier—smile, make eye contact to establish a connection, and have a brief conversation.”

The journal article reports that participants instructed to make a “genuine connection” had more positive affect and less negative affect than those instructed to avoid unnecessary conversation. Unfortunately, unlike the Epley experiment, we are not given any comparison with a control condition, which would’ve clarified whether the effect was primarily due to instructing participants to have a “genuine connection” or to avoid conversation.

Then there are Professor Dunn’s student Jordi Quoidbach’s chocolate experiments, which have been promoted not only in this WSJ article, but in Dunn’s op-ed in the New York Times, “Don’t indulge. Be happy.”

The first of the studies was:

Quoidbach, J., Dunn, E. W., Petrides, K. V., & Mikolajczak, M. (2010). Money Giveth, Money Taketh Away: The Dual Effect of Wealth on Happiness. Psychological Science.

stack-50-euro-bills-isolated-white-19404122The study involved priming participants with a reminder of wealth – a photo of a large stack of Euro bills – or a similar photo that was blurred beyond recognition in the control condition. The 40 participants were then instructed to eat a piece of chocolate and complete a follow-up questionnaire.

As seen in the other studies, simple analyses were suppressed in favor of a more complex analysis. Namely, preliminary examination of the data revealed that female participants savored chocolate more than males. So, rather than simple t-test, analyses of covariance were conducted with gender and prior attitude towards chocolate as control variables. Note that they were only 20 participants per group to begin with, and so results of these multivariate analyses are quite dubious. Participants primed with the money photo spent less time eating the piece of chocolate and were rated by observers as enjoying it less.

Studies involving priming participants with seemingly irrelevant, but suggestive stimuli such as this one are now held in low regard and some feel they have contributed to the crisis of confidence in social psychology. Nobel Prize winner Daniel Kahneman suggests the lack of replicability of social priming research is “a train wreck looming” for social psychology. Results are often too good to be true and cannot be replicated. Jordi Quoidbach and her colleagues cite studies by Vohs(1,2) as supporting the validity of the manipulation, however, the two primary studies by Vohs could not be independently replicated.

Overall, this is an underpowered-study with results that probably depended on flexible analyses rather than simple ones. We would probably ignore it, except it appeared in the prestigious journal Psychological Science and has been hyped in the media and positive psychology products.

The second chocolate study was:

Quoidbach, J., & Dunn, E. W. (2013). Give It Up: A Strategy for Combating Hedonic Adaptation. Social Psychological and Personality Science, 4(5), 563-568.

chocolateThe study involved asking 64 participants to eat two pieces of chocolate into lab sessions separated by a week. Analyses were based on the 55 who showed up for the second session. Participants had been randomized to one of three conditions: restricted access (n = 16) in which participants were told not to buy any chocolate until the next lab session; abundant access (n = 18) in which participants were given two pounds of chocolate and told to eat as much as they comfortably could before the next lab session; and a control condition in which no explicit instructions (n = 21) were given. In the second lab session, all participants were given a second piece of chocolate.

Once again we have a small study in which the authors deny readers an opportunity to examine simple statistical tests of what should be a simple hypothesis: that restricted versus free access has an effect on enjoyment of a piece of chocolate. Instead of a simple one-way analysis of variance, the authors looked at their data and decided to do the (unnecessarily) more complex analysis of covariance. Nonetheless, we can still see that in pairwise comparisons between groups, there are differences between the restricted access and the abundant access and control group. Yet there were no differences between the abundant access instructions and having no instructions for what to do in the week between sessions.

The authors did not provide readers with appropriate analyses of group differences in changes in overall positive affect between the two sessions. Nonetheless, within-group t-tests revealed a decline in overall positive affect only for the abundance condition.

So, another small study in which positive results probably depended on tricky flexible analyses. We would not be discussing this if it were not in a relatively prestigious journal, discussed in the WSJ, and written about by one of the authors in an op-ed piece in the New York Times. I invite your comparison of my analysis to the hyped presentation and exaggerated significance for the study claimed in the op-ed.

This blog post ends quite differently than I originally intended. I wanted to take some highly-promoted findings in the positive psychology literature about the effects of small things on overall well-being. I looked to a WSJl article reporting findings from basically prestigious journals with recognizably big name promoters  of positive psychology.

I had expected that positive psychology people out promoting their work and selling their products could surely come up with some unambiguous findings. I could then discuss how we could decide whether to attempt to translate those findings into strategies in our everyday lives and whether we could expect them to be sustained with any lasting impact on our well-being. Unfortunately, I didn’t get that far. Findings turned out to be not particularly positive despite being presented as such. That became an interesting story in itself, even if I will still have to search for robust findings from the positive psychology literature in order to discuss the likelihood that following “scientific” positive psychology advice will make us happier overall.

Despite heavily-marketed claims to the contrary, positive psychology interventions do not consistently improve mental or physical health and well-being. The myth that these interventions are efficacious is perpetuated by a mutually-admiring, self-promotional collective that protects its claims from independent peer review and scrutiny.

As with the positive psychology intervention literature, it is a quick leap from the authors submitting a manuscript to a peer-reviewed journal to making claims in the media, including op-ed pieces in the New York Times, and then releasing products like workshops and books that are lavishly praised by other members of the positive psychology community.

It is apparently too much to expect that positive psychology advice givers will take time out from their self-promotion to replicate what are essentially pilot studies before hitting the road and writing op eds again. And too much to expect that the Association of Psychological Science journals Psychological Science and Social Psychological and Personality Science will insist on transparent reporting of adequately powered studies as a condition for publication.

The incentives for scientifically sound positive psychology advice just aren’t there.

Special thanks to the Skeptical Cat, who is smarter, more independent,  and less easily led than the Skeptical Dog.skeptical sleuth cat 8 30 14-1

 

Category: depression, happiness, positive psychology, self-help | Tagged , , , , , | 4 Comments

Pay $1000 to criticize a bad ‘blood test for depression’ article?

pay to play-1No way, call for retraction.

Would you pay $1,000 for the right to criticize bad science in the journal in which it originally appeared? That is what it costs to participate in postpublication peer review at the online Nature Publishing Group (NPG) journal, Translational Psychiatry.

Damn, NPG is a high-fashion brand, but peer review is quite fallible, even at an NPG npgxJournal. Should we have to pay to point out the flawed science that even NPG inevitably delivers? You’d think we were doing them a favor in terms of quality control.

Put differently, should the self-correction on which scientific progress so thoroughly depends require critics be willing to pay, presumably out of their own personal funds? Sure, granting agencies now reimburse publication costs for the research they fund, but a critique is unlikely to qualify.

Take another perspective: Suppose you have asmall data set of patients for whom you have blood samples.  The limited value of the data set was further comporimsed by substantial, nonrandom loss to follow-up. But you nonetheless want to use it to solicit industry funding for a “blood test for depression.” Would you be willing to pay a premium of $3,600-$3,900 to publish your results in a prestigious NPG journal, with the added knowledge that it would be insulated from critics?

I was curious just who would get so worked up about an article that they would pay $1,000 to complain.

So, I put Translational Psychiatry in PUBLICATION NAME at Web of Science. It yielded 379 entries. I then applied the restriction CORRESPONDENCE and that left only two entries.

Both were presenting original data and did not even cite another article in Translational Psychiatry.  Maybe the authors were trying to get a publication into an NPG journal on the cheap, at a discount of $2,600.

P2PinvestIt appears that nobody has ever published a letter to the editor in Translational Psychiatry. Does that mean that there has never ever been anything about which to complain? Is everything we find in Translational Psychiatry perfectly trustworthy?

I recently posted at Mind the Brain and elsewhere about a carefully-orchestrated media campaign promoting some bad science published in Translational Psychiatry. An extraordinary publicity effort disseminated a Northwestern University press release and video to numerous media outlets. There was an explicit appeal for industry funding for the development of what was supposedly a nearly clinic-ready inexpensive blood test for depression.

The Translational Psychiatry website where I learned of these publication costs displays the standard NPG message that becomes mocking by a paywall that effectively blocks critics:

“A key strength of NPG is its close relationship with the scientific community. Working closely with scientists, listening to what they say, and always placing emphasis on quality rather than quantity, has made NPG the leading scientific publisher at finding innovative solutions to scientists’ information needs.”

The website also contains the standard NPG assurances about authors’ disclosures of conflicts of interest:

“The statement must contain an explicit and unambiguous statement describing any potential conflict of interest, or lack thereof, for any of the authors as it relates to the subject of the report”

The authors of this particular paper declared:

“EER is named as an inventor on two pending patent applications, filed and owned by Northwestern University. The remaining authors declare no conflict of interest.”

Does this disclosure give readers much clarity concerning the authors’ potential financial conflict of interest? Check out this marketing effort exploiting the Translational Psychiatry article.

Northwestern Researchers Develop RT-qPCR Assay for Depression Biomarkers, Seek Industry Partners

I have also raised questions about a lack of disclosures of conflicts of interest from promoters of Triple P Parenting. The developers claimed earlier that their program was owned by the University of Queensland, so there was no conflict of interest to declare. Further investigation  of the university website revealed that the promoters got a lucrative third of proceeds. Once that was revealed, a flood of erratum notices disclosing the financial conflicts of interest of Triple P promoters followed – at least 10 so far. For instance

triple P erratum PNG

Please Click to Enlarge

How bad is the bad science?

You can find the full Translational Psychiatry article here. The abstract provides a technical but misleading summary of results:

“Abundance of the DGKA, KIAA1539 and RAPH1 transcripts remained significantly different between subjects with MDD and ND controls even after post-CBT remission (defined as PHQ-9 <5). The ROC area under the curve for these transcripts demonstrated high discriminative ability between MDD and ND participants, regardless of their current clinical status. Before CBT, significant co-expression network of specific transcripts existed in MDD subjects who subsequently remitted in response to CBT, but not in those who remained depressed. Thus, blood levels of different transcript panels may identify the depressed from the nondepressed among primary care patients, during a depressive episode or in remission, or follow and predict response to CBT in depressed individuals.”

This was simplified in a press release that echoed in shamelessly churnalized media coverage. For instance:

“If the levels of five specific RNA markers line up together, that suggests that the patient will probably respond well to cognitive behavioral therapy, Redei said. “This is the first time that we can predict a response to psychotherapy,” she added.”

The unacknowledged problems of the article began with the authors only having 32 depressed primary-care patients at baseline and their diagnostic status not having been  confirmed by gold standard semi-structured interviews by professionals.

But the problems get worse. For the critical comparison of patients who recovered in cognitive behavioral therapy versus those that did not occurred in the subsample of nine recovered versus 13 unrecovered patients remaining after a loss-to-follow-up of 10 patients. Baseline results for the 9 +13= 22 patients in the follow-up sample did not even generalize back to the original full sample. How, then, could the authors argue that the results apply to the 23 million or so depressed patients in the United States? Well, they apparently felt they could better-generalize back to the original sample, if not the United States, by introducing an analysis of covariance that controlled for age, race and sex.  (For those of you who are tracking the more technical aspects of this discussion, contemplate the implications of controlling for three variables in a between-groups comparison of nine versus 13 patients. Apparently the authors believe that readers would accept the adjusted analyses in place of the unadjusted analyses which had obvious problems of generalizability. The reviewers apparently accepted this.).

Finally, treatment with cognitive behavior therapy was confounded with uncontrolled treatment with antidepressants.

I won’t discuss here the other problems of the study noted in my earlier blog posts. But I think you can see that these are analyses of a small data set truly unsuitable for publication in Translational Psychiatry and serving as a basis for seeking industry funding for a blood test for depression.

As I sometimes do, I tried to move from blog posts about what I considered problematic to a formal letter to the editor to which the authors would have an opportunity to reply. It was then that I discovered the publication costs.

So what are the alternatives to a letter to the editor?

Letters to the editor are a particularly weak form of post-publication peer review. There is little evidence that they serve as an effective self-correction mechanism for science. Letters to the editor seldom alter the patterns of citations of the articles about which they complain.

pay to paly2Even if I paid the $1,000 fee, I would only have been entitled to 700 words to make my case that this article is scientifically flawed and misleading. I’m not sure that a similar fee would be required from the authors to reply. Maybe responding to critics is part of the original package that they purchased from NPG. We cannot tell from what appears in the journal because the necessity of responding to a critic has not yet occurred.

It is quite typical across journals, even those not charging for a discussion of published papers, to limit the exchanges to a single letter per correspondent and a single response from the authors. And the window for acceptance of letters is typically limited to a few weeks or months after an article has appeared. While letters to the editor are often peer-reviewed, replies from authors typically do not receive peer review.

A different outcome, maybe

I recently followed up blogging about the serious flaws of a paper published in PNAS by Fredrickson and colleagues with a letter to the editor. They in turn responded. Compare our letters to see why the uninformed reader might infer that only confusion had been generated by either of them. But stay tuned…

The two letters would have normally ended any exchange.

However, this time my co-authors and I thoroughly re-analyzed the Fredrickson et al data and PNAS allowed us to publish our results. This time, we did not mince words:

“Not only is Fredrickson et al.’s article conceptually deficient, but more crucially statistical analyses are fatally flawed, to the point that their claimed results are in fact essentially meaningless.”

In the supplementary materials, we provided in excruciating detail our analytic strategy and results. The authors’ response was again dismissive and confusing.

The authors next refused our offer for an adversarial collaboration in which both parties would lay responses to each other with a mediator in order to allow readers to reach some resolution. However, the strengths of our arguments and reanalysis – which included thousands of regression equations, some with randomly generated data – are such others have now calling for a retraction of the original Fredrickson and Cole paper. If that occurs, it would be an extraordinarily rare event.

Limits on journals impose on post-peer-review commentary severely constrain the ability of science to self-correct.

The Reproducibility Project: Psychology is widely being hailed as a needed corrective for the crisis of credibility in science. But replications of studies such as this one involving pre-post sampling of genomic expression from an intervention trial are costly and unlikely to be undertaken. And why attempt a “replication” of findings that have no merit in the first place? After all, the authors’ results for baseline assessments did not replicate in the baseline results of patients still available at follow-up. That suggests a table problem, and that attempts at replication would be futile.

plosThe PLOS journals have introduced the innovation of allowing comments to be placed directly at the journal article’s webpage, with their existence acknowledged on the article itself. Anyone can respond and participate in a post-publication peer review process that can go on for the life of the interest in a particular article. The next stage in furthering post-publication peer review is that such comments be indexed and citable and counted in traditional metrics, as well as altmetrics. This would recognize citizen scientists’ contributions to cleaning up what appears to be a high rate of false positives and outright nonsense in the current literature.

pubmedcommonsPubMed Commons offers the opportunity to post comments on any of the over 23 million entries in PubMed, expanding the PLOS initiative to all journals, even those of the Nature Publishing Group. Currently, the only restriction is that someone attempting to place a comment have authored any of the 23,000,000+ entries in PubMed, even a letter to the editor. This represents progress.

But similar to the PLOS initiative, PubMed Commons will get more traction when it can provide conventional academic credit –countable citations– to contributors identifying and critiquing bad science. Currently authors can get credit for putting bad science into the literature that no one can get for  helping getting it recognized as such.

So, the authors of this particular article have made indefensibly bad claims about having made substantial progress toward developing an inexpensive blood test for depression. It’s not unreasonable to assume their motive is to cultivate financial support from industry for further development. What’s a critic to do?

In this case, the science is bad enough and the damage to the public and professionals’ perception of the state of the science of ‘blood test for depression’ sufficient for  a retraction is warranted. Stay tuned – unless Nature Publishing Group requires a $1,000 payment for investigating whether an article warrants retraction.

Postscript: As I was finishing this post, I discovered that the journals published by the  Modern Language Society requires  payment of a $3,000 membership fee to publish a letter to the editor in one of their journals. I guess they need to keep the discussion within the club. 

Views expressed in this blog post are entirely those of the author and not necessarily those of PLOS or its staff.

Special thanks to Skeptical Cat.skeptical sleuth cat 8 30 14-1

Category: blood test, Conflict of interest, depression, ethics, mental health care, open access, Peer review | Tagged , , , | 4 Comments