Is psychotherapy for depression any better than a sugar pill?

placebo.We now know  that estimates of the efficacy of antidepressants that were once readily accepted were exaggerated. Sure, the estimates came from meta analyses of the carefully searched published literature, but that literature was skewed to make antidepressants look better than they were. Pharmaceutical companies had withheld publication of negative and weak trials and they had selectively promoted positive trials, even buying reprints to distribute journal articles that reported them. Some worried that these sometimes huge purchases gave journals an added incentive for the journals to participate in the confirmatory bias.

A much more balanced perspective on how well antidepressants worked was contained in reports submitted to the Food and Drug Administration (FDA). Since 1997 Drug Approval Packages had been available for download by anyone. For drugs approved before then, reports were available on request. But no one had yet made effective use of these reports in a published evaluation of the efficacy of antidepressants. Either those who were potentially interested in doing so did not know how to gain access or they had been discouraged by what had been the FDA’s notoriously slow response to such requests.fda-logo

Erick Turner once worked for the FDA, but is now a Veterans Administration Health System psychiatrist. He made use of these data in a provocative paper. Whereas the published literature up until then suggested that almost all randomized trials involving antidepressants were positive, the FDA records he obtained revealed that only half were positive. He found that data submitted to the FDA that was positive was 12 times more likely to end up published than negative trials. So, just as many suspected, but could not quantify, there was a considerable bias going on in the evidence being made available to clinicians and policymakers. Turner re-analyzed the data concerning the efficacy of antidepressants, and found that when the suppressed FDA data were included, the effect size (mean standardized difference) dropped from 0.41 to 0.31. Turner did not claim that antidepressants lacked efficacy in comparison to pill placebos, but he did point out that previous estimates of their advantage were exaggerated.

Turner is a cautious, understated guy. As I noted in another blog post, Rich Bonin, a CBS Morning News producer approached him to appear in a segment about whether antidepressants were no more effective than sugar pills.  By Turner’s account, his careful explanation of his work left the CBS producer thinking out loud with his co-producer about whether they should proceed with the story, “or is it too murky…Is it good murky, or is it just murky?”

The producer resolved his dilemma by leaving Turner out of the program. Better a dramatic even if inaccurate message rather than a “murky” one.

60 minutesIrving Kirsch, a psychologist conducted analyses on four of the 12 antidepressants Turner had analyzed, and obtained essentially the same result, an effect size of 0.32. But Kirsch had much more of a flair for grabbing media attention than Turner did, and some would say much more of a flair for hyperbole. Kirsch had already become well known for his relentless declarations that the differences between antidepressants and a placebo pill were trivial. As in the 60 Minutes segment from which Turner was excluded:

Irving Kirsch: The difference between the effect of a placebo and the effect of an antidepressant is minimal for most people.

Lesley Stahl: So you’re saying if they took a sugar pill, they’d have the same effect?

Irving Kirsch: They’d have almost as large an effect and whatever difference there would be would be clinically insignificant.

Kirsch based his claims on the arbitrary requirement that in order to be clinically significant, an effect size has to be 0.50 and the differences between an antidepressant and a pill placebo on the rating scale that the FDA required, the Hamilton Rating Scale for Depression, had to be three points. This standard had previously surfaced in evaluations by the UK’s National Institute for Clinical Excellence  (NICE), but had already been abandoned by the time Kirsch made his claims. I asked Simon Gilbody, a respected British psychiatrist who does lots of meta-analyses what he thought of the now rejected NICE criterion of 0.50 as a sharp boundary between effective and ineffective treatments. He said

“I never believe any study these days which finds an ES > 0.5. it usually means the science is wrong and we haven’t found the bias(es) just yet.”

What would an effect size of 0.5 represent? It would indicate that the mean of the treatment group is at the 69th percentile of the control group and that the non-overlap is 33%.

Turner may not have made it to CBS 60 Minutes, but he had been invited to discuss in BMJ the differences between his and Kirsch’s interpretation of getting basically the same results. In a previous blog post, I commented on what Turner said.

He noted that the NICE criterion of an effect size of 0.5 had been taken from Jacob Cohen’s designation of it as “medium”, but Cohen himself had distanced himself from  his own designation of effect sizes as “small,” “medium,” or “large” with “The values chosen had no more reliable a basis than my own intuition.” Turner suggested that Cohen would undoubtedly have rejected NICE’s rigid distinction of 0.5 as a categorical cutoff between ineffective versus effective treatments. Indeed, Cohen who is now deceased, offered no hint that he would have welcomed becoming the arbiter of categorical judgments of clinical significance and famously lampooned the categorical p<.05 for deciding the world is flat rather than round.

Turner went on:

It seems unfair that pharmacological, and not psychotherapeutic, treatment has become the usual first line approach to depression merely for economic reasons. But before we embrace any treatment as first line, it is prudent to ask whether its efficacy is beyond question. For psychotherapy trials, there is no equivalent of the FDA whose records we can examine, so how can we be sure that selective publication is not occurring here as well?

In hindsight, Turner’s challenge has become particularly important now that Ben Goldacre has caused such a stir about hidden pharmaceutical trials. Goldacre showed that Pharma had successfully promoted the drug reboxetine, as safe and effective as an antidepressant, when data withheld from publication actually indicated it was dangerous for some people and, on average, ineffective. Does psychotherapy have a problem with hidden trials? In a future blog post, I will demonstrate that indeed it does.

But more immediately, Turner left me puzzling about what if we apply the same requirement of an effect size of 0.50 to comparisons between psychotherapy and pill placebo. Fair is fair. If we’re going to judge antidepressants as no better than a sugar pill, what if psychotherapy was no better than a sugar pill by the same standard?

Kirsch attracted a lot of attention, especially among those who already wanted to believe that antidepressants were ineffective and maybe even dangerous – the growing anti-psychiatry crowd that had almost become mainstream, but also clinical psychologists who were frustrated with their declining role in treating all the depression that was now being detected in primary care. When I raised in blog posts the issue that Kirsch’s interpretation of the uselessness of antidepressants might also be extended to the uselessness of psychotherapy, something none of us wanted to do, I found my blogs attacked by trolls and spammed repeatedly, even with praise for what Scientology had done for psychiatry. This was not the first time and I am sure it will not be the last in which I hit on an emotional issue in which the opposing sides were invoking “evidence”, but really did not want to look more closely to see whether the evidence actually supported their views.

Nonetheless, I became determined to put the issue of psychotherapy versus pill placebo to an empirical test. I recruited Erick Turner, but also Pim Cuijpers who had an excellent database about psychotherapy for depression that he constantly updated with citation alerts and renewed searches. You can find a list of the meta analyses he has produced here. We assembled a writing team that included Erick, Pim, and myself, but also David Mohr, Stefan Hofmann, Gerhard Andersson, and Matthias Berking. We did a fresh search of the literature, Pim performed the meta analysis, and together the group drafted the manuscript that became the 2013 Psychological Medicine article that I am now going to discuss.

The Question We Asked.

We asked whether if we integrated all of the available data that we could find, would psychotherapy be more effective than pill placebo.

Facing the Criticism That We Should Not Have Even Done This Meta-Analysis

When we first sent our manuscript out for review, the reviewers were critical about our even having undertaken this meta-analysis. Our responding to this criticism in revising the paper strengthened it. The basic criticism concerned a lack of blindness for either clinicians administering psychotherapy versus pill placebo or to their patients receiving it. In double-blind pharmaceutical trials, clinicians and patients can both see that the patients are given a pill, but they don’t know whether its an antidepressant or a placebo, because both are kept in the dark. On the other hand, in a comparison between psychotherapy and pill placebo, it’s obvious which treatment patients are being given. There is lots of room for bias to creep into the study in the form of patients having a strong preference for one treatment or the other and that influencing their response either by skewing the answers they give in assessments or even in having them a more positive expectation about one treatment affecting their actual response.

Elements of the bias are present even in comparisons of psychotherapy and antidepressants. In an earlier study, my colleagues and I showed that over half of patients enrolling in a study in improving the care for depression in primary care would have preferred psychotherapy to the antidepressants, and those with a strong preference did less well with the antidepressant. The bias is present any time you compare psychotherapy to a pill, and so it’s not exactly equivalent to comparing an active treatment contained in a pill in an inert one, especially when neither the clinician nor the patient knows which pill is which. The reviewers were correct that in important ways psychotherapy versus pill placebo comparisons were different then antidepressants versus pill placebo comparisons.

If so, why do the meta-analysis? Because it allowed us to address important policy issues. Policymakers, clinicians, and patient/consumers were making decisions based on arguments that the difference between antidepressants and pill placebo were trivial, and jumping to the conclusion that therefore psychotherapy would be a more effective treatment. We wanted to put that judgment to a test.

A pill placebo condition is not a sugar pill

I phrased the title of this blog post to parallel a lot of the discussions that have gone on about antidepressants versus pill placebos. But I should be clear that a lot more goes into the effect size calculated for such a trial. Patients receiving either treatment benefit from the expectation that they are getting an active treatment and there is a lot of attention that goes on in terms of an active clinical management protocol in which patients are regularly being asked about how well they are doing and provided a lot of social support. These nonspecific factors can then themselves lead to substantial improvement, but in a clinical trial data provided to all conditions. So, the effect size that is calculated takes into account both conditions having this advantage.

This brings us to a point that is not well understood in lots of popular discussions of how well treatments work.. We often talk about efficacy in terms of effect sizes, but we need to keep in mind that effect sizes do not characterize a treatment, but rather a comparison of treatments in a particular context. So, when we are discussing a pill placebo condition, this a lot more involved in producing the effect size than the pill itself, namely, the comparison condition and the context in which the treatment and control condition were administered.

It’s sad that many antidepressants are prescribed in routine depression care in the community without the support and follow-up that patients enrolled in a clinical trial get with their active clinical management. Instead, patients in routine care are simply given their prescription and sent off, often without anyone checking in on them further. When their prescriptions need refilling, that can be accomplished simply with a telephone call to the prescribing physician’s office, without anybody asking how they are doing or whether they require adjustments in dosage or changing medications. Studies show that about 50% of patients prescribed antidepressant require such an inquiry and possible adjustment or different treatment to get any effect.

I wouldn’t be surprised if the benefits of getting an antidepressant in the context of primary care are less than the benefits obtained from getting a pill placebo within the support and attention of a clinical trial. For that matter, many patients getting non-evidence-based psychotherapy in the community can come back month after month without any formal assessment of whether they are getting any benefit. But these are stories for another time.

We went into our meta-analysis fully expecting that psychotherapy might not perform as well as some would like when compared to a pill placebo. As we said in the published article,

Because pill placebo more plausibly controls for positive expectations, support and attention, it is conceivable that this approach will lead to effect sizes that are smaller than those historically found using waitlist and no-treatment control conditions. This would have important implications for clinical and policy decisions concerning provision of psychotherapy versus antidepressants for depression. Because earlier research based on head-to-head comparisons of psychotherapy and pharmacotherapy have found that their effects on depression are comparable (Cuijpers et al., 2008), our hypothesis is that the effect size of psychotherapy is comparable to that of pharmacotherapy, i.e. about g = 0.3 (Turner et al., 2008).

So, rather than 0.5 that Kirsch had proposed has a cutoff for a nontrivial difference, we anticipated that psychotherapy versus pill placebo would be about the same as antidepressants versus placebo. The effect size we anticipated, would indicate that patients getting either psychotherapy or antidepressants would be at the 62nd percentile of those getting a pill placebo and the non-overlap between either of the two active treatments and a pill placebo would be 21%.

What We Found.

We had the requirement that any study that we included had to rely on a standardized assessment of depression such as the Beck Depression Inventory or Hamilton Depression Rating Scale. An exhaustive search only revealed 10 randomized trials that yielded comparisons between psychotherapy and pill placebo. Thats because researchers typically do not use pill placebo as the comparison condition for psychotherapy. Rather, they are more interested in comparing psychotherapy to an antidepressant and a placebo condition is there only to establish that the antidepressant was effectively delivered. That would be shown in an antidepressant/pill placebo difference. So, we were taking advantage of comparisons that were not necessarily intended by investigators.

At the end of clinical trials, the effect size for psychotherapy compared to pill placebo was g = 0.25 [what does this mean?]. If we translate that into practical terms of Number Needed to Treat (NNT), 7.14 psychotherapy patients had to be treated in order to get assured of getting one who did better than getting a pill placebo. Patients in the psychotherapy conditions scored 2.66 points lower on the Hamilton depression rating scale than those assigned to pill placebo. These differences are well within the range of the differences found between antidepressants and pill placebo in the FDA registered trials.

Essentially, when compared to pill placebo, psychotherapy did as well or, if you’d like, as poorly as an antidepressant. So, inferring that psychotherapy is the preferred treatment based simply on the basis of the small differences between antidepressants and pill placebos is not warranted.

The FDA data on which both Erick Turner and Irving Kirsch drew did not include psychotherapy conditions. Thanks to the regulatory requirements of the FDA, we can be assured that there are not many trials being hidden. But we cannot make that assumption about psychotherapy trials because psychotherapy investigators do not consistently register their trials, and can simply leave unpublished trials in which the psychotherapy does not perform well. Add to that that much psychotherapy is conducted by persons who would benefit from a positive result in terms of either their academic career or selling of their treatment on the workshop circuit. There are well-known strong investigator allegiance effects so that it isunusual for published clinical trials to obtain results counter to the interest of the investigators.

Thus, it may have been a draw between the effects observed for psychotherapy versus pill placebo in a meta-analyses versus the effects observed in other meta-analyses of data obtained from the FDA concerning antidepressants versus pill placebos. But our analyses may have overestimated the advantages of psychotherapy.

Elsewhere, I referred to the controversies about antidepressants as being part of the “antidepressant wars.” Combatants in these wars have given a lot of attention to the lack of strong differences between antidepressants and pill placebo. But I think both sides should recognize that neither psychotherapy nor meidcation have the efficacy that we would like to obtain from them in treating depression, especially when they are ineffectively delivered in routine care in the community. Some have pointed out that the efficacy of neither treatment has greatly changed in the past 50 years. Pharmaceutical companies have stopped investing large amounts of money in research seeking to discover new treatments for depression. Similarly, Pim Cuijpers has shown that past research strongly indicates that no one active psychotherapy treatment for depression is consistently superior to its evidence-based alternatives, and so theres little reason to expect there is some breakthrough treatment on the horizon. We are stuck trying to do the best we can with existing treatments and improving upon their in adequate delivery in the community. If anywhere, for now at least, that is where progress is to be made.

Acknowledgments: Some of my co-authors on the Psychological Medicine provided extremely helpful feedback on earlier drafts of this post, Stefan Hofmann, Gerhard Andersson, and Matthias Berking, but especially Erick Turner. However, the views expressed here do not necessarily express theirs.

My interest in pursuing a comparison between psychotherapy and pill placebos developed has I worked on earlier blog posts on the efficacy of antidepressants. I drew liberally from these earlier blog posts in writing this one, among them 1, 2, 3

 

 

Creative Commons License
This work, unless otherwise expressly stated, is licensed under a Creative Commons Attribution 3.0 Unported License.

This entry was posted in antidepressants, Psychiatry and tagged , , , , . Bookmark the permalink.

19 Responses to Is psychotherapy for depression any better than a sugar pill?

  1. anon says:

    1. How can you compare a treatment that aims to change beliefs and instill hope (psychotherapy) with a treatment that… may change beliefs and instill hope (placebo)? Surely psychotherapy is just an honest placebo? It just doesn’t make sense to draw conclusions about absolute efficacy from a comparison of two psychological interventions.

    2. Comparing effect sizes across meta-analyses is a pretty limited way of comparing treatments, not least because of the non-random differences between the various groups of trials (eg different patient types, different study durations, different cohorts). The most valid comparison involves looking at individual trials comparing psychotherapy and antidepressants. Why not just look at them? Seems a roundabout way of looking at the question… i can see why you sent it to psychological medicine!!

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • The assumption of some people like Irving Kirsch is that antidepressants work almost entirely by changing belief and instilling hope and he concluded that the evidence supported this view. We were interested in seeing how psychotherapy fared.

      Unfortunately, there are not good ways of determining absolute efficacy, so we are mostly stuck with comparing relative efficacy and trying to control for some nuisance sources of differences.

      How would we summarize differences between treatments across studies, except with meta analysis? At least with meta analysis we can test whether differences across studies matter and how much, when we test for heterogeneity. This is certainly not foolproof, however.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  2. RRRR says:

    Thanks for this interesting post.

    “These nonspecific factors can then themselves lead to substantial improvement”

    How confident can we be that this is true, and not the result of biases? I’ve noticed how often the placebo affect seems to have a positive response upon questionnaire scores, but not things like employment data, or other more objective measures of outcome. (A 2010 Cochrane review found: “We did not find that placebo interventions have important clinical effects in general. However, in certain settings placebo interventions can influence patient-reported outcomes, especially pain and nausea, though it is difficult to distinguish patient-reported effects of placebo from biased reporting.”). Controlling for biases in psychotherapy trials seems incredibly difficult when we’re relying upon questionnaires as outcome measures (although other outcome measures have their own dangers too).

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Your quote from the 2010 Cochrane review is an important one. For many clinical problems, we are entirely dependent on patient reports, and these are usually most influenced by patient expectations and preferences. It is tough to get around this with psychotherapy research. We tried to do so in a complex trial which some patients got medication, some psychotherapy, and others got their preference of medication or therapy. Results were not clear cut, and so we apparently were not so clever as we thought.

      http://www.karger.com/Article/Abstract/318772

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
      • RRRR says:

        Thanks for that link.

        I’d like to see more trials using a wide range of outcome measures, ideally without those designing/carrying out treatments being aware as to how patients would be assessed. I don’t see any easy way of distinguishing genuine improvements from reporting bias though.

        I also worry that undue confidence in the ‘real’ power of placebo leads to an acceptance of misleading claims about the efficacy of treatments, and the belief that a promotion of positive views about treatments is a form of ‘treatment’ in itself, which can be legitimately performed without informed consent. As you mentioned above, their are a number of reasons to believe that the interventions (pharmacological and psychological) patients receive are likely to be less effective than is commonly claimed. This discrepancy can be harmful in a number of ways, not just by misleading patients, but also by affecting the way in which society views those who do not improve with treatment.

        Recent biopsychosocial reforms to the UK disability benefits system were founded on false assumptions about the extent to which psychosocial interventions could lead to the sick and disabled gaining employment, and we are now seeing many people being pushed into poverty, or having to go through long and arduous appeals processes. Insurance companies (many of which were involved in promoting BPS reforms through the UK government) have also been using similar tactics to deny payments to those with a wide range of health problems. Attempts to promote pragmatically devised narratives often serve to harm those who are in positions of weakness.

        VA:F [1.9.22_1171]
        Rating: 0 (from 0 votes)
        • Good points. Surely, few depressed patients enter treatment seeking a 3 point drop in their Hamilton and even those who get it rarely fail restored to “old self” or “back on track”. Don’t get wrong, depression treatment makes a difference, particularly when a clinician is willing to consider switching when it is not working. But unrealistic to assume will necessarily return unemployed to work or keep a cardiac patient from having another heart attack. Treatments not great, but better than nothing except they are often delivered ineffectually in community.

          VN:F [1.9.22_1171]
          Rating: +1 (from 1 vote)
  3. Faraaz Rasul says:

    James,
    You write that…
    “that no one active psychotherapy treatment for depression is consistently superior to its evidence-based alternatives, and so there’s little reason to expect there is some breakthrough treatment on the horizon”.

    This assumes that the different psychotherapuetic interventions have been compared against each other. I’d be surprised if they had.

    Ascertaining superiority of competing treatments is a common problem in drug trials where often there is an absence of “head to head ” trials to ascertain superiority. This makes it difficult for regulatory authorities to make licensing, economic effectiveness decisions based upon the marginal effectiveness of the new intervention. To get around this many regulatory authorities now expect “indirect comparisons” of alternative interventions to be carried out in their regulatory submissions to help them understand how the companies product compares against other similar products in efficacy / effectiveness.

    Not withstanding the cautionary notes made by above Anon such methods as indirect comparisons, multiple treatment meta-analysis, particularly using a Bayesian analytic framework may be useful in evaluating competing psychotherpuetic interventions and help to elucidate the effectiveness of various “flavours” of psychotherapy and other complex interventions.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  4. Faraaz Rasul says:

    Following on from my earlier post Re: indirect comparisons, multiple treatment meta analysis for different psychotherapuetic interventions please see or look out for the following article

    Barth J, Munder T, Gerger H, Nuesch E, Trelle S, Znoj H, Juni P, Cuijpers P (2013). Comparative efficacy of seven psychotherapeutic interventions for depressed patients: A network meta-analysis. Plos Medicine, in press.

    Faraaz

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Faraaz, Thank you for your interesting observations. In my blog post above, you can click on Pim Cuijpers’ highlighted name near the end and it’ll take you to a paper of his in which he indicates he and his colleagues’ meta-analyses of existing head-to-head comparisons of psychotherapies suggest only modest differences among them that are not likely to be clinically significant. He estimates that a future head-to-head noninferiority trial would require about 1000 patients and would hardly be worth the effort. I have not seen the paper of Barth and colleagues, but Pim is an author on it (as he is on the psychotherapy/pill placebo meta-analysis on which I am commenting in the blog). I would be surprised if this forthcoming paper will contradict what Pim has recently said elsewhere, but I am always open to that.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  5. Pingback: What If a Sugar Pill Was Just as Effective As Psychotherapy? | World of Psychology

  6. Pingback: Is psychotherapy for depression any better than a sugar pill?

  7. James says:

    I do prefer therapies than antidepressants. Ive heard one time that antidepressant pills have so much side effects, I heard it can even cause Alzheimer

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • There is certainly room for patient preference in the treatment of depression. Choose from an array of evidence-based treatments. But best to investigate if the basis of your preference is accurate. I doubt antidepressants have been identified as a cause of Alzheimer’s.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  8. Pingback: Friday Links | Meta Rabbit

  9. Discussant says:

    It would be interesting to see the results of trials comparing psychotherapy with a licensed professional for 45 minutes per week to talking with an empathetic/sympathetic friend, unpaid/unlicensed volunteer (perhaps having undergone a one-day training workshop), peer, or pet for 45 minutes per week.

    VA:F [1.9.22_1171]
    Rating: +1 (from 1 vote)
  10. Camilla says:

    I apologise for being off topic but I was wondering whether you’ve ever considered writing about ECT and the extent to which, if any, it can be said to be effective. Like with antidepressants, it seems impossible for to understand what actually the case is. The debate is polarised but it just can’t be the case ECT is both very effective and not effective at all.

    VA:F [1.9.22_1171]
    Rating: +1 (from 1 vote)
  11. Pingback: Post Of The Week – Thursday 19th September | DHSG Psychology Research Digest

  12. Fergus Kane says:

    Nice article, thanks.

    One thing that is worth considering is the effect of psychological therapy in preventing depressive relapse (and in improving non symptom based quality of life measures). This is currently an under-explored but very significant aspect of therapies for depression. This is also an important avenue for improvement in psychological therapy. I don’t have a lot of references to hand, but this classic one might be of interest:

    http://www-psych.stanford.edu/~pgoldin/Buddhism/MBCTrelapsedepressionTeasdale2000JCCP.pdf

    VA:F [1.9.22_1171]
    Rating: +1 (from 1 vote)
  13. Pingback: A2 Depression – Critical Questions On Explanations And Therapies | DHSG Psychology Research Digest

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>