Positive psychology in the schools: the UK Resilience Project

Why it could be predicted that the large scale school-based intervention would failcard_3_monkeys_see_no_evil_hear_no_evil_see-ra33d04ad8edf4f008e5230ac381ec8b0_xvuak_8byvr_512.

An important new study was greeted with a resounding silence  from the positive psychology community, and notably on the Friends of Positive Psychology listserv, but…

Results of the largest ever evaluation of a school-based positive psychology program, the UK resilience project are now available at Journal of Consulting and Clinical Psychology. The results are, uh, not impressive.

thrive_large_logoThe intervention, the 16-hour UK Resilience Programme (UKRP), was carefully based on the Penn Resiliency Program (PRP) for Children and Adolescents.  Jane E. Gillham, the corresponding author for the UK study was also one of the developers of the Penn program.

The study is behind a pay wall, but here’s an abstract. I’m sure that you can obtain a full copy from Dr. Gillham, E-mail: <jgillha1@swarthmore.edu>.

The study enrolled almost 3,000 students, with 1,000 students in the intervention group. The UK study is thus larger than the 17 previous studies combined. The largest past study had a total of only 697 students.

The authors reported that students receiving the intervention reported lower levels of depressive symptoms than students assigned to the control group, but the effect was small and did not persist to 1-year or 2-year follow-ups. There was no significant effect of the intervention on symptoms of anxiety or behavior at any point.

The authors concluded that the UKRP produced small, short-term effects on depressive symptoms and that

These findings suggest that interventions may produce reduced impacts when rolled out and taught by regular school staff.

In this blog post, I’m going to be arguing that

  • What the authors represent as weak findings may be even weaker than they portrayed.
  • There is nothing particularly new or positive psychology about the intervention package. It is a rehash of conventional (dare we say, bad old negative psychology?) treatment of depression applied to a student population in which the levels of depressive symptoms were low.
  • Under these circumstances, the intervention could not be expected to have an effect.
  • If we are truly committed to improving the well-being of students, we need to rethink the nature and focus of such interventions, and whether students should be required or coaxed to attend. As this intervention stands, it wastes staff and student time that could better be used for other ways of improving student well-being.

But first, some more details of the study:

Sample.

  • The 2,844 students were ages 11–12, 49% were female, 67% were white and they were drawn from 16 schools.
  • Students were not randomly assigned, but entire classes of students were arbitrarily enrolled in the intervention (UKRP) or control (usual school) conditions based on class timetables.
  • There were some baseline differences between the intervention and control groups and between schools. Some schools assigned students of above average academic achievement to the intervention groups, whereas other schools assigned students the intervention group because of concern about their emotional well-being or behavior.

Outcome measures.

Three standardized, normed self-report measures were used to evaluate the intervention:

Assessments were administered at baseline, immediately after the intervention, and at 1-year and 2-year follow-up.

The intervention package.resiliency

The article provides a web link to obtain more information about the intervention. When I went to the site, extensive information was requested that would be associated with me actually using the manual in a study. However, the description of the curriculum is available here.

The curriculum teaches cognitive-behavioral and social problem-solving skills and is based in part on cognitive-behavioral theories of depression by Aaron Beck, Albert Ellis, and Martin Seligman (Abramson, Seligman, & Teasdale, 1978; Beck, 1967, 1976; Ellis, 1962). Central to PRP is Ellis’ Adversity-Consequences-Beliefs (ABC) model, the notion that our beliefs about events mediate their impact on our emotions and behavior. Through this model, students learn to detect inaccurate thoughts, to evaluate the accuracy of those thoughts, and to challenge negative beliefs by considering alternative interpretations. PRP also teaches a variety of strategies that can be used for solving problems and coping with difficult situations and emotions. Students learn techniques for assertiveness, negotiation, decision-making, social problem-solving, and relaxation. The skills taught in the program can be applied to many contexts of life, including relationships with peers and family members as well as achievement in academics or other activities.

The control group.

The intervention received by the control group varied across the schools, but was generally Personal, Social and Health Education (PSHE) classes. In some of the schools, the control group was regular academic lessons.

Were effects of the intervention even weaker than presented?

Confirmation bias is common in presentation of results of test of interventions, especially when one of the developers of the intervention is among the authors or a consultant. To reduce the risk of bias, investigators are commonly required to preregister their design, including their plans for analysis of data. This commits investigators to a particular choice of outcomes and assessment points for evaluating the intervention. The alternative is that investigators can undertake a full range of analyses and report those that make the intervention looked strongest. This trial was apparently not preregistered.

Another check on risk of bias in reporting the results of a study are including all participants who were assigned to the intervention or control group in the primary analyses. The risk of not doing what is called an intent-to-treat analysis is a bias because selective retention on dropout of participants may affect results. In this particular study, results were quite weak and the appearance of significance could be influenced by even a small loss of participants from the analysis. If there is such a loss, a variety of techniques are available for adjusting.

Contrary to what the investigators say in the article analyses were not true intent to treat. Participants were excluded if they did not complete follow up assessments. Analyses indicate that students who came from special education classes or had initial high scores on depressive symptoms were less likely to complete subsequent assessments. The effect was bigger than the difference between intervention and control groups. No effort for compensating for loss of participants from follow up was reported. They were simply dropped.

For practical reasons, the study was not a true randomized trial, and the means of selecting participants resulted in differences in baseline characteristics. The investigators attempted to compensate these differences with statistical control. If there were any differences between the intervention and control groups, this could prove inadequate. Ideally, in such situations, investigators provide results without such corrections and then with them. If the two sets of results agree, it is more reassuring that apparent effects were not simply due to baseline differences between the intervention and control groups.

The article does not present simple differences in depressive symptoms, anxiety, and behavior problems at the end of the intervention. It is possible that already small differences between the intervention and control groups would disappear in a presentation of the simple analysis.

For their primary analysis, the investigators compared the intervention and control group and overall level of depressive symptoms. There were no significant differences. That would usually rule out continuing onto subgroup analyses examining the different time points. However, the investigators went on to look at depressive symptoms at each of the three post-assessment time points, and found a small difference at the first assessment that did not persist. This provided the basis for their bragging rights for having found a small, rather no effect, which is emphasized in their abstract and discussion.

Thus, by conventional standards, it could be concluded that  UKRP produced no significant effects, not merely small effects12359023-burst-balloons-white-background.

How is this intervention a positive psychology intervention?

In a Great Debate article, Howard Tennen and I complained  about proponents of positive psychology often drawing a false distinction between what is special about positive psychology versus the rest of conventional, “negative psychology” (Seligman, 2002).

Positive psychology articulates a role for hope, wisdom, courage, spirituality, responsibility, and perseverance in human adaptation in sharp contrast, proponents claim, to the negative biases of a conventional psychology that is too focused on distress and psychopathology to the exclusion of positive experiences.

Elsewhere in debates and on listserves and Facebook, I have argued that much is what effective about so-called positive psychology interventions is not new, and what is new about them is not effective.

This intervention is a warmed-over set of  “negative psychology”  interventions developed decades ago.

The UKRP intervention was carefully modeled after the Penn Resiliency Project and a key developer of the Penn project provided training and consultation and was the corresponding author for this article. Along with the Comprehensive Soldier Fitness Program, the Penn Resiliency intervention represents a premier positive psychology intervention package. But how does this intervention represent the distinctive ideas of positive psychology?

The article describes the intervention as

promoting resilience broadly and promoting adaptive thinking and coping.

Yet, key elements of the intervention come directly from Aaron T. Beck’s cognitive theory of depression and Albert Ellis’ Rational-Emotive Therapy (RET), or as Ellis later called it, his Adversity-Consequences-Beliefs (ABC) model. Both are conventional models of depression and its treatment that predate positive psychology by decades.

The primary outcome was a reduction in depressive symptoms, not any improvement in a characteristic positive psychology outcome, such as positive well-being or flourishing. As far as I can see, the only thing new about this intervention is that was taken out of its usual context of a treatment for  clinical depression and put into the schools where it was provided to all students, who happened, as a group, to be low in depressive symptoms. If any students actually showed high risk of clinical depression, they were evaluated and potential referred to conventional depression treatment.

So, does this important test of positive psychology in the schools merely examine whether conventional treatments for depression will produce lower levels of depressive symptoms subsequent to students receiving the intervention?

Why the intervention could not be expected to have an effect.

There was on average so little elevation in depressive symptoms, so the intervention could not be expected to have much of an effect. The investigators state:

  • At baseline, 60% of students in our sample scored 8 or below (average or below- average levels of symptoms), and 12% scored 0 or 1.
  • Only 6% scored above 19, indicating significant symptoms of depression.
  • Because of this, we encounter a strong floor effect: Students without many symptoms and with low risk of depression do not have much room for improvement.

For the time span covered by the intervention and the follow-up periods, depressive symptoms are relatively stable. Even students assigned to the control group are unlikely to face situations in which whatever is provided by the intervention would be of much use up to them, in terms of avoiding an increase in depressive symptoms.

First do no harm (Primum non nocere)

  • The study  required students to participate in a 16 hour intervention.
  • Most of the students who were present could not be expected to benefit from the intervention.

There is the possibility that post hoc (unplanned and after the fact) subgroup analyses would suggest that some subgroup had benefited. But given normative data suggesting that the intervention would be ineffectivewhy subject a large group of students to such intervention?

With only weak or probably no effects, the UK Resilience Programme cannot be presumed to be cost-effective. And in calculating the costs, we need to consider lost opportunities for the students enrolled in the program.

Arguably, students at risk for depressive symptoms would include those who had academic deficits which are readily identifiable. Why not devote the week and a half to remedying those deficits?

Is it ethical to require that students submit to a program that is unlikely to demonstrate benefits in the primary outcomes by which the program is evaluated?

The rollout continues…Roll_Out

As evidence of the practicality and sustainability of the intervention, there are now 85 schools teaching it in the United Kingdom, with over 800 teachers trained at 10 training courses. At least 250 of these teachers will have had their places funded entirely by the schools they work for, with the remainder being funded by some combination of school and LA [local area] funding. This demonstrates that schools and LAs are able and willing to provide the financial backing for the program.

 

 

Creative Commons License
This work, unless otherwise expressly stated, is licensed under a Creative Commons Attribution 3.0 Unported License.

This entry was posted in depression, positive psychology, Uncategorized and tagged , . Bookmark the permalink.

7 Responses to Positive psychology in the schools: the UK Resilience Project

  1. Pingback: Positive psychology in the schools: the UK resilience project – PLoS Blogs (blog) | Depression Therapy:What Suits You Best

  2. Jane Gillham says:

    This blog contains a number of serious inaccuracies and misunderstandings. Those interested in our peer reviewed work are welcome to contact me.

    Jane Gillham, Ph.D.
    Email: jgillha1 at swarthmore.edu

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Dr. Gillham, thanks for your interest in my blog post.
      Unfortunately, you did not preregister this clinical trial, which would have involved creating a public record of your precommitment to particular plans for analyzing your data and not others.
      In the absence of that public record, I think inferences on my part are quite fair about how you got from what could have been negative findings to at least weakly positive. You did, after all, report no overall results for depressive symptoms and yet went on to at which follow up point, significant results occurred. Usually no overall effects would preclude further analyses.
      Regardless, these findings could have been anticipated, and would not have justified such a large scale effort across the UK schools or the demands on student time.
      If you disagree, you can certainly point out where I am wrong.
      You fall back on the study being “peer-reviewed.” It did indeed received prepublication review in which an editor and maybe two or three reviewers agree to its publication. But that process is of course fallible. There are recent demonstrations of positive psychology studies getting through such peer review without serious problems being corrected. I note as examples the work of Barbara Fredrickson on positivity ratios in American Psychologist and her work on genomic expression in Proceedings of the National Academy of Science, both of which were so seriously flawed that retraction should be considered.
      Fortunately peer review is no longer limited to the hidden processes by which an editor and reviewers decide on the merits of a paper and their prepublication judgment is no longer final. PubMed Commons now allows continuous post publication peer review for the life of interest in an article.
      Readers who are not yet familiar can learn more at my blog post. All readers need to qualify is being among the authors on one of the over 23 million articles listed in PubMed. After registering, they can participate in the peer review of your paper and my comments with neither of us having the last word.
      I’ve taken the liberty of posting a link to my blog and a commentary at the PubMed listing for your article. I’ve also sent you an invitation to participate in whatever post publication peer review occurs there. I hope you will join in.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  3. donald klein says:

    A major problem is called reactivity of measurement. The therapy teaches you to avoid all words signifying dispirited wimp. The evaluations use this vocabulary”Do you????” Ans. “What me, (I know that is systematically discouraged)”.
    consonant with small effects ,decreasing in time.
    Poses an interesting methodological quandary.

    Don Klein

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • I very much agree. Particularly bright students can figure out the point of these lessons and tendencies to want to please the instructors can take over.

      VN:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
  4. Pingback: Meta-analysis in education: Some cautionary tales from other disciplines |

  5. Jessie Sun says:

    After familiarising myself with the school-based universal depression prevention literature recently I’ve come to pretty much the same conclusions. It’s a shame because universal prevention avoids the problems of stigma and labelling associated with targeted interventions – but it just doesn’t seem to work. How ironic that researching depression prevention could be so depressing.

    I think an even sadder story is the beyondblue Schools Research Initiative which you may be familiar with: http://www.ncbi.nlm.nih.gov/pubmed/15948913

    This was a large-scale (N = 5633 across 50 schools!!), 3-year intervention with 30 classroom sessions (problem solving, coping & social skills + optimistic cognitive style) (10 sessions/year) AND efforts to improve the school climate AND facilitating access to support and professional services. But across the 3 years of the program and 2-year follow-up, not a single positive finding on any of their measures (depressive symptoms, explanatory style, interpersonal competence, coping actions, perceived social support, school climate).

    To my mind, these two large-scale (and highly-powered) studies combined are perhaps the most informative about the practical utility of school-based CBT-type interventions. But I’m curious as to what you think about this – is it that universal prevention in general is an idea ready to be put to rest, or is it just that we haven’t yet come up with an intervention that is both efficacious and effective (e.g., that CBT is inappropriate)? What are your thoughts on depression prevention in general – and the role of psychoeducation in school settings?

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
Add Comment Register



Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>