Less research is needed

ResearchBlogging.org

Guest blogger Trish Greenhalgh suggests its time for less research and more thinking.

The most over-used and under-analyzed statement in the academic vocabulary is surely “more research is needed”.  These four words, occasionally justified when they appear as the last sentence in a Masters dissertation, are as often to be found as the coda for a mega-trial that consumed the lion’s share of a national research budget, or that of a Cochrane review which began with dozens or even hundreds of primary studies and progressively excluded most of them on the grounds that they were “methodologically flawed”. Yet however large the trial or however comprehensive the review, the answer always seems to lie just around the next empirical corner.

With due respect to all those who have used “more research is needed” to sum up months or years of their own work on a topic, this ultimate academic cliché is usually an indicator that serious scholarly thinking on the topic has ceased. It is almost never the only logical conclusion that can be drawn from a set of negative, ambiguous, incomplete or contradictory data.

Recall the classic cartoon sketch from your childhood. Kitty-cat, who seeks to trap little bird Tweety Pie, tries to fly through the air.  After a pregnant mid-air pause reflecting the cartoon laws of physics, he falls to the ground and lies with eyes askew and stars circling round his silly head, to the evident amusement of his prey. But next frame, we see Kitty-cat launching himself into the air from an even greater height.  “More attempts at flight are needed”, he implicitly concludes.

Image Credit: flickr, breahn

On my first day in (laboratory) research, I was told that if there is a genuine and important phenomenon to be detected, it will become evident after taking no more than six readings from the instrument.  If after ten readings, my supervisor warned, your data have not reached statistical significance, you should [a] ask a different question; [b] design a radically different study; or [c] change the assumptions on which your hypothesis was based.

In health services research, we often seem to take the opposite view. We hold our assumptions to be self-evident. We consider our methodological hierarchy and quality criteria unassailable. And we define the research priorities of tomorrow by extrapolating uncritically from those of yesteryear.  Furthermore, this intellectual rigidity is formalized and ossified by research networks, funding bodies, publishers and the increasingly technocratic system of academic peer review.

Here is a quote from a typical genome-wide association study:

“Genome-wide association (GWA) studies on coronary artery disease (CAD) have been very successful, identifying a total of 32 susceptibility loci so far. Although these loci have provided valuable insights into the etiology of CAD, their cumulative effect explains surprisingly little of the total CAD heritability.”  [1]

The authors conclude that not only is more research needed into the genomic loci putatively linked to coronary artery disease, but that – precisely because the model they developed was so weak – further sets of variables (“genetic, epigenetic, transcriptomic, proteomic, metabolic and intermediate outcome variables”) should be added to it. By adding in more and more sets of variables, the authors suggest, we will progressively and substantially reduce the uncertainty about the multiple and complex gene-environment interactions that lead to coronary artery disease.

If the Kitty-cat analogy seems inappropriate to illustrate the flaws in this line of reasoning, let me offer another parallel. We predict tomorrow’s weather, more or less accurately, by measuring dynamic trends in today’s air temperature, wind speed, humidity, barometric pressure and a host of other meteorological variables. But when we try to predict what the weather will be next month, the accuracy of our prediction falls to little better than random. Perhaps we should spend huge sums of money on a more sophisticated weather-prediction model, incorporating the tides on the seas of Mars and the flutter of butterflies’ wings? Of course we shouldn’t. Not only would such a hyper-inclusive model fail to improve the accuracy of our predictive modeling, there are good statistical and operational reasons why it could well make it less accurate.

Whereas in the past, any observer could tell that an experiment had not ‘worked’, the knowledge generated by today’s multi-variable mega-studies remains opaque until months or years of analysis have rendered the findings – apparently at least – accessible and meaningful. This kind of research typically requires input from many vested interests: industry, policymakers, academic groupings and patient interest groups, all of whom have different reasons to invest hope in the outcome of the study. As Nic Brown has argued, debates around such complex and expensive research seem increasingly to be framed not by régimes of truth (what people know or claim to know) but by ‘régimes of hope’ (speculative predictions about what the world will be like once the desired knowledge is finally obtained). Lack of hard evidence to support the original hypothesis gets reframed as evidence that investment efforts need to be redoubled.[2] And so, instead of concluding that less research is needed, we collude with other interest groups to argue that tomorrow’s research investments should be pitched into precisely the same patch of long grass as yesterday’s.

Here are some intellectual fallacies based on the more-research-is-needed assumption (I am sure readers will use the comments box to add more examples).

  1. Despite dozens of randomized controlled trials of self-efficacy training (the ‘expert patient’ intervention) in chronic illness, most people (especially those with low socio-economic status and/or low health literacy) still do not self-manage their condition effectively. Therefore we need more randomized trials of self-efficacy training.
  2. Despite conflicting interpretations (based largely on the value attached to benefits versus those attached to harms) of the numerous large, population-wide breast cancer screening studies undertaken to date, we need more large, population-wide breast cancer screening studies.
  3. Despite the almost complete absence of ‘complex interventions’ for which a clinically as well as statistically significant effect size has been demonstrated and which have proved both transferable and affordable in the real world, the randomized controlled trial of the ‘complex intervention’ (as defined, for example, by the UK Medical Research Council [3]) should remain the gold standard when researching complex psychological, social and organizational influences on health outcomes.
  4. Despite consistent and repeated evidence that electronic patient record systems can be expensive, resource-hungry, failure-prone and unfit for purpose, we need more studies to ‘prove’ what we know to be the case: that replacing paper with technology will inevitably save money, improve health outcomes, assure safety and empower staff and patients.

Last year, Rodger Kessler and Russ Glasgow published a paper arguing for a ten-year moratorium on randomized controlled trials on the grounds that it was time to think smarter about the kind of research we need and the kind of study designs that are appropriate for different kinds of question.[4] I think we need to extend this moratorium substantially. For every paper that concludes “more research is needed”, funding for related studies should immediately cease until researchers can answer a question modeled on this one: “why should we continue to fund Kitty-cat’s attempts at flight”?

 This blog was informed by contributions to my Twitter page @trishgreenhalgh

Trish Greenhalgh is Professor of Primary Health Care at Barts and the London School of Medicine and Dentistry, London, UK, and also a general practitioner in north London.

[1] Prins BP, Lagou V, Asselbergs FW, Snieder H, & Fu J (2012). Genetics of coronary artery disease: Genome-wide association studies and beyond. Atherosclerosis PMID: 22698794

[2] Brown N (2007). Shifting Tenses: Reconnecting Regimes of Truth and Hope Configurations DOI: 10.1353/con.2007.0019

[3] Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M, & Medical Research Council Guidance (2008). Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ (Clinical research ed.), 337 PMID: 18824488

[4] Kessler R, & Glasgow RE (2011). A proposal to speed translation of healthcare research into practice: dramatic change is needed. American journal of preventive medicine, 40 (6), 637-44 PMID: 21565657

This entry was posted in General. Bookmark the permalink.

20 Responses to Less research is needed

  1. Boaz says:

    Hahaha, ““why should we continue to fund Kitty-cat’s attempts at flight”? …perhaps because it helps to feed the “prestigiousness” of the beasts who have plenty to profit from Kitty’s suicidal attempts? 

    VA:F [1.9.22_1171]
    Rating: +1 (from 1 vote)
  2. Andrew Miller says:

    Interesting post, thanks Trish. What is your view in same context of the rise of the ‘methodologically sound’ OA mega journals in STM and their value to scientific process?

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  3. Dave Curtis says:

    Quite right too. The really annoying studies are the ones which produce a clear result, a negative one, and which then still say that more research is needed. It’s like you have to keep researching the subject not until you get an answer but till you get an answer which you like the look of.

    You’re more than right about the genome wide association studies. In situations like this what is needed more is not more information but different information. Along with some pretty clear thinking about exactly what kind of phenomenena one is trying to understand and what the possible benefits and applications might be.

    You’re quite right about the electronic patient record too but I can’t bear to say any more on this subject because I’d only get upset.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  4. If we took Trish’ advice, it would put the USEPA and the IPCC out of business.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  5. Ryan Tracey says:

    But Trish, if we extended a moratorium to the “more research is needed” crutch, then researchers would become accountable for their work – and more importantly, their results.

    Less facetiously, I think a major driver of this phenomenon is that too many research studies *are* methodologically flawed. Consequently, more research *is* needed, and so the merry-go-round continues.

    I completely agree that more serious scholarly thinking is required, but instead of coinciding it with less research, I would suggest complementing it with better research.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  6. You say

    “Despite the almost complete absence of ‘complex interventions’ for which a clinically as well as statistically significant effect size has been demonstrated and which have proved both transferable and affordable in the real world, the randomized controlled trial of the ‘complex intervention’ (as defined, for example, by the UK Medical Research Council [3]) should remain the gold standard when researching complex psychological, social and organizational influences on health outcomes.”

    I think that this implies a non sequiter. The reason why RCTs of complex interventions often fail may well be that most complex interventions don’t work very well. That’s worth knowing, It is a reason for using RCTs, not abandoning them as you seem to suggest.

    That being said, I agree with much that you say. The field in which this tendency is worst is alternative medicine, Every time a well-designed trial shows they don’t work (the usual outcome), authors fail to draw the obvious conclusion -drop the treatment -and call for more research.

    There may even be some analogy. Most alternative therapies are desperately implausible. They are dreamt up almost at random, so it isn’t surprising that they mostly fail. Complex interventions mostly lack any clear rational basis too. That’s why they have to be properly tested with RCTs, and perhaps it is why many of them fail.

    We have to use RCTs to avoid fooling ourselves, and patients. The fact of the matter is that it isn’t easy to find treatments that work. That’s disappointing, but it’s better that we know it ans admit it.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Mark Hawker says:

      You say:

      I’m not convinced that that’s what’s being said here. (At least, that’s not what I think is being said.) If we go back to the original example of (laboratory) research the story was: [a] ask a different question; [b] design a radically different study; or [c] change the assumptions on which your hypothesis was based.

      How this would apply to health services research is, not to abandon RCTs as you seem to think is being implied (I appreciate that [a] and [b] may lend themselves to that interpretation), to do [c] first. (Reading through your comments it seems an equivocation is being made between “health services research” and “treatments”.)

      For example, there is an assumption in most (health services) research that involves technology that it is a “black box” that will be appropriated uncritically by those who it is “imposed upon”. Research (e.g. RCTs) that adopts this assumption will continue to show “failure” and that “more research is needed” unless this it is challenged. (No matter how much one can profess that the “random” nature of an RCT would “neutralise” this assumption they will fail to appreciate what is going on in the human-technology interaction.) Science and technology studies (e.g. the social shaping of technology) challenge the “black box” assumption in a big way. What an appreciation of the evidence from science and technology studies would do is contribute to [b] and then to [a].

      Here, a social science discipline (i.e. science and technology studies) could be used to support further research. This further research could well be an RCT. But, this further research could well include a social science element that aims to understand what is really going on on-the-ground e.g. “what are the experiences of people using this technology? What effect does the technology have on working practices? How do the perceptions of technology influence its adoption or rejection? These questions are different to those that would be answered by an RCT but are, arguably, just as important if not more so. Social science, then, could be used prior or during “other research”.

      I think the ordering of the original example of (laboratory) research is the wrong way up and that may have led to your interpretation. When you say that “[c]omplex interventions mostly lack any clear rational basis too” I think that what you may be highlighting is those who think up complex interventions fail to challenge common assumptions. That is what is being argued in the original post.

      NB: I have not engaged with your “alternative medicine” train of thought as I didn’t find it helpful in this discussion.

      VA:F [1.9.22_1171]
      Rating: 0 (from 0 votes)
      • Mark Hawker says:

        Comment above should read:

        You say:

        That’s worth knowing, It is a reason for using RCTs, not abandoning them as you seem to suggest.

        VA:F [1.9.22_1171]
        Rating: 0 (from 0 votes)
  7. While agreeing with much of what you say, this bit worries me.

    “Despite the almost complete absence of ‘complex interventions’ for which a clinically as well as statistically significant effect size has been demonstrated and which have proved both transferable and affordable in the real world, the randomized controlled trial of the ‘complex intervention’ (as defined, for example, by the UK Medical Research Council [3]) should remain the gold standard when researching complex psychological, social and organizational influences on health outcomes.” 

    It seems to suggest that, because RCTs have failed to show useful effects of many complex interventions, that is a reason not to use RCTs. Exactly the contrary is true. It is very useful to know that an intervention doesn’t work. The only proper way to test it is with an RCT, as pointed out so eloquently in the recent Cabinet Office paper. It’s "thinking" that dreams up the hypotheses, and most of them turn out to be wrong.
    I agree, of course, that once well-designed trials have shown that an intervention doesn’t have worthwhile benefit, than the idea, however good it sounded on paper, should be dropped. There’s a parallel with alternative medicine. That too fails to show benefit in almost every well-designed trial. Yet rather than drawing the obvoius conclusion that it should be dropped, the universal conclusion is that "more research is needed". Vested interests and vanity seem to make it hard to accept that your pet idea didn’t work, both for sociologists and quacks. Sociologists, at least, generate their hypotheses by thinking, but often they don’t work. That’s why they must be tested properly.
    Thinking is to generate ideas, but ideas have to tested. That’s the research bit.

    VA:F [1.9.22_1171]
    Rating: +1 (from 1 vote)
  8. Luke says:

    There is an important comment left out from this post; namely that the approach to statistical significance and methodology is somewhat different in lab-based work than in clinical research. For instance, using Fairly homogenous engineered animals or cell lines and looking for physiological activity that is fairly uniform means that one might be satisfied with finding significance after 6 observations. In clinical research, however, this would be called a case study and may be a “fluke”. I realise that this is a simplistic explanation but the point of difference is important. It does start to explain why we have and need so many RCTs and systematic analyses and reviews of reviews.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  9. David Flint says:

    You raise real concerns but I’d pick one area where we do need more research – that’s in the performance of the research funding system. Since there are many funders operating many sets of criteria it ought to be possible to start by comparing outcomes.

    (How’s that as a way to offend almost everyone?)

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  10. I apologise for dual posting, The first got lost somewhere, so I re-wrote it (rather less well).. Then the original rematerialised.

    I must admit that I’m having difficulty in understanding what point Mark Hawker is trying to make. I was talking about how we know whether or not any sort of intervention is effective. It has been explained very clearly in the Cabinet Office paper that the only way to test tsocial interventions (and drugs) properly is with RCTs.

    It would help the discussion if Mark Hawker could explain which part of that paper he disagrees with.

    It’s obvious that Hawker doesn’t like my alternative medicine analogy. but once again it is relevant. Hawker sa\ys

    "But, this further research could well include a social science element that aims to understand what is really going on on-the-ground"

    This is exactly analagous to acupuncturists and homeopaths who love to speculate about how their magic works before they have established that there is any phenomenon to explain. Usually the phenomenon isn’t there, so there is nothing to explain.

    The first thing one has to do is to show (with RCTs) that the intervention has a useful effect. If it doesn’t that should be the end of the story, though, as Greenhalgh points out, it often isn’t.  If it does work then by all means go on to speculate about mechanisms. But such speculation isn’t really necessary from the point of view of patients. In the, perhaps simpler, case of testing a new drug the same process happens. First you show (or fail to show) a useful effect, then you can speculate about how the effect might come about, though such speculations often turn out to be wrong.

    It seems to be that social sciences will never be taken seriously until they learn how to test their ideas properly. Until that time, they don’t deserve the ‘science’ part of theor description.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
    • Mark Hawker says:

      David,

      It has been explained very clearly in the Cabinet Office paper that the only way to test tsocial interventions (and drugs) properly is with RCTs. It would help the discussion if Mark Hawker could explain which part of that paper he disagrees with.

      I’m not convinced this approach would answer the “why?” question. It would do fantastic at answering the “what?” question of a social intervention. What is missing from the paper is a discussion of how one would explore an ‘ineffective’ social intervention. No matter how well you can answer the “what?” question you wouldn’t get very far at answering the “why?” question. For example, there are numerous reasons why assistive technologies are not used e.g. personal, device-related, environment-related and intervention-related factors. I can’t see how an RCT would have surfaced that information. All it would have done is highlighted that a certain percentage were not used but how would that be useful for future assistive technology?

      It’s obvious that Hawker doesn’t like my alternative medicine analogy. but once again it is relevant.

      That is because it is biological and not social. A social intervention would include the biological and the social.

      This is exactly analagous to acupuncturists and homeopaths who love to speculate about how their magic works before they have established that there is any phenomenon to explain. Usually the phenomenon isn’t there, so there is nothing to explain.

      No, no it’s not. I am in total agreement with you that that may in fact happen in the alternative medicine world. But, again, that is a biological intervention that requires, as you rightly say, biological evidence to back it up. And, as you say, an RCT has been shown to be the gold standard for that. So, in fact, I have no issue at all with your alternative medicine analogy it just misses the other half of a more complex story for social interventions. Social interventions, particularly those that involve technology, are not like pills.

      The first thing one has to do is to show (with RCTs) that the intervention has a useful effect.

      And if it doesn’t?

      If it doesn’t that should be the end of the story, though, as Greenhalgh points out, it often isn’t.

      I disagree with that interpretation of the original post. If it doesn’t, then one would want to explore “why?”, surely? You seem to want to only explore ‘successful’ interventions. I think, for social interventions, that is a mistake and where your thinking about biological interventions is only part of the whole story.

      In the, perhaps simpler, case of testing a new drug the same process happens. First you show (or fail to show) a useful effect, then you can speculate about how the effect might come about, though such speculations often turn out to be wrong.

      And that is great, for biological interventions. That’s your area, I presume, and I agree with you!

      It seems to be that social sciences will never be taken seriously until they learn how to test their ideas properly. Until that time, they don’t deserve the ‘science’ part of theor description.

      I’m not sure how this is helpful? A social science would explore “why?” a social intervention was successful or unsuccessful and then that could well be re-tested using an RCT. It is not something dreamt up in an ivory tower, it is grounded very much in the data made available within an RCT. Data that is ‘missed’ by focusing exclusively on effectiveness.

      VA:F [1.9.22_1171]
      Rating: +1 (from 1 vote)
  11. Richard Smith says:

    Are you implying, Trish, from your comment on electronic records that we should stay with paper records or that we shouldn’t do research on electronic records? I couldn’t imagine most enterprises–for example, airlines–doing RCTs on electronic systems, failing to show their benefit, and then deciding to stick with paper systems. They just changed. Similarly almost all journals adopted electronic forms without RCT evidence of benefit.
    As I once wrote, we don’t have RCT evidence of the benefit of professors of primary care. Does that mean we shouldn’t have them? Perhaps it does.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  12. Karen Bloor says:

    I agree with David Colquhoun on complex interventions – if RCTs show no effect this doesn’t mean RCTs are the wrong approach.

    But economics can help with your broader question – when is ‘more research needed’ and when do we need to stop? Value of information analysis explores exactly that. See, for example, ‘When is evidence sufficient’ in Health Affairs at http://content.healthaffairs.org/content/24/1/93.full

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  13. The corollary to this excellent post is that we should abandon treatments that continue to fail to show clear benefit in RCTs. For instance: PSA screening tests, platelet rich plasma, spinal fusion for uncomplicated low back pain, vertebroplasty, etc, etc, etc. If the treatment doesn’t have a clear, unambiguous benefit, there’s a good chance it only causes harm.

    Why don’t we do so? I think because both physicians and patients are often so anti-scientific they simply cannot believe that something that seems like a good idea is, in fact, not. For amusement’s sake, see Dr. David Newman riff on brain natriuretic peptide and the tendency of his emergency department residents to use tests without understanding the evidence. He calls it “sciencey-ness,” the attribute of ideas in medicine that are so good, so scientific-seeming, they have just got to be true. http://avoidablecare.org/sciencey-ness-by-dr-david-newman/

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  14. Jim Woodgett says:

    An uncomfortable corollary to this excellent essay is that the scientific machine has developed a self-perpetuating system of research demand. When a third or so of published papers never attract a single citation, is that an indicator that we have reached a surplus state, whereby the need to publish becomes the end and not the means? As we plead our case to governments for more money for research, should we not be sure what is being given is being used effectively?

    There is no doubt that our ignorance still massively exceeds our knowledge but that, in itself, does not justify *more* research. It needs to be the best research with better questions.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  15. Pingback: Midlife is not too late to start exercising and reaping the benefits | Dr Sarb Johal – Complete Coach

  16. David Flint says:

    There are cases where we need research to determine why something didn’t work. The obvious cases are where it’s based on solid science or where it’s worked elsewhere, or where something similar has worked on similar problems in the past. That’s not an argument for continuing to use treatments that don’t work in anything other than a research context.

    VA:F [1.9.22_1171]
    Rating: 0 (from 0 votes)
  17. Pingback: It’s time for less research and more thinking @drsilenzi @trishgreenhalgh @muirgray « Carlo Favaretti