Systematic Reviews & Meta-Analyses: A 5-Step Checkup

 

It’s easy to be a little blinded by the specialized statistical techniques in systematic reviews and meta-analyses. As with any type of study, though, there are bad ones that can lead you down a garden path.

So let’s look at some simple – or at least less-than-hellishly-complicated! – ways to spot reviews that are worth taking more seriously than others.

 

 

You don’t start with the statistics. No amount of fancy statistical legwork can save a study that’s biased by its design. That’s true of individual studies, and it’s true of meta-studies that analyze groups of them, too.

A bonus of this phenomenon is that even without knowing a lot about methods of analysis, you can whittle down the number of reviews and meta-analyses that could cause you problems.

But first, some shortcuts that I don’t think are a reliable help.

Many argue that there is a “brand” that should give you confidence. I don’t agree (even though I participated in establishing some of those brands).

And don’t believe it must be good just because it says it was “systematic”, or something like “followed PRISMA (or Cochrane) methods/principles”. That alone is no guarantee that the quality is higher. (See for example this study on quality and mentioning PRISMA).

A last proviso before we start: a perfectly well-done study can land on the wrong answer. And a perfectly crappy study can nonetheless “be right”. Still, we really need time-savers, and steering clear of worse studies is a way to cope with an information glut.

Let’s get to my 5-step check-up. Most systematic reviews and meta-analyses won’t pass all 5 of them.

 

1. Are there clear, pre-specified, eligibility criteria for studies being chosen or rejected for the review?

 

Ideally, the review makes it quick and easy for you to answer this one, with a really clear list of inclusion and exclusion points – or at least a clear description of the grounds for accepting or rejecting a study. The criteria might be extremely broad, or very specific.

 

 

The easiest way to be sure details like this were pre-specified is if there was a fully worked out study protocol before the reviewing steps began. That might have been published in a journal, or in a registry like the one specifically for systematic review protocols, PROSPERO. Or they may mention that they had one. Sometimes, the language is the best guide you have to go on.

If there are very clear criteria, but I can’t be sure if they were pre-specified, I tend to give them the benefit of the doubt at this point, but it’s a point for caution.

While it’s a good sign, having a protocol is not enough. You can totally rig a protocol to get you where you want to go. And as with anything else, just because you said you would do things, doesn’t mean you did them – or did them well.

 

2. Did they make a strong effort to find all the studies which could have been eligible?

 

 

Once you have a clear question, and a clear idea of what studies are eligible, then you need a good search strategy to find them.

There should be a good search strategy. That’s an organized plan of what databases and other sources are being searched, and with what terms. It might be in an appendix. A rule of thumb is it should search, at the very least, in more than one place. The less detailed and organized this is, the less confident we can be that this review has really captured the relevant research.

 

3. Were the searches done within the last 5 years?

 

 

This one is a rule of thumb, too. In some areas, even a year ago might be too late, if the field is moving fast. If the only review is an old one, then it might still be useful. There’s a pretty high chance that there’s been a game-changer since, though.

Tip: the date of the review’s publication and the date of the search could be very, very different – the contents of a review can even be years older than the article in which they are published.

 

4. Can you see a list of the studies that were excluded from the review?

 

It really helps if there is a flowchart, something like this one:

 

 

(This flowchart one comes from a systematic review I participated in.)

But while the flowchart is good, looking to find a list of the rejected studies is a good shortcut for assessing the transparency and thoroughness of a review.

 

5. Have they given you some indication of how good they think the studies they included are?

 

 

I’ve written more about this over at Statistically Funny (and here, too, in looking at meta-analyses in psychology). There can be quality problems that affect the whole of a study – for example, whether the researchers assessing outcomes in a clinical trial knew what comparison group a patient was in. And there can be quality problems with a specific question or measurement within studies, too.

You need, though, to know how much weight can be put on the studies in a review or meta-analysis. If you can’t do that, then you really have to go back and look at all the studies yourself, pretty much.

There are many different ways that people could have done this, depending on the kind of review and study types involved. For example, they could assess quality, risk of bias, or particular hurdles a study has to clear. The quality criteria are sometimes also entry criteria for the review. In a case like that, just being included already tells you something.

What’s more, the review’s conclusions should take the quality of the evidence into consideration. This isn’t just a numbers game.

Want to go into this more deeply? The most commonly used method for formally assessing the quality of a systematic review in health care is AMSTAR. Another is ROBIS. Julian Higgins and colleagues developed a checklist for assessing the quality of a meta-analysis. The GRADE Working Group have a range of methods for assessing quality of evidence, including one for assessing syntheses of qualitative research. And Sally Thorne has proposed some interesting questions for qualitative meta-syntheses, too.

Once you’ve gotten as far as a meta-analysis you want to make sense of, check out my tips for understanding data in meta-analyses.

 

Happy sifting!

 

~~~~

The cartoons are my own (CC BY-NC-ND license). (More cartoons at Statistically Funny and on Tumblr.)

 

* The thoughts Hilda Bastian expresses here at Absolutely Maybe are personal.