Battles over vaccine evidence sure can get toxic, can’t they? If you’re a diehard partisan, though, you can look for standard bearers on your side of the barricades, and watch them demolish any studies or arguments that even slightly complicate your side’s narrative.
It’s harder, I think, for the rest of us. The volume of data and bias is supersized!
This vaccine was always going to be a tough call for clinical trials, too. HPV infection causes most cervical cancer, but the cancer is uncommon and mostly takes a decade or two to develop. Which means only very large-scale, decades-long, placebo-controlled trials could unequivocally settle the question – if the researchers could keep track of enough of the women over that period of time. Mission: Impossible!
As if all that’s not already hard enough, once vaccination was shown to reduce HPV infection and the lesions that can be “precancer”, having placebo groups in later studies becomes a likely non-starter for ethics committees.
That ability to detect possible “precancer”, years or even decades before the actual disease, made these vaccine trials feasible. But using these surrogate outcomes for cervical cancer also set up one of the core controversies about the evidence. More on that soon.
This graphic maps out the complexity of HPV infection and cervical cancer. Just because you get the infection, doesn’t mean it won’t clear up and cause no problem. It can persist though, and cause lesions – CIN (cervical intraepithelial neoplasia) or AIS (adenocarcinoma in situ) – collections of abnormal cells called precancerous, pre-invasive, or cancer precursors. But those lesions mostly heal, too, without invading deeper to become actual cervical cancer.
It was always going to be complicated to track the vaccine’s effects via clinical trials – but it’s far worse without them! The goalposts are shifting because of HPV testing. That could improve detection of cervical cancer and its precursors, which could seem like the disease is increasing. [Update: I dig into this in a follow-up post.]
All that complexity is a recipe for perpetual conflict over evidence. Which brings us to the recent firestorm we’re looking at in this post.
It all started with a Cochrane systematic review/meta-analysis in May that concluded the vaccine was, essentially, effective and safe for women (Arbyn 2018). In July, a scathing critique was published by a group of meta-analysts, under the banner of the Nordic Cochrane Centre in Copenhagen (Jørgensen 2018). Skeptical Raptor dubbed them “an antivaccine thought center”, and called their critique “without merit”. On 9 August, the BMJ published a report (Hawkes 2018) with this alarming headline:
HPV vaccine safety: Cochrane launches urgent investigation into review after criticisms.
So what is this storm about, and where does all this leave the evidence for the HPV vaccine?
The Cochrane review undertook the basic steps you need to see in a systematic review. (See for example my 5 step checkup.) The Copenhagen critique focused mostly on how well some of this was done, some methodological choices the reviewers made, and the vaccine trials themselves. Let’s go through their 7 charges.
1. “The Cochrane review missed nearly half of the trials”.
The Cochrane reviewers published a protocol – a detailed research plan – in 2011, and it was revised and republished in 2013. Their search strategy was completed in June 2017. They ended up finding 26 trials that met their pre-specified criteria, including 73,428 female participants.
Meanwhile, the Copenhagen group were working on their own systematic review of HPV vaccines, based on a method they advocate: working with clinical study reports and registry entries rather than searching journals for articles. They published a protocol in January 2017, with a cut-off date for finding studies of 1 July 2017.
They published their list of HPV vaccine trials in January 2018, including ones with male and female participants, completed ones, follow-up studies, ones still in progress, and so on. In their critique of Cochrane’s review, they write:
When we applied the Cochrane review’s inclusion criteria to the 206 studies, we identified 46 completed and eligible trials. The number of randomised participants could be assessed for 42 of the 46 trials and was 121,704.
However, they do not provide a list of these 46 trials in the critique, so this pivotal claim isn’t verifiable. And that’s critical. For example, they do single out a 2015 trial of 9-valent vaccine as one of the “missing” trials they deemed eligible. The 9-valent (or nona-valent) vaccine is aimed at 9 strains of HPV, and the 2015 trial tested it in young women who had already had a full course of the quadrivalent (4-strain) vaccine.
I disagree that this trial was eligible – which puts me in disagreement with both the Cochrane and Copenhagen groups! The Cochrane inclusion criteria stated that the intervention has to be a monovalent, bivalent, or quadrivalent vaccine – so protecting against 1 particular strain, 2 or 4, but not 9. The Cochrane authors, though, wrote this about the 9-valent vaccine trial:
The randomised trial was not included in our review since it compared the nona-valent with the quadrivalent vaccine…
The efficacy and safety of the nona-valent vaccine will be assessed in a future update of this Cochrane review, when results of more trials are available. This update will include also inter-vaccine comparisons without a placebo arm.
The Copenhagen group wrote that this decision was inconsistent with including a trial comparing bivalent and quadrivalent vaccine, which the Cochrane reviewers did. I agree.
Now back to those other 41 to 45 trials said to be eligible.
The list of 206 studies the Copenhagen group published in their article doesn’t tabulate the information you need to assess them yourself, and starting from scratch is a colossal amount of work. You would have to chase down the links or other information they provide for all 206 studies, get copies of whatever’s available, and then dig out the data for every single un-included study that would enable you to check them against the lengthy eligibility criteria.
The BMJ interviewed David Tovey, Cochrane’s editor-in-chief, about this:
The Cochrane team had received a copy of the index, he said, but late in the review process. “In their judgement at the time it did not appear to identify any important eligible studies. Our current investigations appear to show that there may be a handful of missed but potentially eligible studies, but that this falls substantially below ‘nearly half of the eligible trials’.
He promised a fuller response later. “To date, we also have no reason to believe that the main conclusions of the review relating to benefit and serious adverse effects are unsafe.”
What if we just take the Copenhagen finding at face value, though? We don’t know if their list has all the 26 trials in the Cochrane review: the Copenhagen group didn’t search the literature. Assuming they do, though, then Cochrane is missing 20 out of 46 trials (43%), and 20 out of 42 trials with enough data to know how many participants they had (48%).
We don’t know if the 121,704 participants from those 42 trials are all female. We don’t know if there is usable comparable data available for all of them, but the chances of that are vanishingly small.
For example, a trial didn’t have to be an effectiveness trial to be eligible. For the measures of vaccine effects, although there were 26 trials in the Cochrane review, there were only 1 to 8 trials for any measure like “precancerous” lesions. And a couple of those early trials with these clinical measures were huge. When you meta-analyze data, bigger trials carry more weight, and it’s hard to shift the needle against them. (See my backgrounder on understanding data in meta-analyses.)
Here’s some fairly typical data to illustrate the position. Take the primary outcome for the review of “precancerous” lesions (CIN2 or more). Let’s look at the results for women aged 15 to 26 years who had at least 1 dose of vaccine, regardless of what strain of HPV caused the lesion (Analysis 3.7). Only 4 trials had this data: and 2 of them accounted for 97% of the participants (34,562 women out of 35,779).
The Copenhagen group points out that there could be additional outcome data outside the journal publications, which is an important point. But they haven’t revealed any substantial extra outcome data as yet.
Even if the Cochrane review were missing relevant data for 40% of the participants, by the way, rounding that up to “half” as the Copenhagen critique did, is spin.
2. “No included trial in the Cochrane review used a placebo comparator”.
Placebo control groups can be important for various reasons. For example, getting an injection regardless of which group you’ve been assigned to can prevent bias creeping in from participants knowing they’ve had the vaccine. (See my clinical trial critique 101 post to see why that’s important.)
That’s partly the sense in which the Cochrane reviewers mean placebo-controlled. In their eligibility criteria, they define placebos as injections that are inactive for HPV vaccine (bolded to highlight):
Administration of placebo containing no active product or only the adjuvant of the HPV vaccine, without L1 VLP, or another non-HPV vaccine.
An adjuvant is an ingredient added to vaccines (more on that here) and VLP means virus-like particle: L1 VLP is HPV-vaccine related.
The Cochrane reviewers stuck to that definition throughout the review, having defined it right up front. The Copenhagen group, however, took the Cochrane reviewers to task for how they defined placebo:
The Cochrane authors mistakenly used the term placebo to describe the active comparators. They acknowledged that ‘The comparison of the risks of adverse events was compromised by the use of different products (adjuvants and hepatitis vaccines) administered to participants in the control group’. Nevertheless, this statement can easily be overlooked, as it comes after 7500 words about other issues in the discussion…
Why did the Copenhagen group highlight this as a major problem? After all, they say there is only one totally adjuvant- and HPV-free placebo-controlled trial, and that’s the 9-valent one we’ve already talked about.
Placebos are also a strategy for stringently evaluating adverse effects, and that’s the crux of the Copenhagen group’s concern here.
It worried the Cochrane reviewers, too. Their statement about compromised comparators continued,
…the pooled risks of adverse effects associated with HPV vaccines and the assumed risks for control groups must be interpreted cautiously.
The Copenhagen group suggest that the adjuvants (15/26 trials) and the hepatitis vaccines (9/26 trials) used as comparators have adverse effects of their own:
The use of active comparators probably increased the occurrence of harms in the comparator groups and thereby masked harms caused by the HPV vaccines.
Adjuvants are common in vaccines, including the HPV vaccine – and fears about them are a common target for criticizing vaccines, too. Arguing that these ingredients could be responsible for serious adverse effects is a staple of anti-vaccine fear-mongering, so there has been a lot of response on this. The CDC has an explainer about adjuvants, which are regulated and monitored by the FDA.
3. “The included HPV vaccine trials used composite surrogate outcomes for cervical cancer”.
If you read my posts on evaluating outcomes in trials, you’ll know both composite and surrogate outcomes are red flags for me (see an overview post here).
So you might expect me to be all in with the Copenhagen group on this one. But I’m not. They do point out that this is standard for these vaccines, and
The use of such outcomes seemed reasonable for a preliminary assessment of HPV vaccine benefits, but the outcomes can be difficult to interpret. If there were clinically important differences in the severity of the cervical lesions in the two compared groups, they may not have been apparent in the composite outcomes of CIN2+ and CIN3+.
I think the “composite” issue of having CIN2 and worse, and CIN3 and worse, is helpful. I say “composite” in quotes, though, because they aren’t always divided up: historically, CIN2+ has been a single outcome. And although some now advocate managing CIN2 and CIN3 differently, treatment will generally be recommended to you for either diagnosis (see the US Preventive Services Task Force and National Cancer Institute).
That’s why I don’t see CIN2+ as only a surrogate outcome for cervical cancer: it’s a condition that if diagnosed, could cause most women anxiety (or even panic) and invasive treatment that’s painful and comes with its own risks.
The conceptualization of bodily boundaries appeared to change, e.g. through visualization of the previously unfamiliar cervix, pain, vaginal discharge, and bleeding…
And it’s such a lot of women. Back in 2004, Kaiser Permanente extrapolated from their population to the US as a whole, based on about 40 million women having Pap smears in a year. This is before HPV testing could bump up diagnoses, and before HPV vaccination could potentially prevent some lesions. Their conclusion, in a year there would be:
- 2 million abnormal smears, of which half would be false positives;
- 175,000 CIN1 diagnoses;
- 225,000 CIN2+ diagnoses;
- 600,000 women where incomplete follow-up leaves a question mark.
Preventing cervical lesions could reduce a lot of suffering – even without taking into account impact on cervical cancer. In 2014, over 12,500 women were diagnosed with cervical cancer in the US, and the death rate is high: there were over 4,000 deaths from cervical cancer. For countries where late diagnosis and lack of treatment is a common reality, an affordable, effective vaccine could be even more valuable.
The Copenhagen group criticized the reviewers for not addressing cervical cancer in detail. The Cochrane reviewers’ reason was that the trials weren’t powered to detect differences in cervical cancer and the numbers were too low for meaningful analysis. (They concluded that the trials don’t show a reduction.)
4. “The Cochrane review incompletely assessed serious and systemic adverse events”.
This one isn’t a reasonable complaint: assessment of anything is always going to be incomplete, and the Cochrane reviewers point out, rightly, that you can’t resolve uncommon safety issues based solely on trials. In fact, I don’t think the Copenhagen group identifies any serious flaw in this section of their critique.
The Cochrane reviewers are criticized for concluding that results for serious adverse events (SAEs) are “similar”, whether you look at trial registry entries or the published articles. (SAEs aren’t necessarily vaccine-related, by the way: for example, for the specific trial we’ll look at shortly, the SAEs included gunshot wounds.)
This is how the Cochrane reviewers came to their conclusion. Their primary analysis was tallying up the number of women for each trial with reported SAEs from registry sources, and analyzing the combined total.
Then they did a sensitivity analysis for the same data, but this time using what was reported in peer-reviewed journal articles for the trials, instead of what was in the registries. The difference in the total pooled results for each source were not statistically significant.
Here’s what the totals were:
|Source||Women in HPV vaccine groups||Women in control groups|
The numbers aren’t the same, but the overall conclusion (no big difference) is the same, whichever source you use. Fair enough. So what’s the Copenhagen issue with this? Bear with me, here, as it’s worthwhile to see what’s going on.
Here’s what the Copenhagen critique says – with references removed, to make it a bit easier to decipher:
The Cochrane authors reported that they made a ‘Particular effort’ to assess serious adverse events and performed a sensitivity analysis that gave them ‘confidence that published and registry or website-sourced data are similar for the same study’. This seems unlikely. As an example, the PATRICIA trial publication only included two thirds (1400/2028) of the serious adverse events listed on ClinicalTrials.gov. The Cochrane authors included 701 vs 699 serious adverse events (1400) from the PATRICIA trial publication and 835 vs 829 serious adverse events from its ClinicalTrials.gov entry. We found 1046 vs 982 serious adverse events (2028) when we summarised the data from ClinicalTrials.gov.
Sounds really bad, doesn’t it? But it only starts to make sense when you slog through all the data.
The Copenhagen critique is talking about numbers of SAEs. But the Cochrane review, for both its comparisons, as well as the PATRICIA publication, used the women with reported SAEs as the unit of measure – and one woman could obviously have more than one SAE over a period of years. Here’s the original data on numbers of women from ClinicalTrials.gov, with a green arrow pointing this out.
ClinicalTrials.gov also lists the number of events here. I didn’t try to count them up, but I’m assuming that’s where the Copenhagen group got 2,028:
We found 1046 vs 982 serious adverse events (2028) when we summarised the data from ClinicalTrials.gov.
Comparing the number of women to the number of events is just wrong.
The Copenhagen group goes on to criticize the Cochrane reviewers for saying the risk of serious adverse events was similar for HPV-vaccine groups versus controls, when some trials didn’t report events for the whole time period of the trial. I don’t think this was specified for each trial, but prominently early on in the summary of findings, the reviewers state that follow-up for serious adverse events was “6 months to 7 years”. Still, this lack of detail is a small issue compared with the comparators being active – which already led the Cochrane reviewers to conclude adverse effects data need to be interpreted cautiously.
The Copenhagen group criticized the lack of analysis of individual adverse events. But that’s not necessary, given the Cochrane reviewers clearly point out the trials aren’t a good basis for studying these questions – a point on which all agree.
The Copenhagen group then criticize the handling of deaths. There was an increased number of deaths in women over 25 in the vaccinated groups, but none were deemed to be vaccine-related. The Copenhagen group say because there is only data from randomized trials, this conclusion can’t be reached, and, they write,
A death may be coded in a way that does not raise suspicion that the vaccine caused it; for example, a ‘traumatic head injury’ or ‘drowning’ could have been caused by a ‘syncope’, which is a recognised harm.
They cite the product labels of vaccines for this as a recognized harm. But this is what those say (here’s the first one as an example):
Because vaccinees may develop syncope, sometimes resulting in falling with injury, observation for 15 minutes after administration is recommended.
Here’s the CDC on syncope (fainting): it refers to an immediate reaction on getting an injection, and it’s not specific to the HPV vaccine. So it would be related in time to the vaccination, and thus more likely to be scrutinized and considered potentially vaccine-related.
The next criticism is that an analysis of systemic events did not include the events from the PATRICIA trial, for which data are available at ClinicalTrials.gov. I didn’t spend much time on this, but it would be good to know the answer. The group goes on to criticize the reviewers for not doing more to get unpublished data: the Cochrane review reports this was because of lack of resources, and would be done in the future.
Finally, the critique disagrees with trials having been rated as at low risk of bias. That’s par for the course: people often disagree on these ratings.
(Data on adverse effects can be tricky: see tips from me in this post.)
5. “The Cochrane review did not assess HPV vaccine-related safety signals”.
It’s obviously the balance between potential benefits and potential harms that determine whether any drug is “a good guy”, “a bad guy”, or something in between.
But a systematic review of randomized trials can’t answer questions about longterm and rare adverse events, if the trials weren’t designed to answer them. And everyone acknowledges that this was beyond the scope of the trials in this Cochrane review.
The Copenhagen group make it clear they are convinced the HPV vaccine causes harm. The Cochrane reviewers cited WHO and EMA (European Medicines Agency) conclusions about vaccine safety, but the Copenhagen group take issue with these, citing some subsequently published data.
This is outside the scope of the Cochrane review and this post. [Update 26 August] These issues were covered, though, last year by Michael Head and colleagues, when the Nordic Cochrane Centre made similar arguments to the EMA:
However, as the EMA highlight in their detailed rebuttal, the authors ignored the limitations of the cited case reports and introduced basic errors such as accusing ‘the wrong Julie Williams’ of undeclared conflicts of interest.
These arguments about safety are the focus of Skeptical Raptor’s post describing the Copenhagen group as Cochrane’s “anti-vaxxers”, which is a good lead-in to their next critique.
6. “Industry trial funding and other conflicts of interest”.
According to the Cochrane review, all the trials except 1 were funded by the vaccine’s manufacturer. The Copenhagen group point out that this one exception is listed on ClinicalTrials.gov as sponsored by a manufacturer – and indeed, it is (see here). It looks like it’s the ClinicalTrials.gov entry that’s wrong, not the Cochrane review. Here’s what the trial’s publication says, about its NIH National Cancer Institute (NCI) funding:
The Costa Rican Vaccine Trial is a longstanding collaboration between investigators in Costa Rica and NCI. The trial is sponsored and funded by NCI (N01-CP11005) with support from the NIH Office of Research on Women’s Health and conducted in agreement with the Ministry of Health of Costa Rica. The NCI and Costa Rica investigators are responsible for the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation of the manuscript. Vaccine was provided for our trial by GSK Biologicals, under a Clinical Trials Agreement with NCI. GSK also provided support for aspects of the trial associated with regulatory submission needs of the company under FDA BB-IND 7920.
More seriously, they point to industry conflicts of interests among the authors of the first protocol for this review in 2011. Here is what the Cochrane review says:
Authors of this review were assessed by the Cochrane Funding Arbiter Committee after Cochrane received correspondence and feedback on the published protocol. Current authors were approved by this committee based on stringent Cochrane conflict of interest guidelines. A unrestricted grant was provided by Sanofi-Pasteur-MSD to the University of Ghent who co-ordinated the SEHIB study (Surveillance of Effects of HPV Immunisation in Belgium). The grant was given in the framework of the EMA (European Medicine Agency) request to set up post-marketing surveillance of HPV vaccination effects in non-Nordic member states of the European Union. The Sciensano (employer of MA and LX, former name “Scientific Institute of Public Health”) collaborated with the University of Ghent to conduct the SEHIB study.
I’ve written a reader’s guide to authors’ conflicts of interest, and I went back to it for this post. What’s striking here to me is not so much the issue of financial conflicts of interest, but competitive, intellectual, and ideological ones. (I’ve written a personal disclosure of my own interests at the foot of this post.)
7. “Cochrane’s public relations of the review were uncritical”.
This is about Cochrane posting this summary of the UK Science Media Centre’s reaction to the review. Part of the criticism is that only UK experts were invited, but that’s not odd for a UK organization! Internal critiques of this belong with the Science Media Centre, not Cochrane.
That’s it for the Copenhagen critique. (There’s another, if you’re interested, submitted as a comment on the Cochrane review here. [Update] I posted a comment addressing it – a copy of the text is posted here.)
Where do I end up after all this? First, on the evidence: the HPV vaccine looks effective enough at preventing cervical lesions that it could reduce an enormous amount of suffering. And that’s wonderful.
Secondly, the critique: hmmm! When I first glanced at it, I was shocked and disappointed that Cochrane had seemed to mess up so badly on such an important review. But that flipped the more I dug into it, and it was the critique that increasingly shocked me.
Now that I’ve finished? I think it’s a hatchet job by people with several axes to grind, that needed better editorial peer review. I would be surprised if Cochrane’s investigation and update resulted in changed conclusions. But I look forward to reading it.
CODA: Cochrane’s investigative report was published on 3 September (in the UK). They found only 5 eligible trials (including 5,267), of which only 1 included the primary outcome, CIN2+, and it wouldn’t change the main conclusion. Furthermore.
…adding the studies that were missed by limiting the search to published study reports had no impact on the direction of effect for all outcomes reported…
We have analysed the publicly available data from the missing studies and we believe that including them would make no material difference to the Cochrane Review’s results and conclusions….
There is already a formidable and growing anti-vaccination lobby. If the result of this controversy is reduced uptake of the vaccine among young women, this has the potential to lead to women suffering and dying unnecessarily from cervical cancer.
So no outcome flipped. And here’s the before-after primary outcome:
Cochrane’s next steps: update the review – and try to find out from the Copenhagen critique group what 10s of thousands of missing eligible trial participants they were referring to. I’ll keep you posted!
(I started a page here on my website to keep track.)
I did look! Check out my follow-up post:
[Update 4 September 2018] Coda on Cochrane’s response to the Copenhagen critique.
[Update 3 September 2018] I added a link to a copy of the comment I submitted to the Cochrane journal.
[Update 31 August 2018] The post originally criticized the fact that the Cochrane review wasn’t open access. But Cochrane made it free to view, so I deleted the criticism.
[Update 26 August 2018] I added some text about an article by Michael Head and colleagues after the lead author drew my attention to it on Twitter – thanks, Michael!
Disclosures: I led the development of a fact sheet and evaluation of evidence on HPV vaccine for consumers in 2009 for Germany’s national evidence agency, the Institute for Quality and Efficiency in Healthcare (IQWiG), where I was the head of the health information department. We based our advice on this 2007 systematic review including 6 trials with 40,323 women, and an assessment of those trials. The findings were similar to those of the 2018 Cochrane review. I have no financial or other professional conflicts of interest in relation to the HPV vaccine. My personal interest in understanding the evidence about the HPV vaccine is as a grandmother (of a boy and a girl).
I am one of the members of the founding group of the Cochrane Collaboration and was the coordinating editor of a Cochrane review group for 7 years, and coordinator of its Consumer Network for many years. I am no longer a member, although I occasionally contribute peer review on methods. I often butt heads with the Cochrane Collaboration (most recently as a co-signatory to this letter in the BMJ). I have butted heads on the subject of bias with authors of the Copenhagen critique.
26 September: In light of developments, some further disclosures which I did not realize may relate to this controversy and the theories around it, when I first started writing about the Cochrane HPV vaccine review. I was invited to speak at Evidence Live, and my participation was supported by the organizers, a partnership between the BMJ and the Centre for Evidence-Based Medicine (CEBM) at the University of Oxford’s Nuffield Department of Primary Care Health Sciences – the director of the CEBM is the editor of BMJ EBM. Between 2011 and 2018, I worked on PubMed projects at the National Center of Biotechnology Information (NCBI), which is part of the US National Institutes of Health. I am currently working towards a PhD on some factors affecting the validity of systematic reviews.