Always Read the Methods Section
Always Read the Methods Section
How a preprint paper with seemingly good news and interesting findings fueled panic worldwide instead
It’s been an eventful week in pandemic land. The pause in the J&J vaccine is the most dramatic development. There’s a lot to say on the societal aspects of the pause, as it relates to a core issue that I’ve been exploring all last year: decision-making under widespread risk and uncertainty.
As one piece of this puzzle, I want to talk about something that doesn’t always get the attention it deserves: the secondary and global wave of panic and sensationalism that can follow what happens in the United States and Europe, especially the panic in media around the world.
Take this headline from Turkey, which says that the “South African variant could ‘overcome’ BioNTech”:
I saw this because it was retweeted into my timeline, from someone with a lot of followers panicking about it., After looking a bit, I saw that there were multiple versions of the same story:
The second version in Turkish is even more misleading, claiming that “a single dose does not offer protection” and that both doses do not offer “full protection.” Both stories are framed as being originally reported by Reuters
In fact, I had already been seeing these stories in English-language press as well. A few examples of the framing that was widespread:
And here’s Reuters on Twitter:
Their headline is actually worse, if that were possible:
By now, you are probably guessing where this is going, but let’s walk through it anyway.
For background, vaccines mostly have what’s called “breakthrough” infections: people who get infected or sick despite vaccination. Some of the plausible mechanisms for this are weak immune systems (something that happens especially in the elderly), waning of immunity over time (that’s why it’s recommended that adults repeat, for example, tetanus vaccines every ten years), or, obviously of great interest right now, because the pathogen mutated and was able to circumvent some immune system defenses to some level. But as discussed before (and hopefully more in those forthcoming posts by virologists and other experts I keep promising—they are genuinely understandably really busy but the posts are in progress!), the immune system is not a seawall that, once breached, is gone. Even when a breakthrough does happen and cause symptoms, the case might be much milder, not progressing to severe disease as other parts of the immune system are called in.
For further background, there’s been widespread coverage of the potential for “vaccine evasion” from the B.1.351 variant—the so-called South African one. It is, of course, an important topic. In one trial, the AstraZeneca vaccine showed poor ability to prevent breakthrough mild/moderate severe cases. But the trial was small (with very wide confidence intervals so hard to be definitive about the exact effect size) and did not look into severe cases (which, as explained in this past post, are harder to study with sufficient statistical power because they are rare). Since then, there have been a reasonable number of studies showing drops in neutralizing antibodies against this variant compared with the original (wildtype). But there have also been some articles with encouraging results showing that the T cell response—which, unlike the initial neutralizing antibodies comes into play after infection has started, and thus does not prevent mild/moderate symptoms but can help stop progression into severe illness—was still there against all the major new variants, including B.1.351.
After all this, we finally get to the point of the study causing the wave of panic around the world, which is fairly straightforward—something you might have missed from the coverage.
The study is a neat one, and something we should definitely do more of: researchers looked at a large health records database and found a group of people who had tested positive and were partially or fully vaccinated [meaning at least two weeks out from their first dose, though some were further along including two weeks from their second dose] (Note: I clarified this and next paragraph, and added the table below for clarity.)
The researchers then retrospectively matched this with another group from the same sample called “controls”—people who had similar characteristics who could be used as comparisons but who had not been vaccinated. You can see that there is 149 people with “FE” vaccination (meaning both doses) and 247 with “PE” (meaning at least two weeks after first dose, but before the second dose). Equal numbers are selected as controls. (The method section has a lot of details on why this was a challenge! But we can take these as controls for the purposes of this post).
The researchers then looked at these “breakthrough” cases among the vaccinated and compared them with the cases among the unvaccinated controls (also matched as FE and PE, in equal numbers) in terms of which variants had caused the infection. They found that, in the first two weeks following the second dose, there were 8 cases of infection from B.1.351 compared with one from the UK variant B.1.1.7, which is already dominant in Israel at the time of study, while “those infected between two weeks after the first dose and one week after the second dose, were disproportionately infected by B.1.1.7 (odds ratio of 26:10).” There were no infections two weeks after the second dose.
There is a lot more going on with this study, which is fine—it’s a preprint. For example, what was the breakthrough rate in the whole sample? I couldn’t find it calculated explicitly in the study, but if they selected all the breakthrough cases, the total would be 396 in a sample of 4.7 million patients in the database they reference, some (unknown) portion were vaccinated which would be the denominator for that 396. The samples were collected between January 23, 2021 (paper has a typo and says 2020 but the figures make it clear it is 2021) to March 07 2021, during which Israel vaccinated around 50 percent of its citizens. The real calculation would depend on the exact sampling times, but it looks like that 396 would likely be a remarkably low rate of breakthroughs, especially considering the “breakthrough” cases included people just two weeks after first their dose.
Also, one obvious question is if there were clusters among the breakthrough cases (i.e. one family getting infected, naturally, with the same variant), which would complicate how we calculate the statistics of this. When there are such clusters of things that are related to one another (therefore not independent draws from a sample, as assumed by some statistical methods), we use what’s called “multilevel models” (actually these are called something different in almost every discipline but the idea is the same: if people are in the same classroom or family etc., they may be representing a shared environment as well). It’s one thing if there are eight independent infections of one variant versus, say, two clusters with eight people in them.
These are all interesting questions, and I really look forward to reading the full paper.
But you know what the paper does not say, imply, or hint at? There is nothing in the paper to suggest the “South African variant may evade protection from the Pfizer vaccine,” as Reuters or many other news organizations claim, beyond what we already know: there will be a few breakthroughs even with the mRNA vaccines, because that’s just reality with almost all vaccines, and it’s possible more of those breakthroughs will be the South African variant because of its effect on neutralizing antibodies.
The key here is reading the methods section. The researchers selected the positive cases first. They then worked backwards to divide them into the vaccinated and not, and then try to do a comparison among the already-broken-through vaccinated cases with the regular, unvaccinated cases which they “matched” using participant characteristics.
A study that could show if this variant was indeed evading immunity from the Pfizer vaccine would, necessarily, work the other way. It would be a trial design where they immunize some people, but not others, ideally in a double-blind setting, and then see if the vaccine protected from the new variant (the way they did with the AstraZeneca study). That study would work forwards.
Further, as we have been discussing a lot in this newsletter, what researchers mean with “evade” when they use that term is not “leave without any protection,” which is what all the headlines imply and what caused the panic. Full vaccine escape, where vaccinated individuals show no difference from people for whom this virus is completely novel, would certainly be alarming, and necessitate vaccinating everyone, again, with a new vaccine. But that doesn’t even happen fully with influenza, a terrible disease against which we don’t have the kind of excellent vaccines we have against COVID-19.
Plus, as the researcher of the paper herself even tried to explain in a Twitter thread, there was a remarkable enough finding even in this sample: there were no B.1.351 cases at all two weeks after the second dose.
This could well be because this variant isn’t that competitive to begin with, or that the mRNA vaccines are completely protective after both doses, or a combination of the two. If I had to choose a headline from that, though, it would be just that—because it is, in fact, really really good news overall: the breakthrough cases were very few, even if they potentially skewed towards one particular variant as we expected (though the details of this depends on further information about their sampling methods which we will hopefully read in the full paper). And even that ceased completely after two weeks.
So we have a draft paper that hopefully will flesh out into a longer paper with a lot more details to bring clarity to the unanswered parts., But the preprint looked to me like really good news, especially the part about no cases of P.1.351 two weeks after second dose. How did that turn into something that has people in Turkey—and no doubt elsewhere—panicked?
It’s especially baffling since here’s the context for what’s going on in Turkey, where, from what I can tell, the dreaded B.1.1.7 is either dominant or on its way to becoming dominant and plus, only about 10 percent of the population is vaccinated (mostly with Sinovac but some with Pfizer/BioNTech).
This is the cases:
Tragically, deaths are at an all time high as well, even with vaccination having started among the elderly:
This is, sadly, the case around the world in multiple countries (Brazil, India, Turkey, Iran, multiple countries in Europe) where cases are skyrocketing and there just aren’t enough vaccines. We should, indeed, panic and despair about the tragedy of so much death and suffering that will continue if we cannot step up our response, rather than panic over papers reported incorrectly or with wildly misleading headlines.
Here’s a good rule. The first question, always, should be: did the reporter read and understand the methods section? If the answer is not yes, maybe it’s not necessary to write that piece or that headline, yet.
This was a helpful example for showing how understanding methods and carefully reading the actual study are so important and can prevent misinterpretation of results. Venting just a little here, but it has been very frustrating discussing post-vaccination behavior with roommates who are heavily engaged with a lot of news sources that have been heavily emphasizing the potential threat of variants. It has been very difficult to acknowledge that there are real concerns there, but that our vaccinations (particularly the mRNA) have shown little reason to panic to this point or fret that the pandemic will never end. And also there seems to be a poor understanding as you've been saying that a small number of breakthrough cases aren't necessarily cause for alarm either.
I'm not offering analysis here, just expressing some exasperation... and also saying thank you!
I think we need a medical/scientific equivalent of "snopes" to cover scientific papers, which journalists would consult before printing their stories. Over and over again it's clear that journalists are not up to the task of understanding these papers. Perhaps they could come together to fund such an organization which they could all make use of.