Vaccines and Decision-Making with Imperfect Data

It's trade-offs all the way down

The peer-reviewed paper outlining data from the Pfizer/BioNTech vaccine was just released, as at type, the FDA is holding a meeting which will almost certainly provide emergency authorization for this vaccine. As the editorial accompanying the paper said, the results are “a triumph.” It’s only December, and we are about to launch mass vaccination with ~95% efficacy in preventing disease—likely even higher for preventing severe disease. Remember that as late as April of this year, experts (including Dr. Fauci) thought 18 months to two years was too optimistic a timeline for a vaccine, and the FDA was ready to approve a vaccine with only 50% efficacy.

Instead, what we have is a vaccine ready at half the time the most optimistic timeline projected, with twice the efficacy hoped for. It’s remarkable.

There’s a lot more to be said, but here’s something that jumped at me while looking at the paper at The New England Journal of Medicine, as well as the files submitted to the FDA. Look at that chart below, comparing detected disease between those in the vaccine and placebo group. 

Let’s first note: this is an amazing chart. I have a feeling that I’ll be able to remember that chart for a long time—the beginning of the end of the pandemic. Even better, this data seems very similar to Moderna vaccine data, which is similarly an mRNA vaccine—in effect a replication.

But there’s a very important question lurking in this chart, one that goes straight to the heart of decision-making under uncertain conditions involving significant trade-offs. As can be seen, the incidence of Covid-19 drops off dramatically about 10 days after the first dose (look at the chart on the top left). However, the trial is designed to measure the efficacy of two doses: a prime and a booster. Eyeballing (we don’t have the underlying data), the first shot seems to have 65-80% efficacy in the full group—but we don’t know for how long that lasts. Hence, we have durability of immunity data—for three more months, to day 119—only with the booster shot.

Given the shortages, the potential for single dosing—at least for some populations—or more time between the doses is an important question. It’s possible that a single dose—one that can cover twice the number of people—would provide a significant benefit to the recipients, though we would be unsure about whether the immunity protections last as strongly three months later. This is no minor question. The United States is already planning to withhold vaccination from people unless a second shot has already been secured. If we get fifty million doses now, that’s 25 million people who could be vaccinated now who won’t be even though the doses are sitting in storage.

What should the policymakers do? This seems like a scientific question, but it is not pure science because there are trade-offs on all sides. The question is confronting the dilemmas of not vaccinating millions of people versus vaccinating a smaller group with higher efficacy; and risking a second wave if immunity wears off quickly in an unexpected way. What about trust in vaccines? Can we even do anything that veers, even slightly, from the protocols, let alone something as major as booster timing? Is 60-80% efficacy vaccination for 200 million is better than 95% for 100 million? Given how terrible the next few months look to be, how do we prioritize what we do first? Can we space the booster a bit while we wait for longer-term data? We can model some of what this means depending on who’s vaccinated and whether people adhere to distancing or quarantines after vaccination, but we can’t easily decide which is better without bringing in ethical preferences and thorny questions. 

However, at a minimum, we should continue testing as we vaccinate. 

Initial, earlier data suggests that people over 65 may be in most need of the booster shot for these vaccines, and we know that the death rate for this disease increases exponentially with age, making it especially risky to do a trial for a single dose among those most at risk. There’s also immunological reasons why there are boosters for many vaccines. That said, facing shortages, we’ve previously tested and found fractional (much less than standard formulation) doses to be sufficient: for example, in 2016, WHO approved fractional doses for yellow fever. (Full report is here).

In the United States we will also be vaccinating health-care workers in the first round, and data shows that this group is no longer at as high risk in the United States due to much better PPE use and infection control protocols. We could, for example, ask for volunteers from the many millions of people in that group and randomize them as one or two-doses. If we start today, in just three months we’d have as much data as we have now for the two dose schedule—at least for those under 65. We could also launch an immediate trial for one dose—in fact, we probably should, including for the Moderna vaccine, which similarly has remarkable efficacy but is also similarly administered in two doses. We could also try having a booster, say, three months later or six months later for subgroups of volunteers: being able to vaccinate larger numbers early in 2021 would be of significant benefit to the world, but we also don’t want to undermine confidence among the public, given the existing situation.

The trials for Pfizer/BioNTech upon which FDA is about to issue an emergency use authorization have been conducted only on 40,000 people. Many millions will be in the first round of people to receive a vaccine. We need only test on again on tens of thousands to give us a better idea of durability and other questions raised by this data. It’s quite possible that just organizing the trials and calling for volunteers potentially can get us a reasonable answer. 

All this feeds into a bigger problem: we should have been testing a lot more—randomizing and comparing—from day one. We got one of the most important therapeutics, dexamethasone, because of the RECOVERY trial in the United Kingdom where, early on, when doctors around the world were throwing whatever they could find at the virus, they did the right thing: they randomized and followed-up, to see what worked and what did not. We could also consider trials comparing vaccines.

We can also make better use of whatever natural experiments emerge. The Pfizer/BioNTech trial includes about ~1250 people who did not get the second dose in the vaccine arm, with a similar number in the placebo group. Similarly, there will be people who chose not to get the second dose, or who will miss one. These will not be random groups—people who miss second doses are almost certainly different from the group as a total—but it does give us something to look at. It’s a small group, so the lack of infections may not be as comforting, but a huge spike in that group compared to the group that got the booster would help stop us in our tracks. (The ability of positive and negative incidents ability to inform us is asymmetric in this case).

We also should have been bolder in discussing trade-offs and acting quickly even with imperfect data. There’s no avoiding trade-offs in a pandemic. Having millions of people without vaccinations because of shortages is a terrible outcome, too, and one that deserves a consideration as we discuss the alternatives, and how to move forward. There’s no avoiding making decisions with imperfect data, and there is no way to avoid real human costs with shortages, but the least we can do is to add as much clarity as we can to what data we have, and can have in the future.

Meanwhile, we owe a lot of gratitude to every one of those volunteers who offered up an arm. Our challenges are now at the stage where we’re discussing how to vaccinate the most people as quickly as possible. It’s definitely the beginning of the end. Without the volunteers, we wouldn’t be here.