I'm sending this to neighbors in Berkeley who are somewhat panicked by the article in question. What great reporting and analysis.
Also, spot on about placebo effect. Three years ago the good doctors of Beijing Hospital #3 gave me an injection to my left groin, which we all knew was almost certainly injury free -- but so real. (Spinal damage was real, but my groin was fine) The effect was amazing. Care from medical staff that are aware that our heads are in our body: don't leave home without it.
I believe we can say that a journalism degree alone does not prepare one to analyze the medical literature. But the study authors are also guilty of promoting poor science.
I really appreciate you bringing up, even momentarily, the mind-body dualism. I think this is the huge unstudied and ignored area of health sciences. I hope it's something we see a wide range of research efforts give attention to in the coming decades. After all, when I was growing up in the 1980s there was very little understanding of things like trauma and poverty and how the kinds of physiological effects they have throughout life. There is so much more now.
Thanks for this excellent post. I've been following the Long Covid discussions since the beginning and trying to find hard data and this has definitely been one of the most frustrating topics to get clarity on - especially on reasonable estimates of how common it is. The media narrative seems to often completely ignore it (focusing only on deaths and hospitalizations) or make it into a massive problem (10-30% of cases!).
Obviously we are in the middle of a pandemic and longterm effects are of lower priority to study compared to dealing with death and hospitalizations but I'd still have expected to have better data on at after a year. My sense is Long Covid is very much real but that the high estimates (e.g. 20% of people who get COVID have some form of _serious_ lingering effects) are vastly overblown and that the absence of evidence is at least partially evidence of absence. If those numbers were true we'd be seeing in lots of places, e.g. the medical system overrun with such cases and many more stories. But I'd have hoped that I could cite some real studies rather than having to mostly rely on logic like that.
I do know that several studies are being run (e.g.
In the meantime a lot of it is anecdotal or based on heavily biased samples, with very scary sounding numbers and framing. I ran across another example yesterday in the Atlantic:
"Today, informal estimates suggest that 10 to 30 percent of those infected with the novel coronavirus have long-term symptoms. “What people need to know is the pandemic’s toll is likely much higher than we are imagining,” Craig Spencer, the director of global health in emergency medicine at New York–Presbyterian/Columbia University Irving Medical Center, told me. “It is an area that merits urgent attention. There will be people living with the impact of COVID long after the pandemic is over. This is not made up or in the minds of people who are sickly. This is real.”"
Yeah, I don't think all this has been helpful. Clearly, there is a population but is it as high as a third? Much less? It seems terrible that we are not doing this the right way with proper, long-term studies. It's not serving the sufferers or anyone else.
Specious conclusions drawn from poor statistical analyses are rampant. Yet apparent correlations do spark real investigations.
So I take your point to be that publishing and broadcasting apparent correlations as if they were statistically significant conclusions is tantamount to sensationalism and fear-mongering.
So your complaint is purely methodological and rhetorical (once again), not truly analytic of the symptoms.
Part of my concern is that if we reduce Long Covid to this vague and this non-specific set of symptoms that don't even seem to be particular to people who had COVID (as it's totally plausible to me that a comparison group during this time could have a similar profile), we'll be both distracted from the research and help people do need, and just feed into the narrative that it's not real. In this particular study/coverage, I was not even totally clear what the correlation that deserves particular attention? (Non-representative groups can be revealing through internal correlations but I'm not seeing anything reported here that seems to inform the question).
Your third argument might even be stronger than you say. My understanding is that your third argument is actually a critique of the study, not the coverage of it. You quote the study:
**
Symptoms that are prevalent among long-haulers include (descending order): chest pain, dyspnea, anxiety, abdominal pain, cough, low back pain, and fatigue (See Figure 2B)
**
And, if I understand correctly, you argue that the study itself does not include adequate baselines or comparison groups.
But, here is where there is an additional criticism of the Times article. The Times article at a key point chooses to list only four of the symptoms:
**
More than 60 days after their infection, 27 percent, or 382 people, were struggling with post-Covid symptoms like shortness of breath, chest pain, cough or abdominal pain.
**
By not mentioning anxiety and fatigue, which seem like symptoms even more non-specific to COVID (or to any pathogen), the Times article may be accentuating the problem you are calling attention to even beyond the problem in the study itself.
Additionally, in the study’s abstract, the 27% figure appears to include a much longer list of symptoms, which contains not only anxiety and fatigue, but also headaches, palpitations, and insomnia. Again, your point about the lack of baseline and comparison group seems even stronger given that this broad constellation of symptoms is in play.
[disclaimer: I didn't read the original study in its entirety; and I don’t claim any special ability to read the med lit].
By the way, yours is the only individual newsletter I pay money for. In fact, for news and commentary, I pay for the NYT, the WaPo, The Guardian (by donation), and you. Thanks for consistently great analysis.
Thank you! The study itself identifies one of the clusters as an "anxiety-tachycardia" cluster. Again, I consider anxiety to be real and important but turning a cluster of symptoms of anxiety two months after, frankly, a scary diagnosis into "long Covid"—and declaring a third of this group to be from those who were asymptomatic—isn't going to help people with Long Covid. In fact, I'd been hoping that the clear existence of this issue after a pandemic would help people who are similar "long haulers" either from other pathogens/infections or from auto-immune or other regulatory disorders. But sensationalism is often followed by dismissal, so not good before or after.
This is a minor point, but if they're going to insist on these type of studies, I wish they'd at least prioritize a symptom like sudden loss of smell, which by most accounts is very highly correlated with having the virus, is pretty binary and easy to track (either you can smell or you can't), and is idiosyncratic enough that you wouldn't expect it to show up that much in a control group. Maybe it's even possible to extrapolate a crude prevalence estimate from preexisting data on total percentage of cases normally reporting loss of smell. (Or, COVID toes, or one of the other more prominent symptoms). This doesn't address the problem of self-selection of course, but it could offer a partial solution to the second problem of non-specific symptomatology without a baseline control.
I wonder if another very imperfect way of getting at prevalence could be to simply count up all the individuals who have joined any kind of Long COVID support or Facebook groups, and use this the same way death rate has been used to extrapolate total number of infections. I.e., multiply total unique cumulative members by our best estimate (presumably based on systematic group surveys) of the percentage of group members who actually identify as patients/survivors as opposed to being allies or curious followers, then further adjust this figure for the expected fraction of people not on social media etc., and divide that into total number of estimated infections (or confirmed cases) in the countries where those groups are most active. By this point in the pandemic these resources are developed enough that most people severely impacted by long-term sequella might be expected to at least hear about them. Obviously this would yield an underestimate and is even more self-selecting than the study you critiqued - but maybe for a better reason? At the very least it would insure we are targeting bona fide self-identified Long COVID patients, which seems more parsimonious than inferring backward from flawed hospital records and the like.
Do we even know for certain that everyone in the study had a primary care physician whose notes would be in the electronic medical records? Any chance that the study analyzed only the partial medical records that were available to them?
I don't know. The preprint is thin on the details. Maybe it will turn out to be a great study with all that! But it's just indicative of something to me that it's getting this kind of coverage without any of those details.
I am somewhat new to your work and am so appreciative of how you are trying to improve on the ways science is communicated (both by scientists and the general media). I have a few random comments:
1) I agree that the power of the placebo does get nearly enough attention, either through research or in the popular press. This seems like such a fundamental aspect of interpreting many medical observations, i.e. ways in which our expectations influence so much about the outcome. Maybe that can be one of your articles!
2) Are you aware of any studies that have tried to calculate the percentage of "long-Covid sufferers" who experienced moderate to significant Covid sxs first. It is a different type of comparison than what you point out but seems an important piece of the puzzle.
3) You likely discuss this in some of your other work, but I wish there was more attention media paid to how variables are operationalized in research. For instance, "severe Covid" seems to mean different things, depending on the source. Because the word "severe" sounds scary, it seems important to be consistent and to really understand what this means. It seems the medical definition focuses on changes to breathing (both rate and oxygen saturation) alone. How does this correlate then with other significant symptoms and likelihood of recovery? I wish all journalists included how the important variables were defined and measured as part of their "translation" of the studies.
it is my contention that patients who turn up at a doctor’s office complaining of vague subjective symptoms (pain, mental fog, weakness/fatigue) that are not also supported by objective evidence (objective symptoms/blood in stool/rash/etc, abnormal labs, abnormal imaging/EMG/etc) are common at any clinic let alone even a “post-covid clinic.” i should hope that no doctor in their right mind would make it their practice to dismiss any of their patient’s symptoms without at first looking for a piece of evidence that could support their patient’s claims.
to me this is the real conundrum within a concept like “long covid” is that it will invariably include in many cases those whose symptoms at the end of the day could not be backed up by any actual findings (other than say a positive Covid test) and nothing was known about what the state of their health was pre-Covid, and what changes, if any, they made towards their health post-Covid?
to your question how do you propose one even begin to understand the cohort that doesn’t as you say select itself into the study group?
I didn't read the NYT article when it first appeared; I must have used something like Zeynep's "authoritarian muscle memory" to ignore something that if true and then applied to 25 million US cases would result in heretofore unrecognized millions of Americans with debilitating illnesses.
This pandemic has made obvious how much we rely on well functioning institutions, and since the NYT is one of them, it is important to know what went wrong and how to encourage the NYT to fix it.
It is easy to see what went wrong. The reporter, Pam Belluck, was not critical of her own work, referring to other small, flawed studies with overstated conclusions to support her article without making an equal effort to find contrary opinions. Her editor failed, perhaps being unwilling to aggressively challenge a Pulitzer prize winning reporter.
Maybe there is no fix. Maybe reporters and editors, like the rest of us, learn best from our mistakes and the the most useful thing we can do is to point them out.
I have seen enough references in twitter so I sure the NYT is aware. My only concern is that in their defense, the NYT will point to the other low quality studies in their article and thereby make Zeynep (or her research assistant) comment on them as well.
I tried to very carefully word it to make it clear that I consider ME/CFS to be real (I have friends who are long sufferers) *and* that I find the way the medical science tries to dismiss it by claiming it's in people's heads to be terrible and dualistic. That's not at all claiming that it's not physiological, just that I find the way modern medicine tries to present those as unrelated binary to be unhelpful. I am not dismissing your interpretation but I did not write that ME/CFS was solely post-viral sequelae—but sometimes it is suspected to be that—and just that it is understudied and deserves attention, respect and study. Please feel free to email me if there was specific wording that you thought misrepresented it.
For me, the psychological/physiological discussion seemed like a tangent that was only lightly connected to the core topic. It's not surprising that some people would try to read into your post a more solid connection with the core topic.
I understand the sensitivity, so I hope the original commenter lets me know what I may be misstating. Always happy to correct or phrase better to avoid misunderstandings.
That said, I do really think the division/dualism does a disservice to medicine and health: there is no reason to think that those two categories would not interact: i.e. inflammation may play a role in depression and fatigue, and high-stress or other conditions we classify as psychological have huge effects on our health (for example, it might be one of the drivers of pernicious effects of systemic racism: the stress of it on people subjected to it).
Thank you for your measured and thoughtful reply, Zeynep.
I appreciate the compassion imbued within the language used. Your calling it “ME/CFS” instead of Chronic Fatigue Syndrome alone shows you are well ahead of the majority of the medical world.
This post, per usual, cuts through the clutter created by journalists’ serial misrepresentation of studies and redresses the resultant distress.
I'm sending this to neighbors in Berkeley who are somewhat panicked by the article in question. What great reporting and analysis.
Also, spot on about placebo effect. Three years ago the good doctors of Beijing Hospital #3 gave me an injection to my left groin, which we all knew was almost certainly injury free -- but so real. (Spinal damage was real, but my groin was fine) The effect was amazing. Care from medical staff that are aware that our heads are in our body: don't leave home without it.
I believe we can say that a journalism degree alone does not prepare one to analyze the medical literature. But the study authors are also guilty of promoting poor science.
I really appreciate you bringing up, even momentarily, the mind-body dualism. I think this is the huge unstudied and ignored area of health sciences. I hope it's something we see a wide range of research efforts give attention to in the coming decades. After all, when I was growing up in the 1980s there was very little understanding of things like trauma and poverty and how the kinds of physiological effects they have throughout life. There is so much more now.
You touch on comparative scales of causes of deaths for a particular cohort and I can’t resist sharing my favourite visualization for this https://pbs.twimg.com/media/EPpEiIMUcAAqwsy?format=jpg&name=medium
Causes on the top left tend to have smaller counts as young people die less. And yet they are highly impactful and we don’t want to miss them.
That's a very interesting visualization, thank you!
The visualization is generated by this cdc tool https://webappa.cdc.gov/sasweb/ncipc/leadcause.html
Thanks for this excellent post. I've been following the Long Covid discussions since the beginning and trying to find hard data and this has definitely been one of the most frustrating topics to get clarity on - especially on reasonable estimates of how common it is. The media narrative seems to often completely ignore it (focusing only on deaths and hospitalizations) or make it into a massive problem (10-30% of cases!).
Obviously we are in the middle of a pandemic and longterm effects are of lower priority to study compared to dealing with death and hospitalizations but I'd still have expected to have better data on at after a year. My sense is Long Covid is very much real but that the high estimates (e.g. 20% of people who get COVID have some form of _serious_ lingering effects) are vastly overblown and that the absence of evidence is at least partially evidence of absence. If those numbers were true we'd be seeing in lots of places, e.g. the medical system overrun with such cases and many more stories. But I'd have hoped that I could cite some real studies rather than having to mostly rely on logic like that.
I do know that several studies are being run (e.g.
https://www.ons.gov.uk/news/statementsandletters/theprevalenceoflongcovidsymptomsandcovid19complications) but I personally haven't seen anything that's actually far along and gives some reasonable estimates.
In the meantime a lot of it is anecdotal or based on heavily biased samples, with very scary sounding numbers and framing. I ran across another example yesterday in the Atlantic:
https://www.theatlantic.com/magazine/archive/2021/04/unlocking-the-mysteries-of-long-covid/618076/
"Today, informal estimates suggest that 10 to 30 percent of those infected with the novel coronavirus have long-term symptoms. “What people need to know is the pandemic’s toll is likely much higher than we are imagining,” Craig Spencer, the director of global health in emergency medicine at New York–Presbyterian/Columbia University Irving Medical Center, told me. “It is an area that merits urgent attention. There will be people living with the impact of COVID long after the pandemic is over. This is not made up or in the minds of people who are sickly. This is real.”"
Yeah, I don't think all this has been helpful. Clearly, there is a population but is it as high as a third? Much less? It seems terrible that we are not doing this the right way with proper, long-term studies. It's not serving the sufferers or anyone else.
Specious conclusions drawn from poor statistical analyses are rampant. Yet apparent correlations do spark real investigations.
So I take your point to be that publishing and broadcasting apparent correlations as if they were statistically significant conclusions is tantamount to sensationalism and fear-mongering.
So your complaint is purely methodological and rhetorical (once again), not truly analytic of the symptoms.
Part of my concern is that if we reduce Long Covid to this vague and this non-specific set of symptoms that don't even seem to be particular to people who had COVID (as it's totally plausible to me that a comparison group during this time could have a similar profile), we'll be both distracted from the research and help people do need, and just feed into the narrative that it's not real. In this particular study/coverage, I was not even totally clear what the correlation that deserves particular attention? (Non-representative groups can be revealing through internal correlations but I'm not seeing anything reported here that seems to inform the question).
The abduction fallacy. All men are mortal. Socrates is mortal. Therefore Socrates is a man.
It didn’t work in 300BCE, and it still doesn’t.
Indeed, if the sky were falling we'd expect to see a lot more pieces of the sky.
Hi,
Your third argument might even be stronger than you say. My understanding is that your third argument is actually a critique of the study, not the coverage of it. You quote the study:
**
Symptoms that are prevalent among long-haulers include (descending order): chest pain, dyspnea, anxiety, abdominal pain, cough, low back pain, and fatigue (See Figure 2B)
**
And, if I understand correctly, you argue that the study itself does not include adequate baselines or comparison groups.
But, here is where there is an additional criticism of the Times article. The Times article at a key point chooses to list only four of the symptoms:
**
More than 60 days after their infection, 27 percent, or 382 people, were struggling with post-Covid symptoms like shortness of breath, chest pain, cough or abdominal pain.
**
By not mentioning anxiety and fatigue, which seem like symptoms even more non-specific to COVID (or to any pathogen), the Times article may be accentuating the problem you are calling attention to even beyond the problem in the study itself.
Additionally, in the study’s abstract, the 27% figure appears to include a much longer list of symptoms, which contains not only anxiety and fatigue, but also headaches, palpitations, and insomnia. Again, your point about the lack of baseline and comparison group seems even stronger given that this broad constellation of symptoms is in play.
[disclaimer: I didn't read the original study in its entirety; and I don’t claim any special ability to read the med lit].
By the way, yours is the only individual newsletter I pay money for. In fact, for news and commentary, I pay for the NYT, the WaPo, The Guardian (by donation), and you. Thanks for consistently great analysis.
Thank you! The study itself identifies one of the clusters as an "anxiety-tachycardia" cluster. Again, I consider anxiety to be real and important but turning a cluster of symptoms of anxiety two months after, frankly, a scary diagnosis into "long Covid"—and declaring a third of this group to be from those who were asymptomatic—isn't going to help people with Long Covid. In fact, I'd been hoping that the clear existence of this issue after a pandemic would help people who are similar "long haulers" either from other pathogens/infections or from auto-immune or other regulatory disorders. But sensationalism is often followed by dismissal, so not good before or after.
This is a minor point, but if they're going to insist on these type of studies, I wish they'd at least prioritize a symptom like sudden loss of smell, which by most accounts is very highly correlated with having the virus, is pretty binary and easy to track (either you can smell or you can't), and is idiosyncratic enough that you wouldn't expect it to show up that much in a control group. Maybe it's even possible to extrapolate a crude prevalence estimate from preexisting data on total percentage of cases normally reporting loss of smell. (Or, COVID toes, or one of the other more prominent symptoms). This doesn't address the problem of self-selection of course, but it could offer a partial solution to the second problem of non-specific symptomatology without a baseline control.
I wonder if another very imperfect way of getting at prevalence could be to simply count up all the individuals who have joined any kind of Long COVID support or Facebook groups, and use this the same way death rate has been used to extrapolate total number of infections. I.e., multiply total unique cumulative members by our best estimate (presumably based on systematic group surveys) of the percentage of group members who actually identify as patients/survivors as opposed to being allies or curious followers, then further adjust this figure for the expected fraction of people not on social media etc., and divide that into total number of estimated infections (or confirmed cases) in the countries where those groups are most active. By this point in the pandemic these resources are developed enough that most people severely impacted by long-term sequella might be expected to at least hear about them. Obviously this would yield an underestimate and is even more self-selecting than the study you critiqued - but maybe for a better reason? At the very least it would insure we are targeting bona fide self-identified Long COVID patients, which seems more parsimonious than inferring backward from flawed hospital records and the like.
Do we even know for certain that everyone in the study had a primary care physician whose notes would be in the electronic medical records? Any chance that the study analyzed only the partial medical records that were available to them?
I don't know. The preprint is thin on the details. Maybe it will turn out to be a great study with all that! But it's just indicative of something to me that it's getting this kind of coverage without any of those details.
I am somewhat new to your work and am so appreciative of how you are trying to improve on the ways science is communicated (both by scientists and the general media). I have a few random comments:
1) I agree that the power of the placebo does get nearly enough attention, either through research or in the popular press. This seems like such a fundamental aspect of interpreting many medical observations, i.e. ways in which our expectations influence so much about the outcome. Maybe that can be one of your articles!
2) Are you aware of any studies that have tried to calculate the percentage of "long-Covid sufferers" who experienced moderate to significant Covid sxs first. It is a different type of comparison than what you point out but seems an important piece of the puzzle.
3) You likely discuss this in some of your other work, but I wish there was more attention media paid to how variables are operationalized in research. For instance, "severe Covid" seems to mean different things, depending on the source. Because the word "severe" sounds scary, it seems important to be consistent and to really understand what this means. It seems the medical definition focuses on changes to breathing (both rate and oxygen saturation) alone. How does this correlate then with other significant symptoms and likelihood of recovery? I wish all journalists included how the important variables were defined and measured as part of their "translation" of the studies.
it is my contention that patients who turn up at a doctor’s office complaining of vague subjective symptoms (pain, mental fog, weakness/fatigue) that are not also supported by objective evidence (objective symptoms/blood in stool/rash/etc, abnormal labs, abnormal imaging/EMG/etc) are common at any clinic let alone even a “post-covid clinic.” i should hope that no doctor in their right mind would make it their practice to dismiss any of their patient’s symptoms without at first looking for a piece of evidence that could support their patient’s claims.
to me this is the real conundrum within a concept like “long covid” is that it will invariably include in many cases those whose symptoms at the end of the day could not be backed up by any actual findings (other than say a positive Covid test) and nothing was known about what the state of their health was pre-Covid, and what changes, if any, they made towards their health post-Covid?
to your question how do you propose one even begin to understand the cohort that doesn’t as you say select itself into the study group?
I didn't read the NYT article when it first appeared; I must have used something like Zeynep's "authoritarian muscle memory" to ignore something that if true and then applied to 25 million US cases would result in heretofore unrecognized millions of Americans with debilitating illnesses.
This pandemic has made obvious how much we rely on well functioning institutions, and since the NYT is one of them, it is important to know what went wrong and how to encourage the NYT to fix it.
It is easy to see what went wrong. The reporter, Pam Belluck, was not critical of her own work, referring to other small, flawed studies with overstated conclusions to support her article without making an equal effort to find contrary opinions. Her editor failed, perhaps being unwilling to aggressively challenge a Pulitzer prize winning reporter.
Maybe there is no fix. Maybe reporters and editors, like the rest of us, learn best from our mistakes and the the most useful thing we can do is to point them out.
I sure hope you are pointing out these errors to Pam Belluck and her editor.
I have seen enough references in twitter so I sure the NYT is aware. My only concern is that in their defense, the NYT will point to the other low quality studies in their article and thereby make Zeynep (or her research assistant) comment on them as well.
I tried to very carefully word it to make it clear that I consider ME/CFS to be real (I have friends who are long sufferers) *and* that I find the way the medical science tries to dismiss it by claiming it's in people's heads to be terrible and dualistic. That's not at all claiming that it's not physiological, just that I find the way modern medicine tries to present those as unrelated binary to be unhelpful. I am not dismissing your interpretation but I did not write that ME/CFS was solely post-viral sequelae—but sometimes it is suspected to be that—and just that it is understudied and deserves attention, respect and study. Please feel free to email me if there was specific wording that you thought misrepresented it.
For me, the psychological/physiological discussion seemed like a tangent that was only lightly connected to the core topic. It's not surprising that some people would try to read into your post a more solid connection with the core topic.
I understand the sensitivity, so I hope the original commenter lets me know what I may be misstating. Always happy to correct or phrase better to avoid misunderstandings.
That said, I do really think the division/dualism does a disservice to medicine and health: there is no reason to think that those two categories would not interact: i.e. inflammation may play a role in depression and fatigue, and high-stress or other conditions we classify as psychological have huge effects on our health (for example, it might be one of the drivers of pernicious effects of systemic racism: the stress of it on people subjected to it).
Thank you for your measured and thoughtful reply, Zeynep.
I appreciate the compassion imbued within the language used. Your calling it “ME/CFS” instead of Chronic Fatigue Syndrome alone shows you are well ahead of the majority of the medical world.
This post, per usual, cuts through the clutter created by journalists’ serial misrepresentation of studies and redresses the resultant distress.