99.5% of doctors can’t WHAT?
A recent study found that 99.5% of medical specialists can't calculate a basic probability equation. That's not good news for patients.
‘Of the many “how did this happen???” questions that swirl around the manufactured COVID crisis, one of the most troubling has been, “How did doctors fall for all of this?”
We’ve come to expect politicians to lie to us, in order to serve the interests of their real constituents (who plainly do not include you and me). And Yes Minister taught us that we should expect “public servants” to lie to us, either by commission or omission, because it ain’t the public that they’re serving:
But doctors? Aren’t we supposed to be able to trust them to make decisions in the best interests of their patients? And aren’t they supposed to be highly educated men and women of science, deeply versed in the scientific method, and well-equipped to dissect studies and spot their flaws?
My previous article, Your doctor is not your doctor, answered the first question. As I wrote in that article:
“The line between public health and the practise of medicine – a line that has become increasingly blurry over the past decades, as Dr Mukesh Haikerwal tacitly acknowledged when endorsing the Declaration of Geneva as a reaffirmation of the role of doctors as independent professionals who owed a duty of care to their patients – has now been completely erased, and the manufactured COVID-19 crisis was the implement used to erase it.
Now, when you go to see the person whom you think of as ‘your doctor’, there’s an extra party in the consultation room: the state, which has inserted itself into the doctor-patient relationship without your consent.
In fact, it’s fair to say that the forced incorporation of health practitioners into the apparatus of public health has for all intents and purposes destroyed the doctor-patient relationship.”
An article recently published in JAMA Network Open provides at least a partial answer to the second question: over 99 per cent of doctors can’t do basic probability calculations required to assess the likelihood of a particular diagnosis, or the likely success of a treatment, when two or more independent events are involved.
The study, ‘Analysis of Physicians’ Probability Estimates of a Medical Outcome Based on a Sequence of Events' surveyed 215 specialists in the fields of either obstetrics and gynaecology or pulmonology, and asked them to estimate the probability of success of each of two components of a diagnostic or prognostic sequence, as well as the overall probability of success of the two-step sequence1.
The conjunction fallacy
The authors were interested in finding out whether the conjunction fallacy – that is, the misestimation of the overall probability of success when two or more independent events are involved – is a likely source of the errors that doctors make in both diagnosis and prognosis. As they explain:
“The probability of a conjunction of 2 independent events is the product of the probabilities of the 2 components and therefore cannot exceed the probability of either component. A violation of this basic law of probability is called the conjunction fallacy.”
Analysis of Physicians’ Probability Estimates of a Medical Outcome Based on a Sequence of Events
To illustrate with an example that is highly relevant to me right now, if I wanted to calculate the probability that I will be sitting down to a big bowl of lovely ripe mulberries from my mulberry tree tomorrow, I would need to estimate a) the probability that enough of the currently almost-ripe mulberries will be fully-ripe by tomorrow to warrant a mulberry-picking excursion and b) the probability that the birds don’t get them first, and then multiply the two figures by each other.
As anyone with even the most tenuous grasp of mathematics will quickly realise, the overall probability of me getting to enjoy my mulberries tomorrow cannot exceed the probability of either of the independent factors. For example, assuming a 60 per cent (0.6) probability of rapid-action ripening, even if I deploy enough bird-netting, Mylar ribbons and propane-powered cannons (yes, they’re a thing) to deter all but the most bad-ass birds, thus preserving 90 per cent (0.9) of the berries, there still cannot be a greater than 60 per cent probability of me feasting on ripe mulberries tomorrow.
In fact, the overall probability of me getting a bowl full of ripe mulberries tomorrow is 0.9 x 0.6 = 0.54, or 54 per cent.
OK, back to the study.
The ob/gyns were given the scenario of a brow presentation discovered during labour (that is, the baby’s neck is somewhat arched so that the widest part of the head is trying to fit through the birth canal, which is impossible in the circumstances described in the scenario), and asked to estimate the probability that the patient would convert to occiput posterior (a birth position in which the baby’s head is facing forward rather than backwards, rendering vaginal delivery possible but risky) and deliver vaginally.
In order to calculate the probability of a vaginal delivery, the doctors needed to estimate the probability of the two conjuncts: a) that the brow presentation would convert to occiput posterior and b) that the delivery of the now-occiput posterior-presenting baby would be vaginal, and multiply them by each other.
The pulmonologists were presented with a scenario in which a nodule was incidentally discovered in the lung of a female long-term smoker, and asked to estimate the overall combined probability that the nodule is cancerous, and that the cancerous condition would be detected via a properly-conducted needle biopsy.
Once again, to derive the correct answer, the lung specialists needed to multiply the probability of the two conjuncts: a) that the patient had lung cancer and b) that the cancer would be detected by needle biopsy.
Remember, by definition, the probability of the combined event cannot be greater than the probability of either of the two component events.
So how did they do?
“A total of 168 physicians (78.1%) estimated the probability of the 2-step sequence to be greater than the probability of at least 1 of the 2 component events.”
Analysis of Physicians’ Probability Estimates of a Medical Outcome Based on a Sequence of Events
Uh-oh. How bad were their maths skills? Compared with the product of their two estimated components (i.e. component a) multiplied by component b)), the pulmonologists’ estimates of the combined probability of a biopsy revealing cancer in the putative patient were over by 19.8 per cent, while the ob/gyns overestimated the combined probability of a vaginal delivery after conversion of a brow presentation to an occipital posterior presentation by 12.8 per cent.
Neither the respondents’ sex nor the time since obtaining a medical degree were correlated with the magnitude of overestimation; whether green or experienced, male or female, these highly-qualified specialists were similarly likely to commit a basic mathematical error.
When the researchers gave the ob/gyns a chance to identify and correct their own calculation errors by specifically requiring them to consider the conjunction’s components before estimating the overall probability, they did even worse, overestimating the combined probability by 18 per cent.
“Because the 2 component probabilities were clearly relevant to estimating the overall probability, we concluded that the physicians did not know (or did not recognize when to use) the multiplication rule for probability. Because our third substudy [the ob/gyn do-over] asked for the estimates of the components first and because these estimates were often provided in round numbers (eg, 30% or 40%), calculating the conjunctive estimate should have been relatively easy if physicians were aware of the multiplication rule.”
Analysis of Physicians’ Probability Estimates of a Medical Outcome Based on a Sequence of Events
Almost 80 per cent of the highly-trained respondents gave answers which were, in the restrained language of scientific studies, “mathematically incoherent (ie, formally illogical and mathematically incorrect).”
In fact, out of the 215 respondents, “only 1 physician in these 3 substudies correctly estimated the conjunction probability to be exactly equal to the product of the components’ estimates.” Or, to put it another way, 99.53 per cent of doctors got it wrong.
Why does it matter?
Especially if you’re a little mathematically challenged yourself, you might be wondering at this point, “Does it really matter if most doctors are innumerate? How much does it affect their clinical judgment, and the advice they give to patients?” A lot, as it happens. As the authors of the study note,
“The impetus for this research project was a real case involving a medical tragedy. In that case, patient counseling related to probabilities was misguided in a way consistent with the conjunction fallacy, resulting in the loss of a child due to injuries sustained during prolonged labor.”
Analysis of Physicians’ Probability Estimates of a Medical Outcome Based on a Sequence of Events
In other words, a baby died because the mother was not given sufficient information on the chances of her baby surviving if she continued to attempt vaginal delivery. She declined a caesarean section, which, in this case, would probably have saved the baby’s life. Basic maths skills really do matter:
“Overestimation has the potential to reduce the quality of medical care in any of the myriad scenarios in which decisions depend on probability estimates.”
Analysis of Physicians’ Probability Estimates of a Medical Outcome Based on a Sequence of Events
The authors reference other studies providing evidence that – as they delicately put it – “many physicians may not be facile in the calculation of probability or even basic numeracy”. For example, when provided with all the information that they required to calculate the probability that a woman had breast cancer based on a positive result on mammography screening, “95% of physicians responded with a probability that was 10 times higher than the correct answer”.
Take a moment to imagine the consequences of this miscalculation to the woman sitting in the doctor’s office, and you won’t be surprised that a false positive on mammography screening (that is, being recalled for further investigation after something abnormal shows up on your screening mammogram, but then finding that you’re cancer-free) is associated with elevated anxiety up to 18 months after getting the all-clear.
Now, think about how these findings relate to the behaviour of doctors since the outbreak of covidiacy. Many of my clients and subscribers have tried to discuss the serious limitations of PCR tests, the uselessness of face masks, and the unfavourable risk-benefit ratio of COVID-19 injections, with their doctors, often taking scientific papers to their appointments in hopes of eliciting a rational discussion. Needless to say, this has not gone well. The most common response is angry dismissal, closely followed by stony-faced repetition of the COVID catechism. Not one of my contacts has found a doctor willing to engage in discussion of research.
Whilst the serious conflicts of interest that I discussed in Your doctor is not your doctor provide a large part of the explanation for this phenomenon, the finding that most doctors are disturbingly innumerate also factors in. If your sense of legitimacy and authority rests on being smarter than the average bear, it’s intimidating to be confronted by a patient who seems to have a better grasp of basic mathematical concepts than you do.
What should we do with this information?
The authors of the study echo other researchers’ calls to include “greater emphasis on numeracy as well as statistical and probabilistic reasoning in medical education”. Sounds like a good plan, but even if it were implemented, it’s not going to help the patients who are currently receiving inferior care because their doctors are innumerate.
For the foreseeable future, if you or someone you’re responsible for is in need of medical care, it’s going to be up to you to develop some basic mathematical competency so that you can compensate for your doctor’s deficits when it comes to selecting or rejecting tests and treatments.
You could:
Take an introductory class on probability and statistics from Khan Academy.
Watch Dr H. Gilbert Welch’s excellent videos on biostatistics, overdiagnosis, sensitivity and specificity, and a host of other important topics.
Sign up to Dr James Lyons-Weiler’s course on how to read and interpret a scientific study (or any of the other excellent courses at IPAK-EDU).
Or (shameless plug) join my EmpowerEd membership program and learn how to distinguish reliable information from junk science and industry propaganda.
Even if you only learn how to correctly calculate a conjunction probability, you’ll be doing better than 99.5 per cent of doctors.
UPDATE:
Wowsers, I just read the Bad Cat’s incendiary post on IQ, here:
I couldn’t quite believe how low most countries’ students scored on basic maths questions were, so I hopped over to https://www.oecd.org/pisa/test-2012/form/ to take the test for myself and to check out the charts of student performance in different OECD countries. I don’t mean to brag, but none of the questions seemed particularly challenging to me. It’s truly horrifying that the vast majority of the world’s students can’t answer these questions correctly. After the events of the last 2.5+ years, I think it’s valid to ask the question: How many low IQ people are currently occupying positions of power that are vastly incommensurate with their abilities?
For information on my private practice, please visit Empower Total Health. I am a Certified Lifestyle Medicine Practitioner, with an ND, GDCouns, BHSc(Hons) and Fellowship of the Australasian Society of Lifestyle Medicine.
Diagnosis is the process of figuring out what condition a patient has, based on their symptoms, or subjective experiences, and signs, or objective data obtained through examination and testing. Prognosis is the forecast of the probable course or outcome of a disease; or a prediction of the probability of success of a treatment.
Probability equations are "racist, sexist, and bigoted!"
Great article Robyn. It also reminds me of the quote,
“It is difficult to get a man to understand something, when his salary depends on his not understanding it.”
Doctors have no incentive in understanding anything that might contradict what their pharma funded education has taught them, there is also a fair amount of hubris involved, I know better than you, I don't need to listen to you.