Adventures in awful headlines
This may seem off-topic, but trust me it’ll come around to online learning…
You may have seen headlines that looked something like this, a couple of weeks ago:
Screening Procedure Fails to Prevent Colon Cancer Deaths in Large Study seemed to tell a compelling story. That headline comes from Bloomberg News; similar headlines were on CNN, NBC News, and many other mainstream and health-related sources, including at least one which called the study the “gold standard.” Why was this newsworthy? Because:
“Colonoscopy screening exams that are recommended for older US adults failed to reduce the risk of death from colon cancer in a 10-year study that questions the benefits of the common procedure.
While people who underwent the exam were 18% less likely to develop colon cancer, the overall death rate among screened and unscreened people were the same at about 0.3%, researchers from Poland, Norway and Sweden said Sunday in a study published in the New England Journal of Medicine.
‘This relatively small reduction in the risk of colorectal cancer and the non-significant reduction in the risk of death are both surprising and disappointing,’ Jason Dominitz, a gastroenterologist at the University of Washington School of Medicine, and Douglas Robertson of the Dartmouth Geisel School of Medicine said in an accompanying commentary.
But the study did not in fact provide the evidence that the headlines said. The full study is behind a paywall, but the abstract is available, and states:
We performed a pragmatic, randomized trial involving presumptively healthy men and women 55 to 64 years of age drawn from population registries in Poland, Norway, Sweden, and the Netherlands between 2009 and 2014. The participants were randomly assigned in a 1:2 ratio to either receive an invitation to undergo a single screening colonoscopy (the invited group) or to receive no invitation or screening (the usual-care group). The primary end points were the risks of colorectal cancer and related death, and the secondary end point was death from any cause. (emphasis added)
The study was not comparing people who received a screening compared to those who didn’t—it was comparing people who received an invitation to a screening to those who didn’t. And, amazingly, a fair number of people invited to receive a colonoscopy don’t get one!
Fortunately, with the study receiving as much publicity as this one, there was further analysis and commentary. It took National Public Radio about three days to run this commentary on the study:
“It turns out that more than half of the research participants who were 'invited' to get a colonoscopy never showed up for the procedure.
"A colonoscopy will only work if a patient gets one," says Bret Petersen, a gastroenterologist at Mayo Clinic and president of the American Society for Gastrointestinal Endoscopy, a leading group of GI doctors.”
And, in fact, once you account for the people who didn’t respond to the invitation, the numbers of cancers discovered and lives saved lined up pretty well with prior estimates.
So, in this case, the system worked, in a way. It still seems non-sensical that the New England Journal of Medicine ran this study, and the first articles were poor, but enough people took a closer look to correct the error.
But how many people saw the correction, and how many people are now misinformed about screenings?
And more importantly, how many poor studies get headlines without any pushback because they’re not important enough to challenge?
Which leads me to The Surprising Ways Teachers’ Biases Play Out in Virtual Classrooms, an Education Week article published on October 13.
There are only two major problems with this study and headline.
Nothing about this is surprising for anyone passingly familiar with subconscious bias, and
The study has nothing to do with virtual classrooms.
From the article:
“When asked to evaluate identical student math work in a simulated virtual Zoom classroom, teachers were more likely to recommend that Black students get tested for special education than white students and that boys get tested for gifted programs more than girls, the researchers found. <snip>
For the study, teachers were shown the photos, names, and math work of 12 students in what looked like a Zoom classroom. The teachers were asked to evaluate the students’ work, which remained the same for every teacher, but the student images—which were a mix of Black and white students—and names attached to the work were randomly assigned and different for each teacher.
To sum up: teachers were shown pictures of student in a simulated Zoom classroom, asked questions very similar to the type that have demonstrated subconscious bias in other settings, and they again showed these same biases.
To be very clear—I’m not disputing the demonstrated biases, as I’ve done enough reading on the topic to understand these types of studies. But that’s exactly why this is not at all a surprising study.
And to say this is how bias plays out in a virtual classroom, as the headline does, is misleading at best. It’s also insulting to every online teacher because it suggests that online teachers know their students no better than they would if they had nothing to go on but a single picture of that student.
What’s the lesson here? It’s that skepticism is warranted, even with generally reputable sources like the New England Journal of Medicine and Education Week. And if you don’t have time to read beyond the headline—and in some cases go to the original study—you should be careful about how you use the new information.
(Thanks to Ray Rose for sending me the Ed Week article with his usual biting commentary.)