“No significant difference” confirmed by a new study from AIR

We’ve had this text on our Digital Learning Collaborative website for two years:

“Among the most common questions related to online, blended, and digital learning is "Does it work?" Usually the questioner means "Is it as good as or better than face-to-face instruction?"

There is no simple answer to this question, for a couple of reasons.
 
First, there are so many types of online, blended, and digital learning that no single answer can cover them all. 

Second, comparisons of different school types or instructional models are notoriously difficult…
 
Third, many studies are very narrow, for example, looking at a single grade level or subject area, and the generalizability or transferability to other contexts is unclear or not methodologically sound.”

With all those caveats in mind, the short answer to the “is online as good as f2f” question has been, for many years, “it can be.” The longer answer is that both online and f2f courses have a large range of outcomes, and if you imagine the set of outcomes from online and f2f courses as two circles, those circles overlap, mostly if not entirely.
 
A recent study from the American Institutes for Research leads to essentially the same conclusion, but is valuable nonetheless for several reasons.
 
First, the background on the research:

“The study focuses on first-year high school students who failed Algebra 1 or ninth-grade English (English 9) and retook the course during the summer before their second year of high school.
 
Within each participating school, the study used a lottery to determine whether each student took the school’s typical teacher-directed class or a class that used an online learning model…The credit recovery classes for the study took place during summer 2018 and summer 2019 in 24 LAUSD high schools… 

To examine the effectiveness of the online class, we compared it to the typical teacher-directed credit recovery class students take at each high school in the study. As with the online classes, a credentialed teacher led the teacher-directed classes, which met for 2.5 hours each day during the summer session. Unlike the online classes, teachers had more flexibility to determine the instructional materials, which were typically paper based. Teachers also had more freedom to determine the pace of student progress through the course content."

An important feature of this study is that, as explained above, it explores the attributes of the online credit recovery courses in depth. Students in the study took the course at school, with an in-person teacher available daily, and determined their pace through the course. See Figure 1 for a summary and other details.
 
Core findings are:

• “For Algebra 1, performance on the algebra test and credit recovery rates were similar in the online and teacher-directed classes. For English 9, student performance on the English test was similar in both types of classes, but the credit recovery rate was significantly lower in the online classes than the teacher-directed classes.”

• “When compared with a school’s typical teacher-directed credit recovery class, the Algebra 1 and English 9 findings suggest that students learn about the same content in both types of classes, and the English 9 findings suggest that students are less likely to pass the online class.”

The second of these findings may be a bit confusing. How/why did the online students learn about the same content in both cases, yet were less likely to pass the English credit recovery course?
 
The answer is AIR administered an exam that it created, as part of the research. Online and f2f students scored the same on that exam, on average. Passing the course, however, was determined by the teacher. For reasons that are unclear, the students in the online classes were given passing grades at a lower rate than students in the f2f class. (Many more details are here.)
 
Finally, the researchers explain the uncertainty and caveats:

“…the credit recovery landscape is, like most issues in education, diverse. Effects may be different across different online learning models, and even across different subjects using the same model. Also, implementation can matter. Even though the online learning model examined in this study included an explicit role for in-class teacher support, the online classes did not consistently provide more personalized instruction for students. Furthermore, many students in the online classes did not use the online program as much as intended— a finding that reinforces what other studies of online learning report. In addition, in both the online and teacher-directed classes, many students struggled to master the content, which points to broader challenges in developing effective credit recovery courses, regardless of whether they are online or teacher directed.”

Even though teachers gave passing grades at a higher rate in the f2f class, overall this study is in line with the general finding of “no significant difference” between online and f2f.
 
Why is this research valuable given that it doesn’t change our understanding of the efficacy of online courses? Three reasons:
 
First, it’s always useful to have a large, well-designed study to test our current hypothesis—and in this case confirm our present understanding.
 
Second, this study does a very good job of explaining the nuance of what it is studying. As such, it provides a valuable model to other researchers.
 
Third, the researchers have published a brief comparing costs of the online courses compared to the f2f courses, which is an under-studied area. Our next blog post will look at the cost and resource issues more closely.