Research into efficacy of online, blended, and digital learning
Updated October 1, 2018
Among the most common questions related to online, blended, and digital learning is "Does it work?" Usually the questioner means "Is it as good as or better than face-to-face instruction?"
There is no simple answer to this question, for a couple of reasons.
First, there are so many types of online, blended, and digital learning that no single answer can cover them all.
Second, comparisons of different school types or instructional models are notoriously difficult. For example, so many studies have been published on charter school efficacy that it's not hard for anyone to cherry-pick a subset of reports to support any preconceived notion.
Third, many studies are very narrow, for example, looking at a single grade level or subject area, and the generalizability or transferability to other contexts is unclear or not methodologically sound.
Still, a few themes emerge from the studies that have been done and the experiences of educators.
1. Online courses that fill a gap in course availability have a lower bar to demonstrating effectiveness
Many students are taking online courses because they don't have access to a face-to-face version of the same course. In these instances, the question of "is it better than face-to-face?" is nonsensical, because the student doesn't have a face-to-face option. It is still appropriate to track measures such as Advanced Placement exam scores, college credits in dual-enrollment classes, and course completion rates, to ensure that these online courses are cost-effective and a productive use of students' time.
2. Given that online, blended, and digital learning are being used by schools to meet students’ needs, valuable research focuses on determining promising practices and related questions.
Many researchers are focused on helping educators improve their practices by comparing different instructional strategies and methods, rather than trying to determine if one school model is "better" than another. The question then becomes “under what conditions does online learning, or other educational technology, work?” Dr. Rick Ferdig is among the first and most prominent researchers to make this point.
3. Researchers looking at the use of technology in traditional classrooms and schools at scale demonstrate that technology is only as good as a successful implementation, which relies heavily on teachers, school leadership, professional learning, student support, and many other factors.
No studies have ever demonstrated a technology showing sustained and scale-able success without a mix of these and other elements. When conducting research in digital environments, taking a whole system approach is essential to understanding the many interacting elements and varying contexts.
The use of technology in education does not automatically result in a change in educational outcomes. Some studies show positive outcomes while other studies show no significant difference between the use of technology and traditional instruction; and some studies show negative outcomes. The research that is highlighted on this page is not meant to suggest that the application of technology always improves student outcomes but, instead, intends to show the ways in which technology can be applied successfully. We included some prominent studies that show limited or no impact of the use of technology and other studies that suggest ways in which computers, digital content, technology platforms, and other systems can demonstrate impact on schools and students.
Notable studies
Below we highlight a few studies of interest because they:
Were published relatively recently;
Are relatively high profile, often because they were commissioned or published by well-known organizations, people, or journals; and/or
Focus on comparatively broad topics of interest to a wide audience, instead of very specific issues.
Handbook of Research on K-12 Online and Blended Learning (Second Edition)
2018
This Handbook, now in its second edition, is published every four to five years and serves as a research for those interested in the field of K-12 online and blended learning. The volume provides an empirical resource that includes a syntheses of work in the field to help strengthen the field by identifying clear evidence about what is known and what is yet to be known. In this edition, there are nine sections, including (1) A Background and Historical Perspective; (2) Research on Learning and Learners; (3) Research on Teaching; (4) K-12 Online & Blended Learning in the Content Domains; (5) Research on Student Support Structures; (6) Research on Instructional Design; (7) Research on Learning Environments; (8) K-12 Online Learning Around the World; and (9) Emerging Issues. The chapters are updated each edition, and more chapters are included in each section as the book continues to expand. The Handbook is freely accessible through Carnegie Mellon’s ETC Press.
Suggested citation: Kennedy, K., & Ferdig, R. (Eds.). (2018). Handbook of Research on K-12 Online and Blended Learning. Pittsburgh, PA: ETC Press. https://figshare.com/articles/Handbook_of_Research_on_K-12_Online_and_Blended_Learning_Second_Edition_/6686813
As part of the Institute of Education Sciences Virtual Learning Lab line of grants (R305C160004), the University of Florida Lastinger Center was awarded a five-year grant to study the effects of Algebra Nation, a free math tool customized to help students improve their math skills. The resource is aligned to the standards within the state it is being implemented. Preliminary results from the Florida implementation in the PDF linked to the report title above.
Suggested citation: Leite, W. L., Cetin-Berber, D., et al. (2018). Impact of the Algebra Nation tutoring program on the performance of students who retake the end-of-course exam. Presentation at the annual American Educational Research Association (AERA), New York City, NY.
Combining data and text mining to develop an early warning system using a deep learning approach
2018
Learning analytics and data mining have become a prominent area of exploration in K-12 digital learning. Understanding a student’s progress and what supports they need to be successful in their learning process is the foundation for learning analytics. In this report, Hung and Rice explore the development of an early warning system to support students if they’re struggling in their online learning experience. Using a deep learning (DL) approach, they found an accurate prediction power with both high levels of generalizability and stability. The Michigan Virtual Learning Research Institute has confirmed that Hung and Rice are looking at an even larger dataset in the coming year.
Suggested citation: Hung, A & Rice, K. (2018). Combining data and text mining to develop an early warning system using a deep learning approach. Lansing, MI: Michigan Virtual University. Retrieved from https://www.mvlri.org/research/publications/combining-data-and-text-mining-to-develop-an-early-warning-system-using-a-deep-learning-approach/
Full-Time Virtual and Blended Schools: Enrollment, Student Characteristics, and Performance
2018
This study specifically delves into full-time online schools and blended schools. A few of the key findings include that in 2016-17, 429 full-time virtual schools enrolled 295,518 students, increasing from the previous two years by 17,000. The blended schools had an 80,000 enrollment increase, and from 296 blended schools, the enrollments were at 116,716. Only a little under a half of the schools involved were charter schools. A totlal of 34 states had full-time virtual schools while 29 states had blended schools.
Suggested citation: Miron, G., Shank, C. & Davidson, C. (2018). Full-Time Virtual and Blended Schools: Enrollment, Student Characteristics, and Performance. Boulder, CO: National Education Policy Center. Retrieved from http://nepc.colorado.edu/publication/virtual-schoolsannual-2018.
Michigan’s K-12 Virtual Learning Effectiveness Report, 2016-17
2018
Data collection at the state level is critical to informing policy and practice. One of the first and only states doing this at a comprehensive level specific to K-12 online learning is Michigan. As part of the legislative directives given to the Michigan Virtual Learning Research Institute, the research arm of Michigan Virtual, Michigan’s K-12 Virtual Learning Effectiveness Report, now in its fifth year of production with the 2016-17 installment, looks at the state reported data for students who enroll in online learning to understand what’s working, what’s not working, and what needs to be done for continuous improvement. Freidhoff works with the Michigan Department of Education and the Center for Educational Performance and Information in order to secure the dataset each year.
Suggested citation: Freidhoff,J. R. (2018). Michigan’s K-12 virtual learning effectiveness report, 2016-17. Lansing, MI: Michigan Virtual University. Retrieved from https://mvlri.org/research/effectiveness-report/
UPDATE: Online Learning
2017
Similar to the Michigan report, this report provides an overview of online learning in the state of Washington. The report services as one fulfilling a Legislative request. The findings of this report include but are not limited to the following:
The number of online courses, students taking online courses, and the schools integrating online courses have increased.
White students make up the most amount of online learners.
Lacking in numbers were English language learners, low income students, and students with disabilities.
A majority of students (71%) took fewer than five courses.
Success rate in 2014-15 was 74.7% while in 2015-16 it was 79%. Anything above a D is considered a “success.”
Reports like this one and the one from Michigan help the state, districts, and online learning providers understand what is happening year to year with online learning and what issues need to be addressed to better the experience for all stakeholders involved, especially the students.
Suggested citation: Nelson, R., & Hunter, L. (2017). UPDATE: Online Learning. Retrieved from www.k12.wa.us/LegisGov/2017documents/2017-2-OnlineLearningUPDATE.pdf
The Getting Back on Track Study: Online Courses for Credit Recovery
2017
In a series of six reports, the American Institutes for Research share their research on online credit recovery, a topic that surfaces often in the field as it is one of the most common uses of online learning. The first report focuses on a comparison of face-to-face and online courses. The second explores the role of an in-person instructional support as students engage in the online courses. The third report describes the profile of a student who enrolls in credit recovery courses. The fourth report dives into the content and how that is taught in an online and face-to-face mode of delivery. The fifth report looks at high school credit accumulation and graduation in alignment with the online and face-to-face credit recovery courses. The final report explains how students progress through courses, the courses they take along the way, and their journey to high school graduation. This series is an example of looking at the interplay of elements in the context being studied.
Suggested citation: Heppen, J., et al. (2017). The Getting Back on Track Study: Online courses for credit recovery. Washington, DC: American Institutes for Research. Retrieved from https://www.air.org/project/back-track-study-using-online-courses-credit-recovery
While specifically examining the learning environment is critical, educators and researchers also need to focus on the role policy plays in the implementation of digital technology initiatives. In this article, Archambault, Kennedy, and Freidhoff examined questions of accountability and the need for policy shifts regarding the various stakeholders involved in the K-12 online learning experience. The research provided mini case studies of other states that employ accountability measures as well as a deep dive into the landscape of accountability within Michigan specifically. The research was legislatively directed to provide guided recommendations for the Michigan Legislature pertaining to the implementation of online learning. The researchers were awarded the Association for Educational Technology & Communications 2018 Division of Distance Learning Journal Article Award. Since the journal in which the article is published is not freely accessible, we have linked to the original Michigan Virtual Learning Research Institute report.
Suggested citation: Archambault, L., Kennedy, K., Freidhoff, J., Bruno, J. DeBruler, K. & Stimson, R. (2015). Accountability in K-12 online learning course access programs: Stakeholder recommendations for policy and practice. Lansing, MI: Michigan Virtual Learning Research Institute. Retrieved from http://media.mivu.org/institute/pdf/Accountability_2015.pdf
Students, Computers and Learning: Making the Connection
2015
Sharing data from the international perspective, this study received attention from the media and from critics of technology in education. The study compares scores on the PISA test in different countries and associates the scores with the level of computer usage in their schools. Key findings include:
“Despite the pervasiveness of information and communication technologies (ICT) in our daily lives, these technologies have not yet been as widely adopted in formal education.”
The impact of technology on student performance is mixed. Test results “show no appreciable improvements in student achievement in reading, mathematics or science in the countries that had invested heavily in ICT for education.”
“The real contributions ICT can make to teaching and learning have yet to be fully realised and exploited.”
Perhaps most importantly, the report does not suggest that the use of technology in education should be reduced. Instead, it stresses that “We need to get this right in order to provide educators with learning environments that support 21st-century pedagogies and provide children with the 21st-century skills they need to succeed in tomorrow’s world. Technology is the only way to dramatically expand access to knowledge. Why should students be limited to a textbook that was printed two years ago, and maybe designed ten years ago, when they could have access to the world’s best and most up-to-date textbook? Equally important, technology allows teachers and students to access specialized materials well beyond textbooks, in multiple formats, with little time and space constraints. Technology provides great platforms for collaboration in knowledge creation where teachers can share and enrich teaching materials. Perhaps most importantly, technology can support new pedagogies that focus on learners as active participants with tools for inquiry-based pedagogies and collaborative workspaces.”
Suggested citation: OECD (2015), Students, Computers and Learning: Making the Connection, PISA, OECD Publishing. http://dx.doi.org/10.1787/9789264239555-en
Online Charter School Study
2015
A study done by CREDO at Stanford looks at online charter school performance. The analysis looks at the impact of attending an online charter school on students’ academic progress. Academic growth is examined also using students’ race-ethnicity, poverty status, and students who have special education needs. There is also a comparison of students who took online courses and those who took face-to-face. The study itself and its methods raised some controversy in how the student matching process was done when comparing students who took online versus those who took face-to-face courses.
Suggested citation: Woodworth, J. L., Raymond, M. E., Chirbas, K., Gonzalez, M., Negassi, Y., Snow, W., & Van Donge, C. (2015). Online Charter School Study. Stanford, CA: Center for Research on Education Outcomes. Retrieved from https://credo.stanford.edu/pdfs/Online%20Charter%20Study%20Final.pdf
Evaluation of LAUSD’s Instructional Technology Initiative
2015
The Los Angeles Unified School District has experienced widely publicized problems with its move to the use of iPads and digital curriculum that in part led to the resignation of the district superintendent. This report, from the American Institutes for Research (AIR), explores some of the key problems with the initiative, and also shows how the district has made changes to improve the initiative in its second year. The executive summary of this and the first year report provide valuable examples of both potential technology pitfalls, and solutions.
AIR found that in the second year the district made significant progress in its use of tablets, but that problems remain in the following areas:
Deploying devices in a timely manner,
Communicating with schools,
Coordinating efforts with other instructional initiatives, and
Clarifying a vision fr technology use in instruction.
The report includes recommendations that are valuable to any district considering a large technology initiative or one intended to grow in time.
Suggested citation: Margolin, J., Heppen, J., et al. (2015). Evaluation of LAUSD’s Instructional Technology Initiative: Year 2 Report. Washington, D.C.: American Institutes for Research. Retrieved from http://www.air.org/sites/default/files/downloads/report/LAUSD-Instructional-Technology-Initiative-August%202015.pdf
Published on Best Evidence Encyclopedia, created by the Johns Hopkins University School of Education's Center for Data-Driven Reform in Education under funding from the Institute of Education Sciences, U.S. Department of Education.
This is the first of two related and targeted meta-analyses of education technology applications published with support from the U.S. Department of education. This research reviewed the results of 84 studies examining the use of technology to support reading instruction in both elementary and secondary schools. Overall, the analysis found a small positive effect across the use of various types of education technology. However, the authors found that “innovative technology applications and integrated literacy interventions with the support of extensive professional development” drove most of the positive findings, and other types of technology did not have a significant effect.
Suggested citation: Cheung, A. C. K., & Slavin, R. E. (2012). Effectiveness of educational technology applications for enhancing reading achievement in K-12 classrooms: A meta-analysis. Retrieved from http://www.bestevidence.org/word/tech_read_April_25_2012.pdf
Published on Best Evidence Encyclopedia, created by the Johns Hopkins University School of Education's Center for Data-Driven Reform in Education under funding from the Institute of Education Sciences, U.S. Department of Education.
This, the second of two meta-analyses, reviewed 74 studies examining technology to support math instruction in both elementary and secondary schools. Similar to the reading study, the authors found a small positive effect in total. Their review, and past findings, “found no trend toward more positive results in recent years.” This report was published prior to the large-scale RAND study of impact of Carnegie Learning on math students; given the size of the RAND study it would likely have a significant impact on the meta-analysis of math studies.
Suggested citation: Cheung, A. C. K., & Slavin, R. E. (2011). The effectiveness of educational technology applications for enhancing mathematics achievement in K-12 classrooms: A meta-analysis. Best Evidence Encyclopedia: Empowering Educators with Evidence on Proven Programs. Retrieved from http://www.bestevidence.org/word/tech_math_Apr_11_2012.pdf
Our research scan includes several meta-analyses in order to filter the statistical noise of many small individual studies. We include this study as an important second-order meta-analysis (essentially a study of meta-analyses) that looked at 25 different studies reviewing 1,055 primary studies, collectively with about 100,000 participants, which had been published since 1985. The primary studies covered a wide range of technology applications in K-12 education. Overall the report found a “significant positive small to moderate effect size” resulting from the use of technology.
Similar to other researchers, the authors call for moving from comparisons of technology use versus no technology use, to better understanding conditions under which the use of technology is likely to be successful. “We feel that we are at a place where a shift from technology versus no technology studies to more nuanced studies comparing different conditions…would help the field progress.”
Suggested citation: Tamim, R. M., Mohammed, H. B., et al. (2011). What forty years of research says about the impact of technology on learning: A second-order meta-analysis and validation study. Review of Educational Research, 81(1), 4-28. Retrieved from https://www.researchgate.net/publication/258182794_What_Forty_Years_of_Research_Says_About_the_Impact_of_Technology_on_Learning_A_Second-Order_Meta-Analysis_and_Validation_Study
This extensive review of education technology research concludes with the notion that determining effectiveness is not the bes use of research efforts, which instead should focus on what works and why. This is an increasingly common view among researchers that is explained well in this article.
“As technology continues to rapidly gain increased usage and importance in K-12 education, the next decade will undoubtedly offer unprecedented opportunities for research findings to inform practices for enhancing teaching and learning. To achieve that goal, we encourage researchers to reduce efforts to prove the “effectiveness” of technology, while focusing on conducting rigorous and relevant mixed-methods studies to explicate which technology applications work to facilitate learning, in what ways, in which contexts, for whom, and why.”
Suggested citation: Ross, S. M., Morrison, G. R., Lowther, D. L. (2010). Educational technology research past and present: Balancing rigor and relevance to impact school learning. Contemporary Educational Technology, 1(1), 17-25. Retrieved from http://www.dergipark.gov.tr/download/article-file/252115
Although this study is about online learning and does not include classroom-based technology, we include it because it is the seminal recent study reviewing online learning. Its conclusions are:
“on average, students in online learning conditions performed modestly better than those receiving face-to-face instruction.”
Blended learning (a combination of online and face-to-face instruction) showed the best results.
However, “blended classes often included additional learning time and instructional elements not received by students in control conditions. This finding suggests that the positive effects associated with blended learning should not be attributed to the media, per se.”
The number of studies was small, so the results include many studies from post-secondary and other educational institutions, so generalizing to K-12 education should be done with caution.
This study is important in part because with the growth of 1:1 computing initiatives, low-cost home Internet access, and bring-your-own-device programs, the line between classroom-based technology and online learning is blurring. This is particularly true among middle school and high school students. District-level technology initiatives in particular are increasingly considering the role of students accessing digital content, and entire courses, from outside the school.
Suggested citation: Means, B., Toyama, Y., Murphy, R., Bakie, M., & Jones, K. (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. Washington, DC: U.S. Department of Education.