COVID-19 school closures and educational achievement gaps in Canada: Lessons from Ontario summer learning research.

The 2020 COVID-19 pandemic closed most Canadian public schools for six consecutive months between March and September. This paper explores possible impacts of that closure on student achievement. Longstanding research suggests that lengthy periods of time out of school generally create losses of literacy and numeracy skills and widen student achievement gaps. New American studies have attributed sizeable learning losses to the COVID-19 closures. In lieu of comparable Canadian data, this paper extrapolates from summer learning research to estimate likely shortfalls in literacy and numeracy skills. We draw on data from 14 cohorts of Ontario primary-grade students collected between 2010 and 2015 in which 3,723 attended summer programs and 12,290 served as controls. Across three plausible scenarios, we use meta analyses and OLS and quintile regression models to predict learning losses of 3.5 and 6.5 months among typically-performing and lower-performing students respectively, and achievement gaps that grow up to 1.5 years among same grade peers. After qualifying these predictions, we recommend that provincial ministries offer targeted supplementary programs during the summer and synchronous instruction in the event of future school closures.

INTRODUCTION: THE IMPACT OF COVID‐19 SCHOOL CLOSURES

….[T]he COVID‐19 Pandemic is likely to generate the greatest disruption in educational opportunity worldwide in a generation…. It is imperative, for this reason, that education leaders take immediate steps to develop and implement strategies which mitigate the educational impact of the Pandemic (OECD, 2020, p. 4)

At the height of the COVID‐19 pandemic in 2020, more than 190 countries closed schools, affecting 1.6 billion children and youth, or 90 percent of the world's student population (UNESCO, 2020). Canada was no exception. By mid‐March, most provinces closed schools, prompting boards across the country to shift to emergency‐response remote learning, to distribute technology and other resources and to students, and to develop a variety of contingency plans in light of evolving information about COVID‐19. While two provinces fully or partially re‐opened their schools in June, schools in other provinces remained closed for the remainder of the academic year and did not re‐open until mid‐September (People for Education, 2020). These closures were followed by periodic shutdowns or shifts to remote learning as COVID‐19 cases spiked during the second wave.

Modes of teaching and learning varied widely during the initial period of the school closure; they included ‘no‐tech’ options (e.g., distribution of hard‐copy educational materials), the use of various types of established digital technology (e.g., text, emails), private resources (e.g., Google Classroom), and public or non‐profit resources (e.g., TVO Kids). Across Canada, weekly instructional hours between students and teachers ranged from one to twelve for kindergarten to Grade nine and two to three hours for students in Grades ten to twelve (Campbell, 2020; Gorbet et al., 2020). However, many teachers reported losing regular contact with their students altogether. A survey of 18,000 teachers conducted by the Canadian Teachers Federation (2020) found that only a small minority reported that “almost all” of their students were “checking in weekly.” Thirty‐five percent reported having regular contact with only a quarter of their students, while 64% of teachers said they were in regular contact with half of their students. Other surveys reported that teachers were concerned about “getting students what they need to be successful with online instruction,” (Alberta Teachers Association, 2020a, p. 16), “growing inequalities,” “curricular gaps upon a return to public schools” and “declining skill development (social, emotional, cognitive, physical, behavioral”) (Alberta Teachers Association, 2020b, p. 4). These survey results suggest that frontline teachers were worried that school closures were widening inequalities and achievement gaps.

However, these surveys notwithstanding, we are unaware of Canadian data that directly measure impacts of COVID‐19 school closures on student achievement. To our knowledge, Canada lacks high‐quality and large‐scale data that measured achievement immediately before and after the school closures. Compounding this problem, provinces like Ontario cancelled their planned standardized testing in 2020 and 2021, further restricting opportunities to assess the impacts of those school closures. To compensate for this data shortage, this paper extrapolates from our research on “summer setback”– losses of literacy and numeracy skill during summer vacation – to predict likely shortfalls in learning and widened achievement gaps among elementary‐level students caused by the COVID‐19 school closures. Summer vacations have several parallels to the COVID‐19 closures, and seasonal learning designs provide a useful framework for predicting learning losses and the widening of achievement gaps. In the next section, we review complementary research and then present our Ontario data on summer learning which we use to predict a series of learning shortfalls and gaps across 3 scenarios. We conclude by qualifying our predictions and then recommend that provincial ministries offer or continue to offer targeted literacy and numery supplementary summer programs and high quality synchronous learning in the event of further closures.

RESEARCH ON LEARNING IN SUMMER AND OTHER NON‐SCHOOL TIMES

A large sociological literature has examined gaps between top and low achieving students, noting greater variation within rather than between schools. This theme harkens back to the famous Coleman Report of the 1960s (Bradbury et al., 2015; Coleman et al., 1966; see also Caro et al., 2009; Downey, 2020; Simard‐Duplain & St‐Denis, 2020; Turcotte, 2011). These disparities highlight a key sociological mechanism: that unequal distributions of learning opportunities across home environments can be prime generators of achievement gaps. Research consistently finds that children's non‐school time can produce achievement gaps and other educational disparities (reviewed in Downey, 2020). Educational policies typically regulate school learning resources – by standardizing curricula, monitoring student‐teacher ratios, equalizing per student spending, and providing common training to teachers – but student homes and neighborhoods are far more disparate. The fact that most Canadian children spend more than 80 percent of their time outside of school points to the potency of non‐school environments (authors’ calculation).

For most school‐age Canadians, the 2020 COVID‐19 closures created an amount of non‐school time unprecedented in modern history. When school re‐started in mid‐to‐late September 2020, most students had been out of school for six months. While most schools provided at‐home learning platforms in the spring of 2020, their consistency, duration, and effectiveness are unclear. To provide a framework for pondering the possible impacts of being out of regular schooling for six months on children's academic achievement, we next review literatures on four major sources of non‐school time (see also Stewart et al., 2018).

The first major source is children's pre‐school years. School readiness is “an essential precursor to successful content‐based learning” and positive interactions with peers and teachers (Perry et al., 2018, p.1544; see also Denham et al., 2013). Gaps in readiness have been traced to systematic differences in children's early environments (Hertzman, 2009). Low SES children are more likely to start school with lower levels of reading and math skills (Duncan et al., 2014) and are less “school ready” in the areas of physical health, social confidence, emotional maturity, and language, cognitive, and communication skills (Janus & Duku, 2007; Janus & Offord, 2007; Janus et al., 2021). Longitudinal studies find that early gaps in school readiness tend to persist as children progress through school, generating not only unequal outcomes in education, but also inequalities in health, employment, and judicial involvement (Duncan et al., 2007, 2014; Perry, Braren & Blair, 2018).

The second source of non‐school time is generated by varying levels of school attendance. In Canada, 10–20 percent of elementary school children are chronically absent, defined as missing 18–19 days of school per year (see CBC, 2019; Office of the Child & Youth Advocate, 2019). Chronic absenteeism has been linked to poor academic achievement, school failure, and dropping out (Allison et al., 2019). School absenteeism tends to compound the effects of related circumstances like illness, poor mental health, exposure to domestic violence, unstable housing, and the need to care for family members. Importantly, schools’ potential equalizing effects can be nullified if children attend school irregularly. American research finds that low SES children are most likely to suffer chronic absences (Ready, 2010, p. 281) and that children living in poverty are most likely to miss three or more days of school per month (García & Weiss, 2018). Research also finds that, conversely, socioeconomically disadvantaged children with good attendance rates gain more literacy skills than their higher SES peers during kindergarten and first grade (Ready, 2010, p. 271). This research suggests that time in and out of school can be pivotal for the most vulnerable children.

The third source emerges from learning supplements provided by families, including quality preschools, extracurricular activities, and private tutoring (e.g., Aurini, 2012; Aurini & Davies, 2004; Davies, 2004; Park et al., 2016). These supplements are not distributed evenly among families. American research shows that over recent decades, higher SES parents have been spending nearly seven times as much on their children's development as do lower‐income families, up from four times in the early 1970s (Kornrich & Furstenberg, 2013). Further, higher SES parents spend approximately one hour more per day directly involved with their children than do lower SES parents (Putman et al., 2012, p. 13). By the age of six, children from higher‐income homes spend 400 more hours in literacy‐related activities and 1,300 more hours with their parents outside of the home in ‘non‐routine’ contexts such as parks or libraries (Phillips & Lowenstein, 2011; see also Snellman et al., 2015). These quantitative studies align with a sizable qualitative literature on the various ‘home advantages’ provided by families in the realm of schooling (Lareau, 2011; See Aurini et al., 2020).

The fourth source of achievement gaps, and most germane to this paper, is summertime. Summer vacations are the longest stretches of non‐school time that most children encounter after preschool. To capture their effects, sociologists have developed “seasonal learning designs” that compare learning gains and losses during the school year to those that emerge during the summer months. This design allows researchers to identify the timings by which achievement gaps stabilize, shrink, or grow among different groups of students.

A vibrant body of summer learning research has emerged over the past 40+ years. The most notable studies were those conducted by Barbara Heyns (1978) in Atlanta, Karl Alexander and his colleagues (2001, 2007, 2016) in Baltimore, and Downey, von Hippel, and Broh's (2004) use of nationally‐representative data. Those and other studies found that many children experience summer learning losses, that socioeconomic and racial achievement gaps tended to grow in the summertime, and that a sizeable portion of accumulated achievement gaps by high school could be traced to previous summers. Further, they concluded that achievement gaps tend to widen as students age, with disadvantaged students falling further behind as the progress through school. Meta‐analyses of American studies up to the mid‐1990s estimated that students lost on average 1 month in both literacy and numeracy during the summer months, that gaps between high SES and low SES children grew two to three months, but also that summer interventions had average effect sizes ranging from .2 to .3 (Cooper et al., 1996; Cooper Charlton et al., 2000).

Those researchers reasoned that summer vacations widened achievement gaps because they expose children to unequal out‐of‐school learning opportunities. Formal schooling tends to equalize learning resources during the school year, but children's routines vary far more during their summer vacations, as some are left to entertain themselves and shut off the “faucet” of learning entirely, while others are readily engaged in a variety of skill‐enhancing practices (Entwisle, Alexander & Olson, 2001). These researchers characterized public schools as “great equalizers” due to their capacity to provide considerable parity in learning opportunities during regular academic months, only to have much of their work undone during the following summer. They reasoned that achievement gaps would be far worse if not for formal schooling (e.g., Downey, von Hippel & Broh, 2004).

Karl Alexander and his colleagues’ twenty‐five‐year longitudinal study of Baltimore students was particularly instrumental for raising the profile of summer learning research among sociologists, teachers, and policy makers. They found that learning rates were roughly equal among students from different socioeconomic and racial backgrounds during their school years but became far more disparate over the summer months. Lower SES children tended to lose numeracy and literacy skills, while middle class children gained (Alexander, 2016; Alexander et al., 2007). They attributed two‐thirds of the achievement gap between low income and their middle income peers by ninth grade, as well as similar gaps in high school track placements, high school completion, and four‐year college attendance, to learning differentials that built up over previous summers. More recently, researchers with access to better measures of learning have questioned whether indeed race and SES gaps grow during the summer months (e.g., Kuhheld, Condron & Downey, 2020; von Hippel, Workman & Downey, 2018). However, these researchers continue to find that, overall, rates of summer learning vary considerably more than do school‐year rates, suggesting that summertimes are periods that generate greater stratification in learning.

Research on learning during COVID‐19 school closures

A new series of papers have adopted seasonal learning designs to predict and measure learning stemming from the COVID‐19 school closures. In April 2020, Kuhfeld et al. (2020) drew on massive seasonal learning data in the United States to predict learning shortfalls across 3 different scenarios. Hanushek and Woessmann (2020) extrapolated from research on school attendance and labor‐market outcomes to predict possible shortfalls in learning and subsequent lifetime earnings in a range of countries. Kuhfeld et al. (2020) and Renaissance Learning (2020) drew on achievement data from the fall of 2020 to assess the impact of the school closures on American student learning. These studies found some of their most drastic prior predictions to be off the mark. However, two studies found that compared to previous cohorts, the 2020 cohort was on average almost 2 months behind in reading and 3 months behind in math. Neither study found that pre‐existing lower achievers fell further behind; rather, their losses were similar to those in other achievement groups (see also Jæger & Blaabæk, 2020; Haeck & Lefebvre, 2020).​ However, a recent large‐scale study in the Netherlands found not only that most students experienced considerably learning shortfalls, and but also that learning losses were most pronounced among students from disadvantaged homes (Engzell, Frey & Verhagen, 2021).

RESEARCH QUESTIONS

Building on these recent American studies, and compensating for the lack of comparable Canadian data, this paper uses Ontario summer learning data to generate predictions about the possible impacts of the COVID‐19 school closures on Canadian student achievement. We use learning rates of summer program attendees and their control group peers to simulate plausible learning rates across a range of scenarios that make different assumptions about the effectiveness of schools’ online instruction provided in the spring of 2020. We pose the following questions:

What were the likely learning shortfalls among typical elementary‐level students over the 6‐month public school closure? To answer this question we draw on 6 years of data from our summer learning project, conduct a meta‐analysis to anchor our estimates, and specify three distinct scenarios to acknowledge the uncertain extent to which children were meaningfully engaged with school‐sponsored instruction during the COVID‐19 closures. These scenarios involve adding learning rates across 2 segments of time: the initial 3.5 month school closure period from mid‐March to late‐June, and the 2.5 month summer vacation from early July to mid‐September. Each scenario involves different assumptions about the first interval, and adds estimates of typical summer learning for the second.

What is the likely range of these shortfalls? Since research has shown rates of summer learning to be high among students, we predicted mean values as well as those for the 25th and 75th percentiles of summer learners.

We are unaware of data from the spring of 2020 that definitely measures the effectiveness of school‐provided instruction or parental involvement with their children's learning. Instead, we propose three plausible scenarios that each assume different levels of effectiveness and involvement.

Scenario #1: Learning as usual

This ‘best case’ scenario assumes that children learned at typical school‐year rates during the first interval, and typical summer rates during the second interval. It assumes that teachers and students quickly adjusted to online formats and were readily engaged in school work, were supported by their parents at levels typical of other years in the spring, and that children learned at typical summer rates in the months of July and August. Some might consider these assumptions to be heroic. The spring and summer of 2020 were quite unlike previous years. Parents and guardians experienced different consequences from the pandemic: some were laid off, others lost their jobs, while many essential workers had extended hours (Stanford, 2020). Unlike previous summers, fewer children attended day camps and related summer programs. It is likely that more parents, guardians, and children spent longer periods of time at home during the summer of 2020 than they did in previous summers, though it is unknown whether that time altered patterns of learning.

Scenario #2: An extended summer

This ‘worst case’ scenario assumes that instruction provided during the first interval was entirely ineffective and managed to generate learning similar to that generated by students’ home environments. Building on reports cited above, it assumes that remote learning was improvised and implemented haphazardly from board to board, school to school, classroom to classroom and that most students drifted and were largely disengaged from school‐sponsored learning for the entire period. It further notes that few teachers had prior training with remote learning, that many online instructional materials were created as supplements, not as regular curricula, and that in some jurisdictions, teacher associations resisted synchronous learning (e.g., Clarke, 2020). Many schools stopped formal assessment in mid‐March, so that any subsequent schoolwork was essentially completed ‘voluntarily’ by students.

Further, this scenario assumes that few guardians were in a position to boost their children's learning beyond that of a typical summer, whether due to unemployment, having to work at home, or due to other challenges. The COVID‐19 closures generated a wide range of home circumstances (Stanford, 2020). Some continued to work similar (or more) hours from home, while others continued to go to workplaces daily, leaving their children at home or with care providers. While parents who were unemployed or shifted their employment tasks to home had, in theory, extra time to spend on their children's schoolwork, such variation is also present in a typical summer. Ineffective school instruction could have motivated guardians to place more focus on children's learning. However, this scenario assumes that since few parents are as effective as professional teachers (Campbell, 2020: 13; see also Alberta Teachers Association, 2020a, 2020b) most children learned little more than they would in a typical summer.

Scenario #3: Online learning as akin to a summer intervention

A third scenario assumes that online instruction during the first interval was only partially effective, generating weaker learning rates than in typical school years, but better than those in the worst‐case scenario. This scenario assumes that schools eventually provided online instruction, but only after several weeks, and that teachers lost contact with many of their students soon afterwards. It thus likens student participation in the spring 2020 to that of 3‐week full‐day summer intervention over a two‐month period (we multiply those effects by 1.35 to extend them to a 3.5 period). Reports suggest that most students were not engaged in online provision until April, that few were engaged for full days, and that most lost contact by the end of May. Many students thus appeared to have about 8 weeks of half‐day learning between early April to the end of May, which is broadly comparable to 3 × 1.35 weeks of full‐day learning achieved in summer programs. To that estimate we add a typical summer rate for the remaining 2.5 month period.

DATA AND METHODS

Background on the Ontario summer learning project

The Ontario Summer Learning project was the largest summer learning study conducted outside of the United States. Our data is drawn from multiple regions across the province, 3 grade transitions, and six years of data collection between 2010 and 2015. In partnership with Ontario's Ministry of Education, the project began as a small pilot study but soon grew. In 2010 it encompassed 55 summer classes in French and English literacy for Grades 1, 2, and 3 in 28 school boards across the province. In 2011, numeracy programs were added in both languages. In 2014, over 300 classes across 65 of the 72 school boards in Ontario offered summer programs. By then many programs had waiting lists, prompting the Ministry to increase funding to keep pace with demand.

The project proceeded in an organic manner, evolving on a yearly basis as dictated by practical and changing Ministry priorities. The standard summer intervention consisted of 3 weeks of full‐day classes in French or English, totaling 45 hours of instruction alongside a recreational component, in classes of 15 students overseen by one certified teacher and one teaching assistant. Many boards provided healthy meals and snacks. Overall, program coordinators attempted to create an enjoyable ‘camp’ atmosphere. Programs were free of charge, voluntary, and officially open to all children at participating boards.

Research design and sample selection

Over six years, we collected data on a total of 14 unique cohorts of students as defined by combinations of year, language, and test type (see Table 1). In total, 3,723 students attended summer programs and 12,290 students served as controls. We evaluated those programs using a quasi‐experimental seasonal learning design. Summer program attendees were designated as the treated group, while control groups were comprised of their school‐year classmates who did not attend those programs.

TABLE 1

Summary meta‐analysis of language

English Language Studies:
Study	Hedges's g		[95% Conf. Interval] % Weight
Longitudinal Literacy 2015	−0.084	−0.293	0.125 7.02
English Literacy 2013	0.142	0.009	0.274 7.76
English Numeracy 2013	0.125	0.024	0.227 7.99
English Literacy 2011	0.100	0.009	0.191 8.06
English Literacy 2012	0.104	0.022	0.186 8.11
English Numeracy 2012	0.320	0.125	0.515 7.16
English Literacy 2010	−0.078	−0.177	0.020 8.01
Longitudinal Numeracy 2014	0.028	−0.163	0.220 7.20
Longitudinal Numeracy 2015	0.063	−0.123	0.248 7.26
Longitudinal Literacy 2014	0.049	−0.105	0.202 7.57
Theta	0.076	0.014	0.138

Note: Total Number of studies = 14. Hedges G based on random‐effects models estimated using restricted maximum likelihood methods.

We could not select random, representative or stratified samples since Ministry officials created their own practical and subjective criteria for selecting participating boards, schools, and students. The Ministry invited boards perceived to be in need, usually indicated by below‐average scores on provincial standardized tests, while also ensuring geographic representation of all Ontario regions, as well as French language boards. In turn, boards selected schools using similar criteria. Teachers were instructed to encourage students they saw as ‘needing a boost.’ But the programs were officially open to all; no student could be compelled to attend, but space permitting, no interested children were turned away. Since the Ministry added extra classes to schools with over‐subscribed enrolments, rather than use a lottery to ration access, we could not add a randomizing element to the treatment assignment.

The resulting samples were somewhat disadvantaged academically compared to the provincial population. Three‐quarters of participating schools had average pass rates on standardized tests below the provincial average and many were located in economically and socially challenged neighborhoods. Summer program attendees had significantly weaker academic profiles and came from families with significantly lower levels of parental education and income than their classmates (e.g., Davies & Aurini, 2013). Since disproportionate numbers of participants came from relatively disadvantaged homes, our estimates of learning losses may be biased downwards in comparison the full population of Ontario students.

Measures

We collected three sources of quantitative data. First, we measured English language learning using Star Reading and Star Math online tests, which have many desirable measurement properties (Renaissance Learning, 2020; von Hippel et al., 2018). We conducted pre‐tests mainly in the latter half of June and post‐tests mainly in mid‐September. These timings were quite favorable compared to most summer learning studies which are often contaminated by sizeable numbers of school days (Attebury & McEachin, 2020). We used a standardized Grade Equivalent (GE) metric that converts raw scores into months of learning in typical student trajectories. By this metric, a typical (American) student learns at a rate of 0.1 years each month during the school year. We measured summer learning by calculating change scores between the spring and fall. To measure French language learning, we used the province's “GB+” tests. Second, to provide baselines, we obtained report card measures of student grades, attendance, and grade level during the previous school year. Third, we conducted parent surveys for measures of student demographics and family practices.

Statistical procedures

To generate predicted learning rates we proceeded in multiple steps. First, for each cohort we predicted mean and quartile learning scores for both treated and non‐treated students using OLS and quantile regression models that adjusted for students’ prior achievement, test interval, grade level and school board. We did not use more complex model specifications that could have included variables from our survey data, since those surveys had considerable response rate bias in favor of summer program participants, and because we found, as have American researchers, that student demographics were surprisingly weak predictors of summer learning (for a recent review see Atteberry & McEachin, 2020). Our adjustments accounted for pre‐existing differences between control and treated students.

Second, since we detected considerable heterogeneity in patterns of summer learning across cohorts, we conducted meta‐analyses to systematically account for that variability and build uncertainty into our estimates. We created a meta‐analysis dataset that recorded average summer learning scores across all 14 cohorts (see Table 1 and Figure 1), with covariate‐adjusted means and quartiles, along with each cohort's sample size, standard deviations, year, language, and type of test. Third, to measure different learning rates between treated and untreated students we calculated Hedges's G standardized mean difference to measure effect sizes for each cohort, and also estimated Theta to summarize all effect sizes across cohorts. The latter used restricted maximum likelihood (REML) methods which assume that the distribution of random effects is normal. These meta‐analyses showed that treated and untreated students had statistically significantly different summer learning rates, justifying our decision to use them in our different scenarios. However, they also revealed greatly heterogeneous effect sizes by language and much greater reliability among the English cohorts. For those reasons, along with the superior interpretability of the English test metric, we used results only from the ten English cohorts for our subsequent predictions, weighted by cohort sample size. Finally, we applied those predictions across the three scenarios for students at achievement means and quartiles.

FIGURE 1

Forestplot of meta‐analysis [Color figure can be viewed at wileyonlinelibrary.com]

FINDINGS

We next summarize three groups of findings. First, we present our meta‐analysis of 14 cohorts. Second, we use covariate‐adjusted and weighted means and quartiles to estimate mean summer learning rates for treated and untreated students. Third, we apply those predictions to the three scenarios detailed above.

Section 1: Meta‐analysis

The Ontario summer programs generally had positive and significant effects on summer learning, though those effects were modest overall with considerable heterogeneity. Table 1 displays the theta score that summarizes all effect sizes across the 14 cohorts (each calculated by Hedges’ G) weighted by sample size . Theta for the entire project was .20, a moderate effect by the standards of elementary school interventions (Hill et al., 2008), and similar to Cooper et al.’s (2000) meta‐analysis of American summer interventions that estimated thetas of .2 to .3. Thus, since this meta‐analysis suggests that Ontario summer programs had moderately positive effects on learning, we can use separate estimates for treated and untreated students across our three scenarios.

However, the meta‐analysis also uncovered great heterogeneity in effect sizes by language. Both Table 1 and Figure 1 show that the effect sizes for all 4 French cohorts were larger than those for all 10 English–language cohorts. The French theta was .61, with very large Hedges G's ranging from .35 to 1.1. In contrast, the English‐language cohorts’ theta was far smaller at .076, though positive and statistically significant, with effect sizes ranging from ‐.08 to .32. Only half of the English cohorts had significantly positive effect sizes, and 2 cohorts had non‐significant negative effects. This sizeable range of effect sizes is actually common among reviews of summer interventions (e.g., Cooper et al., 2000; Kim & Quinn, 2013; McCombs et al., 2020), and illustrates the inconsistent effectiveness of summer programs. Further, a likely reason for the smaller theta score among our English cohorts is that our meta‐analysis does not suffer from publication bias – it draws on data from all cohorts, including those from which we are yet to publish studies. A prime limitation of meta‐analyses is that studies with null or negative findings are likelier to be unpublished or unreported than those with significant and positive effects, a bias that was likely particularly strong before the advent of powerful search engines twenty years ago when Cooper et al. (2000) published their meta‐analysis.

The stark contrast in effect sizes between the French and English cohorts is a likely product of 3 factors: language of instruction, sample size, and curriculum‐test alignment. By definition, the French programs were conducted in the French language, and treated a very distinct student population. However, we doubt there was anything inherent in the language of instruction that generated far larger effect sizes. The second factor was that the samples were considerably smaller among the French cohorts. Reflecting the fact that students in Ontario's French language boards comprise only 5% of the provincial school body, the number of treated students in the French cohorts ranged from 45 to 172, compared to a corresponding range among English cohorts of 98 to 604 (see Figure 1). All other things equal, smaller studies tend to generate larger effect sizes (Slavin & Smith, 2009). The third factor, which we believe is the most potent, was curriculum‐test alignments. The French boards used literacy and numeracy tests (the GB+) that were developed for the Ontario French language curriculum. In contrast, the English cohorts used Star Math and Star Reading tests (Renaissance Learning, 2020) that were developed to tap relatively generic measures of skills and have no particular connection to Ontario's English language curriculum. It is quite likely that the instruction in the French summer programs was aimed at the French standardized assessments, and that the skills learned in those programs were more readily captured by the GB+ in comparison to the English program instruction and the Star tests. The weaker curriculum‐test alignment among the English programs likely served to underestimate the true effect sizes among the English cohorts.

Nevertheless, the greatly differing French and English effect sizes prompted a key decision for estimating average rates of summer learning. As mentioned in the previous section, we elected to use the English cohorts only for several reasons. First, their results were far more reliable than those for the French cohorts. Second, the English results are similar to those found recently in a massive American study that also used the Star tests (Renaissance Learning, 2020), which offers confidence in their reliability. Third, the standardized GE metric is readily interpretable, unlike the French GB+ tests. Fourth, the English cohorts come from the vast majority of Ontario's public school population. Finally, using the weaker effects found among the English cohorts follows standard statistical practice of erring on the side of conservativism.

Section 2: Estimates of summer learning

In descriptive statistics not shown, summer learning was consistently normally distributed across all 10 English cohorts, with medians centered on zero or negligibly just above or below. Most students broke even over the summer months, returning to school in September with approximately the same amount of numeracy and/or literacy skill that they had in June. Figure 1 displays adjusted means for both treated and control students. Exceptional rates of learning gains were found among 2 treated cohorts (the 2014 longitudinal literacy and 2013 literacy) that had large mean gains of 2 to 3 months, and one control cohort (2014 longitudinal literacy) that gained an average of 2 months. Two control cohorts had average learning losses of more than 1 month (2013 English literacy and 2013 English numeracy). Otherwise, summertime had negligible impacts on the achievement of typical children across our cohorts.

The real action, however, occurred at the tails of the distributions. Children scoring at the 25th percentile in summer learning tended to suffer substantial learning losses ranging from 2 to 5 months. Students in the top quartile gained comparable amounts, essentially learning at school‐year rates during the summer months. These results illustrate that at least one quarter of the control groups suffered from sizeable summer setbacks, and that therefore the summer months tend to generate a considerable amount of achievement stratification, a finding recently replicated in large scale American data (Atteberry & McEachin, 2020).

Table 2 displays covariate adjusted and weighted means and quartiles, along with confidence intervals, for both treated and control students. These statistics were predicted from OLS regression models described in the previous section. The weighted mean for control students is negligibly below zero (but not statistically different from zero), while the weighted mean for the treated is 0.4 months, also not statistically significant. Table 2 also displays predicted values at the 25th and 75th quartiles, derived from quantile regression models. Those values illustrate considerable stratification in summer achievement. Untreated students at the lower quartile lost 3.3 months of learning, while their counterparts at the upper quartile gained a similar amount. Among the treated, students at the lower quartile lost 2.7 months, while their counterparts at the upper quartile gained 3.6 months. These statistics portray considerable achievement stratification, particularly when examining lower and upper bounds estimates. The lower bound estimate for the lowest achieving group (lower quartile controls) is a loss of more than 4 months, while the upper bound for the high achieving group (upper quartile treated) is a gain of more than 4 months. While this 8‐month gap over a mere summer vacation may seem extreme, they accord with recent American studies of millions of students (Atteberry & McEachin, 2020).

TABLE 2

Covariate adjusted weighted means and confidence intervals (English Cohorts only)

	Mean (CI)	Lower Quartile (CI)	Upper Quartile (CI)
Control	−.004 (−.066, .058)	−.331 (−.416, −.246)	.337 (.288, .386)
Treated	.040 (−.028, .107)	−.274 (−.348, −.200)	.362 (.297, .427)

Note: Predicted means are weighted averages for each of 10 English language cohorts; each predicted mean was derived from 10 regression models that controlled for summer program attendance, test interval, prior achievement and school board dummy variables. Predicted quartiles are weighted averages from quintile regression models with the same control variables for each of 10 English language cohorts.

Section 3: Predictions across 3 scenarios

This section applies the above estimates to three scenarios. The top row of Table 3 displays predictions for the first ‘best case’ scenario in which children's rates of learning during the spring of 2020 were like those of other school years, followed by typical summer rates thereafter. It provides a baseline for the other scenarios. The mean predicted rate is a gain of roughly 3.5 months; the predictions at the quartiles are roughly zero and almost 7 months respectively. Thus, even in the best‐case scenario we predict a difference of almost 7 months of learning over a 6‐month period.

TABLE 3

Three scenarios for predicted learning gains/losses and confidence intervals between mid‐March and mid‐September 2020, years of learning

	Students at Mean	25th percentile	75th percentile
Scenario 1: Regular school rates + summer program rates	.346 (.284,.408)	.019 (−.066,.104)	.687 (.638,.736)
Scenario 2: Summer control rates + summer control rates	−.009 (−.143,.126)	−.718 (−.903, −.534)	.731 (.625,.838)
Scenario 3: Summer program rates + summer control rates	.043 (−.099,.183)	−.651 (−.823, −.480)	.760 (.635,.886)

Note: Predictions based on adding estimates of two sets of rates: 1) those from the 3.5‐month COVID school closure from mid‐March to the end of June, and 2) those from the 2‐month summer break from the beginning of July to the beginning of September. Scenario 1 combines standard learning rates (3.5 months) and standard summer rates. Scenario 2 combines summer program rates and standard summer rates. Scenario 3 substitutes standard summer rates for both periods. Each cell value was derived from the corresponding estimates presented in Table 2.

The next row of Table 3 displays predictions for the second or ‘worst case’ scenario which assumes that children's learning during both the spring and summer months resembled those of untreated summer students. It predicts that students at the mean broke even, while those at the quartiles had respective losses and gains of more than 7 months, creating a gap of approximately 1.5 years. Predictions in this scenario suggest that the COVID‐19 school closures greatly polarized learning rates.

Scenario three, presented in the bottom row of Table 3, offers a less extreme prediction. It assumes that the online instruction provided by schools in the spring 2020 was less effective than normal in‐person instruction, but had impacts akin to those in summer programs. These predictions tell a largely similar story, nonetheless. Students at the means were predicted to have small gains, while those at the quartiles were predicted to have losses and gains smaller though comparable to those in the second scenario.

How do these predictions compare to the findings of recent American studies? In November 2020, Kuhfeld et al. (2020) and Renaissance Learning (2020) assessed the impact of the spring school closures by amassing data from the spring and fall of 2020, and comparing learning growth of the 2020 cohort to previous cohorts. Both studies found that the closures had negative impacts. Using metrics similar to ours’, Renaissance Learning estimated that typical students had shortfalls of 2 months in reading and 3 months in math. Those estimates are very close to our predictions for Scenario three, which assumed that the online instruction provided by schools in the spring of 2020 was considerably less effective that in‐person instruction provided in normal years.

Overall, our predictions suggest that the 2020 school closures likely generated learning shortfalls of approximately 4 months among average performing Canadian students, losses of 7 months among lower‐performers, and almost certainly widened pre‐existing achievement gaps by large amounts. Our predictions could be exaggerated, however, if the spring 2020 online instruction was more effective than has been commonly reported, and if instruction was more effective in Canada than it was in the USA.

DISCUSSION AND CONCLUSION

Several bodies of research show that long stretches of non‐school time can generate learning losses and sometimes exacerbate achievement gaps. In the spring of 2020, a series of converging forces altered opportunity structures for student learning: most schools closed, most instruction switched to online, albeit inconsistently, and many families re‐arranged their schedules. During normal times, families vary greatly in their resources and capacities to promote children's learning, and it is unlikely that the added pressures of COVID‐19 altered that variation.

This paper estimates learning shortfalls among Canadian elementary students that may have been generated by those forces. Two recent studies measured those impacts on American students, but they cannot be extrapolated to Canada, where student achievement is somewhat higher on average and less stratified (Davies et al., 2016). We used data from our Ontario summer learning project to generate predictions for elementary‐level students. We systematized our predictions via meta‐analyses to account for sampling variability without any publication bias. The meta‐analysis was used to predict learning shortfalls across three scenarios that varied by assumed student uptakes of school‐sponsored instruction during the spring. In the best‐case scenario, students would fare the same as they would in other years, typically gaining 3 months of learning, with gaps of about 6.5 months between the quartiles. In the worst‐case scenario, the average student would have a 3‐month learning shortfall compared to a regular school year, with 1.5 years gaps between the quartiles.

These predictions might exaggerate or under‐estimate the true impact of the COVID‐19 school closures. They might exaggerate learning losses if parents reacted to the COVID‐19 closures in ways unlike typical summers. Once schools rolled out their ‘learning at home’ models in April, May and June 2020, most children were nominally receiving some formal instruction. Further, many parents and guardians were spending longer periods of time at home during the school closures than they did prior to the pandemic. If many parents were highly involved in their children's school work, they may have mitigated learning losses, and perhaps even generated gains. During typical summers in contrast, many families, adopt a “vacation” mindset in which they do not pressure their children to engage in activities that could build their skills. During the COVID‐19 closure, however, many parents and guardians may have urged their children to complete assigned school work, and may have not adopted a vacation mentality until the summer months. If those conditions did indeed materialize, then Scenario two would underestimate learning gains if children had greater access to effective learning opportunities than imagined.

Conversely, there are plausible reasons to expect that our scenarios underestimate the impact of the school closures. Our estimates were derived in part from linear extrapolations of summer learning rates that emerge over a 2 or 3‐month period of non‐school time. However, at some unknown threshold of non‐school time beyond a standard summer vacation, learning losses may become non‐linear and more extreme, accelerating the atrophy of skills, particularly among struggling students. Just what that threshold may be is unknown. But given that the 6‐month school closure of 2020 is equivalent to 2 to 3 consecutive summer vacations, it is plausible that at least some students may have reached their threshold, and would be unable to ‘break even’ after nearly a half year out of school, at least without other sources of learning. Given that the length of COVID‐19 school closures are unprecedented in modern history, we do not know of a body of research that could provide empirical guidelines for such estimates.

If our predications are broadly accurate, however, what is to be done? We offer two recommendations. First, since we doubt the remote learning models adopted in the spring of 2020 were as effective as regular in‐school instruction, we recommend that school authorities offer or continue to offer high‐quality and targeted supplementary interventions in the summer in order to compensate. These supplements would shift some of the load of children's learning off parents, many of whom may need to work extended hours in the summer of 2021 to compensate for their own income losses from the previous year (for an international argument along these lines, see OECD, 2020). These programs should continue in 2022 and beyond to further chip away at learning losses. Our research shows that summer programs can have positive if modest benefits, halting learning losses and reducing achievement gaps. Ministries of Education are recognizing that summer supplements are relatively inexpensive, can be high‐quality, can concentrate their instruction on one or two key subject areas, and are popular with students and parents. Our qualitative research also found that summer programs had secondary benefits such as strengthening home‐school relationships. We also recommend that they experiment with some 6‐week programs, which have been found to generate greater gains, at least in the short run (National Summer Learning Association, 2009; McCombs et al., 2020).

Our second recommendation is that school online instruction models build in more real‐time interactions between students and teachers, particularly if additional COVID‐19 ‘waves’ force more students to move online beyond the 2020‐2021 school year. Ministries of Education responded to the COVID‐19 pandemic in the fall of 2020 with a wide range of delivery models, modifying their classroom routines, altering timetables to support social distancing, staggering re‐openings, providing blended options, and prioritizing certain students and subjects (People for Education, 2020). If school closures continue to be a reality (either as a full‐time option or intermittently), surveys suggest that students and parents want their teachers to be available and ‘present’ during the school day (e.g., Hamilton‐Wentworth District School Board, 2020). More online ‘live’ interaction between teachers and students can bridge learning gaps. Overall, our recommendations suggest a need to move from an emergency response mindset to learning how to effectively live and school with COVID‐19 in the longer term (e.g., Jones, 2020).