Estimating the Effects of Non-Pharmaceutical Interventions and Population Mobility on Daily COVID-19 Cases: Evidence from Ontario.

This study uses coronavirus disease 2019 (COVID-19) case counts and Google mobility data for 12 of Ontario's largest Public Health Units from Spring 2020 until the end of January 2021 to evaluate the effects of non-pharmaceutical interventions (NPIs; policy restrictions on business operations and social gatherings) and population mobility on daily cases. Instrumental variables (IV) estimation is used to account for potential simultaneity bias, because both daily COVID-19 cases and NPIs are dependent on lagged case numbers. IV estimates based on differences in lag lengths to infer causal estimates imply that the implementation of stricter NPIs and indoor mask mandates are associated with reductions in COVID-19 cases. Moreover, estimates based on Google mobility data suggest that increases in workplace attendance are correlated with higher case counts. Finally, from October 2020 to January 2021, daily Ontario forecasts from Box-Jenkins time-series models are more accurate than official forecasts and forecasts from a susceptible-infected-removed epidemiology model. © Canadian Public Policy / Analyse de politiques.

Introduction

With the enactment of stringent restrictions on public mobility and rising vaccination rates, all Canadian provinces began to experience a downward trend in daily coronavirus disease 2019 (COVID-19) cases from June 2021 onward. A limited number of studies have used econometric modelling to evaluate the effects of non-pharmaceutical interventions (NPIs) on daily cases across Canadian provinces and sub-provincial jurisdictions.[1] Before vaccines, public health officials maintained that reduced social contact, mobility, and access to businesses would be the best way to reduce COVID-19 cases. This article provides some evidence about the possible magnitude of these effects.

This study uses a policy stringency index developed by Karaivanov et al. (2021) to evaluate the effects of NPIs and population mobility on daily COVID-19 cases from 2 April–30 September 2020 and across the 12 largest public health units (PHUs) in Ontario.[2] Using PHU-level data enables an evaluation of the effects of business closures and restrictions on public gatherings while controlling for potentially confounding unobserved jurisdiction-specific and time-invariant characteristics. Publicly available Google data are used to estimate the effects of population mobility on daily new cases. Although NPIs reduce the spread of infections through decreases in population mobility, it is important to study the impacts of overall social mobility on daily case counts because the effects of stricter policies on population movements may diminish over time with lower public compliance. Moreover, the use of Google data enables an assessment of the effects of public mobility to destinations commonly frequented by individuals and households, such as grocery and retail stores and workplaces. Matching these mobility measures to corresponding trends in COVID-19 cases should be useful to policy-makers in deciding specific types of economic and social lockdowns, because a paucity of knowledge exists on which types of population mobility are the most responsible for the spread of COVID-19.

The challenge with identifying causal policy effects in this exercise is that ordinary least squares estimates might be confounded and biased downward, because increases in daily cases are also likely to lead to more stringent policies. We attempt to identify a causal interpretation by using lagged cases as instruments, under the assumption that the policy implementation lag is likely longer than the disease transmission lag. Specifically, although current daily cases are affected by recent daily trends, the impact of successive daily case counts on current case counts should diminish over time. However, there is a higher probability that the implementation of stricter restrictions on population mobility in response to surges in daily cases is not as immediate and takes a longer time period to occur. Although we do not claim that such identification is unimpeachable, standard statistical tests of instrument strength and of overidentification yield statistics suggesting that the approach may have some validity, and the resulting instrumental variable (IV) coefficients indeed suggest stronger policy effects than corresponding single equation estimates.

Some studies have used Google mobility data to understand the spread and propagation of COVID-19 cases in Canada.[3] The study most similar to ours is that of Karaivanov et al. (2021), who use data across Ontario PHUs and Canadian provinces to estimate the impact of mask mandates and other NPIs on COVID-19 case growth in Canada. Similar to our research, they attempt to account for behavioural responses by using Google mobility data. However, they average values across different Google mobility measures and focus on the effects on case growth rates as opposed to the incidence of daily cases. Chu and Qureshi (2020) study the relationship between COVID-19 confirmed cases and Google mobility patterns by province and state level in Canada and the United States. They find evidence of a lagged relationship between Google mobility indicators and case counts. However, it is not clear which types of social mobility are the most responsible for variation in daily cases because they only consider an aggregate measure of mobility, rather than each individual Google mobility index. Moreover, both Karaivanov et al. (2021) and Chu and Qureshi (2020) do not use IVs to account for possible simultaneity bias. Sen (forthcoming) focuses on the lagged effects of different Google social mobility indicators through individual time-series regressions for different Ontario health regions as opposed to pooling data across jurisdictions and over time.

Finally, this study also contributes to the evolving literature on forecasting daily COVID-19 cases by investigating the predictive power of Google population mobility indicators. Although different research institutes offer long-term forecasts based on epidemiological models, the amount of corresponding research on short-term predictions is much more limited. Altieri et al. (2021), Bryant and Elofsson (2020), and Liu, Moon, and Schorfheide (2020) are examples of research that has focused on constructing models generating one- and two-week-ahead forecasts. However, the methods used in these studies are computationally intensive, involving either different types of linear and exponential predictors or Bayesian methods that are not easily replicable or interpretable. Chu and Qureshi (2020) find that time-series models with a quartic trend function can generate comparable short-term out-of-sample forecasts for one- to seven-day logarithmic case counts relative to the classic epidemiological susceptible, infected, and recovered (SIR) approach.[4] Chen et al. (2021) use smooth transition autoregressive models, neural network (NN) models, and a SIR model to predict cumulative daily case counts for Ontario, Alberta, British Columbia, and Quebec and find that NN models outperform other approaches in terms of prediction accuracy. However, these studies do not investigate the usefulness of mobility patterns in generating accurate forecasts. This research evaluates the efficacy of a wide range of Box–Jenkins models (Box et al. 2015). The models used here should be useful for policy purposes because they are easily interpreted and implementable through standard statistical software packages such as R, STATA, IBM SPSS, and Excel.

In terms of primary findings, weighted least squares (WLS) estimates of the COVID-19 policy index are, in most cases, statistically insignificant. However, the corresponding IV estimate is statistically significant. Moreover, increases in the policy index and the implementation of mandatory indoor mask mandates are correlated with reductions in social mobility. Hence, stricter policies may also have an indirect impact on lowering daily cases through decreasing population mobility. The coefficient estimate of mandatory mask mandates is also statistically significant in the IV regression. Both WLS and IV regressions reveal a robust and statistically significant association between increases in workplace mobility and daily COVID-19 cases. With respect to prediction, we construct one-week-ahead forecasts of daily COVID-19 cases from 1 October 2020 to 31 January 2021, updating our models and parameter estimates on a weekly basis. Autoregressive integrated moving average models conditioned on weekly seasonality are able to predict daily COVID-19 cases in Ontario with good accuracy, because our daily forecasts differ on average from the actual daily case numbers by roughly 10 percent. In contrast, predictions generated by a SIR model have an average forecast error of roughly 39 percent.

The remainder of the article is structured as follows. The next two sections discuss the data and the results; the final section concludes with a summary of key findings and policy implications.

Data

Google Mobility Indicators

The mobility data being used in this research have been extracted from the location history associated with Google Maps app use. The information has been passively generated and collected and is now being made available for use by researchers and policy-makers through Google (2021).

The Google mobility data capture total visits to the following specific destinations commonly frequented by individuals and households: (a) grocery and pharmacy stores, which include grocery markets, food warehouses, farmers markets, specialty food shops, drug stores, and pharmacies; (b) parks, which encompass local parks, national parks, public beaches, marinas, dog parks, plazas, and public gardens; (c) transit stations, which consist of subway, bus, and train stations; (d) retail stores and recreation outlets, which includes places such as restaurants, cafes, shopping centers, theme parks, museums, libraries, and movie theaters; and (e) workplaces. With respect to places of residence, Google social mobility represents duration of stay.

As detailed on its website, Google creates these aggregated and anonymized sets of data from users who have turned on the location history setting for Google accounts on their phones and have agreed to share this information. Consequently, a limitation to acknowledge is that Google data on social mobility trends are based on a sample of users who own mobile devices and who choose to share their location history. These data may therefore not be representative of the population. In addition, Google has not made public its precise methodology for calculating social mobility. Hence, there is some ambiguity about the extent to which Google data are representative of population-level trends. However, data on the number of people using Google Maps in the United States indicates that Google social mobility indicators might be fairly representative of overall population trends.[5]

Daily values are aggregations across individuals who have enabled their location history and are available for each province in Canada from 15 February onward. These values are calculated relative to a baseline, which is defined as the median for the corresponding day of the week, during the five-week period 3 January–6 February 2020. Hence, each daily value is the percentage change in the social mobility category relative to its baseline, which shows how visits and length of stay at different destinations have changed since the onset of the pandemic. A visual inspection is useful to evaluate whether trends in social mobility correspond with intuition. A caveat is that although data for all Google social mobility indicators are available at the province level, this is not the case for PHUs, with many missing values for park, transit, and residences.

Source: Google (2021).

Figure 1 displays trends in the different social mobility categories for Ontario from 2 April 2020 to 31 January 2021. Grocery and pharmacy mobility increased over the sample period along with mobility at retail and recreational venues. However, a difference is the decline in retail social mobility from August onward. Residential duration of stay was initially high and then fell during the latter part of the sample period. Unsurprisingly, movements in work and transit mobility are significantly correlated, with both indicators increasing over time. The top spikes in both of these variables are mobility values during the weekend, which did not significantly decline relative to pre-pandemic observations. Finally, the sharp rise and fall of social mobility at parks reflects outdoor activities in the warmer months. In addition, the time-series visualized in Figure 1 all show a strong seasonal day-of-week effect, meaning that every seventh observation is highly correlated.

Figure 1:

Google Social Mobility Data for Ontario from 2 April 2020 to 31 January 2021

Trends in Daily COVID-19 Cases

Figure 2 depicts trends in daily cases over the same period. These data have been made publicly available by the Ontario provincial government via its online data-sharing website.[6] Ontario is the only province in Canada, and one of the few jurisdictions in the world, that publishes daily COVID-19 case data based on date of specimen collection. This is an important qualification to using social mobility data to capture trends in population movements in COVID-19 cases. Specifically, relying exclusively on daily case data constructed using date of confirmation of test results might lead to misleading estimates of the relationship between social mobility and daily cases if there are significant and inconsistent delays in the release of test confirmations.

Figure 2:

Ontario New Daily COVID-19 Case Counts

As can be seen in Figure 2, the number of new cases each day steadily increased in Ontario until the second week of April when the number began to decrease. This decline continued until mid-August, after which a sharp increase in cases began to occur—an increase that continued through the remainder of 2020. This sharp increase, however, was followed by an even sharper decline early in the new year. Increases in mobility seen throughout the summer preceded the corresponding increase in cases we observe in the late summer and into the fall. Likewise, the decline in mobility in the fall preceded the decline in cases observed early in the winter. This suggests that past social mobility information may be useful in forecasting future COVID-19 case counts. From a modelling perspective, we see some notable structure that, when accurately accounted for, may be exploited for purposes of predicting daily new COVID-19 case counts. For instance, the general pattern of increases and decreases just discussed represents a strong non-linear trend that should be accounted for. In addition, just like the Google mobility data, we see a strong seasonal day-of-week effect. Accounting for the weekly seasonality exhibited by both these data and the Google mobility data will be very important. Note that we also observe an increase in volatility in daily case counts as time passes.

Note: COVID-19 = coronavirus disease 2019.

Source: Public Health Ontario (2021).

Our regression analyses exploit differences in daily cases across the 12 largest PHUs in Ontario. In particular, we have data for the following 12 PHUs (with population size in parentheses): Durham (645,862), Hamilton (1,399,073), Halton (548,430), Middlesex–London (455,526), Ottawa (1,306,249), Niagara (447,888), Peel (1,381,744), Simcoe–Muskoka (540,249), Waterloo (535,154), Windsor (398,953), Toronto (2,731,571), and York (1,109,909). Cumulatively, these health units account for more than 85 percent of the province’s population. Other PHUs have much smaller populations and did not experience a significant number of COVID-19 cases. In terms of sample means of daily cases for 2 April–30 September 2020, the PHUs are ranked as follows (with the mean of daily cases in parentheses): Toronto (100.41), Peel (50.86), York (22.06), Ottawa (16.96), Windsor (13.344), Durham (11.273), Waterloo (8.9126), Halton (6.169), Hamilton (5.781), Niagara (5.24), Simcoe–Muskoka (4.754), and Middlesex–London (4.01).

Policy Variables

The effects of NPIs at the province level are measured through the Bank of Canada (BOC) Policy Stringency Index created by Cheung et al. (2021). This index is based on the methodology of the Oxford COVID‑19 Government Response Tracker developed by the University of Oxford’s Blavatnik School of Government.[7] The index is comprehensive in capturing different policies aimed at restricting public mobility and includes school and university closures, workplace and office closures, public event cancellations and restrictions, restrictions on private gatherings, public transport closures, stay-at-home requirements, restrictions on intra-provincial travel (between cities or regions within a province), restrictions on interprovincial travel (between provinces), restrictions on international travel, enforcement mechanisms for individuals, enforcement mechanisms for firms, and public information campaigns.[8] The index ranges from 0 (no restrictions) to 1 (maximum restrictions). We use this index in our forecasting of Ontario-level daily cases.

To estimate the effects of NPIs at the PHU level, we use the policy stringency index created by Karaivanov et al. (2021) for Ontario. Karaivanov et al. (2021) were able to compile restrictions on business operations and compute values (ranging from 0, no restrictions, to 1, highest level of restrictions) capturing the intensity of restrictions for businesses and gatherings. Their index captures restrictions on non-essential and retail business; personal services business; restaurants, bars, and nightclubs; places of worship; events and gatherings; and recreation, gyms, and parks.

As noted, much of our estimation uses the period 2 April–30 September 2020. Accordingly, Table 1 gives some summary statistics for Ontario aggregate data over that period, and Figure 3 decomposes time-series variation in the PHU index for the largest regions of Durham, Toronto, Peel, York, Ottawa, and Hamilton, as well as the corresponding values for the BOC Policy Stringency Index for Ontario.

Table 1:

Summary Statistics 2 April–30 September 2020

Name of Variable	Mean (SD)	Variance	Min-Max
Total daily cases	21.479 (37.660)	1,418.3	0.10000E–08–324.00
7-day lag mask mandate dummy	0.39526 (0.48902)	0.239	0.0000–1.0000
7-day lag policy stringency index	0.566 (0.33985)	0.1155	0.18750–0.99375
7-day lag temperature	16.017 (7.6752)	58.908	−3.9000–29.500
Tuesday dummy	0.14208 (0.34921)	0.122	0.0000–1.0000
Wednesday dummy	0.1475 (0.35473)	0.12583	0.0000–1.0000
Thursday dummy	0.14208 (0.34921)	0.12195	0.0000–1.0000
Friday dummy	0.14208 (0.34921)	0.12195	0.0000–1.0000
Weekend dummy	0.31694 (0.4654)	0.2166	0.0000–1.0000
Google mobility indicators
7-day lag retail and recreation	−30.901 (18.834)	354.72	−86.000–33.000
7-day lag grocery and pharmacy	−7.9167 (14.985)	224.55	−83.000–48.000
7-day lag work	−42.099 (19.657)	386.40	−89.000–5.000

Source: Authors’ computations using data available at Google (2021) and Public Health Ontario (2021).

Figure 3:

Large Public Health Units and Bank of Canada Policy Stringency Index

Figure 4 depicts movements in the PHU index for the smaller regions of Halton, Middlesex–London, Niagara, Waterloo, Simcoe–Muskoka, and Windsor. As can be seen in Figure 3, the PHU index is identical for Durham and Hamilton and for Peel, Toronto, and York. There is time-series variation for all PHUs as captured by the loosening of restrictions on mobility throughout the sample period. There is also variation across PHUs with relaxation in restrictions for Peel and Toronto in late June and July, after the lifting of restrictions on mobility in other health regions. The BOC Policy Stringency Index follows a similar decline through time and is highly correlated with the PHU index for Toronto (Pearson correlation coefficient of 0.95). Figure 4 shows a similar relaxation of mobility restrictions for smaller PHUs over the sample period. Halton and Niagara have identical trends, as do London, Waterloo, and Simcoe–Muskoka. For these health regions, lifting of policy restrictions occurs later in Windsor relative to other health regions.

Figure 4:

Policy Stringency Index for Smaller Public Health Units

We also construct a dummy variable to represent the implementation of mask mandates in indoor settings. There is time-series variation in their enactment across PHUs. The enactment dates are as follows: Durham, 10 July 2020; Halton, 22 July 2020; Hamilton, 20 July 2020; Middlesex–London, 18 July 2020; Niagara, 31 July 2020; Ottawa, 7 July 2020; Peel, 10 July 2020; Waterloo, 13 July 2020; Simcoe–Muskoka, 13 July 2020; Toronto, 7 July 2020; Windsor–Essex, 26 June 2020; and York, 17 July 2020.[9]

Sources: Karaivanov et al. (2021) and Cheung et al. (2021).

Source: Karaivanov et al. (2021).

Results

Effects of NPIs on Population Mobility

Table 2 reports results of basic WLS regressions with retail and recreational, grocery and pharmacies, and workplace Google mobility indicators as dependent variables.[10] The motivation is to explore the impacts of policy stringency on mobility. In this respect, the local COVID-19 Policy Stringency Index may share a different relationship with population mobility measures, relative to indoor mask mandates. Specifically, increases in the stringency index should be associated with reductions in social mobility because it captures restrictions on public gatherings and access to businesses. The implementation of mask mandates in indoor settings might also be correlated with lower public mobility if individuals view such regulation to be indicative of a heightened risk of infection to the public. However, if individuals feel safer with mask mandates, it is possible that such regulation will result in increased public movements.[11]

Table 2:

Estimates of the Effects of NPIs on Daily Google Mobility across Ontario PHUs

	(1)	(2)	(3)
Explanatory Variables	Retail Mobility	Groceries and Pharmacies Mobility	Workplace Mobility
1-day lagged dependent variable	0.191[***]	−0.003	0.175[***]
	(0.019)	(0.0195)	(0.020)
2-day lagged dependent variable	0.346[***]	0.275[***]	−0.572[***]
	(0.019)	(0.019)	(0.026)
Local COVID-19 Policy Stringency Index	−20.658[***]	−18.977[***]	−36.646[***]
	(1.223)[***]	(1.457)	(1.813)
Mask mandate dummy	−2.372[***]	−4.412[***]	−3.705[***]
	(0.498)	(0.741)	(0.940)
Average daily temperature	0.139[***]	0.237[***]	0.356[***]
	(0.026)	(0.038)	(0.046)
PHU dummies	Yes	Yes	Yes
Day of week dummies	Yes	Yes	Yes
Adjusted R2	0.8994	0.6418	0.7185

Notes: The estimates in this table are based on data from 12 Public Health Units (PHUs) between 2 April-30 September 2020. The dependent variables are different Google mobility variables. Regression estimates are obtained from Weighted Least Squares (WLS) regression where observations are weighted by PHU specific population. Standard errors are in parentheses below coefficient estimates. NPIs = non-pharmaceutical interventions; PHUs = public health units; COVID-19 = coronavirus disease 2019.

* p = 0.1;

** p = 0.05;

p = 0.01.

Sources: Data compiled by the authors from Public Health Ontario (2021), Google (2021), Karaivanov et al. (2021), and Canada (2021) for daily temperatures.

The results reveal that, controlling for other factors, both stricter policies on mobility and mask mandates are significantly correlated (at the 1 percent level) with reduced mobility as measured by all Google variables. In most cases, the lagged dependent variables are statistically significant. The coefficient estimate of the average temperature covariate is positive and also statistically significant, which reflects the association between warmer temperatures and higher social mobility. Although the underlying model is simple, the adjusted R2 is above 0.7 with respect to retail and workplace mobility. Given that an objective of stricter COVID-19 policies is to reduce public mobility, these results suggest that these specific initiatives were successful.

Effects of NPIs and Population Mobility with PHU Data

Table 3 contains WLS regression results based on data pooled across 12 PHUs and over time. The dependent variable is the number of daily cases. Column (1) contains estimates of the local Policy Stringency Index conditioned on one- and two-day lagged cases, PHU dummies, and day-of-week dummies. Column (2) adds the mandatory mask dummy, one-week lagged Google mobility, and average temperature variables, and Column (3) adds three-, four-, five-, six-, and seven-day lagged dependent variables to assess the fit of a more dynamic specification.[12] Column (4) contains second-stage IV results where the Policy Stringency Index is instrumented by 12-, 13-, 14-, and 15-day lags of daily cases.

Table 3:

Estimates of the Effects of NPIs on Daily COVID-19 Cases and Google Mobility across Ontario PHUs

	(1)	(2)	(3)	(4)
Explanatory Variables	WLS	WLS	WLS	IV
7-day lagged local COVID-19 Policy Stringency Index	−2.313[***]	0.178	−1.746	−53.427[***]
	(0.880)	(3.084)	(3.030)	(21.77)
7-day lagged mask mandate dummy variable		−0.664	0.037	−12.255[***]
		(1.177)	(1.154)	(5.004)
1-day lagged cases	0.629[***]	0.611[***]	0.549[***]	0.554[***]
	(0.020)	(0.020)	(0.022)	(0.023)
2-day lagged cases	0.363[***]	0.355[***]	0.234[***]	0.400[***]
	(0.021)	(0.021)	(0.025)	(0.023)
3-day lagged cases			0.177[***]
			(0.026)
4-day lagged cases			−0.027
			(0.026)
5-day lagged cases			0.042
			(0.026)
6-day lagged cases			−0.082[***]
			(0.025)
7-day lagged cases			0.102[***]
			(0.022)
7-day lagged retail mobility		0.281[***]	0.206[***]	−0.801[***]
		(0.087)	(0.086)	(0.378)
7-day lagged grocery and pharmacy mobility		−0.26[***]	−0.202[***]	0.384[***]
		(0.063)	(0.062)	(0.203)
7-day lagged workplace mobility		0.102[***]	0.122[***]	0.0598[***]
		(0.027)	(0.027)	(0.022)
Average daily temperature		−0.396[***]	−0.362[***]	−0.534[***]
		(0.068)	(0.066)	(0.140)
F statistic (p-value) of joint significance of instruments (12-, 13-, 14-, and 15-day lags of daily COVID-19 cases)				13.763
				(0.000)
Sargan Test for overidentifying restrictions				1.142
				(0.331)
PHU dummies	Yes	Yes	Yes	Yes
Day-of-week dummies	Yes	Yes	Yes	Yes
Adjusted R2	0.9451	0.9466	0.9489	0.9151

Notes: The regressions in this table are based on data from 12 PHUs between 2 April and 30 September 2020. The dependent variable is the total number of daily cases. Regression estimates in Columns (1)–(3) are obtained from WLS regression where observations are weighted by PHU-specific population, whereas Column (4) contains IV estimates where the 7-day lagged local COVID-19 Policy Stringency Index is instrumented by 12-, 13-, 14-, and 15-day lags of daily COVID-19 cases. Standard errors are in parentheses below coefficient estimates. NPIs = non-pharmaceutical interventions; COVID-19 = coronavirus 2019; PHUs = public health units; WLS = weighted least squares; IV = instrumental variables

* p = 0.1;

** p = 0.05;

p = 0.01.

Source: Data compiled by the authors from Public Health Ontario (2021), Google (2021), Karaivanov et al. (2021), and Canada (2021) for daily temperatures

The motivation for using IV analysis is to account for the possibility that single equation estimates of the local Policy Stringency Index might be biased downward by simultaneity bias. Single equation models assume that changes in policy exogenously affect daily cases.[13] However, changes in historical daily case trends may also influence the enactment or easing of more stringent policies aimed at restricting public mobility. The 12- to 15-day lagged dependent variables we use are far enough in the past that it can be argued that they should not be strongly correlated with the current daily COVID-19 cases. We used multiple lagged values to lessen the possibility of a spurious correlation and to enable a test of overidentifying restrictions.[14]

The coefficient estimate of the local Policy Stringency Index is negative and statistically significant at the 1 percent level in Column (1) but becomes insignificant in Columns (2) and (3) with the addition of the mask mandate dummy and other control variables. One- and two-day lags of the dependent variable are positive and statistically significant at the 1 percent or 5 percent levels in all columns. The mask mandate dummy is statistically insignificant in Columns (2)–(3). In Column (3), the three-, six-, and seven-day lags are statistically significant at the 1 percent level. Coefficient estimates of seven-day lagged retail mobility are statistically significant at the 5 percent level in Columns (2) and (3) and imply that a 10-percentage-point mobility is correlated with, on average, a roughly two to three daily case increase across PHUs. Coefficient estimates of groceries and pharmacies are significant in Columns (2) and (3), with negative and positive signs in these columns, respectively. Results in the same columns imply that a 10-percentage point rise in work mobility is associated with an increase of approximately one daily case (statistically significant at the 1 percent level). Higher temperatures are significant at the 1 percent level and possess the expected negative signs; an increase in temperatures should result in more outdoor and socially distanced mobility and, therefore, fewer cases.

Results in Column (4) confirm the possibility that WLS estimates of the Policy Stringency Index are likely biased downward by simultaneity bias, because the IV coefficient estimate of the Policy Stringency Index is negative and statistically significant (at the 5 percent level). The F-statistic and p-value of the test of joint significance of the instruments (reported in the table) enable us to reject the null hypothesis that coefficient estimates of 12- to 15-day lags in the dependent variable from the first stage regression are zero. Moreover, the use of multiple instruments allows us to use a Sargan test for overidentifying restrictions. As reported in Table 3, we could not reject the null hypothesis. Nonetheless, the statistical significance of the IV estimate should be treated with caution because it is based on a specific set of lagged dependent variables.

With respect to other findings in Column (4), the mask mandate dummy is negative and statistically significant at the 5 percent level. The lagged grocery and retail mobility variables are statistically significant at either the 1 percent or the 5 percent level. However, their signs are reversed relative to the corresponding WLS estimates. Hence, this article does not provide evidence that retail mobility increases daily cases. In contrast, the coefficient estimate of workplace mobility remains positive and significant at the 1 percent level.[15] In summary, the IV results imply that more stringent policies—as measured through the local Policy Stringency Index and the mask mandate dummy—are correlated with lower daily cases. As demonstrated by the results in Table 2, stricter policies on social mobility are also associated with reduced population movements, which are in turn correlated with lower daily case counts.

Forecasting

Forecasting at the Ontario Level

Given that the time-series of case counts in Ontario exhibits non-stationarity and strong weekly effects, we use Box–Jenkins models for their ability to flexibly model and forecast complex correlation structure. We specifically consider pure seasonal autoregressive integrated moving average (SARIMA) models (which model daily new COVID-19 case data as a function of historical daily cases only) as well as SARIMA models augmented with all Google mobility variables and the BOC Policy Stringency Index.[16] These latter models may also be thought of as regression with SARIMA errors. We should note that in the context of SARIMA models, a seasonal effect is one that recurs predictably with some fixed frequency, independent of the specific frequency, which could be weekly, monthly, or quarterly. Hence, SARIMA models are appropriate to control for the day-of-the-week trends that we observe in our cases and Google mobility data.

We choose training and forecasting periods to avoid the confounding effects of public vaccination programs. Specifically, we partition the available data into training and testing sets where the training data (2 April–30 September 2020) are used to fit the model, and the test data (1 October 2020–31 January 2021) are used to evaluate the accuracy of the model’s forecasts. This partition has also been chosen for illustration because it showcases the model’s ability to accurately forecast the pronounced increase in cases that began in September 2020 as well as the sharp decline that followed in January 2021. Further details of our SARIMA modelling are available in the Appendix.

Figure 5 visualizes the fit and forecasts of a SARIMA(1,1,2)(1,1,2)[7] model without any exogenous information, over the training and testing time periods.[17] The shaded regions represent 95 percent prediction intervals, and the vertical dashed line separates training from testing data. As evidenced by Figure 5, the model fits and forecasts the data very well. We quantify this good performance using the mean absolute error (MAE), which calculates, on average, the absolute difference between a forecast (blue line) and the true count (black line). For this model and these data, we have MAE = 172.8855, meaning that on a typical day, our forecasted daily case count is roughly 173 away from the truth. Quantified another way, the mean absolute percent error (MAPE) is 9.74 percent.

Figure 5:

Observed and Forecasted Daily New COVID-19 Case Counts in Ontario

We may also quantify the efficacy of the methodology by considering the accuracy of the interval forecasts. In particular, we observe whether the prediction intervals (grey shaded area) contain the true daily case count (black line). An especially important time frame to consider is the first week of January. During this time, Ontario saw a dramatic change from increasing to decreasing case numbers in a matter of one week. The blue line (which deviates more than usual from the black line in this time frame) indicates that the model’s forecasts did not immediately predict the sudden downward trend; it was not until the second and third weeks of January that the forecasts re-aligned with actual case numbers. However, the 95 percent prediction intervals captured this sudden and dramatic change in trend, indicating the value of accurate interval estimates.[18]

Although not depicted visually, the predictive accuracy associated with SARIMA models that do include the Google mobility variables and the BOC Policy Stringency Index perform similarly. The top section of Table 4 reports corrected Akaike information criterion (AIC), MAE, and MAPE values for four versions of SARIMA: the pure SARIMA model depicted in Figure 5; the SARIMA model that includes the BOC Policy Stringency Index but not the Google mobility variables, the SARIMA model that includes the Google mobility variables but not the BOC Policy Stringency Index, and the SARIMA model that includes the Google mobility variables and the BOC Policy Stringency Index. The results indicate that the pure SARIMA model is (strictly speaking) superior but that including the exogenous variables does not drastically worsen performance. Although not included here, these conclusions generalize to different train–test partitions.

Table 4:

Comparison and Evaluation of PHU Forecasting Models (1 October 2020–31 January 2021)

Jurisdiction and Model	AICC	MAE	MAPE
Ontario (aggregate)
SARIMA	−145.45	172.89	0.0974
SARIMA + STR	−130.74	193.66	0.1068
SARIMA + GM	−129.81	187.17	0.1029
SARIMA + GM + STR	−127.55	198.43	0.1091
Durham
SARIMA	282.77	14.93	0.2105
SARIMA + STR	273.32	14.22	0.2054
SARIMA + GM	272.00	16.24	0.2342
SARIMA + GM + STR	274.14	15.65	0.2231
Hamilton
SARIMA	372.65	20.17	0.3167
SARIMA + STR	300.80	16.88	0.2688
SARIMA + GM	334.04	17.70	0.2852
SARIMA + GM + STR	334.91	17.00	0.2727
Ottawa
SARIMA	269.75	17.41	0.2510
SARIMA + STR	206.14	18.65	0.2857
SARIMA + GM	205.74	15.83	0.2418
SARIMA + GM + STR	208.04	16.28	0.2503
Peel
SARIMA	82.41	48.55	0.1468
SARIMA + STR	74.23	51.32	0.1566
SARIMA + GM	74.87	49.14	0.1451
SARIMA + GM + STR	75.27	49.19	0.1485
Toronto
SARIMA	−29.7	71.11	0.1415
SARIMA + STR	−14.14	69.19	0.1367
SARIMA + GM	−14.02	66.23	0.1347
SARIMA + GM + STR	−11.88	67.77	0.1379
York
SARIMA	176.89	28.01	0.1622
SARIMA + STR	174.05	25.89	0.1559
SARIMA + GM	209.5	32.3	0.2046
SARIMA + GM + STR	210.55	31.7	0.1970

Notes: PHU = Public Health Unit; AICC = Akaike information criterion; MAE = mean absolute error; MAPE = mean absolute percent error; SARIMA = seasonal autoregressive integrated moving average; STR = Bank of Canada COVID-19 Policy Stringency Index; GM = Google mobility.

Sources: Data compiled by the authors from Public Health Ontario (2021), Google (2021), and Cheung et al. (2021).

Note: COVID-19 = coronavirus disease 2019.

Sources: Authors’ estimates using data available at Public Health Ontario (2021), Google (2021), and Cheung et al. (2021).

Forecasting at the Public Health Unit Level

We also investigated using a SARIMA model like the one specified in the Appendix to forecast daily cases in each of the PHUs individually.[19] However, because of missing data associated with the parks, transit, and residential Google mobility variables at the PHU level, we omit them and focus on retail and recreation mobility, grocery and pharmacy mobility, and workplace mobility. Such models proved to be ineffective for small PHUs with relatively low case counts. As such, we present here the results only for the largest six PHUs: Durham, Toronto, Peel, York, Ottawa, and Hamilton. SARIMA models for the daily cases in each of these PHUs are depicted in Figure 6, and their prediction accuracy is quantified in the bottom six sections of Table 4.

Figure 6:

Observed and Forecasted Daily New COVID-19 Case Counts in PHUs: (a) Durham, (b) Hamilton, (c) Ottawa, (d), Peel, (e) Toronto, and (f) York

For illustration, the models depicted in Figure 6 are SARIMA models that include Google mobility variables but not the Policy Stringency Index. Unfortunately, at the regional level there is not one model specification that is uniformly superior to the others across all jurisdictions, but we see that including either the Google mobility variables or the stringency index is advisable. This is in contrast to forecasts with the Ontario-level data, in which the policy variable does not seem to be important. The forecast errors with these models are on the order of 15 percent to 30 percent, with values of 14 percent to 15 percent for Toronto and Peel and roughly 20 percent for York, which also happen to be the worse-hit PHUs in the province with respect to daily COVID-19 case counts.

A valid question is how our forecasts compare against corresponding government projections. Through the Ontario COVID-19 Science Advisory Table (2020), the province of Ontario collects information and data on COVID-19 health impacts as well as projections of daily cases that are compiled by different experts and researchers, which are also released to the public. Unfortunately, these public briefs do not offer specific numerical daily forecasts but only time trends through graphs. The forecasts are based on a fixed daily percentage increase in COVID-19 cases. Projections available from Ontario (2020) specifically indicate a belief that daily case counts could reach more than 1,000 cases a day during mid-October.[20] On the basis of actual daily cases, this is consistent with a roughly 3.5 percent daily increase in reported cases from mid-September to mid-October. This daily increase results in an absolute forecast error of 16 percent with respect to daily case predictions. Over the same time period, our SARIMA models with Google mobility variables and the BOC Policy Stringency Index produce forecasts over the same time period with forecast errors of roughly 18 percent.

Notes: COVID-19 = coronavirus disease 2019; PHUs = public health unit.

Sources: Authors’ estimates using data available at Public Health Ontario (2021) and Google (2021).

However, the SARIMA model with exogenous variables produces much more accurate daily forecasts between 16 November 2020 and 15 December 2020 compared with predictions generated by the Ontario COVID-19 Science Advisory Table (2020) for this time period. In this report, specific daily case growth rates of 3 percent and 5 percent are assumed from mid-November to 2 December 2020. When compared with actual daily cases, the results have MAPE values of roughly 22 percent and 69 percent, when assuming 3 percent and 5 percent growth rates in daily case counts. Our model produces daily forecasts with a much lower error of approximately 10 percent with Google mobility variables and the BOC Policy Stringency Index. Hence, for this time period, the SARIMA forecasting model with exogenous variables offers more accurate predictions relative to available government projections.

Comparison with a Susceptible, Infected, and Recovered Model at the Ontario Level

As a final sensitivity exercise, we evaluate the performance of the SARIMA model by constructing forecasts from a SIR model over the same period. Despite the widespread use of Box–Jenkins methods for forecasting, SIR models are the dominant methodology to model the spread of epidemics.[21] The SIR model uses a differential equations approach to model changes in the number of infections (I), by incorporating population size (N), the susceptibility of the population to the disease (S), and recovery rates (R). Infections (I) are calculated on the basis of daily cases, and recovery (R) is counted as the number of daily recovered and deceased individuals.[22] S is calculated as a function of β, which is the average number of contacts per infectious person per time unit. We specifically use the approach detailed in Chen et al. (2021). As is the case with our SARIMA modelling, the parameters of the SIR model are updated weekly, and the model is used to construct week-ahead daily predictions. Figure 7 visualizes SIR forecasts for Ontario from 1 October 2020 to 31 January 2021 against predictions generated from the SARIMA model with no exogenous variables and a “naïve model” in which the daily forecast is the average case count in the previous seven days. On average, the SIR model performs poorly against SARIMA, because the MAPE over the testing period is approximately 39 percent. In particular, we note that SARIMA is able to predict the downward trend in daily cases during January, whereas the SIR model forecasts a continuing rising trend. Even the MAPE from the naïve model at roughly 16 percent is lower than the MAPE of SIR forecasts.

Figure 7:

Forecasted Daily New COVID-19 Case Counts by Model (Ontario)

Conclusion

This article studies the effects of NPIs in the form of policy restrictions on businesses and public gatherings and population mobility on daily cases in the 12 largest PHUs in Ontario. These estimates are conditioned on the use of Google mobility data, which are intended to control for the magnitude of population-level movements. Given declining daily case counts across the country and increases in vaccination rates, it is important to gain an understanding of the effects of government policies and public mobility on COVID-19 cases during a time period in which cases were rising rapidly and vaccines were unavailable.

Results from IV regressions based on PHU-level data demonstrate that stricter policies are correlated with reductions in daily COVID-19 case counts. Increases in the local Policy Stringency Index and the enactment of mask mandates are associated with reduced public mobility. We observe a statistically significant positive correlation between mobility at workplaces and daily cases. This is unsurprising given recent evidence that for some regions, surges in COVID-19 infections are associated with congested workplaces, prompting the Toronto and Peel regions to ask all businesses with five or more employees to shut down for a 10-day period (Herhalt 2021; Marotta 2020). We note that the coefficient estimate of mask mandates is statistically significant in our IV regression. Caution should be used in interpreting this result, given that we did not instrument the mask dummy variable. However, the importance of mask mandates cannot be dismissed given the robust correlation between such regulation and reduced case growth obtained by Karaivanov et al. (2021). A conservative interpretation of the effects of indoor mask mandates can be obtained by contextualizing the IV coefficient of −12.25 against the mean of daily cases in the Toronto and Peel regions, the two PHUs with the highest sample means of 100.41 and 50.86, respectively. These summary statistics imply that mask mandates are associated with roughly 12 percent and 24 percent declines in daily cases in the Toronto and Peel regions, respectively.

Notes: COVID-19 = coronavirus disease 2019; MAPE = mean absolute percent error; SIR = susceptible, infected, and recovered; SARIMA = seasonal autoregressive integrated moving average.

Sources: Authors’ estimates using data available at Public Health Ontario (2021) and Google (2021).

Another objective of this research was to develop time-series models that are capable of forecasting daily new COVID-19 cases. In this respect, SARIMA models fit daily Ontario data very well and provide accurate forecasts over the four-month period from 1 October 2020 to 31 January 2021 that are roughly 10 percent different from actual values. Google mobility variables and the BOC Policy Stringency Index do not offer much help in improving seven-day forecast accuracy for aggregated Ontario data, but at the PHU level, and over longer forecasting time frames, these exogenous variables do indeed help to improve forecast accuracy.

When compared against available evidence, forecasts based on SARIMA models with exogenous variables are comparable to government projections from mid-September to mid-October but superior to corresponding predictions between mid-November and mid-December. Finally, we benchmark the SARIMA forecasts against corresponding forecasts generated from a SIR model. On average, the MAPE in SIR forecasts is 39 percent, which is much higher than the MAPE in SARIMA predictions (10 percent).