Evaluating the reliability of mobility metrics from aggregated mobile phone data as proxies for SARS-CoV-2 transmission in the USA: a population-based study.

BACKGROUND: In early 2020, the response to the SARS-CoV-2 pandemic focused on non-pharmaceutical interventions, some of which aimed to reduce transmission by changing mixing patterns between people. Aggregated location data from mobile phones are an important source of real-time information about human mobility on a population level, but the degree to which these mobility metrics capture the relevant contact patterns of individuals at risk of transmitting SARS-CoV-2 is not clear. In this study we describe changes in the relationship between mobile phone data and SARS-CoV-2 transmission in the USA.
METHODS: In this population-based study, we collected epidemiological data on COVID-19 cases and deaths, as well as human mobility metrics collated by advertisement technology that was derived from global positioning systems, from 1396 counties across the USA that had at least 100 laboratory-confirmed cases of COVID-19. We grouped these counties into six ordinal categories, defined by the National Center for Health Statistics (NCHS) and graded from urban to rural, and quantified the changes in COVID-19 transmission using estimates of the effective reproduction number (Rt) between Jan 22 and July 9, 2020, to investigate the relationship between aggregated mobility metrics and epidemic trajectory. For each county, we model the time series of Rt values with mobility proxies.
FINDINGS: We show that the reproduction number is most strongly associated with mobility proxies for change in the travel into counties (0·757 [95% CI 0·689 to 0·857]), but this relationship primarily holds for counties in the three most urban categories as defined by the NCHS. This relationship weakens considerably after the initial 15 weeks of the epidemic (0·442 [-0·492 to -0·392]), consistent with the emergence of more complex local policies and behaviours, including masking.
INTERPRETATION: Our study shows that the integration of mobility metrics into retrospective modelling efforts can be useful in identifying links between these metrics and Rt. Importantly, we highlight potential issues in the data generation process for transmission indicators derived from mobile phone data, representativeness, and equity of access, which must be addressed to improve the interpretability of these data in public health. FUNDING: There was no funding source for this study.

Introduction

Global efforts to prevent the spread of the SARS-CoV-2 pandemic in early 2020 focused on non-pharmaceutical interventions such as physical distancing; policies that aim to reduce transmission by changing mixing patterns between people.1, 2, 3 As countries have implemented these interventions, aggregated location data from mobile phones have become an important source of real-time information about human mobility and behavioural changes on a population level. An unprecedented amount of mobile phone data have become available as a result, and mobility metrics have not only been widely used by researchers modelling SARS-CoV-2 dynamics, but also by policy makers as an indicator of social activities that drive transmission.4, 5, 6 These metrics have also been used extensively to identify spatiotemporal and individual transmission dynamics of the pandemic.7, 8, 9 It remains unclear, however, whether these metrics are reliable proxies of the face-to-face contact patterns that underlie SARS-CoV-2 transmission.10, 11, 12

Human activity measured using mobile phones reflects the aggregate behaviour of a subset of people, and although metrics of mobility are related to patterns of social contact that spread SARS-CoV-2, they do not provide a direct measure of metrics of epidemiological interest such as the contact rate. In March, 2020, during the first lockdowns due to COVID-19 in cities like New York and Seattle, the early epicentres of transmission in the USA, substantial changes in mobility were observed in many different data streams from mobile operators and social media. Emergency declarations, shelter-in-place orders, school closures, and the cancellation of mass gatherings led to reductions in travel, increased time spent at home, and affected the number of visits to commercial areas and stores. Furthermore, following this dramatic change in mobility, upticks in movement patterns and travel associated with lockdown fatigue and reopening preceded an increase in confirmed COVID-19 cases, hospitalisations, and deaths.13, 15 These near real-time data streams therefore provided a useful indicator of the societal changes that resulted in more direct indicators of disease burden, such as diagnoses and hospitalisations, and have been widely used to better understand the contribution of human mobility to transmission dynamics during the pandemic.

Evidence before this study

In early 2020, at the onset of the SARS-CoV-2 pandemic in the USA, various advertisement technology companies made anonymised mobility data available to researchers. These data have been used extensively to understand how population movement affects the transmission dynamics of the virus. On March 26, 2021, we used Google Scholar and PubMed and searched using key search terms (“mobility” AND “Ad-tech providers“ AND “public health” OR “epidemiology”) to identify articles of interest published in all languages between Jan 1, 2019, and Dec 31, 2020. Early studies showed promise, suggesting a role for these metrics as proxies for transmission parameters of interests such as the effective reproduction number (R t). Subsequent research and replication studies signalled a weakening utility of mobility metrics as a viable proxy; however, these data continue to be used extensively in public health research.

Added value of this study

We collected a nationally representative dataset of anonymised mobility metrics and evaluated their temporal and spatially varying relationship with R t. We show that the relationship between mobility metrics and R t vary considerably by urbanicity, and that this relationship changes over epidemic period, waning over the course of our study period of interest. Given the continued use of mobility metrics at face value by policy makers and researchers, our study urges nuance in interpretation, and transparency in the data generation process of these mobility metrics.

Implications of all the available evidence

Mobility metrics can be unreliable proxies of the transmission dynamics of SARS-CoV-2. In most cases, these data are collected, calculated, and optimised for commercial purposes. Further collaboration with providers of mobility metrics is needed to ensure that researchers are better able to understand how these data are generated and evaluate the utility of metrics in elucidating the transmission dynamics of an infectious disease.

However, the multiple sources of mobility data—for example from Facebook, Google, mobile operators, or advertisement technology companies like Safegraph or Unacast—are aggregated in different ways by different providers, and have not been rigorously shown to reflect local contact rates or behaviour that are proportional to risk of transmission. Furthermore, the heterogeneous processing steps these data providers take to compute mobility metrics obscure what different proxies mean with respect to transmission. In particular, it remains unclear which mobility indicators to use as robust metrics for making policy decisions about SARS-CoV-2 transmission in any particular location. This knowledge is essential, given the appetite policy makers and the general public have for leading indicators of COVID-19 spread that can be used to guide personal behaviour and government responses.

The effective reproduction number (R t) represents the expected number of infections caused by a single infectious case, accounting for existing immunity in the population. R t provides an important measure of epidemic trajectory and indicates whether incidence is growing, shrinking, or holding steady. COVID-19 case notifications, hospitalisations, and deaths provide observable measures of epidemic severity and trend, but these observations provide only a delayed and incomplete picture of the number of incident infections happening at any point in time. R t is not always straightforward to estimate, and case notifications are unreliable and probably biased in different ways in different locations. Efforts to nowcast the effective reproductive number have relied on available data and subject matter expertise on data generation processes. In the simplest epidemiological models, the reproduction number is linearly related to the contact rate, for which mobility metrics are a proxy. We set out to estimate the relationship between human mobility metrics and estimated R t for 1396 US counties between Jan 22 and July 9, 2020.

Methods

Study population

In this population-based study, we collected data on 1396 counties across the USA that had at least 100 cases of COVID-19 on June 30, 2020. We grouped these counties into six ordinal categories, defined by the National Center for Health Statistics (NCHS) and graded from urban to rural (figure 1 A). For our analysis, we considered NCHS category 1 (large central metro) to be the most urban counties and NCHS category 6 (non-core) to be the most rural counties.

Figure 1

Counties included in the study by their NCHS category (A) and the number of observations of counties with at least 100 cases with available estimates, by state and epidemiological week (B)

NCHS=National Center for Health Statistics.

Data sources and metrics

For each county, we collected epidemiological data on COVID-19 cases and deaths from the COVID-19 data repository from the Center for Systems Science and Engineering at Johns Hopkins University, as well as human mobility metrics collated by advertisement technology that was derived from global positioning systems (GPS) trace data provided by Camber Systems (Washington, DC, USA), an aggregator of mobility data from various providers in the USA (figure 2 ).18, 20 Due to the geographical heterogeneity in initial outbreaks, varying amounts of data are available by state (figure 1B). We collected case data from Feb 15 to June 15, 2020, for the case data and we collected the R t estimates from March 1 to June 15, 2020. With the advertisement technology user data, we calculated mobility metrics including the percentage change in movement into counties, percentage change in movement out of counties, and percentage change in movement within counties, as well as the average radius of gyration, average number of locations users visited, and average entropy of movement for users across all counties, every day from March 1 to June 15, 2020. Movement was categorised in 8-h blocks of time with a vector of movement defined as occurring from the county in which an individual spent most of their time in the preceding 8-h block to the county in which an individual spent most of their time in the current 8-h block. We used the average movement in the month preceding the first day for each county for which we have data as our baseline conditioning for time of day and day of the week. Descriptions of these metrics and their uses have been previously described. All metrics were calculated using GPS coordinates published by mobile phone applications for commercial purposes (advertisement technology). Further descriptions of derived metrics are provided in the appendix (p 1).

Figure 2

Example of metrics used for analysis for Shelby County, Tennessee, which is categorised as a large central metro by the NCHS with a LOESS curve added

NCHS=National Center for Health Statistics. LOESS=locally estimated scatterplot smoothing. % change in=percentage change of movement into counties. % change out=percentage change of movement out of counties. % change within=percentage change of movement within counties. Dwell=average number of locations users visited. Entropy=average entropy of movement. Rt=effective reproduction number.

Statistical analysis

We used day-specific and county-specific estimates of the effective reproductive number, generated using a Bayesian nowcasting procedure that incorporated the effects of reporting delays and change in how cases were defined over time and in different regions.21, 22 We aggregated all of our data points to their average log transformed value for each epidemiological week, in which epidemiological week is as defined by the US Centres for Disease Control and Prevention. We specified models to predict the modelled R t values using the mobility data. First, we evaluated the six candidate mobility metrics and ensured that no highly correlated pairs of metrics remained in our model. Given the nature of the data generation process for both the independent variables and the outcome, we used a linear model that allowed for autoregressive correlated errors. This method is supported by the time-dependent structure of residuals in models in which within-county correlations are not taken into account. We evaluated our model of best fit by comparing both the fit (Pearson correlation of original and predicted R t) and Akaike information criterion (AIC). Given the difference in quality of advertising technology data between rural and urban areas, we estimated the model separately for each of the NCHS categories.

We hypothesised that the association between mobility metrics and SAR-CoV-2 might change over time and, in alternate specifications, included an interaction term to differentiate between the first and second half of our study period. Finally, in our model of best fit, we also allowed the coefficients of the mobility metrics to change as a smoothed function of time to ensure that any failure of performance in our model would not be due to stringent model specifications. We evaluated the performance of our model of best fit by comparing the predicted R t to the nowcasted R t. Finally, we evaluated the operational effectiveness of our model of best fit at various thresholds of R t to determine when they could identify epidemic surges (value of R t above a given threshold of 1) using both agreement and Cohen's kappa. We tested the robustness of our model of best fit against both daily and weekly measures of input data.

Role of the funding source

There was no funding source for this study.

Results

We collected epidemiological and mobility data from 1396 US counties between March 1 and June 15, 2020, including 1 907 098 epidemiological cases from Jan 22 to June 9, 2020 (from which we calculated the R t). In initial data exploration we identified that weekly percentage change in movement into counties and percentage change in movement out of counties were very highly correlated, and so we removed percentage change out from our analyses (appendix p 2). We also included an interaction term between weekly average radius of gyration and entropy as, in practice, these metrics are difficult to interpret individually. Our full model, therefore, includes the estimated R t as the dependent variable and percentage change of movement into counties, percentage change of movement within counties, average radius of gyration, average number of locations users visited, and average entropy of movement as our independent variables. We log transformed both dependent and independent variables before analysis.

Using the full model, we evaluated the relationship between the independent and dependent variables for subsets of the data defined by each NCHS category. Both the fit and performance of our full model dropped off considerably in rural areas (NCHS categories 4–6), relative to urban areas. This effect persists in comparisons that only use the data before or after week 15 (figure 3 ). To avoid false confidence in model results for non-urban centres we restricted our analyses to counties in the three most urban NCHS categories (large central metro, large fringe metro, and medium metro). Next, we tested regression specifications containing subsets of the predictors in the full model, to identify a parsimonious set of independent variables with reasonable predictive performance.

Figure 3

Simple correlation between estimated Rt and modelled Rt with the prediction variance of the fully specified model for each subset of NCHS category

NCHS=National Center for Health Statistics.

We evaluated all possible combinations of independent variables and identified the best models as those that minimise AIC. The top five models in terms of AIC are described in the table . Specific coefficient estimates for all models are presented in the appendix (p 4). The best performing model contains all variables except for radius of gyration and includes an interaction term differentiating the period of time before and after week 15. In this model we found that percentage change of travel into counties was most strongly associated with R t 0·757 (95% CI 0·689–0·857); however, this association was reduced by 0·442 (0·392–0·492) after week 15. In evaluating our model of best fit in a random selection of counties from each NCHS category, we found that all model predictions overlap with the estimated R t; however, this fit changes over time, with poorer fit in later weeks (figure 4 ). As expected, there is an increase in overall model error and decreased fit as we move from urban to rural regions.

Table

Model specifications, AIC, and residual variance for the top five models based on AIC overall along with the overall residual variance (σ)

	AIC	Delta AIC	σ
“% change in” + “% change within” + “dwell” + “entropy” + “dwell * entropy” + “time flag” + “% change in × time flag” + “% change within × time flag” + “dwell × time flag” + “entropy × time flag” + “dwell” × “entropy” × “time flag”	−16 347	0	0·226
“% change in” + “% change within” + “dwell” + “entropy” + “dwell × entropy”	−15 184	1163	0·249
“% change in” + “% change within” + “RoG” + “dwell” + “entropy” + “dwell × entropy”	−15 179	1168	0·249
“% change in” + “% change within” + “dwell” + “entropy”	−15 144	1203	0·246
“% change in” + “% change within” + “dwell”	−15 131	1216	0·248

AIC=Akaike information criterion. % change in=percentage change of movement into counties. % change within=percentage change of movement within counties. Dwell=average number of locations users visited. Entropy=average entropy of movement. Time flag=before or after 15 weeks. RoG=radius of gyration.

Figure 4

Rt estimates and uncertainty bounds from the best model (red) compared with the nowcasted Rt for three randomly selected counties in each of the three most urban National Center for Health Statistics categories.

Rt=effective reproduction number.

To further evaluate the value of mobility metrics as indicators of transmission, we calculated the percent agreement and Cohen's kappa between our model estimates of R t and the estimated R t in varying scenarios (appendix p 6). To further investigate the deterioration in model performance we evaluated the distribution of predicted R t compared with the nowcasted R t (figure 5 ). We found that the reduction in model performance in the second half of the study period is driven by the reduction in variability of the nowcasted R t estimates. An explanation for the reduced explanatory power of the mobility metrics is in the appendix (p 7), in a figure that shows large systematic trends in each mobility metric throughout the study period, with similar trends observed for most counties. In contrast, average log (R t) stabilises with little systematic trend after week 15, but with substantial differences at country level.

Figure 5

Correlation between log (Rt) and the modelled log (Rt) over the course of the study period

Rt=effective reproduction number.

Allowing the coefficients for all predictors in the fully specified model to vary over time, we found that percentage change of movement into counties and percentage change of movement within counties largely drive the performance of the model, with changes in covariate estimate for percentage change of movement into counties coinciding with the decrease in predictive power of the model (appendix p 3).

Discussion

Mobility data have been used extensively during the SARS-CoV-2 pandemic to understand disease transmission between populations, identify hotspots of transmission, and guide policy interventions. In this study we evaluated the relationship between mobility metrics derived from advertisement technology data and the effective reproductive number of SARS-CoV-2, using data from counties in the USA during the initial period of the epidemic, from epidemiological weeks 9 to 24, in 2020. Our model performs well for the first half of the study period with high agreement under various scenarios until the end of epidemiological week 13. As shown in the appendix (p 6), epidemiological weeks 12–13 reflect the period in which, nationally, the effective reproductive number decreased from its earlier highs and showed little change throughout the rest of the study period. For example, in the three most urban classifications of counties, on average, the log (R t) decreased by 0·24 in week 12 when compared with week 11. However, from week 15 onwards, the average log (R t) didn't change by more than 0·05 compared with the previous week. On the other hand, the advertising technology-derived mobility metrics continued to change during the course of the study period.

In our model of best fit, during the first half of our study period, and for the counties for which we have data, a 10% increase in the percentage change of number of individuals travelling into a county is associated with a 5% increase in the effective reproductive number, accounting for the effects of the average number of locations users visited, the average entropy of movement, and the percentage change in the number of individuals moving within a county. By allowing our coefficients to vary over time we can see that this effect was larger during the first half of our study period, peaking at week 13, and diminished slightly over time (appendix p 3). These findings are useful for future use and interpretation of mobility metrics. We caution that, because the relationship between the aggregated mobility patterns picked up by mobile phone data of this kind could have shifting associations with the face-to-face interactions driving transmission, these mobility metrics are unlikely to have sustained, robust predictive power for understanding R t.

We noted a marked reduction in model performance in the three most rural county classifications in the USA. Although this reduction could be a function of the delayed outbreaks in these counties compared with earlier outbreaks in locations such as New York, it is important to note that the data generation process for mobility data probably differs greatly between urban and rural counties. Subsequent analyses on previous studies have reported similar phenomena. The generation of raw GPS data is affected by the types of mobile applications that individuals use, the factors that lead to users sharing location information, and essential trips that users take; all of which can vary immensely from county to county. The implicit and perhaps unavoidable selection of users who generate these data might also complicate the interpretability of these metrics. Furthermore, an intricate market of publishers, providers, and aggregators clean, transform, and extract value from these data, often making simple connections to human behaviour difficult. Finally, the advertising technology pipeline is generally built and optimised for commercial purposes, which means that behaviours that could be important for public health purposes might not be those that are captured on device or enriched post extraction. Much of this information on user demographics, coverage, mobile application usage, and post-extraction transformation is opaque, making comparisons between counties with varying urbanity or even socioeconomic status difficult. These data can be potentially useful, but further research, validation, and standardised frameworks for data generation are needed to better understand how variable processing algorithms of raw GPS traces relate to population mobility, and what this means for the interpretation of mobility metrics.

Evaluating parsimonious models allowed us to infer some general rules of thumb regarding mobility metrics and their association with R t while identifying major limitations in the creation and interpretation of these metrics. As well as percentage change in the number of individuals travelling into a county, entropy (average entropy of movement) and dwell (average number of locations users visited) remain in our parsimonious model. As entropy, or the unpredictability of individuals' movement increases, R t also increases. Anecdotally, this process makes sense as individuals travelling to more locations than usual might be increasing their contact network. However, a decrease in the number of locations that a user visits can also be considered unpredictable and therefore result in higher entropy, underscoring the effect of appropriate baselines. Changes in entropy from week to week with no exogenous interventions should be interpreted in a different light than changes in entropy during the implementation of a travel restriction. The average number of locations an individual in a county spends at least 5 min had an unexpected negative relationship with R t in our parsimonious model, probably because of artifacts of the way in which these data are collected on devices. In following orders for travel restrictions or physical distancing, one might reasonably expect that individuals might travel to fewer locations and therefore that this metric might decrease. However, in our data, the average number of locations users visited increased almost universally in the aftermath of such interventions. This finding is again probably due to individuals travelling by car less, and taking more walks with their mobile devices, than they might do before travel restrictions are implemented; generating more locations in which they spend at least 5 min. These issues again highlight the importance of researchers being able to interrogate the data generation process and understand the rationale behind default conditions (such as spatial limits that define a unique location, minimum number of interactions needed to generate information, or the amount of time a user needs to spend in a location to register as being there) which are used to define metrics. An optimal threshold of defining locations every 5 min versus 10 min can dramatically change the metric output with no clear understanding of the effect on public health inference.

We found that the strength of the relationship between mobility metrics and R t declines dramatically over time. Allowing the coefficients of our mobility metrics to change over time did not strengthen the relationship. It is unlikely that this issue is unique to the specific data aggregator, as they consolidate sources used by a variety of other providers that, as described previously, can have opaque and overlapping user bases. There are several possible reasons why the relationship between mobility metrics and R t weakened over time. The first is exogenous universal public health measures such as masking and physical distancing. These measures intrinsically change the relationship between human movement and disease dynamics, as contact rates between individuals might stay the same, but the probability of transmission given contact can decrease with appropriate masking and distancing. The relationship between human mobility and the effective reproductive number might also be confounded by unmeasured components of human behaviour that drive both mobility and adherence to public health measures, resulting in a decrease of R t. It is also possible that, with the smaller changes in R t observed after the initial epidemic response, the magnitude of signal decreased relative to the inherent noise of the mobility metrics, and estimation uncertainty consequently increased. If this hypothesis is true, mobility metrics might be dependable indicators of gross changes in transmission behaviour, but less useful for identifying smaller changes.

It is possible that a more complex model, inclusion of other variables, and information on demographics or other details about human behaviour might better explain the relationship between human mobility metrics and R t. However, the non-independence of mobility data across time and locations means that there are fewer degrees of freedom available than indicated by the massive amount of data available, and fitting flexible models to these data risks producing models with good in-sample fit but poor predictive validity and difficult interpretation. Given the opacity around the data generation process, uncertainty about the populations that these metrics represent, and the absence of frameworks that optimise the generation of these metrics for epidemiological inference, we chose to fit simple models, while accounting for the within-county correlation inherent in both the mobility and R t data. Finally, we used the CovidEstim package to calculate R t, which incorporates death data into its algorithm. This includes the implicit assumption these data will not face biases in reporting lags and changing definitions that the case data faced early in the pandemic. Nevertheless, exogenous factors may affect the validity of the death data which may bias our estimate of R t.

An abundance of human mobility metrics from various providers are now available and, in some cases, actively integrated into both modelling and public policy efforts. These data provide information on human behaviour at a level previously reserved for the data generators and their commercial interests. Our study shows that the integration of these metrics into retrospective modelling efforts can be fruitful in some cases, linking these metrics directly with public health outcomes of interest such as the effective reproductive number. However, using these metrics exclusively without incorporating key changes in human behaviour such as masking, physical distancing or vaccinations can make the relationship between mobility and transmission difficult to disentangle. In the early period of the epidemic in the USA, we infer a clear relationship between an increase in the number of individuals travelling into a county compared to a pre-pandemic baseline and an increase in the effective reproductive number. These data showed impressive potential for future public health efforts during the early stages of the SARS-CoV-2 pandemic in the USA. Nevertheless, transparency in the data generation process, investigation into representative populations, continuous revaluation of these metrics, standards for optimisation of metrics for public health purposes and, perhaps most importantly, frameworks for equity of access to these data are still needed to truly unlock their potential.

Data sharing

All data used in this study is saved in a static repository. Access to these data is available to researchers once they have established a data use agreement (DUA) with Camber Systems. These datasets include daily estimates of the mobility metrics described in this study for all counties in the USA over our study period of interest. Once a DUA is established, researchers can access the static repository of data used in this research or access the dynamic, up-to-date repository made available by Camber Systems. All code used in the pre-processing, modelling, and production of figures will be available in a static code repository.

Declaration of interests

NV was the chief technology officer of Camber Systems, an advertisement technology company, at the time this research was conducted and provided access to the mobility data. Neither NV nor Camber Systems influenced the final results or the decision to submit this manuscript. All other authors declare no competing interests.