The pitfalls of modelling the effects of COVID-19 on gender-based violence: lessons learnt and ways forward.

Since early 2020, global stakeholders have highlighted the significant gendered consequences of the COVID-19 pandemic, including increases in the risk of gender-based violence (GBV). Researchers have sought to inform the pandemic response through a diverse set of methodologies, including early efforts modelling anticipated increases in GBV. For example, in April 2020, a highly cited modelling effort by the United Nations Population Fund (UNFPA) and partners projected headline global figures of 31 million additional cases of intimate partner violence due to 6 months of lockdown, and an additional 13 million child marriages by 2030. In this paper, we discuss the rationale for using modelling to make projections about GBV, and use the projections released by UNFPA to draw attention to the assumptions and biases underlying model-based projections. We raise five key critiques: (1) reducing complex issues to simplified, linear cause-effect relationships, (2) reliance on a small number of studies to generate global estimates, (3) assuming that the pandemic results in the complete service disruption for existing interventions, (4) lack of clarity in indicators used and sources of estimates, and (5) failure to account for margins of uncertainty. We argue that there is a need to consider the motivations and consequences of using modelling data as a planning tool for complex issues like GBV, and conclude by suggesting key considerations for policymakers and practitioners in using and commissioning such projections. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Mathematical modelling has been used to predict the impact of COVID-19 on a range of issues, including gender-based violence (GBV).

United Nations Population Fund (UNFPA) and partners’ modelling suggests headline figures of 31 million additional cases of intimate partner violence due to 6 months of lockdown, and an additional 13 million child marriages by 2030.

UNFPA and partners’ modelling overlooks variations between and within countries, combines estimates from multiple studies and fails to provide sufficient detail about modelling strategies and underlying assumptions.

Policymakers and practitioners need to carefully consider the limitations of modelling for GBV and the implications of releasing and citing estimates without fully documenting and disclosing the same.

Introduction

Since early 2020, various stakeholders, including development organisations and humanitarian agencies, have argued for the need to consider and address the anticipated gendered impacts of COVID-19. These include increased risk of gender-based violence (GBV) -- including intimate partner violence (IPV), violence against children, and child marriage -- as well as decreased access to sexual and reproductive health services, adverse schooling outcomes for adolescent girls and a disproportionate economic impact and care burden on women.1–4 Advocacy and guidance materials for GBV emphasised the need to ensure GBV response services were not diverted during the pandemic, were classified as essential services, and were adapted to include mobile service delivery, helplines and prevention using digital technology.5

Amidst the mounting fears surrounding a GBV shadow pandemic, in April 2020, the United Nations Population Fund (UNFPA) and collaborators released a technical note projecting the impacts of COVID-19 on GBV. Findings suggested a 6-month lockdown would contribute to a global increase in 31 million cases of IPV and service provision disruptions as well as impacts on poverty would result in 13 million child marriages by 2030.6 These projections have been widely cited and repeated by other United Nations (UN) agencies, non-governmental organisations (NGOs), gender and development actors and international media outlets. For example, in July 2020, the UN Secretary General António Guterres drew on estimates to urge member states to consider GBV services as essential, and in October 2020, Angelina Jolie cited estimates in her TIME magazine op-ed to advocate for support to marginalised women.7–11 Since the UNFPA projections were released, other modelling efforts have published. For example, in October 2020, Save the Children estimated that an additional 2.5 million girls would be married by 2025 due to the economic downturn caused by the pandemic.12 In March 2021, UNICEF estimated 10 million more girls would marry by 2030.13 Each modelling effort relied on different underlying assumptions and parameters, resulting in different estimates.

This reliance on modelling is not unique to GBV. In addition to modelling the spread of SARS-CoV-2, modelling has been used during the COVID-19 pandemic to forecast social and behavioural outcomes, including educational outcomes, mental health and suicide.14 15 Modelling can be a powerful tool to explore impacts of policy decisions and assist in planning and resource allocation. Despite their widespread use, models have been critiqued for being overly simplistic, ahistorical and for uncritically applying estimates from one context to another. Researchers have called attention to the limited accuracy of models, the lack of transparency and inherent gender biases in data sources, limitations in assumptions and methods, and the danger of using either flawed estimates or a single estimate to inform policy.16–22

The politics of GBV data, as well as a preoccupation with quantification for policy attention and resources, may contribute to the motivation underlying GBV modelling.23 Gender-related issues and GBV have often been excluded from administrative and survey data and have also often been viewed as outside mainstream work on global health and development, leading to very limited data on GBV being available in most countries. At the same time, demands for quantitative evidence on the necessity, effectiveness or cost of interventions far outpaced evidence from qualitative or ethnographic research. Obtaining reliable data on GBV to make the case for action was already challenging prior to COVID-19, pushing organisations to produce other kinds of evidence to develop a strong rationale for continued or increased investments. The competition for limited funding both prior to, and during, the pandemic intensifies the demand for quantitative evidence and modelled projections may be positioned as required evidence for the investment case. While we recognise that such estimates can draw attention to GBV, there is a need for critical analyses of how to generate accurate and reliable estimates that can better serve efforts to prevent and respond to GBV.

In this paper, we examine the limitations of modelling the effects of COVID-19 on GBV. We use projections of global child marriage and IPV developed by UNFPA, the first and most widely cited GBV modelling effort, as an illustrative example to discuss the limitations and challenges of modelling GBV, and conclude with recommendations to improve the quality and accuracy of mathematical models on GBV for better investment and more effective public health responses.

The limits of mathematical modelling for GBV: an example using UNFPA estimates

Child marriage

UNFPA’s technical note projected that an additional 13 million child marriages would occur between 2020 and 2030 due to service disruption and the economic consequences of COVID-19. This estimate is based on two mechanisms linking COVID-19 and child marriage. The first mechanism assumes that a 1-year disruption to ‘a well-defined package of interventions to reduce child marriage’ will lead to 7.4 million more child marriages by 2030. The second mechanism assumes that increased poverty related to the economic impacts of COVID-19, particularly in low-income countries with high child marriage prevalence, will lead to 5.4 million more child marriages by 2030. The figures from each mechanism are combined to arrive at the estimate of 13 million additional child marriages.

The first mechanism relies on a modelling study by Rasmussen and colleagues which estimates the cost-benefit of interventions to reduce child marriage drawing on data from 31 low and middle-income countries (LMICs), focused on two kinds of interventions: (A) education-related and (B) child marriage-specific interventions.24 The education-related interventions come from a meta-analysis of interventions for adolescents (eg, girls’ access to latrines, cash transfers to keep girls in school, pedagogical training for teachers).25 The child marriage-specific interventions come from two systematic reviews, which identified studies designed to delay or reduce child marriage, including life skills education, community mobilisation and conditional economic incentives.26 27 These were individual or community-level interventions with mixed evidence on effectiveness. Rasmussen and colleagues assume one ‘dose’ is sufficient for long-term impacts and that this intervention package results in 50% fewer child marriages by 2050. Although Rasmussen and colleagues identify limitations in relying on a small number of studies to generate global predictions and observe that they only model a subset of all child marriage interventions, the UNFPA technical brief does not acknowledge these—or other—limitations. Instead, the UNFPA model draws on this study and assumes that all child marriage interventions are effective and will be delayed by 1 year due to social distancing requirements and travel restrictions which prevent technical staff from conducting teaching and training. This assumption suggests that all child marriage interventions require travel and external international expertise. However, even in early 2020, many organisations had already adapted programmes to deliver activities remotely or adjusted them to comply with safety measures. Assuming a delay of 1 year does not do justice to adaptation and innovation potential. Further, projecting child marriages over a 10-year period (2020–2030) neglects other social change processes that may affect child marriage.

The second mechanism is related to economic impacts and relies on two studies. One study estimates the impact of COVID-19 on global poverty and offers three scenarios of gross domestic product (GDP) per capita reduction (5%, 10%, 20%).28 Offering limited justification, the UNFPA projection draws on the mid-range of a 10% reduction in GDP from this study, alongside uncited World Bank projections, to arrive at a 7% increase in poverty among 50% of the global population. The second study draws on India’s 2015–2016 Demographic and Health Survey to examine the prevalence and correlates of child marriage. Findings show that the ‘poverty variable alone [household wealth quintiles] explain between 5% (middle) to 32% (poorest) in total variation’ in child marriage.29 UNFPA used these figures to project increases in child marriage due to increases in poverty ranging from 2.2% for the poorest segment to 0.5% for the richer segment. The UNFPA projection uncritically combines global estimates of reductions in GDP per capita with predictions from a study in India to conclude that the child marriage rate globally would increase by 1.4%. The relationship between household wealth and child marriage in India cannot be extrapolated to the relationship between GDP reduction and child marriage across the world; these constitute different ways to measure economic standing. This assumption collapses contexts with different income distributions and assumes a singular, invariant universal relationship between GDP, household wealth and child marriage.

In addition to the specific limitations of how each mechanism is operationalised, the model has several overarching limitations. First, the UNFPA brief is not explicit about the conceptual framework for the mechanisms between COVID-19 and children marriage to arrive at the total figure of 13 million additional child marriages, and instead uses two drivers of child marriage in specific settings to generate global projections. In contrast to this, a recent UNICEF model identified five mechanisms which were all included in projections of child marriage.13 This approach is more in line with existing evidence identifying multiple, context-specific drivers of child marriage.30–39 Finally, the brief aggregates different estimates from multiple studies that vary in size, methodology and geographical context without accounting for variability either of intervention effectiveness, or of the relationship between types and levels of poverty and child marriage.

Intimate partner violence

An additional outcome examined in the UNFPA technical brief is IPV (GBV is used in the brief; however, background documents indicate the hypotheses is IPV specific). Modelling is based on the assumption that COVID-19 would: (1) reduce prevention and protection efforts, social services and care as funding and attention are refocused on the pandemic; and (2) increase the incidence of IPV during the pandemic. The first mechanism assumes a 2-year delay in scale-up of prevention activities (ie, delay during 2020 and 2021), after which programming would rapidly expand. This delay is estimated to lead to 2 million additional instances of IPV during 2020 and 2021 and nearly ‘200 million fewer cases of violence averted by 2030’ (equating to a reduction of approximately one-third in progress towards ending IPV). In addition, the pandemic is assumed to increase levels of IPV in the immediate pandemic period due to stay-at-home and movement restrictions, economic stressors and household tensions, among others. Projections of a 20% increase in IPV during lockdowns equate to 31 million additional cases of IPV within a 6-month period, and for every 3 months the lockdown continues (up to a year total), an additional 15 million cases of IPV are expected.

The assumptions this projection relies on are far from straightforward. First, definitional components in the technical brief require clarification. Among these is the use of GBV, a broad term, encompassing numerous types of violence (non-partner rape, violence against children, sexual harassment, femicide, psychological harm and more). While the technical note refers to GBV, the modelling relies on IPV specifically, with reference to a case of IPV as incidence of physical and/or sexual IPV in the last 12 months.40 This distinction is not trivial. Although no combined, rigorous, global estimates exist for 12-month incidence of GBV, it is presumably orders of magnitude higher than global estimates of 12-month physical and/or sexual IPV. In addition, the fact that a case is any new occurrence, without reference to frequency or severity—within a 12-month period—is important to consider, given many women experience ongoing, rather than isolated episodes of IPV.

The modelling for the delay in IPV prevention activities relies on a complementary analysis conducted in 2020 to estimate the cost of ending IPV by 2030 in 123 LMICs (rather than globally).40 This analysis assumed a basic package of prevention and treatment activities, in addition to NGO strengthening and support functions. While it was acknowledged that the evidence base was not strong enough to generalise impact levels by type of intervention, the package is assumed to be customised by country and potentially consist of: economic empowerment, community mobilisation, education and sensitivity training for adolescents, gender sensitivity training for male youth, enabling environments for sex work, and gender perspectives in health services, as well as counselling and treatment for rape survivors. Headline estimates show a total of $42 billion is needed to end IPV, $32.5 billion more than the $9.5 billion expected spending. Setting aside assumptions around the cost incidence and cost-effectiveness of IPV interventions (as evidence remains extremely thin41), it is not clear the pandemic will delay these efforts by 1–2 years. Early in the pandemic, governments and service providers were quick to adapt and innovate to provide basic services and complementary support in virtual or via contagion-safe methods. The United Nations Development Programme and UN Women ‘COVID-19 Global Gender Response Tracker’ catalogued over 700 action measures taken for violence against women prevention and response within the first 9 months of 2020.42 While investment costs of these new efforts are unknown, the assumed 2-year disruption in services (or projected services) is not clear cut. Second, the translation to concrete numbers of IPV cases associated with these interventions requires broad assumptions across settings, intervention and cost-effectiveness that the evidence does not support.

The modelling of increased IPV due to the pandemic also requires a complex interplay of assumptions across settings and over time; however, almost no information is given regarding underlying analysis. Is a 20% increase in IPV during periods of lockdown (with no definition of what a lockdown entails) a reasonable assumption? No specific study or reference is given for this figure; however, analysis to date of over 74 papers examining the relationship between COVID-19 restrictions and violence against women and children shows a complex picture.4 While the majority of papers that examine trends find increases, the magnitude of change varies widely based on the setting and data source, with evidence of fluctuation over time. These estimates are complicated by the fact that COVID-19 is likely to affect data quality and propensity of women to seek help, and thus affect the incidence of reported data. While this evidence was not available at the start of the pandemic, accounting for uncertainty (eg, use of confidence intervals (CIs), or scenarios of low, medium and high lockdowns) would greatly increase confidence in estimates—as has been done in other modelling efforts.43 Finally, due to the existing and ongoing complex relationships of survivors, assuming 15 million additional women will experience IPV every 3 months ignores realities of existing behavioural patterns and minimises ongoing changes in frequency and severity of IPV.

Conclusion

Our critique of UNFPA estimates predicting increases in child marriage and IPV during COVID-19 and recommendations to improve GBV models are summarised in table 1. While recognising that no modelling effort is infallible and that there is a need for evidence to inform investment and prioritise interventions, insufficient information, oversimplified assumptions and limited data undermine the validity of models, the accuracy of results, and may diminish the credibility of evidence on GBV.

Table 1

Summary of critiques and recommendations for modelling gender-based violence (GBV) during COVID-19

Critique		Example from models estimating effect of COVID-19 on GBV*	Recommendation and/or opportunity to strengthen GBV modelling
(1)	Oversimplification of a complex phenomenon	Lack of theoretical framework. Assuming a singular or limited number of pathways/factors link COVID-19 and GBV. No discussion of which pathways/factors are missing in the model or how each pathway/factor is measured. Including or combining predictors/variables (eg, poverty and GDP) at multiple levels (household, institutional, national) without consideration of how these interact with each other to influence GBV.	Develop or adapt an evidence-based theory of change to account for, and model multiple underlying pathways/factors linking COVID-19 and GBV. Explicitly define and cite data sources, variables and estimates from prior studies, with attention to level of measurement. If focusing on a singular pathway/factor, properly acknowledge and state this assumption/limitation in discussion or results.
(2)	Overgeneralisation of estimates and results across geographies and intervention typologies	Applying estimates from a small number of intervention studies to make global projections. Applying results on pathways/factors linking COVID-19 to GBV in one country to extrapolate global drivers. Limited consideration of differences in the prevalence of GBV and in access to interventions prior to the COVID-19 pandemic.	Draw on local or country-level secondary data if possible and consider building models and presenting estimates of country-specific scenarios, or focusing on regional estimates from similar income group classifications. Be transparent about and acknowledge the limitations of applying estimates from one country to other contexts. Acknowledge the limitations of country-level models which can obscure subnational effects of COVID-19 on GBV.
(3)	Not accounting for changes and adaptations in service provision during COVID-19	Assuming complete service disruption of GBV-related prevention and response due to COVID-19 in all settings. Only modelling one lockdown scenario for service disruptions.	Acknowledge and account for changes in service delivery innovation and adaptation, for example, remote or virtual services and social distancing precautions. Acknowledge and account for changes in funding for GBV prevention and response (both increases and decreases), or mention these uncertainties or lack of reliable data as limitations. Acknowledge which populations and locations may not benefit from remote service delivery or may be most impacted by changes in service provision.
(4)	Unclear definitions of GBV and sourcing of estimates	Lack of clarity in GBV indicator definitions. Lack of clarity in citations of estimates or assumed changes in GBV.	Clearly define each GBV indicator, including if estimates account for ongoing versus new cases (as applicable). Cite all sources for relevant GBV estimates, or clearly state when figures are not evidence derived.
(5)	Inflexible linear model of change over time, without accounting for margins of uncertainty	Modelling GBV over the long term (eg, 10 years) without accounting for macrotrends over time. Modelling impacts of COVID-19 without accounting for changes over time in severity of the pandemic or restrictions. No CIs, error margins, sensitivity checks or other tests of uncertainty.	Account for macrotrends for GBV—where applicable—or acknowledge uncertainty in long-term estimates. Provide estimates with CIs, including those to account for low-medium-high severity of COVID-19 or lockdowns.

*Critiques are based on United Nations Population Fund (UNFPA) estimates of intimate partner violence (IPV) and child marriage. We use ‘GBV’ as an umbrella term that includes IPV and child marriage.

GDP, gross domestic product.

Our analysis suggests that three key issues need to be considered when deciding whether to develop a mathematical model to estimate complex, contextual events like GBV. First, the motivations for generating projections using modelling need to be considered. Do we need quantitative evidence showing increases during pandemics like COVID-19 in order to know how to act? If results are needed to demonstrate GBV financing should be prioritised, there is existing evidence which shows that GBV has increased during epidemics and pandemics in many contexts.44 Furthermore, guidelines from the Inter-Agency Standing Committee and others specify that we should assume GBV occurs during humanitarian emergencies and there is never a need to prove GBV.45 Given this guidance, what motivates the need for global-level modelling? The results of the UNFPA models cannot be used to prioritise groups/interventions over others, or guide prevention or response, and indeed the data underlying them would be insufficient to do so. These models are unable to account for the changing nature of the pandemic and do not account for uncertainty, variability within or between countries in GBV, or in the response to COVID-19. Additionally, these models cannot answer how best to prevent GBV, how best to deliver services to survivors or how to improve GBV help seeking during COVID-19.

Second, the consequences of producing and citing models need to be considered. At times, providing headline numbers—if they rely on incomplete data or inaccurate assumptions—may do more harm than good. Perhaps when (more accurate) numbers later show smaller increases, or numbers are lower in future emergencies, stakeholders think the issue is less important or urgent as before.46 Additionally, lack of clarity on the limitations of secondary data or prior studies used in projections may lead to modelling data being misunderstood or overemphasised. The credibility of the sector may be undermined by reliance on inaccurate estimates or without transparency about their limitations. The topic of GBV might also become further sidelined if methodologies for generating projections are not clear. It is also important to reflect on the sensationalist power of modelling projections on issues like GBV, and carefully weigh the benefits of headlines against the validity of context and reputational risk of global actors.

Third, the data used in models matter: ‘[a] mathematical model is as good as the data it uses.’47 Global projections based on few individual country-specific studies will not generate useful data for global policy and planning purposes. Modelling at the national level with more granular data increases potential for accuracy, if the sample, indicators and quality of the data are high. In such cases, where the original data proposed for a model are limited, intervention planning could rely on using secondary, administrative or programme data, conducting rapid assessments or consulting community organisations. This reinforces the urgent need to invest in improving collection of gender data and indicators. If models are to be used, they should be transparent, include measures of variability, and be validated, tested, and peer reviewed before widespread use or media distribution. Most importantly, limitations and assumptions should be clearly articulated and described, including which populations and locations the underlying data include (or exclude).

Models do teach us one important lesson: that over the longer term, investment in better data systems, including surveillance systems, survey data and administrative data, will undoubtedly improve the quality of real-time data and evidence on GBV. Additionally, in uncertain and turbulent periods, local, context-specific GBV knowledge may make the most important contribution to rapid prevention and response efforts.