Reassessing the evidence for universal school-age BCG vaccination in England and Wales: re-evaluating and updating a modelling study.

OBJECTIVES: In 2005, England and Wales switched from universal BCG vaccination against tuberculosis (TB) disease for school-age children to targeted vaccination of neonates. We aimed to recreate and re-evaluate a previously published model, the results of which informed this policy change.
DESIGN: We recreated an approach for estimating the impact of ending the BCG schools scheme, correcting a methodological flaw in the model, updating the model with parameter uncertainty and improving parameter estimates where possible. We investigated scenarios for the assumed annual decrease in TB incidence rates considered by the UK's Joint Committee on Vaccination and Immunisation and explored alternative scenarios using notification data.
SETTING: England and Wales. OUTCOME MEASURES: The number of vaccines needed to prevent a single notification and the average annual additional notifications caused by ending the policy change.
RESULTS: The previously published model was found to contain a methodological flaw and to be spuriously precise. It greatly underestimated the impact of ending school-age vaccination compared with our updated, corrected model. The updated model produced predictions with wide CIs when parameter uncertainty was included. Model estimates based on an assumption of an annual decrease in TB incidence rates of 1.9% were closest to those estimated using notification data. Using this assumption, we estimate that 1600 (2.5; 97.5% quantiles: 1300, 2000) vaccines would have been required to prevent a single notification in 2004.
CONCLUSIONS: The impact of ending the BCG schools scheme was found to be greater than previously thought when notification data were used. Our results highlight the importance of independent evaluations of modelling evidence, including uncertainty, and evaluating multiple scenarios when forecasting the impact of changes in vaccination policy. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This study re-evaluates a key piece of the quantitative evidence used to motivate the change in BCG vaccination policy in 2005, correcting a methodological flaw in the original model.

The inclusion of parameter uncertainty and measurement error allowed the uncertainty in the final estimates to be presented. Previously published estimates may have been spuriously precise.

As this study used a historical approach, the model used is not the accurate method for assessing the impact of ending the BCG schools scheme. However, it provides an estimate that is based on the available data and on the framework used to inform policy making. This allowed the strength of some the quantitative evidence used in the decision-making process to be assessed.

A weakness of the modelling framework used in this study is that it did not include the whole population or age groups outside those directly affected by vaccination. The exclusion of these factors means that our results are likely to underestimate the impact of ending the BCG schools scheme.

This study only considered the impact of ending the BCG schools scheme and not the impact of the introduction of the targeted neonatal vaccination programme. This should be considered when evaluating the change in policy as a whole.

Introduction

The BCG vaccine remains the only licensed vaccine for use against tuberculosis (TB), although its use globally is controversial due to evidence of variable effectiveness1 and waning protection 10–15 years after vaccination.2 Global usage of the BCG varies between no vaccination, universal vaccination and high-risk group vaccination, and may target either neonates or school-age children.3 4 The WHO recommends vaccination for all neonates as early as possible after birth in high-burden settings, with vaccination in low-burden settings being dependent on the country-specific epidemiology of TB.5 This recommendation is based on the strong evidence that the BCG is highly protective in children,6 7 while its effectiveness has been shown to vary with latitude when given later in life.1

In England and Wales, universal school-age (at 13–14 years old) vaccination (hereafter referred to as the BCG schools scheme) was introduced after an Medical Research Council (MRC) trial in the 1950s estimated BCG’s effectiveness at 80% in the ethnic white UK-born population.8 The policy remained in place until 2005, when England and Wales changed to targeted vaccination of high-risk neonates. The 2005 change in BCG vaccination policy was motivated by evidence of decreased transmission of TB, an increasing proportion of TB cases occurring in the non-UK-born9 and modelling evidence that suggested stopping the BCG schools scheme would have minimal long-term effects on incidence rates.10 Due to the complex nature of both TB and the BCG vaccine, the ongoing impact of this change in policy is hard to directly estimate, with decision makers relying on expert opinion, evidence from surveillance data and insight from modelling studies.

In 1987, an assessment of the school-age vaccination programme was carried out in England and Wales.10 This study was used, combined with a sensitivity analysis of notification rates, as supporting evidence by the Joint Committee on Vaccination and Immunisation (JCVI) BCG subgroup for the change in vaccination policy.11 12 This paper aimed to re-evaluate this modelling and re-estimate the predicted impact of stopping the schools scheme. Re-evaluating this work allows for the strength of the evidence used in decision making to be assessed and may highlight any issues with the approach used.

Methods

Modelling the impact of ending the BCG schools scheme

We implemented and updated Sutherland and Springett’s10 model for estimating the impact of ending the BCG schools scheme, which is outlined briefly as follows. This model was based on data from TB notification surveys conducted in 1973, 1978 and 1983.13 These were used to estimate incidence rates, stratified by BCG vaccination status, in the ethnic white UK-born population of England and Wales aged 15–19 years old, 20–24 years old and 25–29 years old. Future incidence rates were forecast by assuming an annual decrease in incidence rates, which was based on historic trends.10 14 Primary impacts from ending the schools scheme, including the number of vaccines required to prevent a single notification, were estimated by calculating the difference in incidence rates between the vaccinated and unvaccinated populations. Additional notifications from TB transmission were then calculated using a transmission chain model and combined with the primary impact estimates, to calculate the number of annual additional notifications due to ending BCG vaccination. Based on data availability, the model used a 5-year time step.

Original estimation of notification rates

The effectiveness of the BCG vaccine was originally estimated by an MRC trial in 1953 at 78% in the UK.8 As a follow-up to this trial, members of the MRC biostatistics unit conducted a series of notification surveys attempting to ascertain any change in effectiveness, as well as acting as an estimate of notification rates across different demographics.13 Surveys of those aged 15–24 years were carried out at 5-year intervals in 1973, 1978 and 1983 in England and Wales. For the 1983 survey, records of BCG status, tuberculin status and ethnicity were extracted from the records of notifying physicians and the records of the local health and education authorities. Total notifications across the study period were then aggregated for the following groups: tuberculin negative and BCG vaccinated, tuberculin negative and BCG unvaccinated, tuberculin positive and not vaccinated, and those who did not participate. These totals were then combined with the population estimates for each cohort at 13 years of age to estimate the ethnic makeup of the population and to construct notification rates for each category. Data were drawn from a range of sources including Office of National Statistics data, annual local authority returns for total tuberculin test results, BCG vaccinations in the schools scheme and the Labour Force Survey (1983).

For 1983, this corresponded to 874 notifications in ethnic white UK-born persons aged 15–24 years old in England and Wales; survey participation was 80%. As the number of tuberculin-negative subjects not given BCG was unreported, this was estimated at 1.9% of those vaccinated with the BCG.13 See Sutherland and Springett13 for full details of the survey and the additional assumptions used to give similar estimates for both the 1973 and 1978 surveys. The findings of these surveys were as follows: in the ethnic white population, notification rates had fallen by an annual average of 9%, and BCG efficacy had remained high.10 14

Evidence suggests that the BCG vaccine has a high efficacy for at least the first 15 years after vaccination; therefore, Sutherland and Springett extrapolated from the data on the cohort aged 20–24 years old to a theoretical cohort aged 25–29 years old. Data on the notifications in 25–29 year olds were available for the first 6 months of the 1983 survey, and this was then scaled up to a yearly estimate using the ratio of notifications from this age group against the total number of notifications recorded in that year. Population estimates for the cohort aged 25–29 years old were based on data from the cohort aged 0–24 years old adjusted for all causes of mortality (0.34%). Migration was ignored. The tuberculin-positive cohort had a sharp decline in the previous two age cohorts; therefore, it was assumed that this continued. Lastly, the efficacy was estimated as being that seen in the cohort aged 20–24 years old but with the same decline in protection seen between the last two cohorts. These assumptions allowed notification rates to be estimated for the population aged 25–29 years old, resulting in a complete cohort over the projected 15 years of BCG effectiveness.

Original construction of forward estimates

Based on these estimated notification rates, Sutherland and Springett then sought to quantify the ongoing risk of developing notified TB, projected forward in time, for both the vaccinated and unvaccinated populations. To construct these estimates, several key assumptions, based on the results seen in the previous surveys, were made. First, it was assumed that efficacy was not degrading within the ethnic white population, and therefore historic estimates would continue to apply into the future. Additionally, it was assumed that the annual decay of 9% in notification rates, across all ethnic white populations, would continue indefinitely.

These assumptions allowed the notification rates in both the BCG vaccinated and unvaccinated groups to be projected forward in time. By assuming that the schools scheme is responsible for the observed variation between vaccinated and unvaccinated rates, the rate of prevented cases can then be estimated. By scaling this against a cohort of 100 000 13-year-old individuals, the number of prevented cases over a 15-year period can be projected for each cohort. By dividing the total number in a given cohort by the number of prevented cases, the estimated number of vaccines required to prevent a single case in the 15-year period can then be calculated.

To estimate the total number of prevented notifications for each cohort in England and Wales, the total number receiving the BCG and the coverage of the schools scheme were required. The coverage of the BCG schools scheme was estimated from annual reports of the Department of Health and Social Security (DHSS) and was assumed to be 75% for all years. The number of BCG vaccines given each year was estimated from the DHSS returns for the years 1967–1981; it was then taken as 75% of the estimated ethnic white population aged 13 years from 1982 to 1996; for each 5-year period thereafter, it was assumed to be 2.1 million.

Using the data on BCG coverage, the number of vaccines given each year and the projected differences between vaccinated and unvaccinated notification rates allowed the number of prevented notifications, due to vaccination, for each age group to be found for each year. These estimates can then be used to give the total number of prevented notifications for those aged between 15 and 29 years. To understand these estimates, estimates of the projected yearly notifications if the scheme continues were required. These totals were derived from the vaccinated and unvaccinated rates supplemented with similar projections from the tuberculin-positive or otherwise ineligible sourced from the 1983 BCG survey.13

Original transmission chain model

Sutherland and Springett defined their TB transmission model as follows:

The total expected number of secondary notifications () arising from any single primary notification was estimated as

where is the percentage annual decay in notification rates, and is the average interval between the notification of any individual and the notification of the patient who infected them.

The total expected number of secondary notifications arising from any single primary notification () is related to the number of notifications in each generation using the relative generation size () with the following power series:

The expected average interval between each primary notification and all secondary notifications () is defined to be the sum of time to all notifications, weighted by the fraction in each generation, divided by the sum of all notifications.

Both 2 and 3 are only valid when .

Updating the transmission chain model

If we assume a constant decay rate of % per year and that the next generation of secondary cases is notified years after the person who infected them, then 1 notification in year 0 will result in secondary notifications years later. This is therefore the relative generation size (), not the total number of secondary notifications (), as (incorrectly) stated by Sutherland and Springett. Therefore, to correct this we revised 1. to the following in our updated model,

Sutherland and Springett gave annual estimates of when secondary cases occurred. We were unable to reproduce this using the original model and therefore added an additional model step for validation purposes. This did not impact our results as we used the the total number of secondary notifications occurring due to primary notifications rather than an annual estimate as our measure of impact. Implementing the model also required several additional assumptions not detailed in.10 See the online supplemental information for details.

Updating model parameter estimates and incorporating parameter uncertainty

Incidence rates were included as point estimates in Sutherland and Springett10; in the updated model, we included uncertainty in these rates. We did this by first estimating notifications for 1973, 1978 and 1983 using published incidence rates and population estimates. Samples were then generated using a Poisson distribution.10 13 These samples were then used to estimate a distribution of incidence rates to replace the point estimates used in the original analysis. Sutherland and Springett assumed a serial interval of 2 years between linked notifications. Using a newly available literature source, we updated this assumption with an estimate of 1.44 (95% CI 1.29 to 1.63) years.15

We considered the original assumption of a 9% annual decrease in incidence rates as well as three scenarios based on those considered by the JCVI BCG subgroup11 12: these were a 3.9% decrease, a 1.9% decrease and no change annual in incidence rates. Data on the annual decrease in incidence rates in the ethnic white UK population were not available, so we used two proxy measures. The first proxy measure was the annual change in notifications in England and Wales, which was estimated using data from Public Health England. The SD of this measure was then calculated using the pr;op.ts;t function in R.16 The second proxy used was the annual decrease in the UK born age-specific incidence rates in the English population. These were calculated using notification data from the Enhanced TB Surveillance System (ETS) and the June Labour Force Survey.9 Incidence rates were estimated using the epiR package.17 Uncertainty was incorporated by sampling from a normal distribution for both proxy measures. Data collection for the ETS began in 2000, and prior to this notification, data were only available in years with notifications surveys (1973, 1978 and 1983). We therefore estimated incidence rates between 1984 and 1999, and for the years between notification surveys (1974–1977 and 1979–1982) using locally estimated scatter plot smoothing (LOESS) regression fitted to incidence rates published in Sutherland and Springett10 and the estimated incidence rates from 2000 onwards. LOESS is a local regression method that combines multiple regression models in a k-nearest neighbours meta-model.18 This approach allows non-linear trends to be fitted using a series of linear models. For years prior to 1973, the annual decreases were assumed to be the mean of the annual decreases from the previous 3 years. For both proxy measures, the annual decreases in incidence rates post-2016 were assumed to be the average of the estimates in 2013–2015.

Statistical analysis

For each scenario, we ran the model for 69 years (1969–2028) with 10 000 parameter samples. We tested the difference between scenarios using the Mann-Whitney test for the number of vaccines needed to prevent a single case in 15 years after vaccination for a cohort aged 13–14 years old at vaccination. As in Sutherland and Springett,10 a 15-year time horizon was used with 5-year intervals. The year closest to the year of the change in vaccination policy (2005), which had model estimates, was used as the baseline.

Patient and public involvement

We did not involve patients or the public in the design or planning of this study.

Results

Model validation

Our model produced results that were comparable with those from Sutherland and Springett10 (online supplemental table S1) when the original model structure and assumptions were used. When estimating the total notifications from ending the BCG schools scheme at different times in ethnic white UK-born adults aged 15–29 years old in England and Wales, our model had a median absolute error of 2.03% (2.5; 97.5% quantiles (Q): 0.00%, 3.72%) and a maximum absolute error of 3.91% when compared with Sutherland and Springett.10

Annual change in TB incidence rates

We found that the assumption of a 9% annual decrease in incidence rates in the ethnic white UK-born was not comparable to estimates using either notification data or age-specific incidence rates in the time period studied (figure 1).

Figure 1

Annual percentage change in ethnic white UK-born incidence rates for those aged 15–19, 20–24 and 25–29 years old under different scenarios. For the notification and incidence rate scenarios, each line represents the median of 10 000 parameter samples.

Vaccines required to prevent a single notification

We found that incorporating uncertainty did not alter the number of vaccines required to prevent a single notification within 15 years in a cohort vaccinated at school age, when the annual decrease in incidence rates was assumed to be 9% (online supplemental table S2). However, the updated estimate had a wide range (15 000 (2.5; 97.5% Q: 12000, 19000) vaccines required in 2004). As the assumed annual decrease in incidence rates was reduced the number of vaccines required to prevent a single notification was also reduced. Assuming an annual decrease of 1.9% (one of the scenarios evaluated by the JCVI) resulted in an estimate of 1600 (2.5; 97.5% Q: 1300, 2000) vaccines required to prevent a single notification in 2004. This assumption was the most comparable, although not equivalent, to estimates derived using notifications (1400 (2.5; 97.5% Q: 1100, 1700), p=0.077) and age-specific incidence rates (1500 (2.5; 97.5% Q: 460, 4900), p=0.083). The estimate using incidence rates had a high degree of uncertainty (figure 2). The number of vaccines required increased slightly over time with 1800 (2.5; 97.5% Q: 1500, 2200) required in 2009, 2000 (2.5; 97.5% Q: 1600, 2500) required in 2014 and 2200 (2.5; 97.5% Q: 1800, 2700) required in 2019 when an annual decrease of 1.9% in incidence rates was assumed.

Figure 2

Vaccines required in a cohort of those vaccinated at school age to prevent a single case of tuberculosis within 15 years of vaccination in 2004, 2009, 2014 or 2019. The years presented were dictated by the 5-year timestep of the model. The percentage annual decrease scenarios considered were based on those considered by the Joint Committee on Vaccination and Immunisation BCG subgroup, with the addition of a scenario using aggregate notification data and a scenario using estimates of age-specific incidence rates in the UK born. Each boxplot summarises the output of 10 000 model simulations for each scenario.

Average annual additional cases from ending the BCG schools scheme at various dates

We found that updating parameter values and incorporating uncertainty did not alter the average annual primary additional notifications from stopping the BCG schools scheme when the annual decrease was assumed to be 9%. However, when these changes were combined with the updated transmission model, we found that the impact of ending BCG vaccination was greater than previously reported with an increase in the number of estimated cases due to onward transmission (online supplemental tables S3 and S4). These estimates were uncertain with 94 (2.5; 97.5% Q: 72, 119) additional annual notifications if vaccination was stopped in 2001. As the assumed annual decrease in incidence rates was reduced, the annual number of additional notifications increased with 6099 (2.5; 97.5% Q: 4691, 7719) notifications when the annual decrease was assumed to be 1.9% and vaccination stopped in 2001. The number of annual notifications reduced with time: 5314 (2.5; 97.5% Q: 4082, 6725) from ending vaccination in 2006; 4327 (2.5; 97.5% Q: 3315, 5496) from ending vaccination in 2011; and 2852 (2.5; 97.5% Q: 2074, 3741) from ending vaccination in 2016 (figure 3).

Figure 3

Annual additional (stratified into primary and secondary) notifications in 15–29 year olds from stopping the BCG schools scheme in 2006, and 2011–2028. The years presented were dictated by the 5-year timestep of the model. The percentage annual decrease scenarios considered were based on those considered by the JCVI BCG subgroup. Data-based scenarios and the JCVI 0% decrease scenario were not presented here as the updated transmission model could not support these scenarios. Each boxplot summarises the output of 10 000 model simulations for each scenario. Secondary notifications are reported, assuming they occurred in the same year as the primary notifications that caused them. JCVI, Joint Committee on Vaccination and Immunisation.

Discussion

The existing method for estimating the impact of the BCG schools scheme produced uncertain estimates of the impact of ending the scheme in all years evaluated when parameter uncertainty was included. The approach used to estimate additional notifications due to transmission was found to be incorrect. Once corrected, the transmission model produced much higher estimates of additional notifications caused by ending BCG vaccination than previously reported. Further updating the model with the annual decrease in TB notifications based on both notifications and using age-specific incidence rates resulted in a decrease in the number of vaccines needed to prevent a single case in all years considered. A scenario with a 1.9% annual decrease in incidence rates was most comparable to our results based on notifications. Using this scenario, we found that the number of TB notifications arising from ending school-age BCG vaccination was much greater than originally estimated using the scenario considered in Sutherland and Springett.

This study reassesses a key piece of the quantitative evidence used to motivate the change in BCG vaccination policy in 2005.

In addition to identifying that the public health impact of the change in vaccine policy was likely much larger than originally estimated, our results also provide new insights into the uncertainty of the previously published model predictions by including parameter uncertainty and measurement error and update these predictions using newly available data. As historical data on incidence rates in the ethnic white UK-born in England and Wales were not available, we considered two approaches to proxy them and investigated multiple scenarios based on those explored by the JCVI BCG subgroup. The simulation approach used here, although updated where possible, is not the most accurate method for estimating the impact of ending the BCG schools scheme as it relies on numerous assumptions based on the available knowledge in 1987 and does not account for the role of non-white and non-UK-born cases. However, the strength of this work is that the estimates are based on the framework used to inform policy making. This allowed the strength of the model used in the decision-making process to be assessed once parameter uncertainty had been incorporated and for flaws in the model to be identified. This would not have been possible if the impact had been assessed using only the observed data or with an alternative model. It also allowed estimates based on updated data to be compared with historic estimates within the same framework. This would also not have been possible if a different framework had been used. As mentioned, a weakness of the model used in this study is that it did not include the whole population or age groups outside those directly affected by vaccination. Heterogeneous mixing between these groups is also likely to be an important consideration. The exclusion of these factors means that our results likely underestimate the impact of ending the BCG schools scheme. A final limitation is that this study considers only the impact of ending the BCG schools scheme and not the impact of the introduction of the targeted neonatal vaccination programme. This should be considered when evaluating the change in policy as a whole.

Little work has been done to assess the impact of the 2005 change in BCG vaccination policy or to assess the quantitative evidence used in decision making. However, multiple studies have evaluated the cost effectiveness of various BCG programmes and the impact of switching between them. A cluster-randomised trial in Brazil found that BCG vaccination of those at school age was cheaper than treatment and would prevent one TB case per 381 vaccinations even with a vaccine effectiveness of only 34% (8%–53%).19 This is substantially fewer than our estimate of 2000 (2.5; 97.5% Q: 1600, 2500) (the most comparable year from our study). However, the same trial found that for regions close to the equator, BCG effectiveness was low in school-age children but unchanged in neonates,20 highlighting the importance of considering the BCG vaccines reduced effectiveness near the equator when determining vaccination policy.21 There is also some research which supports universal revaccination of those at school age, in countries with high incidence and universal vaccination of neonates, as it may be cost effective when BCG effectiveness is moderate to high.20 22 There is some evidence that targeted vaccination of high-risk neonates may be more cost effective than universal vaccination of neonates.23 24 However, a study in Sweden found that incidence rates in Swedish-born children increased slightly after universal vaccination of neonates was discontinued in favour of targeted vaccination.25 In France, which switched from universal vaccination of neonates to targeted vaccination in 2007, it has also been shown that targeted vaccination reduced coverage in those most at risk.26 Targeted vaccination may not be more cost effective than universal vaccination when possible reductions in transmission are considered. Our study indicated that a substantial number of cases due to transmission may be preventable if universal school-age BCG vaccination was still in place. This result is dependent on the effectiveness of BCG vaccination when given later in life, for which there is good evidence in the ethnic white UK-born.8 We did not consider neonatal vaccination, which would be less impacted by BCG’s effectiveness reducing when given later in life, but may also be less likely to result in the same reductions in ongoing transmission.

This study indicates that some of the evidence used to justify the 2005 change in BCG vaccination policy may have depended on a methodologically flawed model, resulting in the impact of ending BCG vaccination being underestimated. Modelling evidence can often be complex and difficult to reproduce; it is important that policy makers, or those who work with them, have the skills to assess its quality. This study also highlights the importance of including both parameter and measurement error, as excluding these sources of variation may lead to spuriously precise results. Spurious precision is problematic for policy makers as the worst-case scenario often needs to be considered when making policy decisions. In addition, our exploration of the assumptions used to estimate the annual change in TB incidence rates in the ethnic white UK-born illustrates the structural impact of assuming an annual decrease in TB incidence rates. More realistic estimates of the annual decrease in incidence rates resulted in a greatly increased impact of ending the BCG schools scheme. Policy makers should consider these updated estimates when assessing the role of BCG vaccination in those at school age. However, decisions regarding vaccine policy in the UK require economic evaluation, which discounts costs and benefits in the future; discounting has not been applied in this study, which estimates the epidemiological impact of vaccination only.

This study has reassessed some of the evidence previously used in decision making, correcting the transmission model used and updating the approach with new data. However, as 15 years of detailed surveillance data have been collected since the ending of the BCG schools scheme, it is now possible to use regression-based approaches to estimate the direct impact on incidence rates of ending the BCG schools scheme.27 These approaches could also be used to estimate the impact of vaccinating high-risk neonates, which may outweigh any negative impacts of ending the BCG schools scheme. In addition, the development and use of a transmission dynamic model would allow a more accurate estimation of indirect effects and the forecasting of long-term impacts.