Vaccine-Preventable Disease Incidence Based on Clinically, Radiologically, and Etiologically Confirmed Outcomes: Systematic Literature Review and Re-analysis of Pneumococcal Conjugate Vaccine Efficacy Trials.

BACKGROUND: Vaccine regulatory decision making is based on vaccine efficacy against etiologically confirmed outcomes, which may underestimate the preventable disease burden. To quantify this underestimation, we compared vaccine-preventable disease incidence (VPDI) of clinically defined outcomes with radiologically/etiologically confirmed outcomes.
METHODS: We performed a systematic review of efficacy trials for several vaccines (1997-2019) and report results for pneumococcal conjugate vaccines. Data were extracted for outcomes within a clinical syndrome, organized from most sensitive to most specific. VPDI was determined for each outcome, and VPDI ratios were calculated, with a clinically defined outcome (numerator) and a radiologically/etiologically confirmed outcome (denominator).
RESULTS: Among 9 studies, we calculated 27 VPDI ratios; 24 had a value >1. Among children, VPDI ratios for clinically defined versus vaccine serotype otitis media were 0.6 (95% CI not calculable), 2.1 (1.5-3.0), and 3.7 (1.0-10.2); the VPDI ratios comparing clinically defined with radiologically confirmed pneumonia ranged from not calculable to 2.7 (1.2-10.4); the VPDI ratio comparing clinically suspected invasive pneumococcal disease (IPD) with laboratory-confirmed IPD was 3.8 (95% CI not calculable). Among adults, the ratio comparing clinically defined with radiologically confirmed pneumonia was 1.9 (-6.0 to 9.1) and with vaccine serotype-confirmed pneumonia was 2.9 (.5-7.8).
CONCLUSIONS: While there is substantial uncertainty around individual point estimates, there is a consistent trend in VPDI ratios, most commonly showing under-ascertainment of 1.5- to 4-fold, indicating that use of clinically defined outcomes is likely to provide a more accurate estimate of a pneumococcal conjugate vaccine's public health value.

Pneumococcal disease is a major cause of morbidity and mortality worldwide and affects predominantly young children and older adults. According to the 2016 Global Burden of Diseases [1], Streptococcus pneumoniae was the leading cause of lower respiratory infection morbidity and mortality globally, resulting in more deaths than all other etiologies combined in 2016: 1.2 million deaths due to S. pneumoniae including 0.34 million deaths in children younger than 5 years old, and 0.50 million deaths in elderly adults older than 70 years. Pneumococcal conjugate vaccine (PCV) clinical trials have demonstrated vaccine efficacy (VE) against vaccine-serotype (VST) etiologically confirmed outcomes, including those within the category of radiologically confirmed pneumonia [2]. Different PCVs have been licensed since 2000 that have been implemented worldwide in national immunization programs. When a new vaccine is licensed, policy makers and vaccine technical committees mostly rely on data from efficacy trials to make decisions on clinical recommendations or vaccine introduction. Nevertheless, a vaccine-preventable disease burden that is solely based on etiologically and radiologically confirmed outcomes will always underestimate the full public health impact of a PCV vaccination program. However, the extent of this underestimation, and hence its relevance for policy making, has not been systematically assessed.

First, VE is a relative measure not accounting for background disease incidence, and as such does not quantify the amount of preventable disease. A complementary measure to VE is the vaccine-preventable disease incidence (VPDI), also known as vaccine-attributable rate reduction, absolute rate reduction, or incidence rate reduction. The VPDI measures the difference in an outcome’s incidence between unvaccinated and vaccinated populations in a given epidemiological context [3] (which is mathematically equivalent to VE × unvaccinated group incidence). Second, etiologically confirmed outcomes, while specific and thus useful for estimating VE, lack sensitivity due to diagnostic under-ascertainment (eg, patients may not get a test or the test sensitivity may be low), in turn leading to an underestimation of the true VPDI for the disease syndrome in question.

The objective of this study was to quantitatively assess the degree to which the PCV-associated VPDI for pneumococcal outcomes is underestimated by using etiologically or radiologically defined outcomes versus clinically defined outcomes, as assessed in phase III or IV randomized controlled trials (RCTs) in either children or adults. Our hypothesis was that VDPI would be substantially greater for clinically defined versus etiologically or radiologically confirmed outcomes, and thus should be used to inform decision making in vaccine policy. The current article reports our findings among non-risk (healthy) subjects.

METHODS

We conducted a systematic literature review and re-analysis of phase III/phase IV trials for PCVs. We searched for all reported trial outcomes, ranging from more sensitive but less specific outcomes (eg, all-cause clinically defined disease) to more specific but less sensitive outcomes (eg, etiologically confirmed disease, such as VST pneumococcal outcome, or radiologically defined outcomes). For each trial, we calculated VPDI ratios within the same clinical syndrome by comparing the VPDI value of a clinically defined outcome with the VPDI value of an etiologically or radiologically confirmed outcome.

Search Strategy

A systematic literature review was performed for phase III/phase IV trials for conjugate vaccines against pneumococcus, a subset of the full protocol that has been prospectively registered at PROSPERO (CRD42019145268). We searched MEDLINE (PubMed), EMBASE, and clinicaltrials.gov for relevant articles published between 1 January 1997 and 31 July 2019. Search terms included those for the organism, study design, the vaccine, and efficacy or effectiveness. Details of the search strategy are provided in the protocol. The systematic literature review was performed using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines [4]. Citations from eligible articles were also searched to identify other relevant studies. In the present publication, we focus on PCV studies, including PCV candidates that were not subsequently submitted for licensure.

Inclusion Criteria

PCV studies had to meet the following PICOTS inclusion criteria—(1) Population: persons of any age and gender, participating in phase III/phase IV trials, excluding risk groups, when possible, to improve homogeneity of study populations; (2) Intervention: PCV; (3) Comparisons: subjects exposed versus unexposed to PCV (the unexposed group could have been exposed to another vaccine if this vaccine had no expected impact on the outcome of interest); (4) Outcomes: clinically defined, radiologically confirmed, or etiologically confirmed outcomes. Publications in the following languages were included: English, French, Spanish, Portuguese, Dutch, German, and Italian.

Study Selection Process and Data Extraction

A single author (M. B.) screened studies through the electronic searches based on titles and abstracts, using Rayyan [5]. A second author (G. H.) conducted a full-text review of the initially selected pneumococcal studies to make a final selection of eligible studies for data extraction. Data were extracted by a single reviewer (G. H.) and quality control of data extraction was performed by a second reviewer (K. B.) through re-extraction of 10% of the papers. Uncertainties were settled through discussion with the entire study team. The reasons for exclusion refer to the PICOTS presented above under “Search Strategy.” These were documented (see Figure 1 and Supplementary Material, Protocol) in accordance with PRISMA guidelines [4] and included: (1) no original publication, (2) inappropriate population, (3) inappropriate intervention, (4) inappropriate comparator, (5) inappropriate outcome, (6) inappropriate study design, (7) no multiple outcomes, (8) no clinically diagnosed outcome, and (9) incomplete information.

Figure 1.

PRISMA diagram. Abbreviation: PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-Analyses.

Variables for data extraction were predefined in the protocol and included the following: clinical trial reference number, first author name, journal, year of publication, country/ies, study period, clinical trial phase, study design, study population description, intervention vaccine, intervention schedule (including age at vaccination), comparator group, case ascertainment, study outcomes, follow-up duration, number of events and person-time in the intervention and comparator groups, and VE with associated 95% confidence intervals (CIs). Data corresponding to the intention-to-treat (ITT) analysis were prioritized for extraction but, when not available, per-protocol (PP) analysis data were extracted.

Selection of Outcomes

We extracted first and all episode data for outcomes belonging to 1 of the 3 clinical syndromes most associated with pneumococcal disease: (1) otitis media (OM), (2) pneumonia, and (3) invasive pneumococcal disease (IPD). Outcomes within the same vaccine trial were then hierarchically organized from most sensitive to most specific within a clinical syndrome.

Exclusion From Analysis

Data were subsequently excluded from analysis in case of (1) absence of multiple, hierarchically organized outcomes within a clinical syndrome, (2) absence of at least 1 clinically defined outcome and at least 1 radiologically or etiologically confirmed outcome, or (3) insufficient information to obtain person-time by comparison group.

Quality Assessment

Quality assessment of the selected trials was done using the revised Cochrane risk-of-bias tool for randomized trials (RoB 2) [6]. This tool is structured into 5 domains pertinent to the design and conduct of randomized parallel-group trials: randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result. Within each domain, a series of questions (“signaling questions”) aims to elicit information about features of the trial that are relevant to risk of bias. Based on an algorithm, the risk of bias is then categorized as “low risk,” “some concerns,” or “high risk.”

Statistical Analysis

For each trial and each outcome within the trial, the VPDI (per 1000 person-years) was calculated as the incidence in the comparator group minus the incidence in the PCV group. The corresponding 95% CIs were calculated using the test-based method [7]. For all outcomes within the same trial that could be hierarchically organized within a clinical syndrome from most sensitive to most specific, VPDI ratios were calculated with clinically defined outcomes as numerator and with radiologically or etiologically confirmed outcomes as denominator. A VPDI ratio above 1 indicates that—within the same clinical syndrome—the VPDI value estimate for the more sensitive outcome is greater than the VPDI value estimate for the more specific outcome, and thus the VPDI ratio provides the degree to which a more sensitive outcome identifies more vaccine-preventable disease. For the VPDI ratios, 95% CIs were calculated using a nonparametric bootstrap procedure that accounted for the hierarchical structure of the data [8]. Starting from the reported number of cases per outcome, event profiles were created reflecting this hierarchical structure. These event profiles were then resampled, and for each bootstrap sample the VPDI ratio was calculated while keeping person-time fixed (because the individual-level data were not available to us). For each study, we generated 10 000 bootstrap samples, upon which the 95% bootstrap percentile CIs were calculated [8]. All analyses were done with R (version 3.6.1; R Foundation for Statistical Computing).

RESULTS

Study Selection

We identified 371 articles during the selection process (Figure 1). Of the 371 articles, 29 were considered relevant following abstract or full-text review. Of these, 10 articles reporting on 10 PCV trials had data that could be used for analysis (See Supplementary Table 1 for studies excluded, by reason, after full-text review). Nine articles reported pediatric outcomes, and 1 article reported adult outcomes (Table 1) [9-18]. By clinical syndrome, 1 article reported on multiple clinical syndromes, 3 articles reported on OM, 7 on pneumonia, and 1 reported on IPD (Table 2). The hierarchical organization of the outcomes retained for analysis is graphically represented in Figure 2. While we used a common terminology to describe the different outcomes across studies, the precise outcome definitions provided in the source manuscripts are summarized in Supplementary Table 2. For each outcome within each trial, the number of events, person-time, VE, VPDI, and 95% CIs are provided in Supplementary Table 3.

Table 1.

Characterization of Randomized Placebo-Controlled Pneumococcal Conjugate Vaccine Trials

Study [Ref]	Trial Number	Country	Vaccine Formulationa	Schedule	Analysis	Study Period	Pediatric NIPb	Study Population
Black, 2002 [9]	N.S.	US	PCV7 (Pfizer)	3p + 1 (2, 4, 6, 12–15 m)	ITT	1995–1999	No	Children (< 1 y), healthy
Cutts, 2005 [10]	ISRCTN46147225	GM	PCV9 (Pfizer)c	3p + 0 (6, 10, 14 wks)	PP	2000–2005	No	Children (< 1 y), healthy
Eskola, 2001 [12]	NCT00378417	FI	PCV7 (Pfizer)	3p + 1 (2, 4, 6, 12 m)	PP	1995–1999	No	Children (< 1 y)
Gessner, 2019 [18]	NCT744263	NL	PCV13 (Pfizer)	1 dose (≥65 y)	ITT	2008–2013	Yes	Adults (≥65 y), not i.c.
Kilpi, 2018 [13]d	NCT00861380/NCT00839254	FI	PCV10 (GSK)	2p + 1 or 3p + 1 (<7 m at d1)	ITT	2009–2011	No	Children (< 1 y)
Lucero, 2009 [14]	ISRCTN62323832	PH	PCV11 (Sanofi)c	3p + 0 (6, 10, 14 wks)	ITT	2000–2004	No	Children (< 1 y), healthy
Madhi, 2005 [15]	N.S.	ZA	PCV9 (Pfizer)c	3p + 0 (6, 10, 14 wks)	ITT	1998–2001	No	Children (<1 y), HIV-negative e
Palmu, 2018 [16]d	NCT00861380/NCT00839254	FI	PCV10 (GSK)	2p + 1 or 3p + 1 (<7 m at d1)	ITT	2009–2012	No	Children (< 1 y)
Prymula, 2006 [17]	NCT00119743	CZ, SK	PCV11 (GSK)c	3p + 1 (2, 4, 6, 12–15 m)	PP	2000–2002	No	Children (< 1 y), healthy
Tregnaghi, 2014 [11]	NCT00466947	PA, AR, CO	PCV10 (GSK)	3p + 1 (2, 4, 6, 12–15 m)	ITT	2007–2011	No	Children (< 1 y), healthy

Abbreviations: AR, Argentina; CO, Colombia; CZ, Czech Republic; d, dose; FI, Finland; GM, Gambia; i.c., immunocompromised; ITT, intention-to-treat; m, months; NIP, National Immunization Program; NL, Netherlands; N.S., not specified; p, primary doses; PA, Panama; PCV, pneumococcal conjugate vaccine; PH, Philippines; PP, per-protocol; Ref, reference; SK, Slovakia; wks, weeks; y, year; ZA, South Africa.

aPCV7, PCV9, PCV10, PCV11, and PCV13 refer to 7-, 9-, 10-, 11-, and 13-valent pneumococcal conjugate vaccines; vaccine serotypes: PCV7 [4, 6B, 9V, 14, 18C, 19F, 23F]; PCV9 [PCV7 + 1, 5]; PCV10 [PCV9 + 7F]; PCV11 [PCV9 + 3, 7F]; PCV13 [PCV11 + 6A, 19A].

bNational immunization program (PCV7, PCV9) at the time of the study.

cVaccine formulations that were not subsequently licensed.

dStudies reporting on the same trial.

eStudy involving human immunodeficiency virus (HIV)–positive and HIV-negative subjects. This review reports on HIV-negative subjects only.

Table 2.

Outcomes of Randomized Placebo-Controlled Pneumococcal Conjugate Vaccine Trials Organized by Clinical Syndromes and Outcomes

Outcomea	Episodes	OM Clinical Diagnosis (O-5)	Confirmed by MEF Detection (O-4)	Any Bacteria Identification (O-3)	Sp Identification (O-2)	VST Sp Identification (O-1)
Otitis media
Eskola, 2001 [10]	All	X	X		X	X
Prymula, 2006 [16]	All	X	X	X	X	X
Tregnaghi, 2014 [17]	All	X		X	X	X
		LRTI Clinical Diagnosis (P-5)	Pneumonia Clinical Diagnosis (P-4)	Confirmed by X-ray (P-3)	Sp Identification (P-2)	VST Sp Identification (P-1)
Pediatric pneumonia or adult pneumonia
Black, 2002 [8]	First		X	X
Cutts, 2005 [9]	First		X	X
Gessner, 2019 [11]	All		X	X	X	X
Kilpi, 2018 [12]	All		X	X
Lucero, 2009 [13]	First		X	X
Madhi, 2005 [14]	First	X	X	X	X	X
Tregnaghi, 2014 [17]	First		X	X
		ICD-Defined IPD or Unspecified Sepsis (I-2)			Sp Identification (I-1)
Invasive pneumococcal disease
Palmu, 2018 [15]	All	X			X

Abbreviations: ICD, International Classification of Diseases; IPD, invasive pneumococcal disease; LRTI, lower respiratory tract infection; MEF, middle-ear fluid; OM, otitis media; Sp, Streptococcus pneumoniae; VST, vaccine serotype.

aNomenclature used for levels of outcomes hierarchically organized within clinical syndromes is as follows: O for otitis media, P for pneumonia, and I for invasive pneumococcal disease. Numbering varies by clinical syndromes from 1 up to 5, where 1 is the most specific outcome; the highest number within the clinical syndrome corresponds to the most sensitive outcome.

Figure 2.

Illustrations of potential levels for outcomes, hierarchically organized within clinical syndromes. Note. Nomenclature used for potential levels of outcomes hierarchically organized within clinical syndromes is as follows: O for otitis media, P for pneumonia, and I for invasive pneumococcal disease. Numbering varies by clinical syndromes from 1 to 5, where 1 is the least sensitive and most specific outcome. The highest number within the clinical syndrome corresponds to the most sensitive and least specific outcome. Abbreviations: ICD, International Classification of Diseases; IPD, invasive pneumococcal disease; LRTI, lower respiratory tract infection; MEF, middle ear fluid; OM, otitis media; Sens, sensitivity; Sp, Streptococcus pneumoniae; Spec, specificity; VT, vaccine-type; unspec., unspecified.

The risk of bias due to the selection of the reported result was judged to be not applicable for this systematic review because it concerned a re-analysis of all reported outcomes (Supplementary Table 4). For the other 4 domains in our quality assessment, the risk of bias was “low,” except for 3 studies that included PP analysis data only. The PP analysis included only subjects who received all doses. The impact of this on the study results was not considered substantial. As such, these studies were judged as having “some concerns” of bias, due to deviations from the intended interventions, but were not judged as “high risk” of bias.

Syndrome-Specific VPDI Ratios

Three pediatric OM studies were included from which we calculated 10 VPDI ratios, for which 8 VPDI ratios were 1 or greater. The 3 VPDI ratios comparing clinically defined OM (the most sensitive outcome) with VST pneumococcal OM (the most specific outcome) were 0.6 (95% CI not calculable), 2.1 (95% CI: 1.5–3.0), and 3.7 (95% CI: 1.0–10.2) (Table 2) [11, 12, 17].

Among the 6 studies reporting pediatric pneumonia outcomes, 1 study reported a VPDI less than 0 for clinical pneumonia and thus the VPDI ratio was not calculated. The remaining 5 studies allowed calculation of 13 VPDI ratios (Table 3) [14]. The VPDI ratios comparing clinically defined pneumonia (more sensitive outcome) with radiologically confirmed pneumonia (more specific outcome) had values that ranged from 1.1 (95% CI: .3–1.4) to 2.7 (95% CI: 1.2–10.4), with a median of 1.7 [10, 15]. One of the pediatric pneumonia studies reported lower respiratory tract infection (LRTI) as the most sensitive clinically defined outcome [15]. For this study, the VPDI ratio comparing LRTI with radiologically confirmed pneumonia was 1.7 (95% CI: −0.9 to 6.6). Substantially greater VPDI ratios were calculated comparing clinical pneumonia with etiologically confirmed pneumonia (which was based on culture since there is not a validated serotype-specific urinary antigen detection assay for pediatrics) (Table 3) [19].

Table 3.

Vaccine-Preventable Disease Incidence Ratios Calculated From Randomized Placebo-Controlled Pneumococcal Conjugate Vaccine Trials Organized by Clinical Syndromes

		Numerator				Denominator				Ratio
Study [Ref]	Vaccine, Population,a Schedule	Outcome (Levelb)	Vaccine Group,n (py)	Control Group,n (py)	VPDI [95%CI] (/1000 py)	Outcome (Levelb)	Vaccine Group,n (py)	Control Group,n (py)	VPDI [95%CI] (/1000 py)	VPDI ratio [95% CI]
Otitis media
Eskola, 2001 [10]	PCV7, ped., 3p + 1	OM clinical (O-5)	1251 (1078)	1345 (1085)	80.0 [−12.3 to 172.3]	MEF detection (O-4)	1177 (1078)	1267 (1085)	76.7 [−12.9 to 166.3]	1c
						Sp (O-2)	271 (1078)	414 (1085)	130.4 [83.0–177.8]	0.6c
						VST Sp (O-1)	107 (1078)	250 (1085)	131.3 [97.0–165.5]	0.6c
Prymula, 2006 [16]	PCV11, ped., 3p + 1	OM clinical (O-5)	333 (3958)	499 (3882)	44.4 [30–58.9]	MEF detection (O-4)	322 (3958)	474 (3882)	40.8 [26.7–54.9]	1.1 [1.0–1.2]
						Any bacteria (O-3)	178 (3958)	306 (3882)	33.9 [22.9–44.9]	1.3 [1.0–1.7]
						Sp (O-2)	92 (3958)	189 (3882)	25.4 [17.1–33.8]	1.7 [1.3–2.4]
						VST Sp (O-1)	60 (3958)	141 (3882)	21.2 [14.1–28.3]	2.1 [1.5–3.0]
Tregnaghi, 2014 [17]	PCV10, ped., 3p + 1	OM clinical (O-5)	254 (9018)	308 (8835)	6.7 [1.5–11.9]	Any bacteria (O-3)	45 (9018)	67 (8835)	2.6 [.3–4.9]	2.6 [.6–10]
						Sp (O-2)	17 (9018)	38 (8835)	2.4 [.8–4]	2.8 [.8–7.7]
						VST Sp (O-1)	7 (9018)	23 (8835)	1.8 [.6–3]	3.7 [1.0–10.2]
Pediatric pneumonia
Black, 2002 [8]	PCV7, ped., 3p + 1	Pneumo clinical (P-4)	1712 (39 354)	1804 (39 378)	2.3 [−0.6 to 5.3]	X-ray (P-3)	327 (39 354)	398 (39 378)	1.8 [.5–3.1]	1.3 [−.5 to 3.7]
Cutts, 2005 [9]	PCV9, ped., 3p + 1	Pneumo clinical (P-4)	2172 (9382)	2284 (9191)	17 [2.9–31.1]	X-ray (P-3)	333 (12 808)	513 (12 543)	14.9 [10.4–19.4]	1.1 [.3–1.4]
Kilpi, 2018 [12]	PCV10, ped.	Pneumo clinical (P-4)				X-ray (P-3)
<7 mo	2p + 1 or 3p + 1		398 (40 612)	271 (20 376)	3.5 [1.7–5.3]		197 (40 612)	138 (20 294)	1.9 [.7–3.2]	1.8 [1.1–3.7]
7–11 mo	2p + 1		132 (14 831)	82 (7130)	5 [1.0–9]		69 (14 831)	45 (7143)	3.8 [.9–6.8]	1.3 [.4–2.9]
12–18 mo	1p + 1		88 (8713)	65 (4305)	2.6 [−0.2 to 5.4]		45 (8713)	39 (4333)	1.6 [−.4 to 3.7]	1.6 [−3.6 to 6.9]
Lucero, 2009 [13]	PCV11, ped., 3p + 0	Pneumo clinical (P-4)	1093 (10 280)	1080 (10 234)	−0.8 [−9.7 to 8.1]	X-ray (P-3)	119 (10 276)	141 (10 240)	2.2 [−.9 to 5.3]	N.C.d
Madhi, 2005 [14]	PCV9, ped., 3p + 0	LRTI clinical (P-5)	1033 (43 338)	1106 (43 293)	1.7 [−0.4 to 3.8]	X-ray (P-3)	169 (43 338)	212 (43 293)	1.0 [.1–1.9]	1.7 [−.9 to 6.6]
						Sp (P-2)	5 (43 338)	8 (43 293)	0.1 [−.1 to .2]	24.7 [−87.5 to >100]
						VST Sp (P-1)	2 (43 338)	6 (43 293)	0.1 [0−.2]	18.5 [−51.4 to >100]
		Pneumo clinical (P-4)	566 (43 338)	681 (43 293)	2.7 [1.1–4.3]	X-ray (P-3)	169 (43 338)	212 (43 293)	1.0 [.1–1.9]	2.7 [1.2–10.4]
						Sp (P-2)	5 (43 338)	8 (43 293)	0.1 [−.1 to .2]	38.5 [−121.6 to >100]
						VST Sp (P-1)	2 (43 338)	6 (43 293)	0.1 [0–.2]	28.9 [−85.0 to >100]
Tregnaghi, 2014 [17]	PCV10, ped., 3p + 1	Pneumo clinical (P-4)	2667 (31 480)	2880 (31 265)	7.4 [2.7–12]	X-ray (P-3)	919 (31 480)	1015 (31 265)	3.3 [.5−6]	2.3 [1.1–7.5]
						X-ray+e (P-3a)	377 (31 480)	450 (31 265)	2.4 [.6–4.2]	3.1 [1.3–8.9]
Adult pneumonia
Gessner, 2019 [11]	PCV13, ad., 1d	Pneumo clinical (P-4)	1375 (167 874)	1495 (167 748)	0.7 [.1–1.3]	X-ray (P-3)	876 (167 874)	938 (167 748)	0.4 [−.1 to .9]	1.9 [−6.0 to 9.1]
						Sp (P-2)	144 (167 874)	185 (167 748)	0.2 [.0–.5]	2.9 [.2–11.5]
						VST Sp (P-1)	70 (167 874)	112 (167 748)	0.3 [.1–.4]	2.9 [.5–7.8]
Invasive pneumococcal disease
Palmu, 2018 [15]	PCV10, ped., 2p + 1 or 3p + 1	Clin. susp. IPD (I-2)			2.9	Confirmed IPD (I-1)			0.8 [.3–1.2]	3.8 c,f

Abbreviations: ad., adult; CI, confidence interval; Clin. susp., clinically suspected; IPD, invasive pneumococcal disease; LRTI, lower respiratory tract infection; MEF, middle-ear fluid; N.C., not calculated; OM, otitis media; p, primary doses; ped, pediatric; PCV, pneumococcal conjugate vaccine; py, person-year; Ref, reference; Sp, Streptococcus pneumoniae; unsp., unspecified; VPDI, vaccine-preventable disease incidence; VST, vaccine serotype.

aPCV7, PCV9, PCV10, PCV11, and PCV13 refer to 7-, 9-, 11-, and 13-valent pneumococcal conjugate vaccines; vaccine serotypes: PCV7 [4, 6B, 9V, 14, 18C, 19F, 23F]; PCV9 [PCV7 + 1, 5]; PCV10 [PCV9 + 7F]; PCV11 [PCV9 + 3, 7F]; PCV13 [PCV11 + 6A, 19A].

bLevels for ratios are based on illustrations in Figure 2. Numbers of levels vary by clinical syndromes: O for otitis media, P for pneumonia, and I for invasive pneumococcal disease. Numbering varies from 1 up to 5, where 1 is most specific outcome; the highest number within the clinical syndrome corresponds to the most sensitive outcome.

cInsufficient information to compute confidence intervals.

dRatio not calculated as the vaccine-preventable disease incidence was negative for the numerator and/or denominator.

eConfirmed by X-ray plus inflammatory marker/s.

fVPDI ratio was derived from incidences provided in the paper.

For the single adult study, which measured pneumonia outcomes, 3 VPDI ratios could be calculated, all of which were greater than 1. The calculated VPDI ratio of clinically defined to radiologically confirmed pneumonia was 1.9 (95% CI: −6.0 to 9.1) [18]. For adult nonbacteremic pneumonia, a validated serotype-specific urinary antigen detection assay exists [19]; the calculated VPDI ratio for clinically defined versus VST pneumococcal pneumonia was 2.9 (95% CI: .5–7.8).

One study reported clinically suspected IPD without laboratory confirmation (based on International Classification of Diseases [ICD] codes), allowing calculation of 1 VPDI ratio. The VPDI ratio compared laboratory- and non–laboratory-confirmed IPD or unspecified sepsis (as defined by the investigators) with laboratory-confirmed IPD, and was 3.8 (95% CI not calculated) [16].

DISCUSSION

We document here that reliance on radiologically or etiologically confirmed outcomes tends to underestimate the public health benefits of PCVs when compared with clinically defined outcomes, and for the first time quantify the extent of this underestimate. A VPDI ratio above 1 indicates that the preventable burden calculated on a specific outcome of a clinical trial of VE, such as an etiologically confirmed disease, underestimates burden reduction relative to the comparator condition in the same efficacy trial, such as the clinical syndrome that was used to screen subjects for further etiologic testing. Among 10 studies, 9 yielded 27 VPDI ratios (one had a VPDI ratio that could not be calculated); of these 27 VPDI ratios, 24 were greater than 1 and 3 were less than or equal to 1 (all from 1 pediatric OM study). Confidence intervals could be calculated for 23 of the 27 VPDI ratios. While for many individual VPDI ratios, wide CIs prevent definitive conclusions, the consistency of overall results supports the hypothesis that reliance on radiologically or etiologically confirmed outcomes will underestimate in most cases the public health benefits of PCVs when compared with clinically defined outcomes. This consistency in VPDI ratios greater than 1 remained regardless of whether the syndromic outcome was pediatric OM, pediatric or adult pneumonia, or pediatric IPD. Moreover, we have quantified with this re-analysis the degree of under-ascertainment—most commonly approximately 1.5-fold to 4-fold—which may inform immunization program decision making as well as economic models.

While we did not evaluate mechanisms for the VPDI ratios being greater than 1, numerous issues likely contributed, most importantly that chest radiographs and biologic tests have limited sensitivity [20]. For pneumonia chest radiography, sensitivity may be reduced because most studies include only a single view, which increases the risk that a lesion behind the heart will be missed [21]; furthermore, dehydration may reduce the radio-opaque nature of an infectious process [22]. Outcome definitions based on specific types of infiltrates—such as World Health Organization outcome consolidation for pediatrics and a consolidation consistent with community-acquired pneumonia (CAP) for adults [23, 24]—may not include the full spectrum of infiltrates caused by pneumococcus, particularly when influenced by access to medical care, timing of antibiotic use, and ongoing viral respiratory disease epidemics [25]. Similarly, etiologically defined outcomes depend on test sensitivity, which, in turn, may be reduced by early antibiotic use (eg, cultures for OM or IPD) or by the characteristics of a test, such as the serotype-specific urine antigen detection assay that was designed to emphasize specificity over sensitivity [26].

Among the 3 OM studies, 1 study reported all 3 VPDI ratios less than or equal to 1 including 2 values less than 1 [12], and for 1 pediatric pneumonia study, the VPDI for the clinical outcome was just below 0, which would have yielded a negative VPDI ratio [14]. In the absence of serotype replacement, hierarchically organized outcomes always should provide a VPDI ratio greater than or equal to 1, since more specific outcomes are logical subsets of more sensitive outcomes. However, PCV-induced pneumococcal serotype replacement, at population and individual levels, might simultaneously reduce VST disease and increase non-VST disease, which could lead to a VPDI ratio less than 1. The degree of measured replacement, in turn, may be related to study design. For example, compared with the control group incidence in the 2 OM studies with a VPDI ratio greater than 1 [11, 17], the OM study with a VPDI ratio less than 1 had a substantially larger incidence value in the control group (Supplementary Table 2). Additionally, general practitioners enrolled cases for the study with a VPDI ratio less than 1 in contrast to ear, nose, and throat specialist referrals for at least 1 of the 2 OM studies with a VPDI ratio greater than 1 (while for the other study details were not provided). Similarly, the outlying pediatric pneumonia study, where the VPDI ratio could not be calculated, enrolled the majority of children with radiologically confirmed pneumonia from hospitals but most of the clinical pneumonia episodes from outpatient clinics [14]; furthermore, it had a control-group clinical pneumonia incidence that was larger than 4 of the 5 other pediatric pneumonia studies (Supplementary Table 3). That both studies (acute OM where the VPDI ratio <1 and pneumonia where the VPDI ratio was not calculated) enrolled a relatively larger number of cases, many of which were clinically nonsevere, raises the possibility that non-VST replacement disease is more likely to cause mild than severe clinical disease. It is also possible that in one or both of the OM studies with a VPDI ratio greater than 1, the more specific outcomes were not a complete subset of more sensitive outcomes, as this was difficult to confirm from the methods provided in the manuscripts. Regardless, however, 24 of 27 calculated VPDI ratios were greater than 1. This suggests that, to the extent that PCV introduction might cause an increase in clinically defined outcomes due to non-VST disease, this is more than compensated for by reductions in clinically defined outcomes due to reductions in VST disease.

Our study had several strengths such as the systematic nature of the review and inclusion of only RCTs. We also had several limitations. Results were not always consistently reported in the original publications (eg, person-time information was missing from some studies), meaning that several high-quality trials had to be excluded, and even within included studies not all outcomes of interest could be calculated. This resulted in the inclusion of only 10 trials, which limits the strength of our conclusion. We hope that publication of this review will prompt investigators in the future to include data necessary to calculate VPDI for clinical outcomes. These clinical trials were conducted mostly in high- and medium-income countries (9/10) as only 1 low-income country was included. Because we did not have access to original databases, we relied on the precision of estimates reported in studies, which may have reduced the precision of calculated results, including VPDI ratios. In 1 study, the sensitive outcome for IPD was based on ICD codes that might involve misclassification of IPD, potentially overestimating the VPDI ratio (eg, by including noninvasive cases). In addition, while our results indicate the additional public health value of considering clinically defined outcomes among directly immunized persons, a full accounting of PCVs’ value also should include indirect protection of unimmunized persons [27]. The data described in this study were mostly obtained from pneumococcal vaccine-naive populations. As introduction of pneumococcal vaccination into pediatric national immunization programs is becoming more widespread globally, and as expanded valency PCVs further reduce pneumococcal disease, VPDI ratios could increase further or decrease depending on availability of more sensitive and specific diagnostic tests, frequency of chest X-ray use, and other factors. The availability of only 1 adult study may limit generalizability of our findings in nonpediatric populations. Finally, while inclusion of only RCTs increased methodological rigor, it has the limitation that results may differ from real-world vaccine use—for example, due to inclusion of all persons regardless of underlying diseases or completion of vaccination schedule. Where possible, we also excluded populations at higher risk of pneumococcal disease. The higher underlying pneumococcal disease burden in this group may result in a higher VPDI for the numerator or the denominator that could result in a higher or lower VPDI ratio. Consequently, our conclusions apply mostly to nonimmunocompromised persons. Nevertheless, 5 of 10 trials included any subject, regardless of immune status.

Our results have several implications. We recognize that VE against etiologically or radiologically confirmed outcomes should remain the primary regulatory outcome for vaccine licensure. To better assess the preventable disease burden, vaccine technical committees might also consider VPDI for the most sensitive available outcomes when informing decision making on immunization policy. As vaccine technical committees need to make decisions based on public health value, vaccine RCTs—whether phase III or IV—should always include sensitive clinical outcomes, even if these are designated as exploratory. To encourage this, regulatory agencies should consider inclusion of broader clinical outcomes in product labels, as was recently done by the European Medicines Agency for PCV13 and adult CAP [28]. In the future, standardized clinical outcome case definitions should be developed to improve comparison and interpretation of results, and studies should seek to understand the specific mechanisms driving the additional disease reduction for clinical outcomes.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Click here for additional data file.