A comparison of the test-negative and the matched case-control study designs for estimation of EV71 vaccine immunological surrogate endpoints from a randomized controlled trial.

Since TND could be an appropriate method to assess vaccine effectiveness, we want to know whether it may be used for the estimation of vaccine immunological surrogate endpoints, like case-control study. We conducted two study designs (test-negative design (TND) VS matched case-control design (MCC)) to evaluate immunological surrogate endpoint against EV71-associated diseases. We calculated sensitivity (proportion of participants with EV71-associated disease who have a titer less than the cutoff at day 56), specificity (proportion of matched controls who have a titer equal or greater than the cutoff at day 56), and corresponding Youden index ([sensitivity + specificity] - 1). Then, we compared them between TND and MCC. In test-negative design, we totally enrolled 7029 subjects, 49 tested positive as cases and 6980 tested negative as controls in per-protocol population. In matched case-control design, we totally enrolled 305 subjects, 51 as cases, and 254 as controls in whole cohort. In sensitivity and specificity comparison, TND and MCC's results were similar to each other, except for a titer of 1:4. Nonetheless, in Youden index comparison, MCC's results were slightly higher than the TND's, except for a titer of 1:4. EV71 vaccine immunological surrogate endpoints derived from TND was similar to MCC's. Our results supported that TND could become an alternative research design with the progress of surveillance.

Introduction

Enterovirus 71 (EV71) is one of the major pathogens causing hand, foot, and mouth disease (HFMD), causing a substantial disease burden in the Asia-Pacific region particularly in young children under 5 years old.[1] Currently, there is no specific treatment for HFMD caused by EV71.[2] Three inactivated monovalent EV71 vaccines (developed by Chinese Academy of Medical Sciences(CAMS), Sinovac Biotech, and Beijing Vigoo Biological) were licensed in China since 2016, which showed high efficacy (94.8–97.4%) against EV71-associated HFMD but no cross-protection against HFMD caused by other serotypes in children.[3-5] Therefore, other kinds of enterovirus vaccine, such as monovalent Coxsackievirus 16(CA-16) vaccine and bivalent EV71 and CA-16 vaccine, are under development.[6,7]

An applicable immunological surrogate of protection would help to avoid new large trials and facilitate getting new products and formulations approved. It also could reduce the sample size or shorten the duration of a clinical trial. In previous studies,[3,5] correlate of immunity against EV71-associated disease was explored. The results from matched case-control study design and a receiver operating characteristics (ROC) curve analysis suggested that a potential titer of neutralizing antibody (NTAb) of 1:16-1:32 at day 56 was proposed to be an immunological surrogate endpoint for the EV71 vaccine protection.

Test-negative design (TND) is considered as a variant of the traditional case–control study and has become popular for post-licensure observational studies of the effectiveness of vaccines, especially for influenza vaccines.[8-10] Several studies have operated a comparison between the test-negative design and traditional case-control design for estimation of vaccine effectiveness.[11-13] These studies prompt that TND, compared with traditional case–control study, is a convenient and reliable alternative for estimation of vaccine effectiveness.

Since TND could be an appropriate method to assess vaccine effectiveness, we want to know whether it may be used for estimation of vaccine immunological surrogate endpoints, like case-control study. Hence, we practice a comparison of immunological surrogate endpoints derived from test-negative design and matched case-control study design.

Method

Data source

The database is from a multi-center, randomized, double-blind, placebo-controlled, phase 3 trial which conducted in China (NCT01508247). This first EV71 vaccine phase 3 trial performed by Vigoo had high quality process and data. The trial enrolled 10,245 children and infants, 5120 participants in vaccine group, and 5125 in placebo group. Details can be found in the previous publication.[5]

Matched case-control study design

To evaluate the correlation between EV71 neutralizing antibody levels and disease protection, we assessed correlates of immunity for all participants with confirmed EV71 and their matched case-free participants who had data for antibody titer at day 56. EV71 vaccine immune procedure is at day 0 and 28 with intramuscular injection. There is a popular belief that antibody levels approach the peak level at 28 days (or a month) after a complete immune procedure. Therefore, we chose Day 56 as the cutoff day. The diseases caused by EV71 infection were diagnosed as the case group, and each case was matched at 1:5 after the initial diagnosis to form a subgroup. The matching conditions were as follows: (1) during the period from 0 days to 14 months after the first dose of vaccination, the subjects who had no diseases caused by EV71, and if a control group found diseases caused by EV71 infection during the subsequent surveillance period, the control group was converted to a case and was excluded from the control group; (2) the age gap between the case and the case was less than 3 months and lived in the same village or neighboring village.

Test-negative design

In test-negative design, patients seeking medical care for a predefined clinical condition are tested for a specific viral infection by using a highly sensitive and specific laboratory test. Those tested positive are cases and controls are tested negative for infection but meeting the same enrollment criteria. In origin phase III trial, we did a high-quality combination of active and passive monitoring to capture any suspected EV71 cases. Participants were actively followed after being administered two doses of vaccine or placebo. Guardians were encouraged to take their children to treatment in clinics or hospitals for any illness. The surveillance was from 56 days after first dose to 14 months.

All potential cases were reported by the clinics or hospitals. Throat or rectal swabs or both were taken for pathogen detection within 24 h. These samples were tested for EV71 by real-time PCR (fluorescence assay) with a viral RNA diagnostic kit. A participant was defined as a case if at least one positive test result during follow-up. A participant was defined as a control if at least one negative test results during follow-up and the participant had no positive results.

Measurement of EV71 neutralizing antibody

Serum samples of 3 mL were collected from subjects on day 56 (day 28 after second dose) for determination of EV71 NTAb titers. Serums samples were stored at a temperature of −20∘C or below before tested. A modified cytopathogenic effect method (CPE method) was used to measure EV71 NTAb titers.[14] In brief, samples were 2-fold serially diluted from 1:8 to 1:16384 and NTAb titers were defined as the highest dilution capable of inhibiting 50% of the CPEs. Antibody titers lower than 1:8 was assigned values of 1:4 for calculation. 1219 serum samples from immunogenicity cohort collected on day 56 had been detected by National Institutes for Food and Drug Control (NIFDC), and remaining 8435 serum samples were measured by Beijing Vigoo Biological Ltd later using the same method. The tested original NTAb titers from NIFDC and Beijing Vigoo were standardized according to the NTAb titers (U/mL) of the reference serum (N12: 1000U). If Nab measure were performed with different personnel or multiple times, the results of final measurements would be different and affected our results. Thus, to avoid final measurements affected our results, a standardization method would convert the tested NTAb titers from different laboratories into standardized NTAb titers. This method was adapted from the calibration methods applied for polio previously.[15]

Statistical analysis

In test-negative design, we evaluated immunological surrogate endpoints based on the per-protocol population in common with previous work about immune correlates,[16,17] which consisted of all eligible participants who completed the 2-dose regimen and with available EV71 NTAb titers in serum post-vaccination. Matched case-control study (MCC) analysis has been performed in origin phase III trial.[5]

We calculated sensitivity (proportion of participants with EV71-associated disease who have a titer less than the cutoff at day 56), specificity (proportion of matched controls who have a titer equal or greater than the cutoff at day 56), and corresponding Youden index ([sensitivity + specificity] − 1). The cutoff with the maximum Youden index could be thought of as a surrogate of protection for it could provide the clearest distinction between cases and controls. We used the χ2 test or Fisher’s exact test to analyze categorical data, ANOVA to analyze continuous data. Statistical analyses were done with SAS software (version 9.4). All statistical tests were two-sided, and statistical significance was defined as P < 0.05.

Results

We conducted two study designs to evaluate immunological surrogate endpoint against EV71-associated diseases (Figure 1). In test-negative design, we totally enrolled 7029 subjects, 49 tested positive as cases and 6980 tested negative as controls in per-protocol population. In matched case-control design, we totally enrolled 305 subjects, 51 as cases and 254 as controls in whole cohort. A part of baseline data had been reported in previous research.[18]

Figure 1.

Study plan.

Table 1 showed the characteristics of cases and controls in test-negative design and matched case-control design. There are no significant statistical differences between cases and controls on sex, age, height and weight, both in test-negative design and matched case-control design. However, the case’s GMT and the ratio of received vaccine numbers were significantly lower than the control’s, also both in test-negative design and matched case-control design.

Table 1.

Characteristics of cases and controls in test-negative design and matched case-control design.

		Test-negative design			Matched case-control
	Positive cases (49)	Negative controls (6980)	P value	EV71 cases (51)	Matched controls (254)	P value
Age*	19.8(8.3)	18.4(8.0)	.2268	19.7(8.2)	19.1(8.3)	.6275
Height	83.2(9.0)	81.4(8.1)	.1311	83.0(8.9)	82.0(8.7)	.4622
Weight	13.0(2.7)	12.5(2.2)	.0775	12.9(2.7)	12.4(2.2)	.1309
GMT	5.5(6.7)	109.9(20.1)	<.0001	5.0(6.0)	134.3(20.1)	<.0001
Received Vaccine	8(16.3%)	3469(49.7%)	<.0001	8(15.7%)	135(53.2%)	<.0001
Male	29(59.2%)	3940(56.5%)	.7002	29(56.9%)	133(52.4%)	.5567

*Age, Height, Weight, GMT were described by mean±SD. GMT = Geometric Mean Titer of EV71 NAb. Received vaccine and Male were showed by counts and percent. ANOVA was used to compare means and Chi-square test was used to compare the ratio.

The percentage of controls and cases about TND and MCC with titer ≥1:2 was shown in Figure 2. Clearly, the levels of anti-EV71 neutralizing antibody titers in control group were higher than case groups. Table 2 shows the sensitivity and specificity of different cutoff neutralizing antibody titer against EV71­associated disease by test-negative design and matched case-control design. Except 1:4 neutralizing antibody titer, the TND results of different cutoff were close to the MCC’s results. The maximum Youden index was provided by a titer of 1:32 in TND while the maximum Youden index was reached at a titer of 1:16 in MCC. However, both in TND and MCC, accounting for sensitivity, specificity, and Youden index, a titer of between 1:16 and 1:32 seemed to provide the best protection against EV71-associated diseases.

Figure 2.

Reverse cumulative curves for enterovirus 71 (EV71) neutralizing antibody titers about TND and MCC.

Table 2.

Sensitivity and specificity of different cutoff neutralizing antibody titer against EV71­associated disease by test-negative design and matched case-control design.

		Test-negative design			Matched case-control
Neutralizing antibody titer	Sensitivity*% (95% CI)	Specificity^% (95% CI)	Youden index(sensitivity+specificity-1)	Sensitivity*% (95% CI)	Specificity^% (95% CI)	Youden index(sensitivity+specificity-1)
1:4	71.4(56.7–83.4)	70.3(69.2–71.3)	0.4170	1.9(0.0–10.3)	99.6 (97.8–100.0)	0.0153
1:8	73.5(58.9–85.0)	69.3(68.2–70.4)	0.4277	73.1 (59.0–84.4)	74.3 (68.5–79.6)	0.4739
1:16	85.7(72.8–94.0)	65.9(64.8–67.0)	0.5162	84.6 (71.9–93.1)	70.8 (64.7–76.3)	0.5537
1:32	87.8(75.2–95.4)	63.9(62.8–65.0)	0.5165	88.5 (76.6–95.6)	66.0 (59.8–71.8)	0.5447
1:64	87.8(75.2–95.4)	61.0(59.8–62.1)	0.4871	88.5 (76.6–95.6)	63.2 (57.0–69.2)	0.5170
1:128	87.8(75.2–95.4)	56.3(55.2–57.5)	0.4409	90.4 (79.0–96.8)	57.3 (51.0–63.5)	0.4770
1:256	87.8(75.2–95.4)	48.5(47.3–49.7)	0.3628	90.4 (79.0–96.8)	49.4 (43.1–55.7)	0.3979
1:512	95.1(86.0–99.5)	38.4(37.3–39.6)	0.3434	94.2 (84.1–98.8)	41.9 (35.7–48.2)	0.3613
1:1024	98.0(89.1–99.9)	26.3(25.2–27.3)	0.2422	100.0 (93.2–100.0)	31.2 (25.6–37.3)	0.3123

*Sensitivity was defined as the proportion of EV71 cases having a titer

^Specificity was defined as the proportion of matched controls having a titer ≥the cutoff values at day 56 in total matched controls.

Figure 3 revealed the comparison of sensitivity, specificit,y and Youden index between the TND and MCC. In sensitivity and specificity comparison, TND and MCC’s results were similar to each other. Nonetheless, in Youden index comparison, MCC’s results were slightly higher than the TND’s, except for a titer of 1:4.

Figure 3.

The comparison of sensitivity, specificity, and Youden index between the TND and MCC.

Discussion

We tried to demonstrate the feasibility of TND for estimation of vaccine immunological surrogate endpoints. Taking EV71 vaccine as an example, we did a receiver operating characteristics (ROC) curve analysis in both TND and MCC. We compared the Youden index between TND and MCC and found that TND’s result was similar to MCC’s.

An immune correlate of protection is an immunological response (either humoral or cellular) that reliably predicts the level of vaccine efficacy to evaluate a clinically meaningful endpoint.[19] There have been some studies to evaluate EV71 vaccine immunological surrogate endpoints. Jin and his colleagues fitted scaled logit model to assess the mechanistic correlate of protection.[17] They found that NTAb with a titers of 14.7 U/ml (around 1:16) post-vaccination can be validated as a surrogate associated with the protection of 50% against EV71-associated disease. Besides, Zhu and his colleagues used logistic regression to analyze some other factors as covariates, including the NTAb titer level at baseline, age, and sex, which may affect the efficacy of the EV71 vaccine.[16] They got a slightly higher NTAb titer of 26.6 U/ml (around 1:30) as the immunological surrogate endpoint for EV71 vaccine protection by excluding subjects with a positive baseline in the model. Compared with TND or case-control study, mathematical model could be more precise for estimation of vaccine immunological surrogate endpoints since data in the subset of cohort using ROC curve analysis was limited and potentially biased by the selection of the controls. But we consider that TND and case-control study still could be used as tentative exploration designs.

Many previous studies have discussed the advantages and disadvantages about the TND and traditional case-control study for estimating vaccine effectiveness. Case-control study can be resource-efficient and particularly useful for relatively uncommon diseases because efforts are focused on accurately ascertaining disease status and vaccination history for a relatively small number of cases and controls. Despite being widely used to evaluate vaccine performance, the case-control methodology is susceptible to bias and confounding largely due to controls selection.[20,21] In like manner, TND is often suspicious of validity and accuracy although it is popular because of ease of implementation. However, most vaccine effectiveness studies showed that TND was almost the same as the traditional case-control study, even TND produced less biased vaccine effectiveness estimates.[22,23] Our study results still gained a similar conclusion that EV71 vaccine immunological surrogate endpoint derived from TND was the same as case-control study’s. But TND’s Youden index seems lower than the case-control study’s. Maybe it was because that we did not use the same number of cases or the controls quantity varied greatly.

As far as we know, this is the first analysis for the comparison of the test-negative and the matched case-control study designs for estimation of vaccine immunological surrogate endpoints. But we just did a simple analysis using EV71 vaccine as example to demonstrate the validity of TND for vaccine immunological surrogate endpoints. More study and development of this idea are left to future work. Before our study, Dean A. Follmann. Et detailed an approach to immune correlates analysis under a test negative design using samples collected from subjects received Ebola vaccine (Vesicular Stomatitis Vaccine, VSV).[24] They contributed the idea that an estimate of the effect of the relevant immune response on the risk of disease can be estimated by using an irrelevant immune response based on a prediction model that is estimated in vaccinated controls using samples collected upon arrival at a clinic. In vaccine effectiveness TND study, a key assumption is that the incidence of interest disease is the same in the vaccinated and unvaccinated groups within care-seeking behavior strata and vaccination will not affect non-vaccine pathogen.[25] So similarly, in immune correlates TND study, the key requirement is that the vaccine of interest produce irrelevant immune responses that will be unaffected by the disease of interest, and that these immune responses are correlated with the immune response of interest.[24]

There were some limitations to this study. First, in TND, we did not enroll those subjects who receive only one dose because the previous study which fitted scaled logit model to assess the mechanistic correlate of protection used per-protocol population. But in MCC, we used whole cohorts. Therefore, the number of cases in TND was smaller than that in MCC. It may have a slight effect on the results’ comparison. Second, TND and MCC were different in selecting controls. It may affect the baseline comparability of the two study designs for the negative controls were much more than matched controls. However, it didn’t affect the final result that the optimal Youden index was 1:16-1:32. Third, we excluded those subjects who had no NAb titer data at Day 56. This could cause potential selection bias. Fortunately, missing data accounted for only a small fraction. Finally, this study was a post-hoc analysis and we did not make a sample size estimation.

Compared with vaccine effectiveness TND study which requires vaccination and disease data, immune correlates TND study also requires long-term, uniform, standardized storage of samples from hundreds of thousands of subjects just after vaccination. As a result, the biggest challenge in this approach is whether the antibody data is amenable. And although it is difficult to use TND to evaluate immunological surrogate endpoints after the vaccine is marketed, our results supported that TND could still become an alternative research design, especially with the development of the vaccination system, case surveillance network, and more routine antibody monitoring in the future.