Evaluation of a multiplex immunoassay for bovine respiratory syncytial virus and bovine coronavirus antibodies in bulk tank milk against two indirect ELISAs using latent class analysis.

Bovine respiratory syncytial virus (BRSV) and bovine coronavirus (BCV) are responsible for respiratory disease and diarrhea in cattle worldwide. The Norwegian control program against these infections is based on herd-level diagnosis using a new multiplex immunoassay. The objective of this study was to estimate sensitivity and specificity across different cut-off values for the MVD-Enferplex BCV/BRSV multiplex, by comparing them to a commercially available ELISA, the SVANOVIR® BCV-Ab and SVANOVIR® BRSV-Ab, respectively. We analyzed bulk tank milk samples from 360 herds in a low- and 360 herds in a high-prevalence area. As none of the tests were considered perfect, estimation of test characteristics was performed using Bayesian latent class models. At the manufacturers' recommended cut-off values, the median sensitivity for the BRSV multiplex and the BRSV ELISA was 94.4 [89.8-98.7 95% Posterior Credibility Interval (PCI)] and 99.8 [98.7-100 95% PCI], respectively. The median specificity for the BRSV multiplex was 90.6 [85.5-94.4 95% PCI], but only 57.4 [50.5-64.4 95% PCI] for the BRSV ELISA. However, increasing the cut-off of the BRSV ELISA increased specificity without compromising sensitivity. For the BCV multiplex we found that by using only one of the three antigens included in the test, the specificity increased, without concurrent loss in sensitivity. At the recommended cut-off this resulted in a sensitivity of 99.9 [99.3-100 95% PCI] and specificity of 93.7 [88.8-97.8 95% PCI] for the multiplex and a sensitivity of 99.5 [98.1-100 95% PCI] and a specificity of 99.6 [97.6-100 95% PCI] for the BCV ELISA.

Introduction

Bovine coronavirus (BCV) and bovine respiratory syncytial virus (BRSV) are commonly occurring agents among cattle worldwide (Valarcher and Taylor, 2007; Boileau and Kapil, 2010). They are endemic and prevalent also in the Norwegian dairy herd (Gulliksen et al., 2009; Klem et al., 2014a). BCV causes respiratory disease, calf diarrhea and winter dysentery (contagious diarrhea in adult cattle) (Boileau and Kapil, 2010). BRSV causes respiratory disease mostly in young animals but can affect animals of all ages, and is a common cause of respiratory outbreaks in Norway (Larsen, 2000; Klem et al., 2014a). Consequences of these infections are herd health problems, reduced animal welfare and increased use of antibiotics due to secondary bacterial infections (Larsen, 2000; Valarcher and Taylor, 2007; Boileau and Kapil, 2010). Therapy costs and reduced production entails considerable financial loss for the farmer, and contributes to a present focus in Nordic countries on how to limit the spread of these viruses in the cattle population.

In 2016, a national control program against BRSV and BCV infections was launched in Norway as a joint initiative between the producer organizations. This prompted the need for an easy and cost-effective way to screen dairy herds for a herd level diagnosis for BRSV and BCV. The initial screening in the control program was conducted using bulk tank milk samples (BTM). There are already commercially available indirect enzyme-linked immunosorbent assays (ELISAs) widely used in routine diagnostics and research in the Nordic countries (SVANOVIR® BRSV-Ab and SVANOVIR® BCV-Ab) (Tråvén et al., 1999; Klem et al., 2014b; Toftaker et al., 2016). However, in order to optimize cost-effectiveness of the control program, the development of a new multiplex antibody ELISA was initiated (MVD-Enferplex BCV/BRSV multiplex). The new test allowed screening for both viruses by the use of a single test.

The performance of a diagnostic test is characterized by the test’s sensitivity (Se) and specificity (Sp), where Se is the proportion of true positives correctly classified as positive by the test, and the Sp is the proportion of true negative subjects correctly classified as negative. The true antibody status of each test subject can be determined in two ways: By use of a perfect reference test, or based on populations with known status. However, a perfect reference test (often termed a “gold standard”) is rarely available and for endemic diseases, which is the case for BRSV and BCV in Norway, no reference population with complete certainty regarding disease or disease freedom exists. Consequently, the underlying true infection status for test subjects remains unknown. Test validation studies (erroneously) assuming perfect reference tests are common, even though this has been shown to introduce bias in the estimation of accuracy parameters (Valenstein, 1990; Lijmer et al., 1999). Latent class analysis (LCA) allows for the estimation of test parameters in populations where the underlying true infection status cannot be determined (Hui and Walter, 1980). In LCA the true infection status is treated as an existing, but unknown (latent), variable and test accuracy and prevalence are parameterized according to this latent variable.

As the BRSV/BCV multiplex is a new test, it needs to be validated. Test characteristics are different when a test is used as a herd test, compared to when it is used on individual samples (Christensen and Gardner, 2000) and validation for the relevant application is therefore important. BTM testing is a key component of the Norwegian BRSV/BCV control-program, it is therefore of interest to estimate test accuracy, at different cut-off values, for this application.

The aim of this study was to estimate the test sensitivity and specificity of the newly developed MVD-Enferplex BCV/BRSV multiplex across different cut-off values, for detection of antibodies in BTM. The BCV part of the multiplex was compared to the commercially available SVANOVIR® BCV-Ab, and the BRSV part of the multiplex was compared to the SVANOVIR® BRSV-Ab. As neither test could be considered perfect, the evaluation was done using LCA.

Materials and methods

Study population and sample material

A cross-sectional sampling design was used for the present study. Herds were eligible for inclusion if they delivered milk to the largest dairy company in Norway (TINE SA), and provided a BTM sample during the study period (March 2016). Herds from two counties with an expected difference in true prevalence (TP) were selected in order to meet the model assumptions, described in the LCA section. Using a random numbers generator, 360 samples were randomly chosen from herds in “Oppland” (Pop 1) and 360 from herds in “Sogn og Fjordane” (Pop 2) counties. “Sogn og Fjordane” is located in western Norway, and was assumed to have a relatively low prevalence, based on results from a previous study (Toftaker et al., 2016). Oppland county, located in eastern Norway, was thought to have higher prevalence based on known patterns of animal movements and a history of previous outbreaks of disease (Toftaker et al., 2017).

BTM samples were collected from nearly all Norwegian dairy herds delivering milk to the largest dairy cooperation (TINE SA) during March 2016. The samples were collected as part of the national control program against BRSV and BCV. The milk truck driver collected samples at ordinary milk shipment using standard procedures for BTM sampling. The milk was then stored at 4 °C until received at the laboratory (TINE Mastitis Laboratory, Molde, Norway) where samples were frozen and shipped over-night to the Enfer laboratory in Ireland (Enfer Scientific, Naas, Ireland). Samples were kept frozen until the time of laboratory analysis.

Diagnostic tests

ELISA

The SVANOVIR® BRSV-Ab, hereafter designated the BRSV ELISA, and SVANOVIR® BCV-Ab, hereafter designated the BCV ELISA, were used on all 720 samples, following the manufacturer’s instructions. The optical density (OD) reading of 450 nm was corrected by the subtraction of OD for the negative control antigen, and percent positivity (PP-value) was calculated as (corrected OD/positive control corrected OD) × 100. According to the test manuals, the recommended cut-off values of sample positive >10 PP for both tests were used as a starting point for these tests (Svanova; Svanova). For the BRSV ELISA the Se and Sp provided by the manufacturer were 94% and 100%, respectively. These parameters are calculated from serum samples, and parameters specific for BTM samples have not been reported (Elvander et al., 1995). For the BCV ELISA the test parameters provided by the manufacturer were Se of 84.6% and Sp of 100%, and as for BRSV the calculations are based on serum samples (Alenius et al., 1991).

Multiplex

All 720 samples were analyzed using the MVD-Enferplex BCV/BRSV multiplex, hereafter referred to as the BRSV/BCV multiplex (Enfer Scientific, Naas, Ireland). A panel of three BCV recombinant proteins (BCV A-C), along with a panel of two recombinant proteins and two synthetic peptides for BRSV (BRSV A-D) were used as antigens. Briefly, the antigens were deposited in a multiplex planar array as individual spots into wells of 96 well microtiter plates to produce arrays of antigens. Samples were diluted 1:3 into sample dilution buffer and mixed before added to the well and incubated at 37 °C for 60 min with agitation. After washing procedures, the detection antibody diluted in conjugate buffer was added and plates were incubated (37 °C for 60 min with agitation) before new washing. Finally, the chemiluminescent substrate was added. Relative light units (RLU) were captured (45 s exposure) immediately, using Quansys biosciences imaging system, and data was extracted using Quansys Q view software (v 1.5.4.7). Antigens were combined in a parallel reading, i.e. the test was considered positive when the RLU-value of at least one antigen was above the applied cut-off. Laboratory personnel were not formally blinded to test results, but due to the large volume of samples they were considered blinded for any practical purposes.

Data management and descriptive statistics

Because the multiplex consisted of several antigens each giving a separate response, a separate cut-off value was needed for each antigen. We calculated the proportion of herds that had a positive response to each of the individual antigens within the test-positive group (at manufacturers recommended cut-off values), and defined the antigen with the highest proportion of positive responses as the most influential. This was done for both viruses. When later choosing which cut-off values to assess, changing the cut-off for the most influential antigen for each virus was prioritized. We used an explorative approach to selecting cut-off values, and several different cut-off values were tried for the most influential antigen (Fig. 1 ). Furthermore, we evaluated test performance when including only the single most important antigen. Data preparation and descriptive analysis were performed in Stata (Stata SE/14; Stata Corp., College Station, TX).

Fig. 1

To the left are the different cut-off values (relative light units) for the BRSV antigens (top) and BCV antigens (bottom) included in the BCV/BRSV multiplex. To the right are spider plots of median Se and Sp for the different cut-off alternatives. The BRSV ELISA cut-off was fixed at sample positive >10 PP, except for alternative 9 where sample positive >50 PP was used. For the BCV ELISA the cut-off was fixed at sample positive >10 PP. Test parameters are estimated from a Bayesian LCM analysis.

Latent class analysis

In the present study, we used guidelines for reporting of diagnostic accuracy in studies that use Bayesian LCA (Kostoulas et al., 2017).

The target condition was herds with one or more animals producing BCV/BRSV-antibodies while contributing to the bulk tank. The underlying latent state could be considered as previous exposure, leading to antibodies in BTM.

The use of LCA methodology for diagnostic test evaluation requires a set of assumptions of the tests and test populations to be fulfilled. (1) two or more populations with different prevalence are included, (2) the Se and Sp of the diagnostic tests are the same across the populations, and (3) the tests are conditionally independent (CID) given disease status (Hui and Walter, 1980). We ran the analyses assuming CID between tests; however, we also explored the consequences of relaxing this assumption as explained below. For the CID-models, parameters were estimated for several cut-off values (Fig. 1). Models were fit using Bayesian LCA in the OpenBugs version 3.2.1 rev 781 software. We used non-informative priors in the shape of uniform distributions on the interval between zero and one, modelled using the beta (1, 1) distribution for test properties and sub-population prevalence in all analyses. Models were run with 20,000 iterations, of which 10,000 were used as burn in and discarded. Convergence of the Markov Chain Monte Carlo (MCMC) chains were assessed by visual inspection of history plots, time-series plots and Gelman-Rubin diagnostic plots using three sample chains with different initial values, as suggested by Toft et al. (2007). Posterior inference was done by calculating medians and 95% posterior credibility intervals (PCI) for Se, Sp and true prevalence. The model description is included in Appendix A.

Sensitivity analysis

A correlation between tests, if present, is not possible to estimate in a two tests scenario without including informative priors. We did not have any reliable prior information on test performance or prevalences in the present study. However, the consequences of relaxing the assumption of conditional independence given disease status was first explored by Vacek (1985), who examined the impact of conditional dependence by assuming a fixed proportion of the maximum possible covariance between tests. Following this approach we explored the consequences of conditional dependence between tests for the cut-off values with the preferred test characteristics. (Fig. 1: alternative 2 for BRSV, alternative 8 for BCV.) See Appendix A for details.

We compared the results of the conditional independence model to models allowing 25, 50, 75 and 90% of the maximum possible positive covariance, as well as a negative covariance of −25%.

Results

Descriptive statistics

A combination of different cut-off values for the included antigens, (cut-off alternatives 1–9) are presented in Fig. 1 for the BRSV- and BCV multiplex. For the BRSV multiplex, the BRSV-A antigen was responsible for detecting the majority of the positive samples. For the BCV multiplex the antigen detecting the majority of positive samples was the BCV-A. Counts of test outcomes for the tests are presented in Table 1, Table 2 for the BRSV and BCV tests, respectively.

Table 1

Counts of paired test outcomes in the two sub-populations for the BRSV-antibody tests (BRSV multiplex/BRSV ELISA). For the BRSV multiplex varying cut-off values for the included antigens were used (shown in Fig. 1). The BRSV ELISA cut-off was fixed at sample positive >10 PP, except for alternative 9 where sample positive >50 PP was used.

	BRSV multiplex/BRSV ELISA
	Pop 1				Pop 2
Cut-off alternative	+/+	+/−	−/+	−/−	+/+	+/−	−/+	−/−
1	299	0	35	26	111	16	103	130
2	299	0	35	26	111	16	103	130
3	287	0	47	26	105	16	109	130
4	283	0	51	26	102	16	112	130
5	272	0	62	26	94	12	120	134
6	274	0	60	26	93	15	121	131
7	289	0	45	26	107	12	107	134
8	264	0	70	26	84	11	130	135
9a	295	4	12	49	105	22	18	215

BRSV ELISA cut-off: sample positive >50 PP.

Table 2

Counts of paired test outcomes in the two sub-populations for the BCV-antibody tests (BCV multiplex/BCV ELISA). For the BCV multiplex varying cut-off values for the included antigens were used (shown in Fig. 1). The BCV ELISA cut-off was fixed at sample positive >10 PP.

	BCV multiplex/BCV ELISA
	Pop 1				Pop 2
Cut-off alternative	+/+	+/−	−/+	−/−	+/+	+/−	−/+	−/−
1	336	7	0	17	219	34	0	107
2	335	3	1	21	215	14	4	127
3	334	2	2	22	207	11	12	130
4	330	2	6	22	198	9	21	132
5	329	2	7	22	187	9	32	132
6	324	2	12	22	182	9	37	132
7	301	2	35	22	174	9	45	132
8	336	2	0	22	219	10	0	131
9	321	0	15	24	180	1	39	140

BRSV

Estimates of median Se and Sp and true prevalence in the two sub-populations for the BRSV-multiplex and BRSV ELISA when applying different cut-off values are presented in Table 3 . As a starting point the recommended cut-off values from the test manufacturers were applied (alternative 2 in Fig. 1), resulting in median Se of 94.4 and Sp of 90.6 for the BRSV multiplex, and Se 99.8 and Sp 57.4 for the ELISA. The Sp of the ELISA increased to 99.4 (Se 93.4) when a cut-off of sample positive >50 PP was used. For the multiplex, increasing the cut-off value for the BRSV-A antigen generally resulted in lower Se and higher Sp estimates as could be expected. Discarding all antigens except the BRSV-A resulted in increased specificity, however, with the cost of significantly reduced sensitivity, as can be seen from comparing cut-off alternative 2 and 7 in Table 3. Point estimates (median) of true BRSV antibody prevalence ranged from 84.5 to 87.3 for pop 1, and from 25.2 to 30.5 for pop 2. Results from the COC-models with fixed covariance, showed that allowing for covariance altered specificity estimates for both the ELISA and the multiplex. The change was small for a covariance of 0.25 or less of the maximum possible covariance. The Se estimates were not noticeably affected by allowing for covariance. Results from the sensitivity analysis are presented in Table 4 .

Table 3

Test parameter estimates for the BRSV multiplex and BRSV ELISA: Sensitivity, specificity, and estimates of true prevalence (TP) in the two sub-populations. Cut-off alternative 1- 9 represents different cut-off alternatives for the BRSV multiplex (presented in Table 1). The BRSV ELISA cut-off was fixed at sample positive >10 PP for all alternatives except for alternative 9, where the BRSV ELISA cut-off was increased to sample positive >50 PP.

	Test								Sub-population
	BRSV multiplex				BRSV ELISA				Pop 1		Pop 2
Parameter	Se		Sp		Se		Sp		TP		TP
Cut-offalternative	Median	[95% PCI]	Median	[95% PCI]	Median	[95% PCI]	Median	[95% PCI]	Median	[95% PCI]	Median	[95% PCI]
1	94.4	[89.8;98.7]	90.6	[85.5;94.4]	99.8	[98.7;100]	57.4	[50.5;64.4]	87.2	[81.7;91.5]	29.9	[24.1;35.9]
2	94.4	[89.8;98.7]	90.6	[85.5;94.4]	99.8	[98.7;100]	57.4	[50.5;64.4]	87.2	[81.7;91.5]	29.9	[24.1;35.9]
3	90.7	[85.6;96.0]	90.6	[85.6;94.4]	99.7	[98.6;100]	56.7	[49.7;63.9]	86.8	[83.5;92.8]	29.2	[25.6;39.4]
4	89.5	[84.3;95.0]	90.5	[85.5;94.4]	99.7	[98.6;100]	56.3	[49.2;63.4]	87.0	[80.8;91.4]	28.5	[22.5;34.9]
5	86.3	[80.8;92.0]	92.9	[88.3;96.2]	99.7	[98.5;100]	55.8	[48.8;62.9]	86.8	[80.6;91.4]	27.8	[21.8;34.4]
6	87.0	[80.3;91.2]	91.1	[86.3;94.8]	99.7	[98.6;100]	54.8	[47.7;62.0]	86.6	[80.3;91.2]	26.5	[20.4;33.0]
7a	91.2	[86.3;96.3]	92.9	[88.3;96.2]	99.7	[98.6;100]	57.9	[50.8;65.1]	87.3	[81.4;91.7]	30.5	[24.6;36.7]
8a	84.1	[78.4;90.2]	93.4	[89.1;96.6]	99.7	[98.5;100]	53.9	[47.1;60.8]	86.4	[80.0;91.1]	25.2	[19.3;31.7]
9b	97.0	[94.0;99.2]	91.5	[87.6;94.6]	99.4	[97.5;100]	93.4	[89.0;97.0]	84.5	[80.1;88.2]	30.1	[25.1;35.4]

Only BRSV-A antigen included.

BRSV ELISA cut-off: sample positive >50 PP.

Table 4

Results from the sensitivity analysis (BRSV): Median estimates and 95% posterior credibility intervals (PCI) of the sensitivity (Se) and specificity (Sp) of bulk tank milk BRSV multiplex and BRSV ELISA at the manufacturers’ recommended cut-off (alternative 2, Fig. 1), for the conditionally independent (CID) model and conditionally dependent (COC) models where the covariance is expressed as proportions of maximum possible value.

Conditional covariancea	BRSV multiplex				BRSV ELISA
	Se		Sp		Se		Sp
	Median	[95% PCI]	Median	[95% PCI]	Median	[95% PCI]	Median	[95% PCI]
CID model
0.00	94.4	[89.8;98.7]	90.6	[85.5;94.4]	99.8	[98.7;100]	57.4	[50.5;64.4]
COCSeandSp
0.25	94.1	[89.5;98.6]	87.3	[80.5;92.5]	99.6	[98.1;100]	55.2	[48.3;62.3]
0.50	93.6	[88.8;98.3]	80.7	[70.6;88.6]	99.4	[96.9;100]	50.7	[43.3;58.1]
0.75	92.1	[84.9;97.6]	69.3	[62.5;79.4]	98.5	[93.2;100]	42.7	[36.8;50.3]
0.9	89.8	[82.1;96.6]	67.8	[62.1;75.4]	97.1	[91.7;99.9]	40.6	[35.2;46.8]
−0.5	94.5	[89.9;99.0]	93.7	[90.4;96.3]	99.8	[99.1;100]	59.6	[52.5;66.9]
COCSe
0.25	94.1	[89,5;98.6]	90.5	[85.6;94.4]	99.7	[98.1;100]	57.4	[50.5;64,4]
0.50	93.6	[88,7;98.4]	90.5	[85.5;94.3]	99.4	[96.9;100]	57.4	[50.5;64.6]
0.75	91.8	[84.3;97.5]	90.2	[84.8;94.2]	98.3	[92.8;100]	57.3	[50.2;64.4]
0.9	87.4	[81.1;95.6]	89.4	[83.5;93.7]	95.4	[90.9;99.7]	57.0	[49.9;64.1]
−0.25	94.3	[89.8;98.7]	90.5	[85.5;94.4]	99.8	[98.7;100]	57.4	[50.5;64.5]
COCSp
0.25	94.3	[89.7;98.8]	87.3	[80.6;92.5]	99.8	[98.7;100]	55.2	[48.3;62.3]
0.50	94.3	[89.8;98.7]	80.9	[70.6;88.6]	99.8	[98.6;100]	50.8	[43.3;58.3]
0.75	94.3	[89.7;98.8]	69.6	[62.7;79.5]	99.7	[98.6;100]	43.0	[37.0;50.4]
0.9	94.3	[89.6;98.7]	68.2	[62.3;77.0]	99.7	[98.3;100]	40.8	[35.4;47.7]
−0.25	94.4	[89.7;98.8]	92.0	[87.8;95.3]	99.7	[98.7;100]	58.4	[51.7;65.3]

Proportion of upper limit of conditional covariance.

BCV

Estimates of test parameters and true prevalence in the two sub-populations across different cut-off values for the BCV multiplex and the BCV ELISA are presented in Table 5 . When we applied the cut-off values currently recommended by the test manufacturers (alternative 1 in Fig. 1), the estimated median Se and Sp was 99.9 [99.4–100 95% PCI] and 77.3 [69.8–84.8 95% PCI], for the BCV multiplex, and 99.0 [96.9–100 95% PCI] and 99.5 [97.1–100 95% PCI] for the BCV ELISA, respectively. Similar to what we observed for BRSV, increasing the cut-off for the most important antigen (BCV-A) resulted in a lower Se and a higher Sp for the BCV multiplex. When we used the BCV-A as the sole antigen (cut-off alternative 8, Table 5) the median Sp increased to 93.7 while the median Se remained unchanged (99.9). Point estimates (median) of true BCV antibody prevalence ranged from 91.5 to 94.0 for pop 1, and from 52.4 to 61.5 for pop 2. Results from the sensitivity analysis, i.e. allowing for covariance between tests, showed negligible effect on the estimated test-parameters; less than 5% change in parameters for covariance at 75% of maximum possible (results not shown).

Table 5

Test parameters for the BCV multiplex and BCV ELISA: Sensitivity, specificity, and estimates of true prevalence (TP) in the two sub-populations. Cut-off alternative 1- 9 represents different cut-off alternatives for the BCV multiplex (presented in Table 2). The BCV ELISA cut-off was fixed at sample positive >10 PP for all alternatives.

	Test								Sub-population
	BCV multiplex				BCV ELISA				Pop 1		Pop 2
Parameter	Se		Sp		Se		Sp		TP		TP
Cut-offalternative	Median	[95% PCI]	Median	[95% PCI]	Median	[95% PCI]	Median	[95% PCI]	Median	[95% PCI]	Median	[95% PCI]
1	99.9	[99.4;100]	77.3	[69.8;84.8]	99.0	[96.9;100]	99.5	[97.1;100]	94.0	[91.0;96.3]	61.5	[56.2;66.7]
2	99.6	[98.6;100]	91.1	[85.4;96.0]	99.4	[97.8;100]	97.4	[93.4;99.7]	93.5	[90.5;95.8]	60.3	[55.1;65.5]
3	99.5	[98.1;100]	93.1	[88.0;97.2]	99.5	[98.1;100]	92.3	[87.3;96.7]	93.1	[90.1;95.5]	58.0	[52.7;63.2]
4	98.9	[96.8;100]	94.3	[89.5;98.1]	99.5	[98.0;100]	87.5	[81.1;93.3]	92.6	[89.4;95.2]	55.8	[50.3;61.2]
5	99.0	[96.8;100]	94.3	[89.5;98.1]	99.5	[98.0;100]	81.6	[75.0;87.8]	92.2	[88.9;94.9]	52.4	[47.0;58.0]
6	97.9	[95.1;99.7]	94.3	[89.5;98.1]	99.5	[98.0;100]	79.8	[72.6;86.8]	91.8	[88.2;94.8]	51.5	[45.9;57.3]
7	90.6	[86.6;94.2]	94.3	[89.5;98.1]	99.4	[97.9;100]	82.6	[73.5;92.3]	92.1	[88.3;95.0]	53.3	[46.8;60.1]
8a	99.9	[99.3;100]	93.7	[88.8;97.8]	99.5	[98.1;100]	99.6	[97.6;100]	93.5	[90.6;95.7]	61.2	[56.0;66.2]
9a	97.1	[94.0;99.4]	99.2	[96.9;100]	99.8	[99.1;100]	80.4	[73.1;87.5]	91.5	[87.8;94.5]	51.6	[46.1;57.1]

Only BCV-A antigen included.

Discussion

We estimated the sensitivity and specificity of a new multiplex and two commercial ELISAs for detection of BRSV and BCV antibodies in BTM using LCA. This is the first study evaluating the MVD-Enferplex BRSV/BCV multiplex. The present study is also the first to present test parameters for the SVANOVIR® BRSV-Ab and SVANOVIR® BCV-Ab on BTM. The BRSV multiplex showed a somewhat lower Se, but a much higher Sp than the BRSV ELISA at the recommended cut-off values. However, when we increased the cut-off of the BRSV ELISA to sample positive >50 PP, this resulted in a large increase in Sp without a notable decrease in Se, as shown in Table 3. Our results therefore suggest that a higher cut-off than recommended by the manufacturer might be appropriate when using the SVANOVIR® BRSV-Ab on BTM. For BCV, the specificity of the multiplex was notably lower than the BCV ELISA at the recommended cut-off when using all three antigens. However, when using the BCV-A antigen only, the Sp improved without the cost of reduced Se, and the test performance was then similar to the BCV ELISA. This implies that the extra antigens are adding false positive samples, hence reducing Sp. Overall; the two tests in this study both showed good performance for detection of both BRSV and BCV antibodies. A possible benefit of choosing the multiplex therefore lies in enabling screening for both agents simultaneously as this will reduce screening costs. As the multiplex evaluated in the present study is a new test, there were no relevant studies we could compare estimates to. However, the multiplex technology has been shown useful for bovine tuberculosis in cattle and goats (Clegg et al., 2011; O’Brien et al., 2017). The parameter estimates provided by the manufacturer for the SVANOVIR® BCV-Ab are based on data from a study in which 91 serum samples were analyzed using both the ELISA and a virus neutralization test (VNT) (Alenius et al., 1991). The estimates, Se of 84.6% and Sp of 100%, were calculated using VNT as gold standard. For the SVANOVIR® BRSV-Ab, the Se (94%) and Sp (100%) were calculated in a study comparing the test results to another ELISA in 151 serum samples. Thus, test estimates were relative to the other ELISA (Elvander et al., 1995). Results from the former studies are not comparable to the present study due to different sample material (serum vs. BTM). Even so, it is important to note that in studies assuming a perfect reference test the estimated Se and Sp of the index test will never exceed those of the gold standard, thus the higher Se of both the BRSV and BCV ELISA found in our study was not unexpected.

To explore the effect of different cut-off values on test characteristics we applied a range of cut-off values for the multiplex antigens. Whenever the cut-off is changed this could entail a change in the definition of the latent condition and change the number of true positive and true negative herds. There was relatively little variation in the Se and Sp estimates of the BRSV- and BCV ELISA across the different cut-off values explored, and the change in estimates of true prevalence was minor. The tests generally agreed on the proportion of positive herds indicating that tests had good agreement on the underlying target condition. The explorative approach to choosing cut-offs is a potential weakness of the current study; however, the different scenarios provide examples of expected performance for different cut-offs and do not represent an optimization of the diagnostic tests. The chosen cut-off will likely affect the number of antibody producing animals needed for a positive BTM result, and a positive correlation between within-herd prevalence and OD-value has been shown for other diseases (Muskens et al., 2011; Nekouei et al., 2015). Because the typical Norwegian dairy herd is small (mean herd size 25.7) (Anon., 2015) compared to most other developed countries, this might influence the generalizability of our results: In larger herds antibodies might be diluted in the bulk tank, and hence cause the test Se to decrease. However, larger herds might also have more positive animals.

Careful evaluation of the model assumptions is crucial when performing LCA, as violation of assumptions might lead to biased results. The assumption of different prevalence between populations is central to LCA models, and Toft et al. (2005) showed that the precision of the accuracy parameters improved with increasing difference in prevalence among the populations studied. In the present study, the difference in prevalence between the two sub-populations was relatively large, which in addition to a sufficient sample size, leads to narrow posterior credibility intervals for the Se and Sp estimates.

The second assumption is that the test characteristics are constant in both populations. The Norwegian dairy herd is relatively homogeneous, and the two sub-populations in this study are likely similar in terms of breeds and production systems. A potential source of variation in test characteristics between sub-populations could be antigenic diversity within the Norwegian dairy herd. Findings of antigenic diversity of BCV are summarized by Saif (2010) who concludes that only a single serotype is known based on virus cross-neutralization tests, and that a high level of cross protection has been shown between respiratory and enteric isolates. For BRSV, a Norwegian study found that the current Norwegian strains of BRSV belonged to the same subgroup as other North European isolates, indicating that the within-country diversity is likely to be limited (Klem et al., 2014a). Additionally, cross-reaction is likely to be common, and has even been shown for isolates from different species (Oberst et al., 1993). Even though it seems unlikely that spatial antigenic diversity plays an important role as source of bias it cannot be excluded with complete certainty.

The final assumption to be met is conditional independence of tests given the disease status. Several papers argue that if tests have similar biological basis, this assumption is likely not met (Gardner et al., 2000; Branscum et al., 2005). Conditional independence between tests means that the probability of a positive (or negative) result from one test is the same regardless of the result of the other test, given the true disease state (Enøe et al., 2000; Toft et al., 2005). Conditional dependence would, in terms of false positives, mean that the second test is more likely to pick up a herd as a false positive if it already tested (false) positive on the first test, for instance due to cross-reactivity with other agents. To estimate covariance between tests (γSe and γSp) two extra degrees of freedom are needed. In a two tests, two populations scenario this results in an unidentifiable model i.e. it is not possible to estimate covariance without including prior information. No reliable prior information could be obtained for test parameters or prevalences in the present study. Another approach potentially allowing for estimation of covariance would be to include a third test: either another antibody test, or a test detecting the virus itself (e.g. a qPCR). The first option would not necessarily allow for estimation of covariance unless the third test had some underlying properties substantially different from the two other tests. Adding an antigen test might ensure conditional independence, however, it would change the underlying disease status to involve not only serological response, but also a coherent shedding of virus. We explored the consequences of conditional dependence (sensitivity analysis) by including fixed covariances as proportions of the maximum possible covariance between tests. For the BCV estimates, allowing for covariance in the latent class models had negligible effect on parameter estimates of both tests. As the Se of the BCV multiplex and the Sp of the BCV ELISA is close to one, the small effect of covariance was expected. It can be shown mathematically that test Se (Sp) are conditionally independent whenever one test has Se (Sp) = 1, see Appendix A for details. This was also the situation for BRSV-Se where the Se of the ELISA is close to 1. However, the COC-models with fixed covariances did yield changes in the estimated specificity for BRSV of both tests. This became most notable when the covariance was assumed larger than 25% of maximum. In summary, the effect of covariance was small except for BRSV-Sp for high values of covariance. It is important to note that the sensitivity analysis gives an indication of the effect of covariance if present, but does not answer whether covariance exists. Even though both tests in this study are antibody tests, they differ in the way they are designed. First, the ELISAs uses crude whole virus in the ELISA well, whereas the BRSV/BCV multiplex uses peptides and recombinant proteins. Second, the tests use different techniques for detection. The ELISAs use a chromogenic substrate and results are based on a reading of optical density, whereas the BRSV/BCV multiplex uses a chemiluminescent substrate where results are based on a reading of light emission. These differences make a violation of the conditional independence assumption less likely.

In conclusion, the BRSV/BCV multiplex and the BRSV/BCV ELISA showed similar performance when applied on BTM samples. The Sp of the BCV multiplex can be improved by using the BCV-A antigen only, and the low Sp of the BRSV ELISA can be improved by increasing the cut-off when using this test on BTM.

Conflict of interest

T., N.T., M.S., L.S. and A.N. declare no conflict of interest. G.H. and N.W. are the founders of MV diagnostics and developers of the multiplex test. A.O’B. is also developer of the multiplex test, and an Enfer employee. However, neither MV diagnostics nor Enfer were involved in the data-analysis, and could not have inappropriately influenced this work.