Validation of a new serum granulocyte-macrophage colony-stimulating factor autoantibody testing kit.

Very recently, a modest but significant efficacy of granulocyte-macrophage colony-stimulating factor (GM-CSF) inhalation therapy for the treatment of mild to moderate autoimmune pulmonary alveolar proteinosis (aPAP) has been reported. As the ability to measure the level of GM-CSF autoantibody (GMAb) in the serum is required to decide the indication for this therapy, we developed a high-performance GMAb testing kit for clinical use. As the kit succeeded in reducing nonspecific IgG binding to the ELISA plate, the predictive performance shown in the training study to discriminate aPAP patients from healthy subjects was perfect, providing a cut-off value of 1.65 U·mL-1 in 78 patients with aPAP and 90 healthy subjects in an operator-blinded manner using logistic regression analysis. As in the validation study, serum samples from another 213 patients with aPAP were also blinded and evaluated in an operator-blinded manner against external 207 samples from patients with other types of PAP and patients exhibiting various ground-glass opacities on chest high-resolution computed tomography that require discrimination from PAP. The logistic regression analysis of these validation data sets revealed values of 97.6% and 100% for specificity and sensitivity, respectively. Thus, this new GMAb testing kit is reliable for the diagnosis of aPAP and differential diagnosis of other lung diseases.

Introduction

Pulmonary alveolar proteinosis (PAP) is a rare lung disease characterised by abnormal accumulation of surfactants in the terminal respiratory tract [1, 2]. Autoimmune PAP (aPAP), accounting for 90% of all PAP cases, with an estimated incidence of 1.65 per million [3], is caused by excess production of granulocyte–macrophage colony-stimulating factor autoantibody (GMAb) [3, 4]. GMAb interferes with granulocyte–macrophage colony-stimulating factor (GM-CSF) signalling in alveolar macrophages, causing maturation arrest and dysfunction, thus impairing surfactant catabolism [4, 5].

Measurement of the concentration of GMAb in the serum is increasingly important because it is an essential requirement to designate PAP as an intractable disease and to decide whether there is an indication for GM-CSF inhalation therapy in Japan [6]. Following the latex agglutination test [7], the ELISA became widely used, owing to its cost-effectiveness and the convenience of multisample processing with this method [8, 9]. In 2014, using a polyclonal GMAb purified from the serum of a patient with aPAP as the standard antibody, we optimised the assay components and procedures by evaluating accuracy, precision, reliability, sensitivity, specificity and ruggedness. Using ELISA, the optimal cut-off value for distinguishing aPAP serum from normal serum was 5 U·mL−1 [10].

However, when we consider the clinical use of ELISA, we encounter several problems. Firstly, a polyclonal standard antibody purified from one patient cannot be shared among multiple laboratories. Moreover, contamination of activated cryptic IgG other than GMAb is possible, despite the highly purified standard [11]. Secondly, the evaluation process was not conducted in a double-blinded manner; thus, we could not exclude operator bias. Thirdly, we did not evaluate the cut-off value of 5 U·mL−1 through an external validation study using different samples from the training samples. Therefore, the reliability of the cut-off value could not be guaranteed. For the differential diagnosis of aPAP, the cut-off value was to be validated by measuring the concentration of GMAb in the sera of patients with other lung diseases who exhibited ground-glass opacity (GGO) on high-resolution computed tomography (HRCT). Finally, the cut-off value of 5 U·mL−1 was likely to be excessively high, because we could not eliminate the binding of nonspecific IgG other than GMAb that may be present in the sera of both patients and healthy subjects [12, 13].

Recently, a kit was developed utilising a mouse–human chimaeric monoclonal antibody against GM-CSF. This kit was designed to reduce the serum nonspecific IgG binding to the ELISA plate. Using the kit, we determined the cut-off value in 78 patients with aPAP and 90 healthy subjects in an operator-blinded manner with evaluation using external samples of other types of PAP and patients exhibiting various GGOs on HRCT that require discrimination from PAP.

Methods

Subjects

The institutional review board of the 12 participating study sites (supplementary methods) and internal ethical committee of Medical and Biological Laboratories, Ltd. (MBL; Nagano, Japan) approved this study. All training and validation samples were randomly assigned in a blinded manner, and the data manager at the Clinical and Translational Research Center of Niigata University Hospital (Niigata, Japan) managed the linking table in secret until the key was opened.

For the training study, 78 patients with aPAP were prospectively enrolled at 12 hospitals. The diagnosis of aPAP was reached as described in the supplementary methods. For the control, 90 healthy subjects were enrolled in this study on random basis as age- and sex-matched pairs with patients in this study.

For the validation study, we used sera preserved at −80°C at the Clinical and Translational Research Center. These samples had been sent from various regions in Japan for the purpose of measuring the concentration of GMAb in the serum. The samples included sera from 213 patients with aPAP and 207 patients with conditions other than aPAP. The latter population included 40 and five sera samples from patients with secondary and hereditary PAP, respectively, as well as sera from 162 patients who visited our hospital and exhibited GGO on HRCT but were proven to have lung diseases other than aPAP through bronchoalveolar lavage (BAL) or video-assisted thoracoscopic surgery (VATS). These were retrospectively applied for the measurement of GMAb using the opt-out of the existing specimens (supplementary methods).

Measurement of the concentration of GMAb using ELISA

The concentration of GMAb in the serum samples was measured using the Anti-GM-CSF Autoantibody Measuring Kit (MBL2490023; MBL), which includes microtitre plates coated with recombinant GM-CSF (MBL), according to the instructions provided by the manufacturer (the details are provided in the supplementary methods). All samples were diluted (1/201 dilution) prior to the measurement. Samples with higher GMAb concentration values than the standard top were remeasured after undergoing a 1/2010 dilution. Six points of data for the serially diluted monoclonal anti-GM-CSF antibody standard 33–8F-H (secondary standard, supplementary methdos). Units per millilitre were defined as described in supplementary methods and used to produce a four-parametric logistic curve with Microsoft Excel (Microsoft Corp., Seattle, WA, USA). For quality control, two known GM-CSF-positive serum samples were simultaneously measured in each plate. Anti-aminoacyl transfer RNA synthetase (ARS) and anti-melanoma differentiation-associated protein (MDA)-5 antibody indices were measured in all validation samples [14, 15] (supplementary methods).

The training samples were measured using modified conventional ELISA, as described in the supplementary methods, to compare the new ELISA kit with the conventional system.

Statistical analysis

The detail of the general statistical methods is described in the supplementary methods. A logistic regression model evaluated the cut-off value between two groups. Verification of internal validity of the cut-off value was conducted using a cross-validation method [16] (supplementary methods). Sensitivity, specificity, and positive and negative predictive values were calculated, and receiver operating characteristic curve analysis was performed for each cut-off value [17]. All statistical analyses were conducted using SAS software, version 9.4 (SAS institute, Cary, NC, USA). A p-value <0.05 denoted statistical significance.

Results

Distribution of the concentration of GMAb in the training study

We evaluated the distribution of the concentration of GMAb in 78 patients with aPAP and 90 healthy subjects using a newly developed GMAb measuring kit (MBL2490023, details of the performance of the kit are described in the supplementary methods). There was no difference in the age or sex ratio between patients with aPAP and healthy subjects (table 1). The healthy subject group consisted of 22 Asian and 68 Caucasian subjects, whereas all patients with aPAP were Asian. Values outside the main distribution of the concentration of GMAb in 90 healthy subjects included only one sample at 0.71 U·mL−1. There was no difference in the concentration of GMAb between Asian and Caucasian subjects. The mean±sd concentration of GMAb in 90 healthy subjects and 78 patients with aPAP was 0.151±0.109 U·mL−1 and 91.0±123.0 U·mL−1, respectively (figure 1a). Both the distributions for the healthy subjects and the patients appeared lognormal without overlap (figure 1b).

TABLE 1

Demographic data of subjects in the training study

Characteristics	Healthy subjects	aPAP patients
Subjects n	90	78
Male/female n	37/53	32/46
Age years mean±sd	56.4±14.1	58.7±12.3
Asian/Caucasian n	22/68	78/0
Serum GMAb concentration U·mL−1
Mean±sd	0.151±0.109	90.95±123.0
Maximum	0.71	718.7
Minimum	0.018	2.59

aPAP: autoimmune pulmonary alveolar proteinosis; GMAb: granulocyte–macrophage colony-stimulating factor autoantibody.

FIGURE 1

Distribution of the concentration of granulocyte–macrophage colony-stimulating factor autoantibody (GMAb) in serum samples in the training study. a) A bee swarm plot of the serum GMAb concentrations in 78 patients with autoimmune pulmonary alveolar proteinosis (aPAP) and 90 healthy controls (HC) measured using the newly developed GMAb measuring kit (MBL2490023). b) A histogram of logarithmic serum concentrations in the same patients and healthy subjects as shown in (a).

Comparison of ELISA performance between the newly developed kit and the conventional kit

For the comparison of performance, the same sample sets were evaluated using kit of the modified conventional method (conventional kit) that used the same standard monoclonal anti-GM-CSF antibody, the coated recombinant GM-CSF and the detection antibody as used in the new kit, but used conventional blocking reagent and dilution solution as described in the supplementary methods. As shown in figure 2a and b, in healthy subjects, the concentration of serum GMAb by the new kit was consistently lower than that by the conventional kit (correlation coefficient 0.80) and the maximum value yielded by the new kit was far lower than that observed by the conventional kit (0.71 versus 2.01 U·mL−1, respectively) (figure 2a and b). When the sera were previously incubated with excess recombinant GM-CSF (50 µg·mL−1), the GMAb concentrations by the new kit decreased to 0.13±0.08 U·mL−1, whereas measurement with the conventional kit (0.33±0.27 U·mL−1) did not go down to that level (p<0.001, U test) (figure 2a and b). This suggests that the nonspecific binding by the conventional kit was not completely inhibited by pre-incubation of the sera with excess recombinant GM-CSF.

FIGURE 2

Comparison of the performance of ELISA between the newly developed kit and the conventional method. Granulocyte–macrophage colony-stimulating factor autoantibody (GMAb) concentrations measured a) by the modified conventional ELISA kit or b) by the newly developed ELISA kit in the sera from 90 healthy subjects that were previously incubated with or without excess recombinant granulocyte–macrophage colony-stimulating factor (GM-CSF) (50 mg·mL−1). c) The correlation coefficient of the concentrations between the newly developed and the conventional methods was 0.80 in the healthy subjects and 0.95 in the patients with autoimmune pulmonary alveolar proteinosis. d) Receiver operating characteristic curve analysis (area under the curve 1.0).

Although not as much as samples from healthy subjects, the concentrations of GMAb in aPAP were also affected by renewal of the kit, especially in samples with low concentrations. The minimum value was lower using the new kit than conventional kit (2.59 versus 3.49 U·mL−1, respectively). However, as a whole, data from the two kits were in good agreement, with a correlation coefficient of 0.95 in the patients with aPAP (figure 2c). These data suggested that the new kit succeeded in reducing the nonspecific binding of IgG to the recombinant GM-CSF in both healthy subjects and patients, without greatly changing the values of the patients. This resulted in an excellent predictive performance of the kit. The receiver operating characteristic curve (area under the curve 1.0, 95% CI 1.0–1.0; p<0.001) (figure 2d) supported the kit's excellent predictive performance [17, 18].

Estimation of the cut-off value between healthy subjects and patients

We subsequently estimated the cut-off value between the healthy subjects and patients using the logistic regression method [18] according to the following equation:where P is the vector of probabilities of predicted outcomes for each object that can only have two values (in this study, aPAP or not), X is the design matrix of values of explanatory variables (GMAb values in this study) and β is the vector of the model's coefficients. The cut-off value was estimated to be 1.65 U·mL−1 with a sensitivity and specificity of 100% (figure 3). This was verified using the cross-validation method with positive and negative predictive values of 100% (error rate 0%). For the comparison of performance, modified conventional ELISA showed that the estimated cut-off value for the data was 2.76 U·mL−1. Collectively, the decreased values in the data of healthy subjects using the new kit resulted in reduction of the cut-off value by >1.0 U·mL−1. This effect strengthened the predictive performance of the new kit.

FIGURE 3

Estimation of the cut-off value between the healthy subjects and the patients using the logistic regression methods.

Verification of the cut-off value by the validation study

The concentration of GMAb in patients with aPAP in the validation study, measured and preserved during 2010–2012, 2013–2015 and 2016–2018, were 96.26±139.1 (n=63), 108.6±134.2 (n=80), and 104.4±108.9 U·mL−1 (n=70), respectively, indicating that the period of preservation did not affect the data (p=0.628). The mean concentration in the whole sample (n=213) was 103.6±127.5 U·mL−1 (table 2). All measurements were >1.65 U·mL−1, indicating 100% sensitivity at the cut-off value. Importantly, the distribution matched the lognormal distribution of the training study (p=0.205, U test) (supplementary figure 4).

TABLE 2

Demographic data of cases in the validation study

Characteristics	aPAP	sPAP	hPAP	Other lung diseases
Subjects n	213	40	5	162
Male/female n	122/60	19/21	0/5	86/75
Age years mean±sd	52.6±14.7	57.6±13.3	56.4±14.5	66.2±12.0
Asian/Caucasian n	213/0	40/0	5/0	162/0
Serum GMAb concentration U·mL−1
Mean±sd	103.6±127.5	0.191±0.302	0.148±0.141	0.517±.818
Maximum	899.9	1.9	0.38	48.57
Minimum	5.59	0.04	0.02	0.01
Diagnostic procedures n
BAL	213	36	5	144
VATS	0	4	0	17
Autopsy	0	0	0	1
Other lung diseases n
CTD				45
DIPD				20
IIPs				60
IPF				8
Other IIPs				52
Infectious disease				19
Miscellaneous				18

Connective tissue disease (CTD) cases included five amyopathic dermatomyositis cases, 11 polymyositis/dermatomyositis interstitial lung diseases, 11 rheumatoid arthritis cases, four Sjögren syndromes, five systemic scleroses and nine other diseases. Infectious disease cases included eight Pneumocystis pneumonias, four bacterial pneumonias, three influenza pneumonias and four other infectious diseases. Miscellaneous diseases included four alveolar haemorrhages, three chronic eosinophilic pneumonias, two acute respiratory distress syndromes, two chronic hypersensitivity pneumonias and seven other diseases. aPAP: autoimmune pulmonary alveolar proteinosis; sPAP: secondary pulmonary alveolar proteinosis; hPAP: hereditary pulmonary alveolar proteinosis; GMAb: granulocyte–macrophage colony-stimulating factor autoantibody; BAL: bronchoalveolar lavage; VATS: video-assisted thoracic surgery; DIPD: drug-induced pulmonary disease; IIP: idiopathic interstitial pneumonia; IPF: idiopathic pulmonary fibrosis.

A total of 162 patients with other lung diseases and GGO on HRCT were enrolled. In this study, lung diseases other than aPAP were confirmed based on underlying disorders, and consistent findings of BAL and/or VATS (table 2 and figure 4). These diseases included 45 patients with connective tissue diseases (CTDs); eight patients with idiopathic pulmonary fibrosis; 52 patients with other idiopathic interstitial pneumonias (IIPs); 20 patients with drug-induced pulmonary disease; 19 patients with infectious disease such as atypical, influenza, Legionella, nontuberculosis, Pneumocystis jirovecii and bacterial pneumonia; and 18 patients with miscellaneous lung diseases. We externally verified the cut-off value (1.65 U·mL−1) using samples of aPAP against other lung diseases and other PAP (secondary and hereditary). A total of 213 patients with aPAP were positive for serum GMAb, while four of the 162 patients with other lung diseases tested positive; the antibody was detected with 97.5% specificity and 100% sensitivity, respectively. One of 45 patients with other PAP was positive for serum GMAb, indicating a 97.8% specificity at the cut-off value. As a whole, external validation of the cut-off value using the sera of 213 patients with aPAP versus 207 patients with conditions other than aPAP revealed values of 97.6% and 100% for specificity and sensitivity, respectively.

FIGURE 4

Verification of the cut-off value (1.65 U·mL−1) using data obtained from the sera of patients with autoimmune (aPAP) (n=213), secondary (sPAP) (n=40) and hereditary pulmonary alveolar proteinosis (hPAP) (n=5), and other lung diseases (h=162). GMAb: granulocyte–macrophage colony-stimulating factor autoantibody.

Demographic characteristics of non-aPAP patients with GMAb >1.65 U·mL−1

The demographic characteristics of five out of the 207 patients with conditions other than aPAP, who exhibited a concentration of GMAb in the serum >1.65 U·mL−1, are shown in table 3 and the supplementary results. One patient (case 1) with secondary PAP demonstrated a concentration of 1.9 U·mL−1. Myelodysplastic syndromes with refractory anaemia were the underlying disease. PAP was suspected based on the diffuse homogeneous GGO pattern observed on HRCT (supplementary figure 5A) and pathologically proven through VATS. Four of the 162 patients with GGO observed on HRCT, which had been discriminated from PAP by underlying diseases and findings from BAL and/or VATS, showed low-to-moderate serum GMAb titres ranging from 2.56 to 48.57 U·mL−1 (table 3). Two patients had IIPs (cases 2 and 5), while the other two patients had CTD (cases 3 and 4). Chest HRCT revealed GGO and consistent fibrotic change with (cases 3, 4, and 5) or without consolidation (case 2). Findings of mild emphysema (case 4) and traction bronchiectasis (cases 3 and 4) were noted. Importantly, both the anti-MDA-5 [14] and anti-ARS antibodies [15] were negative in the sera of all five patients. Among 420 subjects (213 with aPAP and 207 others), the former was positive in one patient with secondary PAP complicated with chronic graft versus host disease and five patients with CTDs (two polymyositis/dermatomyositis interstitial lung disease, three clinically amyopathic dermatomyositis); the latter was positive in two aPAP (weakly), one secondary PAP, one drug-induced pulmonary disease, 10 CTDs (six polymyositis/dermatomyositis interstitial lung disease, two systemic sclerosis and one clinically amyopathic dermatomyositis) and 10 IIP patients. Interestingly, every two of three antibodies were exclusive to each other between aPAP and others (figure 5), suggesting that the occurrence of these antibodies was independently case specific.

TABLE 3

Demographic characteristics of patients with granulocyte–macrophage colony-stimulating factor autoantibody (GMAb) >1.65 U·mL−1

Case	Age years	Sex	Diagnosis	GMAb U·mL−1	Anti-MDA-5 Ab	Anti-ARS Ab
1	63	M	sPAP (MDS)	1.90	Negative	Negative
2	76	F	IIPs	2.56	Negative	Negative
3	75	M	CTD	48.57	Negative	Negative
4	68	M	CTD	2.85	Negative	Negative
5	73	F	IIPs	2.85	Negative	Negative

MDA: melanoma differentiation-associated protein; Ab: antibody; ARS: aminoacyl transfer RNA synthetase; M: male; F: female; sPAP: secondary pulmonary alveolar proteinosis; MDS: myelodysplastic syndrome; IIP: idiopathic interstitial pneumonia; CTD: connective tissue disease.

FIGURE 5

Exclusive plots for a) granulocyte–macrophage colony-stimulating factor autoantibody (GMAb) versus anti-melanoma differentiation-associated protein (MDA)-5 antibody (Ab), b) GMAb versus anti-aminoacyl transfer RNA synthetase (ARS) Ab and c) anti-MDA-5 Ab versus anti-ARS Ab. Red: autoimmune pulmonary alveolar proteinosis; blue: other patients.

Discussion

Through a newly developed ELISA kit, we achieved a low cut-off value (1.65 U·mL−1) estimated using samples in the training study by minimising the binding of nonspecific IgG included in the samples. External validation of the cut-off value using the sera of 213 patients with aPAP versus 207 patients with conditions other than aPAP in the validation study revealed high sensitivity and specificity. These findings indicated that the ELISA system is reliable for clinical use.

Previously, our ELISA system, in which polyclonal GMAb was used as the standard antibody and Stabilicoat was used as the blocking reagent, identified a cut-off serum GMAb level of 5 U·mL−1 for distinguishing aPAP serum from healthy serum [10]. This polyclonal standard was purified from the plasma of a single aPAP patient. Thus, it is difficult to generalise the standard concentration. Therefore, we decided to use the polyclonal antibody as the primary standard and developed a mouse–human chimaeric monoclonal antibody (33–8F-H) as the standard antibody in the kit.

The low cut-off value (1.65 U·mL−1) was probably achieved by suppressing the nonspecific binding of serum IgG other than GMAb. Among the sera obtained from 90 healthy subjects, the maximum concentration was 0.71 U·mL−1 with a mean value of 0.15±0.11 U·mL−1. By the conventional test, the difference between the patients' minimum value and healthy subjects' maximum was 1.48 U·mL−1, whereas this was 1.89 U·mL−1 by the new test, indicating that the nonintersection range increased ≥0.41 U·mL−1 (27%). This increased the reliability of GMAb testing in aPAP because we can exclude the possibility of nonspecific binding of IgG, especially in patients with a low concentration of GMAb in the serum. Moreover, this property is advantageous when measuring low concentrations of GMAb in the serum of several diseases such as Crohn's disease [19-21] and myasthenia gravis [22, 23], which were reported to be frequently positive when measured by the conventional methods, so it is better to remeasure using the new kit.

For the practical interests of pulmonary physicians, it will be important to diagnose aPAP noninvasively. According to the previous review articles [24] and descriptions [25], the diagnostic procedures for aPAP may require the inclusion of an invasive examination, such as BAL or VATS. The role of BAL or lung biopsy in the diagnosis of aPAP is still controversial. Future research is necessary for the noninvasive diagnosis of aPAP by the findings of GGO on chest HRCT and positivity for GMAb in the serum described previously [26]. In this regard, we consider that the reliability of serological diagnosis depends on the concentration of GMAb in the serum. Using the present kit to discriminate between healthy and aPAP patients, aPAP can be excluded from the diagnosis in patients with concentrations of GMAb in the serum of <1.65 U·mL−1. In those with a concentration >1.65 U·mL−1, there is a >50% possibility of reaching an aPAP diagnosis. As for our analysis of the present validation data, using the present kit to discriminate between patients with other pulmonary diseases and those with aPAP and GGO observed on chest HRCT, when the serum GMAb was >15 or >21 U·mL−1, the probability of diagnosis was >90% or >99%, respectively.

The presence of a few patients with conditions other than aPAP and positivity for serum GMAb cautions us against diagnosing aPAP based exclusively on GGO findings on HRCT and the concentration of GMAb in the serum, without other clinical features including pathological evidence. As secondary PAP develops in the presence of underlying diseases, the differential diagnosis may not be as challenging [27]. Instead, problems may be encountered in the differential diagnosis of other cases with diffuse lung diseases characterised by positivity for serum GMAb and GGO observed on HRCT.

When the positive and negative likelihood ratios were applied for alternative statistics instead of positive and negative predictive value, the former ratio was 41.4 and the latter ratio was 0, indicating that the cut-value of 1.65 U·mL−1 was reevaluated to be useful for definitive diagnosis [28].

In conclusion, we evaluated the performance of a newly developed ELISA kit measuring the concentration of GMAb in the serum for the diagnosis of aPAP. The predictive performance to discriminate aPAP patients from healthy subjects was perfect, providing a cut-off value of 1.65 U·mL−1 that was externally validated. We believe that this kit will contribute to the definitive diagnosis of aPAP and differential diagnosis of diffuse lung diseases.

Please note: supplementary material is not edited by the Editorial Office, and is uploaded as it has been supplied by the author.

Supplementary material 00259-2019.supp

Figure S1 00259-2019.figureS1

Figure S2 00259-2019.figureS2

Figure S3 00259-2019.figureS3

Figure S4 00259-2019.figureS4

Figure S5 00259-2019.figureS5