Standardisation of ACPA tests: evaluation of a new candidate reference preparation.

INTRODUCTION: Commercial assays measuring antibodies to citrullinated protein/peptide (ACPA) show poor quantitative agreement. The diagnostic industry has never adopted the International Union of Immunological Societies-Centers for Disease Control and Prevention (IUIS-CDC) ACPA reference standard. Recently, the National Institute for Biological Standards and Control (NIBSC) prepared a new candidate ACPA standard (18/204). We evaluated both reference materials using different commercially available ACPA assays.
MATERIALS AND METHODS: This is an international study in which the NIBSC candidate ACPA standard and the IUIS-CDC ACPA reference material were analysed together with 398 diagnostic samples from individuals with rheumatoid arthritis (RA) and in 1073 individuals who did not have RA using nine commercial ACPA assays.
RESULTS: For both reference materials and samples from individuals with RA and individuals who did not have RA, there were large differences in quantitative ACPA results between assays. For most assays, values for the IUIS-CDC standard were lower than values for NIBSC 18/204 and the IUIS-CDC/NIBSC ratio was comparable for several, but not all assays. When NIBSC 18/204 was used as a calibrator, an improvement in alignment of ACPA results across several of the evaluated assays was obtained. Moreover, NIBSC 18/204 could align clinical interpretation for some but not all assays.
CONCLUSION: Adoption of an international standard for ACPA determination is highly desirable. The candidate NIBSC 18/204 standard improved the standardisation and alignment of most ACPA assays and might therefore be recommended to be used as reference in commercial assays. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

Results obtained with commercial antibodies to citrullinated protein/peptide (ACPA) assays show poor quantitative agreement.

Adoption of an international standard for ACPA by the diagnostic industry is highly desirable.

The candidate National Institute for Biological Standards and Control 18/204 standard improved alignment of most, but not all ACPA assays.

Alignment of ACPA assays would be particularly important in the context of the American College of Rheumatology/European Alliance of Associations for Rheumatology 2010 rheumatoid arthritis (RA) classification criteria, where ACPA concentration has a high impact on rRA classification.

Introduction

Antibodies to citrullinated protein/peptide (ACPA) are established biomarkers for diagnosis and classification of rheumatoid arthritis (RA).1 Measurement of ACPA is widely used and several manual and (semi-)automated assays are commercially available. However, there is poor agreement among the currently available ACPA assays, which may have an impact on RA classification of a patient.2 3

An international ACPA reference preparation derived from a single patient donor has been prepared by the International Union for Immunological Societies (IUIS) and Centers for Disease Control and Prevention (CDC) and is available through the Autoantibody Standardisation Committee (www.AutoAb.org).4 A preliminary evaluation of this preparation using 12 ACPA ELISAs and samples from 20 patients with RA and 50 healthy subjects concluded that it could be used as a reference standard.5 However, this preparation has not been adopted by the in vitro diagnostic kit manufacturers as a reference standard for establishing calibration curves in the commercial assays.

Due to the role of ACPA quantification in classification, diagnosis,6 7 risk stratification and prognosis of individuals with RA,8 9 the International Working Group on the Harmonisation of Autoantibody tests of the International Federations of Clinical Chemistry and Laboratory Medicine listed ACPA as one of the antibodies for which the production of a commutable reference material is urged.10 Moreover, traceability to a higher-order reference material (if available) is mandatory according to the In-Vitro Diagnostic Medical Devices Regulation (EU) 2017/746 (IVDR).11

Therefore, the National Institute for Biological Standards and Control (NIBSC) recently prepared a candidate ACPA standard named 18/204 which has been evaluated in a large international collaborative study; the results and conclusions of which will be presented to the WHO in Autumn 2022 as official candidate for the first WHO international ACPA standard (personal communication). The material consists of a serum pool of five individuals with RA and will be made available by NIBSC in due course. A reference material derived from a pool of 5 sera should more closely mimic the polyclonal response than a single donor-derived reference serum.

Here, independently of the NIBSC international study described above, we evaluated NIBSC 18/204 together with the IUIS-CDC ACPA reference material using different commercially available ACPA assays and sera from individuals with RA and individuals who did not have RA (either suffering from another (rheumatic) disease or healthy).

Materials and methods

ACPA assays from nine different manufacturers (Thermo Fisher Scientific, Uppsala, Sweden; Roche Diagnostics, Mannheim, Germany; Svar Life Science, Malmö, Sweden; Immunodiagnostic Systems (IDS), Tyne and Wear, UK; Orgentec, Mainz, Germany; Abbott, Wiesbaden, Germany; Euroimmun, Lübeck, Germany; BioRad Laboratories, Hercules, California, USA; and Siemens Healthineers, Sudbury, UK) encompassing different technological platforms (ELISA, fluoroenzyme immunoassay, chemiluminescence assay and addressable laser bead assay) were included in the study. Details on the different assays are given in online supplemental table 1. The antigens used in all assays are cyclic citrullinated synthetic peptides (second generation) except for the Orgentec assay which uses cyclic citrullinated vimentin peptides.

The IUIS-CDC ACPA reference material was obtained from Plasma Services Group (Moorestown, New Jersey, USA).4 5 The NIBSC 18/204 candidate standard was provided by NIBSC (see online supplemental data ‘Description of NIBSC 18/204’ for details on the preparation and properties of the material). NIBSC 18/204 is intended as reference material for IgG ACPA antibodies, not for IgA ACPA antibodies (NIBSC, personal communication). Both materials were reconstituted according to the guidelines of the provider and aliquoted. Both reference materials were measured in 19 different runs.

Imprecision of all ACPA assays was determined using (1) manufacturer’s internal quality control (iQC) materials and (2) patient serum samples with a low, medium and high ACPA concentration.12 All iQC samples were measured before and after every run during 19 runs.

Linearity was assessed by diluting the IUIS-CDC ACPA and NIBSC 18/204 standards with increasing amounts of phosphate buffered saline. Every dilution was analysed three times in different runs.13

Serum samples from 398 individuals with RA and 1073 individuals who did not have RA were included. Serum samples were obtained from 11 European hospitals: Division of Rheumatology, Medical University of Vienna (Austria), University Hospital of Leuven (Belgium), University Hospital of Ghent (Belgium), OLV Hospital of Aalst (Belgium), National Institute of Rheumatology and Physiotherapy of Budapest (Hungary), Centre Hospitalier de Luxembourg (Luxembourg), University Medical Centre of Ljubljana (Slovenia), Sahlgrenska Academy Hospital of Gothenburg (Sweden), University Hospital of Linköping (Sweden), University Hospital of Basel (Switzerland), and Kantonsspital of Aarau (Switzerland).

The RA cohort (n=398) consisted of consecutive individuals with newly diagnosed RA. The individuals who did not have RA (n=1073) consisted of (1) a rheumatological disease control group (n=656) (ie, consecutive individuals consulting a rheumatology clinic for the first time but in whom RA was eventually excluded); (2) specific disease control cohorts (ie, individuals with established diagnoses of antineutrophil cytoplasmic antibody associated vasculitis with arthritis (n=24), osteoarthritis (n=25), psoriatic arthritis (n=25), reactive arthritis (n=20), spondyloarthritis (n=25), systemic lupus erythematosus (n=50) and primary Sjögren’s syndrome (n=48)) and (3) and healthy individuals (n=200). Sample collection complied with the Word Medical Association Declaration of Helsinki (as revised in 2013). A detailed description of the study groups is provided in14 and in online supplemental table 2 (individuals with RA) and online supplemental table 3 (individuals with no RA). The diagnostic performance of the ACPA assays included in this study based on the samples from individuals with RA and who did not have RA was published previously.14 In short, when the manufacturer’s cut-off was used, the sensitivity ranged from 57.8% to 64.6% and the specificity from 94.9% to 97.8%. When three times the upper limit of normal was used as threshold, the sensitivity ranged from 50.8% to 60.1% and the specificity from 98.0% to 98.5%.14

ROC curves were generated with Analyse-it for Microsoft Excel.

Results

Data on imprecision using patient serum samples with a low, intermediate and high ACPA concentration are given in online supplemental table 4A. Imprecision data obtained with the two reference materials are given in online supplemental table 4B. The highest imprecision was found for ELISAs. Except for the assay from Roche, the CUSUM test for linearity did not reveal significant deviation from linearity (online supplemental figure 1 legend, online supplemental figure 1).

The candidate NIBSC 18/204 ACPA standard and the IUIS-CDC ACPA reference material were measured in 19 different runs. For both reference materials, there were (large) differences in quantitative ACPA results between assays (online supplemental table 4B, figure 1A). With BioRad, values exceeded the measuring range for both reference materials. All assays scored both reference materials as ‘strongly positive’ according to the 2010 American College of Rheumatology (ACR)/European Alliance of Associations for Rheumatology (EULAR) criteria.1

Figure 1

Quantification of candidate NIBSC 18/204 ACPA reference material (A) and IUIS-CDC ACPA reference material (B). The reference materials were reconstituted according to the guidelines, aliquoted, stored frozen (−20°C) on analysis and tested in 19 different runs with every ACPA assay. (A, B) Box-whisker plots of the results obtained. Boxes represent median and IQR, whiskers represent lowest and highest measurement excluding ‘outside’ values (ie, larger than the upper quartile plus 1.5 times the IQR; highlighted in red). The manufacturer’s cut-offs are marked as red bars. The y-axis represents the manufacturer-specific units. (C) Box-whisker plots of the CDC ACPA reference material recalculated taken the reactivity of the candidate NIBSC 18/204 ACPA standard arbitrarily as 100 units. ACPA, antibodies to citrullinated protein/peptide; CDC, Centers for Disease Control and Prevention; IUIS, International Union of Immunological Societies; NIBSC, National Institute for Biological Standards and Control.

For NIBSC 18/204, results obtained with assays from IDS, Abbott and Euroimmun were similar with a median value of 103.7 AU/mL, 110.7 U/mL and 100.6 RU/mL, respectively. Somewhat higher values were obtained with the Siemens (median 130.1 U/mL) and Thermo Fisher (median 206.0 U/mL) assays. The highest values were obtained with the Roche (355.9 U/mL) and Svar Life Science (468.8 U/mL) assays. The lowest results were obtained with the Orgentec assay (40.6 U/mL).

For IUIS-CDC, a comparable spread of results across the different assays similar to that of NIBSC 18/204 was found, except for Orgentec which, in contrast, did not give the lowest result for IUIS-CDC (figure 1B). Results obtained with assays from Thermo Fisher, Abbott, Siemens, Euroimmun, and IDS amounted to 70.7%, 69.0%, 69.7%, 78.8% and 79%, respectively, of those obtained for NIBSC 18/204. Results obtained with assays from Roche, Svar Life Science and Orgentec amounted to 97.0%, 51.5% and 492.8%, respectively, of those obtained for NIBSC 18/204. Thus, for most assays, values for IUIS-CDC were lower than those for NIBSC 18/204 and the ratio of IUIS-CDC/NIBSC was comparable for several, but not all assays. In summary, when NIBSC 18/204 was used as a calibrator, an improvement in the alignment of ACPA results across several of the evaluated assays was obtained (figure 1C). Indeed, significant agreement was found for (1) Siemens, Thermo Fisher and Abbott, (2) Siemens, IDS and Abbott, (3) Euroimmun and IDS (p>0.2, Mann-Whitney U test), but not for Roche, Orgentec and Svar Life Science (p<0.004 for comparison to all other assays).

Figure 2 shows the correlation between different ACPA assays. Full details are shown in online supplemental table5. Results obtained with different dilutions of NIBSC 18/204 are also shown. The best correlations (Spearman’s r 0.823–0.839) were found between IDS and Abbott, IDS and Siemens and Abbott and Siemens. There was a large dispersion of the results for comparisons with assays from Orgentec, Roche and BioRad. There was good commutability of NIBSC 18/204 with patient samples across Siemens, Thermo Fisher, Abbott, Euroimmun, IDS (and Svar Life Science). There was lower commutability for BioRad, Roche and Orgentec. It should be noted that a substantial fraction of patients with RA (range 23.1% (Svar Life Science) – 53.0% (Orgentec); mean 35.8%) and controls (range 0.1% (Svar Life Science) – 1.8% (Orgentec); mean 0.6%) had ACPA values that exceeded the NIBSC 18/204 ACPA level. For comparison, the fraction of patients with RA and controls that had ACPA values that exceeded the measuring range was 8.3% (Svar Life Science) – 34.5% (BioRad) and 0.0% (Svar Life Science) – 0.7% (Roche and BioRad), respectively.

Figure 2

Correlations of individual values of ACPA measured by nine different immunoassays. The immunoassays included were from Thermo Fisher (TF, 1), Roche (R, 2) Svar life science (SV, 3), IDS (I, 4), Orgentec (O, 5), Abbott (A, 6), Euroimmun (E, 7), BioRad (B, 8) and Siemens (S, 9). The samples were from patients with RA (n=398) and (disease) controls (n=1073) obtained in 11 European hospitals. The NIBSC 18/204 candidate ACPA reference preparation and dilutions thereof (0/4-1/4-2/4-3/4-4/4) are represented by triangles. Spearman’s rank correlation coefficients (rs) are shown in the insert on the graph. Detailed statistical data on Spearman’s correlation and Bland-Altman are given in online supplemental table 5. ACPA, antibodies to citrullinated protein/peptide; IDS, immunodiagnostic systems; NIBSC, National Institute for Biological Standards and Control.

In order to explore whether the candidate NIBSC standard can be employed to align clinical interpretation we located the sensitivity and ‘1 – specificity’ associated with a 1:4 dilution of NIBSC 18/204 on the Receiver Operating Characteristics (ROC) curves (generated with 398 individuals with RA and 1073 individuals with no RA) (figure 3). Strikingly, for five assays (Thermo Fisher, Svar Life Science, IDS, Euroimmune and Siemens) the sensitivity / ‘1-specificity’ points almost coincided on the ROC curves. For the Abbott assay, the location of the sensitivity/‘1-specificity’ point was close to those of the 5 above-mentioned assays, whereas for the Roche, BioRad and Orgentec assays, the location was separate. This separate location relates to the non-commutability and/or non-linearity of NIBSC 18/204 with assays from Roche, Orgentec and BioRad (see above).

Figure 3

ROC curve analysis and likelihood ratios for NIBSC 18/204. Left hand pane: ROC for nine different ACPA assays with indication of the sensitivity and ‘1-specificity’ of the result associated with a 1:4 dilution of NIBSC 18/204 (red filled circle surrounded by black line). Right handpanel: likelihood ratio of a test result interval with as centre the result of the candidate NIBSC standard. The interval was chosen such that the number of data points with results higher than the result of NIBSC 18/204 equaled the number of data points with results that were lower than the NIBSC 18/204. The intervals were as follows: thermo Fisher: 9–148 U/mL, Roche: 134–387 U/mL, Svar: 28.6–200 U/mL, IDS: 7.9–677 AU/mL, Orgentec: 3.2–16.3 U/mL, Abbott: 7.3–65.1 U/mL, Euroimmun: 9.7–59.9 U/mL, Siemens: 5.3–69.9 RU/mL. For BioRad, no likelihood ratio was calculated as many results had values exceeding the upper limit. ACPA, antibodies to citrullinated protein/peptide; CCP, cyclic citrullinated synthetic peptides; IDS, immunodiagnostic systems; NIBSC, National Institute for Biological Standards and Control; ROC, receiver operating characteristics.

A similar location on the ROC curve suggests that the likelihood ratios associated with that particular test result are comparable. Next, we determined the likelihood ratio associated with a test result interval with as centre the result obtained with a 1:4 dilution of NIBSC 18/204. For Thermo Fisher, Svar Life Science, IDS, Euroimmune and Siemens, the likelihood ratio associated with such result interval was ~10. By contrast, it was 19, 27 and 0.61 for Abbott, Roche and Orgentec, respectively.

Taken together, the candidate NIBSC standard can be used to align clinical interpretation for five of the nine tested assays. In practical terms, ACPA test results obtained with assays from Thermo Fisher, Svar Life Science, IDS, Euroimmun and Siemens exceeding the result of a 1:4 dilution of NIBSC 18/204 will have an associated likelihood ratio of at least 10. This does not hold for assays from Roche, Abbott, Orgentec or BioRad. It may hold for other assays not included in this study under the condition that for these assays there is good commutability of the reference material with the assays from Thermo Fisher, Svar Life Science, IDS, Euroimmun and/or Siemens. It should be noted that for all assays included in this study, test result interval-specific likelihood ratios have been described.14

Conclusion

NIBSC 18/204 was evaluated as a candidate reference material to standardise ACPA assays. This candidate standard improved the standardisation and alignment of most ACPA assays evaluated in this study. It may also help to align clinical interpretation of test results. However, differences in results between (some) assays still remain. As has been shown for anti-neutrophil cytoplasmic antibodies, using a common reference material does not assure a common clinical interpretation for all assays.15 Factors that might contribute to the non-commutability across assays include non-linearity, difference in antigen recognition and assay configuration.10 16 17 Adoption of an international standard for ACPA, as it has been defined for rheumatoid factor, is highly desirable and would facilitate comparison between ACPA assays of different manufacturers.18 19 This would be particularly important in the context of the ACR/EULAR classification criteria, where ACPA concentration has a high impact on RA classification.1 NIBSC 18/204 could be used as a calibrator by kit manufacturers or as a reference reagent by diagnostic laboratories to standardise the results and line up clinical interpretation.