Intra- and inter-laboratory agreement of the FAM19A4/mir124-2 methylation test: Results from an international study.

BACKGROUND: HPV-based cervical screening detects women at an increased risk of cervical cancer and precancer. To differentiate among HPV-positive women those with (pre)cancer, triage testing is necessary. The detection of cancer-associated host-cell DNA methylation (FAM19A4 and hsa-mir124-2) in cervical samples has shown valuable as triage test. This multicenter study from 6 collaborating European laboratories and one reference laboratory was set out to determine the intra- and inter-laboratory agreement of FAM19A4/mir124-2 DNA methylation analysis utilizing the QIAsure Methylation Test.
METHODS: Agreement analysis for the QIAsure Methylation Test was assessed on high-risk HPV-positive cervical specimens (n = 1680) both at the level of the assay and at the full workflow, including bisulfite conversion.
RESULTS: Intra- and inter-laboratory assay agreement were 91.4% (534/584; 95% CI 88.9-93.5; κ = 0.82) and 92.5% (369/399; 95% CI 90.0-94.7; κ = 0.83), respectively. The inter-laboratory workflow (bisulfite conversion and assay combined) agreement was 90.0% (627/697; 95% CI 87.5%-92.0%; κ = 0.76).
CONCLUSION: These data show that the QIAsure Methylation Test performs robust and reproducible in different laboratory contexts. These results support the use of the QIAsure Methylation Test for full molecular screening for cervical cancer, including primary HPV testing and triage testing by methylation analysis.

INTRODUCTION

HPV‐based cervical screening has a high sensitivity and lower specificity for cervical cancer and cervical intraepithelial neoplasia (CIN) compared to cytology. To improve specificity, triage testing is necessary. FAM19A4 and hsa‐mir124‐2 methylation analysis in exfoliated cervical cell specimens has shown to be a sensitive test for the detection of women with cervical cancer and high‐grade CIN in need of treatment.1, 2, 3, 4, 5, 6 The QIAsure Methylation Test is an in vitro diagnostic assay comprising a multiplex quantitative methylation‐specific PCR (qMSP) that measures the hypermethylation of these two disease‐related genes (FAM19A4 and hsa‐mir124‐2) and the reference gene ACTB. The test can be used to triage women with a positive HPV test, or those with atypical squamous cells of undetermined significance (ASC‐US) on cytology, to determine the need for referral to colposcopy or other follow‐up procedures.

The QIAsure Methylation Test has reported a good overall clinical performance for CIN3 and cancer in high‐risk (hr) HPV‐positive clinician‐taken samples (sensitivity: 67% for CIN3 and 100% for cancer) and self‐collected samples (sensitivity: 66% for CIN3 and 100% for cancer).1, 2, 7 A key aspect is the efficient detection of cervical carcinomas and advanced CIN lesions, that is, CIN2/3 lesions associated with a duration of the preceding hrHPV infection of >5 years, which have increased methylation levels and many chromosomal aberrations (“cancer‐like” (epi)genetic profile), and have therefore been considered to have an expected high short‐term risk of progression to cancer.3, 5, 8 A negative QIAsure Methylation Test, on the other hand, indicates a low cervical cancer risk over the subsequent 14 years in hrHPV‐positive women.9

From a laboratory perspective, established and consistent clinical performance must be supported by a good reproducibility of the diagnostic assay.10 This is pivotal for quality assurance of the diagnostic workflow using the assay in a cervical screening setting. For this purpose, one of the objectives of the Valid‐screen project was to perform a systematic evaluation of agreement and reproducibility related to the QIAsure Methylation Test. The Valid‐screen project is a multicenter, international study designed to validate the clinical performance of the QIAsure Methylation Test (Horizon 2020 Programme, ID 666800). For reproducibility testing, a panel of cervical samples derived from different cohorts collected in four types of sampling media were tested at six different laboratories across Europe and retested in the reference laboratory to determine inter‐laboratory agreement. One cohort was tested and retested in the reference laboratory to determine intra‐laboratory agreement. Here, we report on the intra‐ and inter‐laboratory agreement of the QIAsure Methylation Test and workflow in the Valid‐screen project.

MATERIAL AND METHODS

Study setting

Seven European Union–based national centers and laboratories participated in the Valid‐screen project (SME Instrument in the Horizon 2020 Work Programme of the European Commission (666800)): University of Ljubljana, Slovenia; Catalan institute of Oncology, Spain; Klinikum Wolfsburg, Germany; Hvidovre Hospital, Copenhagen University Hospital, Denmark; University of Edinburgh, UK; DDL Diagnostic Laboratory, the Netherlands; and Amsterdam UMC, Vrije Universiteit Amsterdam, Pathology, Cancer Center Amsterdam, the Netherlands. The latter laboratory served as the reference laboratory.

Study design

Agreement analysis for the QIAsure Methylation Test was performed on hrHPV‐positive cervical specimens. Three independent agreement measures were completed.

Intra‐laboratory agreement of the QIAsure Methylation Test was determined at the reference laboratory by testing bisulfite‐converted DNA from 584 cervical specimens originating from the reference laboratory with subsequent repeat testing of the bisulfite‐converted DNA.

Inter‐laboratory agreement was determined on 399 cervical specimens obtained from one of the external participating centers, and bisulfite‐converted DNA was shipped and subsequently tested in the reference laboratory.

Additionally, given that the bisulfite conversion prior to the methylation testing may influence the assay outcome, the full laboratory workflow was analyzed for inter‐laboratory agreement. For this, each participating laboratory used DNA from cervical specimens from their local study cohort (for numbers, see Table 1) and performed the full workflow. After completion of the testing, DNA of the analyzed samples was sent to the reference laboratory for retesting which included the bisulfite conversion step.

Table 1

Laboratory details on specimen numbers and handling

Laboratory	Collection medium	DNA extraction method	DNA concentration measurement method	HPV test	Agreement analysis	Number of samples (N=)
A	PreservCyt, STM	Biorobot EZ1 or QIAamp Mini Kit	Qubit	RealTime High Risk HPV Assay	Inter‐laboratory assay agreement (test situation 2)	399
					Inter‐laboratory workflow agreement (test situation 3)	99
B	SurePath	MagNA Pure LC96	Qubit	Onclarity HPV Test and CLART2	Inter‐laboratory workflow agreement (test situation 3)	104
C	PreservCyt	QIAamp DNA Mini Kit	Qubit	Hybrid Capture 2	Inter‐laboratory workflow agreement (test situation 3)	159
D	PreservCyt	DNA extractor Maxwell	Qubit or NanoDrop	Hybrid Capture 2	Inter‐laboratory workflow agreement (test situation 3)	140
E	PreservCyt	QIAamp DNA Mini Kit on QIAcube	Qubit	Cobas HPV	Inter‐laboratory workflow agreement (test situation 3)	96
F	PreservCyt	MagNA Pure 96	RNaseP qPCR	Hybrid Capture 2	Inter‐laboratory workflow agreement (test situation 3)	99
Ref	PreservCyt, UCM, or PBS	NucleoMag 96 kit on Hamilton Star platform	Qubit	GP5+/6 + PCR‐EIA	Intra‐laboratory assay agreement (test situation 1)	584
Total						1680

For precision testing, a control sample (QSC1) was incorporated into each test run in all the laboratories, including the bisulfite conversion.

All participating partner laboratories did not have previous experience with qMSP but underwent training on the QIAsure Methylation Test system prior to the start of study.

Specimens

In total, 1680 hrHPV‐positive cervical specimens were analyzed with the QIAsure Methylation Test. The cervical specimens originated from local cohorts organized by the seven different European centers and laboratories and collected in concordance with individual national or regional requirements, due process of governance, and local ethical guidelines. Details about specimen collection medium, handling, numbers per institute, and study setting are outlined in Table 1. Specimens with sufficient leftover material were randomly selected from local cohorts. It was verified within each test situation that the QIAsure Methylation Test positivity rate was comparable to the known positivity rate in an HPV‐positive screening cohort,9 that is, test situation 1:39%; test situation 2:30%; and test situation 3:31%.

The QSC1 sample for precision testing was designed to be QIAsure Methylation Test positive and consists of female genomic DNA (Promega) with 0.5% genomic DNA from the hypermethylation‐positive cervical squamous carcinoma cell line SiHa (ATCC® HTB‐35™). A large batch of QSC1 sample was prepared and tested for performance in the QIAsure Methylation Test in quadruplicate (ie, mean and standard deviation for ΔΔCt values for FAM19A4 of 8.36 and 0.43 and for mir124‐2 of 7.73 and 0.33, respectively). The remaining batch of QSC1 sample was aliquoted in an amount sufficient for one bisulfite conversion and frozen, to ensure that each analysis started with exactly the same DNA input with no difference in freeze‐thawing cycles.

Histology data

Histology data were obtained from the local registries. Histological examination was done locally, and specimens were classified as normal (CIN0), CIN1, CIN2, CIN3, or invasive cancer, according to international criteria.11 Of the 399 samples included in inter‐laboratory assay agreement analysis (ie, test situation 2), all had histology data available. Of the 697 samples used for inter‐laboratory workflow agreement analysis (ie, test situation 3), histology data were available for 373 specimens.

Bisulfite conversion

Bisulfite conversion was performed with the EZ DNA Methylation kit (Zymo Research) according to the manufacturer's specifications. Standard DNA input for bisulfite conversion was 250 ng. Elution was done with 12.5 µL M‐elution buffer yielding 20 ng/µL bisulfite‐converted DNA. For samples with insufficient DNA yield to accomplish an input of 250 ng (16% of the samples; ranging from 0% to 46% per laboratory), a minimal input of 100 ng was used.

QIAsure methylation test

The QIAsure Methylation Test was performed according to the manufacturer's instructions. The QIAsure Methylation Test is designed and manufactured by Self‐screen BV (Amsterdam, the Netherlands), and under an exclusive license distributed by QIAGEN (Hilden, Germany). Sample input in the assay is 2.5 µL bisulfite‐converted DNA. The assay was performed on a Rotor‐Gene Q MDx 5plex HRM instrument. AssayManager software (QIAGEN) controls amplification as well as data analysis and reporting using a fixed assay profile. The AssayManager software calculates ΔΔCt values for both FAM19A4 and hsa‐mir124‐2. A sample is considered “Hypermethylation‐positive” when the ΔΔCt for at least one of the targets is below its cutoff, and “Hypermethylation‐negative” when both targets are above their cutoff. A sample is considered invalid when the housekeeping gene (ACTB) Ct value is above its cutoff.

Statistics

For all settings, test results were blinded and concordance analysis was not performed until all testing was completed. Intra‐ and inter‐laboratory percent of agreement, 95% confidence bounds, and Cohen kappa scores were determined for samples with valid test results from both partner laboratory and reference laboratory. Interpretation of the kappa values was as follows: <0.20: poor; 0.21‐0.40: fair; 0.41‐0.60: moderate; 0.61‐0.80: good; and 0.81‐100: excellent agreement. For the QSC1 sample, the mean and the standard deviation for the ΔΔCt values of the two methylation markers were calculated for each laboratory.

RESULTS

Intra‐laboratory agreement

DNA from a total of 584 cervical specimens was bisulfite‐converted in the reference laboratory and analyzed twice with the QIAsure Methylation Test (ie, test situation 1). The average time between initial testing and repeat testing was 165 days (range 6‐267). The intra‐laboratory agreement of the QIAsure Methylation Test was 91.4% (534/584; 95% CI 88.9‐93.5) with a κ = 0.82, corresponding with an excellent agreement (Table 2).

Table 2

Intra‐laboratory assay agreement

Reference laboratory result 2	Reference laboratory result 1			Agreement (95% CI)	κ
	Hypermethylation‐negative	Hypermethylation‐positive	Total
Hypermethylation‐negative	329	23	352	91.4% (88.9‐93.5)	0.82
Hypermethylation‐positive	27	205	232
Total	356	228	584

Inter‐laboratory agreement

DNA from a total of 399 cervical specimens was bisulfite‐converted in laboratory A and analyzed with the QIAsure Methylation Test (ie, test situation 2). Converted DNA was sent to the reference laboratory, where the QIAsure Methylation Test was repeated. The average time between initial testing and repeat testing was 26 days (range 6‐51). The inter‐laboratory agreement of the QIAsure Methylation Test was 92.5% (369/399; 95% CI 90.0‐94.7) with a κ = 0.83, corresponding with an excellent agreement (Table 3).

Table 3

Inter‐laboratory assay agreement

Test laboratory A	Reference laboratory			Agreement (95% CI)	κ
	Hypermethylation‐negative	Hypermethylation‐positive	Total
Hypermethylation‐negative	251	5	256	92.5% (90.0‐94.7)	0.83
Hypermethylation‐positive	25	118	143
Total	276	123	399

In addition, 697 cervical specimens were analyzed in the six participating laboratories and unconverted DNA was shipped to the reference laboratory for bisulfite conversion and QIAsure Methylation Test (ie, test situation 3). The overall inter‐laboratory workflow agreement was 90.0% (627/697; 95% CI 87.5%‐92.0%) with a κ = 0.76, corresponding with a good agreement (Table 4).

Table 4

Inter‐laboratory workflowa agreement

Test laboratoryb	Reference laboratory			Agreement (95% CI)	κ
	Hypermethylation‐negative	Hypermethylation‐positive	Total
Hypermethylation‐negative	456	45	501	90.0% (87.5‐92.0)	0.76
Hypermethylation‐positive	25	171	196
Total	481	216	697

Result following bisulfite conversion and QIAsure Methylation Test

Pooled data from the six European laboratories

Intra‐ and inter‐laboratory precision of the control sample

A total of 220 measurements were available for the control sample QSC1, ranging from 12 to 97 measurements per laboratory. Overall, the mean and the standard deviation for the ΔΔCt values were 8.46 and 0.49 for FAM19A4, respectively, and 7.90 and 0.80 for hsa‐mir124‐2, respectively (Table 5).

Table 5

Inter‐laboratory precision. Reported are the number of measurements, the mean, and standard deviations for the control sample QSC1

Laboratory	N=	ΔΔCt FAM19A4		ΔΔCt hsa‐mir124‐2
		Mean	Std dev	Mean	Std dev
Reference laboratory	97	8.51	0.48	7.85	0.61
Laboratory A	43	8.63	0.41	8.31	0.65
Laboratory B	13	8.26	0.47	7.34	0.59
Laboratory C	12	8.07	0.47	7.66	0.83
Laboratory D	16	8.38	0.54	8.44	1.45
Laboratory E	19	8.72	0.35	8.36	0.35
Laboratory F	20	7.97	0.20	6.91	0.31
Total	220	8.46	0.49	7.90	0.80

Discrepancy analysis

For intra‐laboratory assay agreement, 50 out of 584 samples (8.5%) had a discrepant test result. For inter‐laboratory assay agreement, 30 out of 399 samples (7.5%) had a discrepant test result. Of these 80 samples, 77 (96%) displayed ΔΔCt values close to the cutoff of the assay on FAM19A4 and/or hsa‐mir124‐2 (ie, within one ΔΔCt from cutoff, either in the reference or in the test laboratory).

For inter‐laboratory workflow agreement, 68 out of 70 discrepant samples (97%) displayed ΔΔCt values close to the cutoff of the assay. For the purpose of root cause analysis a subset of these samples (n = 27) from which sufficient material was left, the analysis was repeated. This resulted in 15 concordant results (56%), indicating that samples with a value close to the cutoff generate less reproducible results.

Histology stratification

The mean ΔΔCt values for the two markers were calculated for the different histology grades (whenever histology data were available). Looking at the full workflow, the methylation levels increased with disease severity, resulting in the lowest mean ΔΔCt values for cancer: 7.2 for FAM19A4 and 6.7 for hsa‐mir124‐2; and the highest mean ΔΔCt values for samples with no (evidence of) disease: 13.0 for FAM19A4 and 10.4 for hsa‐mir124‐2 (Table 6). Samples with CIN1 or CIN2 were in the middle of the spectrum and closer to the assay cutoff. Cancer cases, CIN3, and normal samples were at the outer ends of the spectrum and showed highest agreement values of 100%, 95%, and 92%, respectively (Table 7). For sole QIAsure Methylation Test data, a similar trend was observed, although less pronounced (Table 7).

Table 6

Mean ΔΔCt values stratified for histology

Histology	Mean ΔΔCt FAM19A4	Mean ΔΔCt hsa‐mir124‐2
Cancer (n = 3)	7.2	6.7
CIN3 (n = 41)	9.8	8.0
CIN2 (n = 51)	12.2	9.2
CIN1 (n = 53)	12.6	10.1
Normal or no evidence of CIN (n = 225)	13.0	10.4

Table 7

Inter‐laboratory assay and workflow agreement stratified for histology

Histology	Full workflow			QIAsure methylation test
	Total	Number discrepant results	Agreement	Total	Number discrepant results	Agreement
Cancer	3	0	100%	3	0	100%
CIN3	41	2	95%	75	3	96%
CIN2	51	7	86%	75	8	89%
CIN1	53	9	83%	85	7	92%
Normal or no evidence of CIN	225	19	92%	161	12	93%
Total	373	37	90%	399	30	92%

DISCUSSION

We evaluated the intra‐ and inter‐laboratory agreement of FAM19A4/mir124‐2 DNA methylation analysis utilizing the QIAsure Methylation Test, both at the level of the assay and at the full workflow (including bisulfite conversion), supported through a collaboration across six different European test laboratories and a reference laboratory. To the best of our knowledge, this is the first systematic and international assessment of inter‐laboratory reproducibility of a CE‐IVD qMSP assay for DNA hypermethylation detection. Good‐to‐excellent intra‐ and inter‐laboratory agreement of the assay and the full workflow were observed (ie, kappa value range from 0.76 to 0.83).

The present study confirms that the QIAsure Methylation Test is a reproducible test, which is a key parameter when considering its application in cervical screening. When applying the minimal intra‐ and inter‐laboratory reproducibility criteria from the guidelines for primary HPV DNA test requirements,12 the lower confidence bound for agreement and kappa value in this study are above the threshold of 87% and 0.5, respectively, for intra‐laboratory assay analysis, inter‐laboratory assay analysis, and inter‐laboratory workflow analysis. Although these minimal intra‐ and inter‐laboratory reproducibility thresholds have been set for HPV DNA tests,12 and may not directly be applied to other molecular markers (such as methylation markers), the QIAsure Methylation Test complies with the criteria, supporting robustness of the assay.

After stratification for histology, agreement was the highest among women with cancer (100%, 3/3), followed by CIN3 (95%, 39/41; and 96%, 72/75) and normal histology (92%, 206/225; and 93%, 149/161). This represents true‐positive and true‐negative sample groups characterized by methylation levels most distanced from the assay cutoff. Discordance in test outcome was predominantly associated with methylation levels around the assay cutoff, which is known to be prone to variation, and this observation is consistent with other studies using diagnostic assays with binary output.13, 14

Of note, the participating test laboratories did not have previous experience with methylation testing and many practical variables were included (Table 1), that is, seven different laboratories from six different countries, four types of collection media, six different DNA extraction methods, and assay comparison with or without the bisulfite conversion. The high agreement values therefore indicate that the QIAsure Methylation Test and workflow are resilient to the vagaries of different laboratory and service contexts. A limitation of the study may be that its design does not allow for clinical performance evaluation of the QIAsure Methylation Test. External clinical validation of the QIAsure Methylation Test is the topic of a large ongoing study. Because the QIAsure Methylation Test is an innovative assay, there are no published reproducibility data of comparable assays. As a consequence, results cannot be compared to similar assays nor be related to standard guidelines. When comparing the agreement of the QIAsure Methylation Test to other HPV triage assays, like cytology (κ = 0.46)15 or p16/Ki67 dual‐stain cytology (κ = 0.71),16 a higher agreement is observed for the QIAsure Methylation Test.

In conclusion, the QIAsure Methylation Test is a highly reproducible assay and may be used to discern hrHPV‐positive women with clinically relevant cervical disease. These results support the possibility and feasibility of a full molecular screening for cervical cancer, including primary HPV testing and triage by methylation analysis.

ETHICS STATEMENT

The work with human derived material is conducted under national and international rules and legislation, as well as European standards of research ethics, as it is expressed in the applicable legislation/regulations (the Declaration of Helsinki (informed consent for participation of human subjects in medical and scientific research)) and guidelines for Good Clinical Practice. The study was approved by the local ethics committees.

CONFLICT OF INTEREST

AF, AH, and SD are employed by Self‐screen BV AO was supported by the COHEAHR Network (Grant No. 603019), which was funded by the 7th Framework Programme of DG Research and Innovation, and received reimbursement of travel expenses for attending conferences and honoraria for speaking from Qiagen. EA has no personal conflict, but EA's institution has received research project funding and/or associated consumables to deliver research from the following entities in the last 3 years: Hologic, Cepheid, Qiagen, Euroimmun, LifeRiver, Genomica, and Gene‐First. WQ is shareholder of LBP, subcontractor of Self‐screen as the producer of the QIAsure kit. KC has no personal conflict, but KC's institution has received research project funding and/or associated consumables to deliver research from the following entities in the last 3 years: Hologic, Cepheid, Qiagen, Euroimmun, LifeRiver, Genomica, and Gene‐First. JB has in the past served as paid advisor to Roche and Genomica and unpaid advisor to BD Diagnostics and received honoraria from Hologic/Gen‐Probe, Roche, Qiagen, Genomica, and BD Diagnostics for lectures. JB is subcontractor lead on Hvidovre Hospital's element of project Valid‐screen, EU666‐800. DH has been on the speaker´s bureau of Qiagen, serves occasionally on the scientific advisory board of Pfizer and Bristol‐Myers Squibb, and has minority stake in Self‐screen BV, a spin‐off company of VU University Medical Center (currently known as Amsterdam UMC, Vrije Universiteit Amsterdam). Self‐screen BV holds patents related to the work and has developed and manufactured the methylation assay, which is licensed to Qiagen (QIAsure® Methylation Test). All other authors declare that they have no conflicts of interest.

AUTHORS’ CONTRIBUTION

Project management was done by AF and DH. AO, EA, HP, MTH, and SD performed reproducibility experiments. Data collection was performed by AO, EA, BR, HP, MTH, WQ, KUP, MP, KC, JB, SdS, and MB. SD trained and instructed the participating laboratories. AF and AH performed the data analysis. AF, AH, and DH drafted the manuscript. All other authors reviewed and approved the final manuscript.