External quality assessment for EGFR mutations in Italy: improvements in performances over the time.

External quality assessment (EQA) schemes are essential procedures to assess the quality level of laboratories performing molecular testing of the epidermal growth factor receptor (EGFR) gene in non-small cell lung cancer. The Italian Association of Medical Oncology (AIOM) and the Italian Society of Pathology (SIAPEC-IAP) organise EGFR EQA programmes to ensure that the Italian laboratories achieve the quality standard levels required. Comparing the 2011, 2013 and 2015 EGFR EQA schemes, it was possible to observe improvements in the methodologies used and the outcomes. The use of direct sequencing was reduced from 78.7% in 2011 to only 14.1% in 2015, whereas the use of pyrosequencing and real-time PCR increased. The number of rounds in which centres using direct sequencing failed was significantly higher than the number of rounds that failed using other methods, both when analysing each single scheme and when combining the three EQAs together. In 2011 and 2013, about 29% of the participants failed the first phase of the programmes, compared with the 13% of centres failing in 2015, suggesting that the switch to more sensitive and robust methods could allow to increase the percentage of good performers. Although the molecular analyses are performed with good quality in Italy, the continuous education carried out by AIOM and SIAPEC-IAP remains a fundamental tool to maintain this quality level.

What is already known about this subject?

The importance of quality assessment schemes in molecular diagnostic laboratories is widely acknowledged. Also, the efficacy of continuous education through these schemes and trainings has resulted in an increase in the quality levels reached by Italian laboratories.

What does this study add?

Since the number of biomarkers and methods at laboratories’ disposal continues to increase without further indication on which method should be chosen to perform the tests, the comprehensive evaluation of the results obtained during all the EGFR EQA schemes organised by AIOM and SIAPEC could provide information on the favourable evolution of EGFR testing in Italy.

How might this impact on clinical practice?

These data will help laboratories to improve their diagnostic practice in EGFR testing, and they might also turn useful to update guidelines and educational programmes and to offer the patients high-quality biomarker testing, essential for the identification of the optimal treatment.

Introduction

In the management of advanced patients with non-small cell lung cancer (NSCLC), evaluation of the mutational status of the epidermal growth factor receptor (EGFR) gene has become mandatory for the choice of the optimal therapy because it can predict whether a patient is likely to benefit from an EGFR-targeted therapy. Indeed, it has been largely demonstrated that a prolonged progression-free survival can be reached in patients carrying EGFR activating mutations receiving first-line EGFR tyrosine-kinase inhibitors (TKIs), when compared with conventional treatment with chemotherapy.1–8

Activating mutations are mainly identified in exons 18 through 21 of the tyrosine kinase domain of the EGFR gene. Approximately 90% of all EGFR mutations are detected in exon 19 (small in-frame deletions) and exon 21 (p.L858R) and are usually associated with specific clinical and pathological characteristics: they are generally found in never-smoker female patients with adenocarcinomas.9 10

The patients who respond to EGFR TKIs develop resistance at a certain point, mainly due to the acquisition of the EGFR p.T790M mutation.11 In this scenario, the detection of all these mutations by means of accurate, reproducible and rapid tests plays a key role in patient management. Internal and external quality control schemes are essential procedures for any laboratory aiming to offer a quality service. In particular, external quality assessment (EQA) allows to obtain an external, thus objective, evaluation of laboratory performances.

Since 2010, the permanent joint board created by the Italian Association of Medical Oncology (AIOM) and the Italian Society of Pathology (SIAPEC-IAP) organises EQA programmes for the evaluation of the quality level reached by the Italian laboratories in biomarker testing for solid tumours.

In this article, we will discuss the data obtained during the EGFR EQA schemes organised in 2011, 2013 and 2015 to provide a snapshot of the changes in both methodologies and performances, which partially reflect the improvements in the molecular testing systems reached during the past years.

Materials and methods

The EGFR EQAs were organised by the AIOM and SIAPEC-IAP working group in the years 2011, 2013 and 2015. The aim of the schemes was the evaluation of the analytical phases performed by the participating centres.

Formalin-fixed, paraffin-embedded specimens derived from NSCLC were collected at three referral surgical pathology departments. In each scheme, 10 primary tumour samples were chosen with both an adequate content of tumour cells (≥50%) and a 100% concordance among three independent referral centres, which were selected within the steering committee of the EQAs based on their expertise in molecular pathology and in EQA organisation and on their track of scientific publications. Three out of the 10 samples were prepared to simulate small biopsies.12

In each EQA, Italian laboratories that performed EGFR testing were invited to participate and to register through the dedicated website. The participation in the EQAs was on a volunteer basis. One 10-µm thick slide for each of the 10 samples was sent to the laboratories. Every 20 slides, the samples were reanalysed in order to ensure that the mutant fraction was maintained and that the samples without pathogenic alterations in exons 18–21 of the EGFR gene were constantly negative during the slicing procedures. In 2011, direct sequencing for exons 18–21, fragment analysis of exon 19 deletions, an allelic discrimination-based real-time approach for the p.L858R detection and the Therascreen EGFR RGQ kit were used to analyse the samples. In 2013, next-generation sequencing (NGS) was also introduced as additional confirmatory method to further validate the material used. Starting from this year, only the samples whose sections showed an allelic frequency of mutant constantly ≥20% by NGS were chosen for the programmes.

In each scheme, the laboratories were asked to test for the possible presence of sensitising mutations in exons 18, 19, 20 and 21 of the EGFR gene and to submit the results within 3 weeks after the shipment date.

The participants provided information on the methods chosen to perform the analyses and were asked to submit a molecular pathology report, used only for a qualitative evaluation. Marking of the results was performed using previously established criteria.13 14 However, the AIOM-SIAPEC working group established that both a score ≥18 and the absence of major genotyping errors, consisting of false-positive or false-negative results, were mandatory to pass the schemes because a major genotyping error can cause detrimental effects.12 15

In the first EGFR quality control scheme organised in 2011, 47 laboratories participated, while in 2013, the number of the centres increased to 86. These schemes were organised in two phases: the centres failing the analyses on the first round of 10 samples had the chance to test another round of 10 samples. In this article, only the results obtained during the first phase were taken into account for the statistical comparison of the technologies used with the performances. In 2015, 92 centres took part in the programme, which was composed of a single phase.

Overall, 96 laboratories took part in the three schemes, for a total of 225 rounds of testing. We compared the frequency of the rounds resulting in poor performance according to the specific method. Of the 225 rounds, we did not consider the rounds in which more than a method was used (3 out of 225 rounds) or for which the method(s) was unknown (4 out of 225 rounds). In total, 218 out of 225 rounds were considered for the analyses.

Results

During the three schemes, samples with similar characteristics were shipped to the participating centres. In general, at least four samples had no pathogenic sequence variant detectable in the hotspot regions of EGFR, to evaluate the specificity of the analyses used; the remaining samples were representative of the activating mutations normally detected in patients with NSCLC. In 2015, sections carrying the p.T790M mutation were also included in the set of samples sent, although its detection was considered optional. The distribution of the mutations among the samples is described in table 1.

Table 1

Mutational status of the samples selected for the three EGFR EQA programmes

	Samples shipped in 2011	Samples shipped in 2013	Samples shipped in 2015
1	WT	p.Glu746_Ala750del	WT
2	p.Glu746_Ala750del	p.Leu858Arg	WT
3	WT	p.Leu747_Pro753delinsSer	WT
4	p.Leu858Arg	p.Glu746_Ala750del	WT
5	WT	p.Glu746_Ser752delinsVal	p.Leu858Arg; p.Thr790Met
6	WT	WT	p.Glu746_Ala750del
7	WT	WT	p.Glu746_Ala750del
8	WT	WT	p.Glu746_Ala750del
9	WT	WT	p.Leu858Arg
10	p.Glu746_Ala750del	WT	p.Leu858Arg; p.Thr790Met

WT (wild-type) indicates that no pathogenic sequence variants were detected in exons 18–21 of the EGFR gene.

The three EGFR quality control programmes revealed that a wide range of methodologies were used to perform the molecular analyses. In 2011, direct sequencing was the most chosen technique (78.7%), whereas pyrosequencing and real-time PCR were used to a lesser extent (17% and 4.3%, respectively; figure 1A). In 2013, together with a doubling in the number of participants, which passed from 47 to 86, there was also an increase in the number of methods chosen to perform the tests (figure 1B). In particular, 40.5% of the centres chose direct sequencing, real-time PCR was used more than pyrosequencing (30.4% vs 20.3%) and for the first time, other methods such as MassArray were introduced. The last EGFR quality control scheme organised in 2015 showed a different trend: the most selected method was pyrosequencing (27.2%), followed by a real-time PCR commercial kit (Therascreen) and other real-time PCR approaches no further detailed (25% and 15.2%, respectively). The use of direct sequencing decreased to 14.1% of the centres (figure 1C).

Figure 1

Methods distribution among the three EQAs. (A) Percentage of the methods chosen in the first phase of 2011 EQA (participants: 47). (B) Percentage of the methods used in the first phase of 2013 EQA (participants: 86). (C) Percentage of the methods used in the 2015 EQA (participants: 92).

We gathered together the centres’ performances resulted from the first phase of the 2011 and 2013 schemes and those from the 2015 EQA, whose results are summarised in table 2, comparing them with the methods used based on the successes or failures obtained (Fisher’s exact test). In particular, direct sequencing was used in 82/218 rounds (37.6%) among the three schemes, whereas 136/218 rounds (62.4%) were performed with other techniques. A significant correlation was observed when comparing the performances of the centres on the basis of the methods chosen. The error rate for the direct sequencing was higher than for the other techniques, thus resulting in 33/82 (40.2%) of the rounds with poor performance using direct sequencing, compared with the 17/136 (12.5%) rounds resulting in poor performance when other methods were used (p<0.0001; figure 2A). In addition, we compared singularly the performances obtained during each scheme, to evaluate whether this difference was still observable. The correlation between the performances and the techniques used was still present in each of the scheme, confirming a worse outcome of direct sequencing compared with the other methods (figure 2B: 2011, p=0.022; figure 2C: 2013, p=0.033; figure 2D: 2015, p=0.012). In this respect, the errors made by the laboratories using direct sequencing were both false positives and false negatives. In particular, considering all the 218 rounds performed during the three schemes, 24 false-positive and 26 false-negative results were reported when direct sequencing was used, whereas the number of false positive and false negative was reduced to 9 and 12, respectively, when other methods were chosen. Moreover, in both the rounds conducted with direct sequencing and with other methods, three cases were reported as false positive in an exon and false negative in another one.

Table 2

Summary of the results of the first phase of 2011 and 2013 and of the single phase of 2015 NSCLC quality control schemes

NSCLC quality control scheme	First phase (passed %)
EGFR EQA 2011	70.2
EGFR EQA 2013	70.9
EGFR EQA 2015*	87.0

*One single phase.

Figure 2

Correlation between performances obtained by means of direct sequencing and other methods. (A) The methods performances in the three EQAs (p<0.0001). (B to D) The correlation between direct sequencing and other methods used in 2011 (p=0.022), 2013 (p=0.033) and 2015 (p=0.012), respectively.

Discussion

The implications of the results of molecular testing on the identification of the optimal treatment for a patient are widely known. Targeted therapy in oncology is an evolving field, and it represents a challenge for the continuous identification of relevant biomarkers that play a predictive role and that, for this reason, must be analysed. Considering the need for accurate and reproducible tests, EQA schemes have largely demonstrated their importance for the laboratories performing molecular pathology analyses, in order to guarantee good-quality results. AIOM and SIAPEC organise the quality control scheme for each tumour type (NSCLC, colon and melanoma) every other year. As stated in the Materials and methods section, the EGFR quality control schemes in NSCLC were organised in 2011, 2013 and 2015. Although we acknowledge that EQA should be offered every year, the two scientific societies are planning to organise a ‘panel EQA’ that will be organised yearly.

A variety of methods have been developed for routine molecular diagnostics to detect the different genomic alterations. For a long time, direct sequencing has represented the first choice for the majority of the laboratories, as observable in the first EGFR scheme (78.7%; figure 1A). The limits of this method have been largely described, in particular its low sensitivity, although its strong points are the flexibility and the possibility to describe the particular mutation detected. Sensitivity might be particularly relevant in EGFR testing in NSCLC. In fact, it has been demonstrated that the tissue biopsy from patients with NSCLC usually carries a low fraction of neoplastic cells, thus leading to dilution of the mutant allele within wild-type alleles.16 This phenomenon is also relevant for the mutations associated with acquired resistance such as the p.T790M, which might be heterogeneously expressed within the tumour.17 In this respect, the continuous educational activity conducted during the years by AIOM–SIAPEC allowed the implementation in routine diagnostics of methods displaying higher sensitivity, higher confidence and reproducibility. Indeed, in 2013, the use of direct sequencing was reduced from 78.7% to 40.5%, and in 2015, only 14.1% of the participants chose this method, which was surpassed by pyrosequencing and real-time PCR approaches.

Giving a closer look to the techniques commonly used during the EQA programmes, we observed that in 2011, pyrosequencing and real-time PCR were the only two other methods used. In the subsequent schemes, the use of commercial tests, in particular the Therascreen EGFR kit from Qiagen, has been introduced. The commercial methods have their advantages, as well as their challenges: they are standardised and highly sensitive in the detection of specific known alterations, but they lack flexibility and capacity to detect novel mutations in the regions of interest.

Although in this article the results of the first phase only have been used for statistical comparison, the EQAs in 2011 and 2013 included a second phase of testing that was offered to laboratories that failed the first phase. Interestingly, the presence of two phases allowed a discrete number of laboratories to rapidly improve their performances within the programmes. Eight out of 14 (57.1%) and 8 out of 25 (32%) centres that failed the first phase in the 2011 and 2013 EQAs, respectively, were able to reach a successful performance in the second phase. Because of this procedure and the systematic organisation of quality control schemes, the good-quality level of molecular testing reached could be maintained over time, as proven by the 2015 EQA. Indeed, despite the presence of a single phase, in this scheme, only 12 out of 92 participants failed (13%). The data presented in this article also suggest that such an improvement in performances might be related to the adoption of new techniques by the majority of the Italian centres performing EGFR testing.

The rapid development of precision medicine implies the need to quickly update the technologies available, to offer the optimal methodological approach to the patients. Here, we tried to compare the outcome of the centres choosing direct sequencing with the outcome of centres choosing other techniques, to evaluate whether a difference was observable (figure 2B–D). In each scheme, a significant correlation existed, suggesting that the other methods performed better than direct sequencing. Moreover, this correlation was confirmed when all the rounds were pooled together (p<0.0001; figure 2A). A significantly better outcome of pyrosequencing compared with direct sequencing has been already observed during the EQA organised by AIOM–SIAPEC for the molecular analysis of KRAS, NRAS and BRAF in colorectal carcinoma.15 Direct sequencing is a laboratory-developed technique that has shown a sensitivity between 10% and 30% in different studies.18–20 The sensitivity of direct sequencing is significantly affected by a number of variables, including the quality and the quantity of the input genomic DNA, the efficiency of the PCR, the sequencing platforms employed and the data analysis software. Moreover, the high number of false-positive results observed during the schemes highlights that issues related with sample contamination or with potential artefacts introduced by sequencing procedures might affect the analyses. In this respect, our findings suggest that the use of direct sequencing should be limited to skilled laboratories, should be able to develop highly standardised workflow and should be mainly used as preliminary screening or as a confirmation method.

In conclusion, the collected data confirm that the molecular analyses are performed with good-quality levels in Italy and that the continuous education and trainings carried out by AIOM and SIAPEC-IAP are fundamental tools to maintain the quality services offered. However, the chance to use methods more robust and with higher sensitivity could allow to further increase the percentage of centres, resulting good performers. The participation in external quality assurance programmes represents a possibility to monitor internal quality standards to overcome the possible analytical issues and to improve the quality level of the services offered to the patients.