A multi-centre pilot proficiency programme to assess the quality of molecular detection of respiratory viruses.

OBJECTIVES: To assess the quality of molecular detection of respiratory viruses in clinical diagnostic laboratories. STUDY
DESIGN: Respiratory virus proficiency panels were produced from diluted stocks of respiratory viruses provided and tested by four reference laboratories. The panels consisted of strong positive, positive, low positive and negative samples for influenza viruses A and B, respiratory syncytial virus, parainfluenza viruses 1 and 3, adenovirus serotypes 4 and 7, human rhinovirus serotypes 16, 72 and 90, human coronaviruses OC43 and 229E. The panels were sent to 17 participants; results and information on methodology was collected.
RESULTS: All laboratories returned results, of which five submitted complete data sets. So, for analysis all results were combined. Samples were correctly identified by participants in 93.75%, 76.75% and 47.03% for the high positive, positive and low positive samples, respectively. One false positive was reported for all data sets (1.1%). The overall score for all assays using different methodologies was 78.8%. Laboratory performance was not dependant on methodology as all in-house methodologies could achieve optimal results, but dependant on careful optimisation and procedures specific to the laboratory.
CONCLUSIONS: The first proficiency panel showed that in general all participants performed well. Although, it also highlights areas for improvement for all participants in order to generate robust results for use in clinical diagnostics.

Introduction

Detection of respiratory viruses in clinical samples is important for effective patient management and infection control. Many viruses are involved in respiratory infection and include: influenza viruses A and B, respiratory syncytial virus (RSV), parainfluenza viruses (PIV) 1–4, adenoviruses (ADV), human rhinoviruses (hRV), human coronaviruses (hCoV: OC43, 229E, NL-63, NL-Hong Kong and severe acute respiratory syndrome coronavirus (SARS-CoV)) and human metapneumoviruses (hMPV). Many of the infections are indistinguishable by clinical features alone and require rapid laboratory investigation for optimal patient management and infection control. Detection of respiratory viruses is becoming clinically important as the possibilities for antiviral treatment increase. Some of the viruses (hRVs and hCoVs) were initially considered as causing mild infections but are now more frequently found to be associated with severe infections (Ison et al., 2003, van Elden et al., 2004).

Viral culture is the gold standard for laboratory diagnosis of respiratory viruses. However, some viruses grow poorly in cell culture, and therefore routine diagnosis is sub-optimal as, for example, with hMPV (van den Hoogen et al., 2004). Furthermore, culture is relatively slow which affects the clinical value, and therefore alternatives are employed. Rapid antigen detection tests are available for influenza viruses A and B, PIV 1–3 and RSV and adenoviruses, which are widely used in routine laboratories. These have been shown to be less sensitive and less specific than culture, particularly in adult or elderly populations (Casiano-Colon et al., 2003, Effler et al., 2002, Storch, 2003). Amplification assays are now increasingly being used in the clinical laboratory and have proven to be more sensitive and specific than culture (Syrmis et al., 2004, van Elden et al., 2002, Weinberg et al., 2002, Weinberg et al., 2004). Newer real-time PCR formats also enable rapid test results (Templeton et al., 2004) while some conventional formats can detect a large number of respiratory viruses in a single test (Puppe et al., 2004, Coiras et al., 2004, Gruteke et al., 2004) PCR is often the most appropriate means to diagnose slow growing viruses (van Elden et al., 2004; van den Hoogen et al., 2004) due to the difficulties sometimes observed with culture.

Currently, some commercial assays are available for respiratory virus testing (Henrickson, 2004) but the majority of assays applied in clinical diagnostic laboratories have been developed in-house and standardisation is problematic for all formats. Laboratories performing respiratory molecular tests want to report accurate and reliable results regardless of the type of assay in use and one of the best ways to assess performance is to participate in proficiency programmes, enabling laboratories to evaluate their performance (Schirm et al., 2002, Schloss et al., 2003, Noordhoek et al., 2004, Verkooyen et al., 2003).

A pilot study for the evaluation of nucleic acid amplification technologies (NATs) for detection of respiratory viral pathogens was organised by Quality Control for Molecular Diagnostics (QCMD) (www.qcmd.org). The aim of the pilot programme was to perform a comparative study of the current NATs and protocols used for the molecular detection of a variety of respiratory viruses. The aim of this study was to focus on the sensitivity and specificity of the NAT protocols used, as well as the methodology employed for molecular testing.

Materials and methods

Sample production and in-process testing

A project group comprising QCMD and four European laboratories was established to co-ordinate and produce materials for the proficiency programme panel. Each laboratory was responsible for the production of bulk stocks for a number of defined viruses: Specialist Centre for Virology (SVC), Gartnavel General Hospital, Glasgow, UK (influenza viruses A and B, RSV A); University Medical Centre Utrecht (UMCU), The Netherlands (hRV 16, 72 and 90); Erasmus Medical Center (EMC), Rotterdam, The Netherlands (229E and OC43, hMPV); Leiden University Medical Center (LUMC), Leiden The Netherlands (PIV 1 and 3, ADV 4 and 7).

Preparation of bulk stocks

At SVC stocks of influenza A virus (A/New Caledonia/20/99 (H1N1)) and influenza B virus (B/Victoria/504/00) were prepared on MDCK cells in Medium 199. A RSV A clinical isolate (03.413667) was cultured on HEp-2 cells in MEM (+2%, v/v, FCS). Viral nucleic acid extraction was carried out using the BioRobot Blood DNA kit (Qiagen, Crawley, UK). Real-time PCR with TaqMan probes was used to amplify the influenza virus A NS gene, influenza virus B matrix gene and the RSV nucleocapsid gene on the i-Cycler IQ real-time detection system (Biorad, Hemel Hempstead, UK).

Clinical isolates of ADV 4 and 7 were cultured on A549 cells in MEM (+10%, v/v, FCS) and the subtype was confirmed by serotyping and sequence analysis. Clinical isolates of PIV 1 and 3 were cultured on LLC-MK2 cells and the subtype was confirmed by monoclonal antibody staining and sequence analysis. Virus nucleic acid extraction was carried out with QIAamp Blood DNA mini spin protocol for ADV and the QIAamp viral RNA Mini spin protocol for PIV (Qiagen, Hilden, Germany). Real-time PCR with molecular beacons were used to amplify the ADV hexon gene and the PIV haemagluttinin-neuramididase gene on the i-Cycler IQ real-time detection system (Biorad, Veenendaal, The Netherlands).

ATCC strains of 229E and OC43 and the hMPV isolate 00-1 (van den Hoogen et al., 2001) were cultured on Vero cells in MEM (+10%, v/v, FCS). Virus nucleic acid extraction was carried out using the MagNAPure LC total nucleic acid kit (Roche Applied Science, Almere, The Netherlands). Real-time PCR was performed with TaqMan probes to amplify and detect the nucleocapsid gene of OC43 and 229E and the nucleoprotein of hMPV on the ABI Prism 7700 sequence detection system (Applied Biosystems, Nieuwerkerk aan de IJssel, The Netherlands).

The human rhinoviruses (hRV) were donated by Dr. Tapani Hovi (KTL, Helsinki, Finland) and were originally obtained from ATCC, hRV type 16), RIVM (Bilthoven, The Netherlands, hRV type 72) and Janssen Pharmaceutica (Beerse, Belgium, hRV type 90). They were grown on Hela Ohio cells in roller tube cultures at 33 °C, and characterised by acid-lability testing, serotyping and sequence analysis. For production of stocks for the pilot proficiency panel, viruses were grown in human diploid fibroblast cells. Virus nucleic acid extraction was carried out using the MagNAPure LC total nucleic acid kit and real-time PCR was performed on the ABI Prism 7700 sequence detection system with TaqMan probes to amplify the 5′ non-coding region (NCR) of rhinoviruses.

Preparation of panel stocks

Each laboratory prepared a small-scale dilution series from their bulk stocks and determined the lower limit of detection (LLD) for each virus stock, based on real-time cycle threshold (C t) values (Table 1 ). This process was repeated by a second laboratory. Thereafter, the project group selected three dilutions of each virus for inclusion in the proficiency panel. Bulk dilution stocks were then prepared in virus transport medium (VTM) except for influenza virus B, which was prepared in Medium 199 (+10%, v/v, FCS). The bulk dilution stocks were repeat tested using the same methodology (data not shown), dispensed into 1 ml aliquots and shipped on dry ice to QCMD for coding, packaging and distribution to participants.

Table 1

Panel composition and results of pre-distribution testing at reference laboratories

Virus	Strong positive (A)		Positive (B)		Low positive (C)
	Dilution	Ct	Dilution	Ct	Dilution	Ct
Influenza virus Aa	10E−3	27	10E−5	34.5	10E−6	37.9
Influenza virus Ba	10E−3	25.3	10E−5	33.4	10E−6	37.3
Parainfluenza virus 1b	10E−3	35.8	10E−4	38.9	10E−5	44.9
Parainfluenza virus 3b	10E−3	28.1	10E−4	33.5	10E−5	35.8
Respiratory syncytial virusa	10E−3	27.4	10E−5	35.7	10E−6	38
Human metapneumovirusa	10E−1	24.5	10E−3	32.5	10E−4	37.6
Human rhinovirus 16a	10E−3	23.8	10E−5	31.1	10E−6	36.2
Human rhinovirus 72b	10E−3	24.2	10E−5	30.1	10E−6	34.7
Human rhinovirus 90a	10E−3	26.1	10E−5	32.8	10E−6	36.5
Adenovirus 4b	10E−2	31.8	10E−4	37.7	10E−5	41.8
Adenovirus 7b	10E−2	30.8	10E−4	38.2	10E−5	41.0
Human coronavirus 229Eb	10E−3	27.6	10E−5	34	10E−6	36.6
Human coronavirus OC43a	10E0	30.7	10E−1	33.5	10E−2	37.2

Ct: cycle threshold values.

Ct values shown are a mean of results from two laboratories.

Ct values shown are a mean of results from a single laboratory.

Distribution of the proficiency panels

The panel samples were randomised by QCMD, labelled, packed and distributed to participants on dry ice along with a panel receipt form, an instruction leaflet, a reporting form and a questionnaire for technical details. The panel was distributed in the form of six sub-panels: Panel A: containing influenza viruses A and B; Panel B: PIV 1 and 3; Panel C: RSV A and hMPV; Panel D: hRV 16, 72 and 90; Panel E: ADV 4 and 7; Panel F: hCoV 229E and OC43. Each sub-panel contained a three-member dilution series for each virus, a mixed sample (except for Panel C) and a negative sample.

External quality assessment process

Laboratories who had expressed an interest to QCMD in participating in a proficiency programme for molecular detection of respiratory viruses were asked to complete a questionnaire detailing technical aspects of the assays they had applied. Each participating laboratory was assigned a unique, confidential code. Participants were requested to return the result form and technical questionnaire, either by fax or by e-mail, before the closing date for return of the results, which was 6 weeks post-distribution. The majority of participants did submit results prior to the closing date, but as a number of participants requested additional time to complete testing, the date for return of results was extended to 10 weeks. Results and questionnaire data was collated and validated by QCMD and participants were re-contacted, if further information was required. Result codes letters were sent to all participants after all results had been received.

Scoring system

Results were scored: 2 points for a correct result, 0 for an incorrect and 1 for an equivocal result on a positive sample or for detection of only one sample in a mixed sample in the influenza viruses A and B, PIV 1 and 3 and Coronavirus 229E and OC43 panels. The mixed samples in the adenovirus and rhinovirus panels were not scored, as they required typing results and this was not requested.

Results

Identification of samples

The results of the molecular testing of the samples in the proficiency panel are shown in Table 2 . Three dilutions of each of the 13 different viruses were included in the QC panel. Five of the 17 (29%) participants provided complete data sets on all viruses and none of these five reported 100% correct. The high positive sample was on average correctly identified by 93.8% (range 72.7–100%) by the participants. The mean correct identification of the positive and low positive dilutions was 76.8% (range 53.3–100%) and 47.0% (range 0–85.7%), respectively. Only one false positive was reported from 94 different assays (1.1%).

Table 2

Overview of results for the respiratory virus proficiency panel

Virus	na	Strong positive	Positive	Low positive	Negative sample
Influenza A virus	17	17 (100)	11 (64.7)	10 (58.8)	17 (100)
Influenza B virus	17	17 (100)	17 (100)	11 (64.7)
Parainfluenza virus 1	15	13 (86.7)	9 (60)	2 (13.3)	15 (100)
Parainfluenza virus 3	15	14 (93.3)	8 (53.3)	4 (26.7)
Respiratory syncytial virus	16	14 (87.5)	11 (68.8)	9 (56.3)	16 (100)
Human metapneumovirus	14	14 (100)	13 (92.9)	12 (85.7)	14 (100)
Human rhinovirus 16	11	11 (100)	10 (90.9)	5 (45.5)	10 (90.9)
Human rhinovirus 72	11	8 (72.7)	8 (72.7)	4 (36.4)
Human rhinovirus 90	11	11 (100)	10 (90.9)	5 (45.5)
Adenovirus 4	14	13 (92.9)	12 (85.7)	10 (71.4)	13b (100)
Adenovirus 7	14	12 (85.7)	8 (57.1)	7 (50.0)
Human coronavirus 229E	8	8 (100)	6 (75)	4 (50)	8 (100)
Human coronavirus OC43	7	7 (100)	6 (85.7)	0b (0)	7 (100)
Overall mean (%)		93.75	76.75	47.03	98.7

Numbers indicate correct results and the numbers in parenthesis are percentages of participants with correct results.

n is the number of participating laboratories submitting results for a specific virus.

Results from 13 laboratories only, as one participant did not receive this sample.

In the mixed samples from the influenza virus, PIV and hCoV panels, seven participants reported only a single pathogen. In the influenza virus panel, one participant reported influenza A only and one participant reported influenza virus B only. In the PIV panel, one participant reported PIV 1 only and one participant reported PIV 3 only. In the hCoV panel, three participants reported 229E only.

Only 2 of 11 participants achieved a 100% score for the hRV panel. Three participants did not detect any of the hRV 72 samples, although they could detect the other rhinovirus types 16 and 90, even in the low positive. Three participants who detected hRV 72 as the low positive did not detect both hRV 16 and 90 in the low positive. Four of the 14 participants scored 100% in the adenovirus panel. All participants detected the strong positive adenovirus 4 (subtype E) sample and 10 out of the 14 participants detected the low positive sample. In contrast, only 12 out of the 14 participants detected the strong positive adenovirus 7 (subtype B) sample and 8 out of the 14 participants detected the low positive adenovirus 7 sample. In the hCoV panel, all participants detected the strong positive samples but none of the participants detected the low positive sample for OC43 in contrast, four out of eight participants detected the low positive for 229E.

Performance of laboratories

The performance of the 17 laboratories is shown in Table 3 . The mean overall score was 78.8% with a range from 46.4% to 96.9%. Ninety-four assays were used for all different targets of which 42 (44.5%), 27 (29%) and 25 (26.5%) were real-time PCR, nested PCR and other methodologies, respectively. Six participants used a combination of methodologies. The scores obtained for each of the individual panels are shown in Table 3. The hMPV panel was performed best with 12 out of 14 participants scoring 100% whereas the PIV panel had only one participant scoring 100% and 10 out of 15 scoring <70%. Six participants performed real-time PCR, as the only NAT technology, and their overall rank was 2, 4, 5, 7, 13 and 17. The only commercially available assay was the RV®Chip assay and this had an overall performance of 57.1%. The other assays were all in-house protocols and consist of many different parameters and technical aspects. The correlation between these aspects and a 100% score is given in Table 4 and this shows that a maximum score in the panel could be achieved with any combination of methodology and protocol.

Table 3

Participant scores for individual assays and overall laboratory performance

Panels	Scores (%)
Influenza A and B viruses	882	1001	1003	941	881	884	1001	693	881	882	751	1005	881	882	632	756	631
Parainfluenza viruses 1 and 3	1002	691	753	561	751	634	751	nt	631	882	nt	565	501	632	562	506	501
Respiratory syncytial virus	1002	1001	1003	1001	1001	752	751	1003	1002	502	nt	635	1001	502	752	506	251
Human metapneumovirus	1004	1001	1003	1001	1001	1004	751	1003	1004	1002	504	nt	1001	1002	1002	nt	nt
Human rhinovirus	nt	911	823	1001	731	1002	731	824	732	nt	nt	nt	451	642	752	nt	nt
Adenovirus	1001	nt	nt	881	1001	882	1001	nt	884	754	1002	885	881	502	632	506	381
Human coronaviruses	nt	881	nt	811	881	882	881	nt	nt	nt	nt	nt	811	752	ns7	nt	nt
Overall score (%)	96.9	89.5	88.6	87.2	86.3	86.3	84.3	83.3	81.4	81.3	80	78.6	73.5	68.6	68.6	57.1	46.4
Overall rank	1	2	3	4	5	5	7	8	9	10	11	12	13	14	14	16	17

Methodology used as indicated by integers in superscripts: 1real-time PCR; 2nested PCR; 3nucleic acid sequence amplification (NASBA); 4single PCR; 5multiplex reverse line blot (PCR amplification with detection by reverse line blot); 6RV®Chip assay (manufactured by BCS Biotech SPA, Cagliari, Italy); 7only assay for 229E performed, nt = not tested, ns = not scored.

Table 4

Correlation between technical aspects of assays used and the ability to obtain a maximum on each of the sub-panels

Method	Description	No. of users	No. of sub-panels tested (%)	No. of panels tested with maximum score (%)
Commercial	RV®Chip	1	4	0
In-house		16	90	34 (56.7)
	Published or submitted for publication		65 (72)	21 (32.3)
	Not described in publication		18 (20)	10 (55.6)
	No information provided		7 (8)	3 (42.9)
In-house assays with practical information		15	83	31 (37.3)
Extraction	Roche—MagNApure	3	17 (20.5)	9 (52.9)
	Roche—manual extraction kit	5	28 (34)	12 (35.7)
	Qiagen—extraction kit	4	22 (27)	4 (18.2)
	Biomeriuex—nuclisens	2	9 (11)	5 (55.6)
	Guardidine thiocynate/isopropanol	1	7 (8)	1 (14.3)
Input: extraction	1:>20	3	23 (28)	5 (21.7)
	1:20	3	4 (5)	1 (25)
	1:10	6	39 (47)	15 (38.5)
	1:<10	4	17 (20)	10 (59)
RT enzymea	Promega	3	17 (24)	8 (47)
	Invitrogen	3	15 (21)	7 (46.6)
	Qiagen	3	15 (21)	5 (33.3)
	Roche/ABI	2	8 (11)	1 (12.5)
	Not required (NASBA)	2	8 (11)	5 (62.5)
	No information	3	7 (10)	1 (14)
Amplification	Real-time PCR	5–8	35(42)	13 (37)
	Nested PCR	2–6	27 (33)	9 (33.3)
	Single PCR	1–4	9 (11)	3 (33.3)
	NASBA	1–2	8 (10)	5 (62.5)
	Reverse line blot	1	4 (5)	1 (25)
Platform	ABI prism	3	15 (18)	6 (40)
	Roche Lighcycler	2	4 (5)	1 (25)
	BioRad iCycler	1	9 (11)	4 (44)
	Nuclisens Easy-Q	2	9 (11)	5 (55.5)
	Corbett Rotorgene	1	6 (7)	2 (33.3)
	PCR Thermocycler	6	40 (48)	13 (32.5)

Manufacturer of RT enzyme, excludes adenovirus assays (four of which achieved maximum score in this assay).

The different genes targeted in the NAT assay are shown in Table 5 . Four out of the five NATs targeted to the matrix protein of influenza virus detected the weak positive sample, whereas only two of the five non-structural protein NATs detected the low positive sample. The NAT assays for the HN protein of PIV detected the weak positive samples, whereas the NAT targeting the nucleoprotein and fusion protein only detected the strong positive sample.

Table 5

Number of samples detected correctly as related to the target gene for amplification

	Gene targeta						Number with positive tests/total number
Influenza A virus	MP (5)	NSP (5)	NP (3)	HA (1)	NA (1)	NK (2)
Strong positive	5	5	3	1	1	2	17/17
Positive	4	3	2	0	1	1	11/17
Low positive	4	2	2	0	1	1	10/17
Influenza B virus	MP (4)	NSP (3)	NP (4)	HA (4)	–	NK (2)
Strong positive	4	3	4	4		2	17/17
Positive	4	3	4	4		2	17/17
Low positive	2	2	2	3		2	11/17
Parainfluenza virus 1	HN (13)	NP (1)	–	–	–	NK (1)
Strong positive	11	1				1	13/15
Positive	8	0				1	9/15
Low positive	2	0				0	2/15
Parainfluenza virus 3	HN (9)	F (4)	NP (1)	–	–	NK (1)
Strong positive	9	3	1			1	14/15
Positive	7	0	0			1	8/15
Low positive	4	0	0			0	4/15
Respiratory syncytial virus	F (6)	NP (7)	P (1)	NSP (1)	–	NK (1)
Strong positive	6	5	1	1		1	14/16
Positive	5	4	1	0		1	11/16
Low positive	5	2	1	0		1	9/16
Human metapneumovirus	F (3)	P (5)	NP (3)	5′ NCR (1)	–	NK (2)
Strong positive	3	5	3	1		2	14/14
Positive	2	5	3	1		2	13/14
Low positive	2	5	2	1		2	12/14
Adenovirus	H (12)	MP (1)	–	–	–	NK (1)
Strong positive	11;10	1;1				1;1	13;12/14
Positive	9;6	1;1				1;1	12;8/14
Low positive	8;6	1;0				1;1	10;7/14
Human rhinovirus	5′ NCR (10)					NK (1)
Strong positive	10;7;10					1;1;1	11;8;11/11
Positive	9;7;9					1;1;1	10;8;10/11
Low positive	4;3;4					1;1;1	5;4;5/11
Human coronavirus 229E	NP (4)	P (2)	M (1)	–		NK (1)
Strong positive	4	2	1			1	8/8
Positive	2	2	1			1	6/8
Low positive	1	1	1			1	4/8
Human coronavirus OC43	NP (4)	P (1)	M (1)	–		NK (1)
Strong positive	4	1	1			1	7/7
Positive	3	1	1			1	6/7
Low positive	0	0	0			0	0/7

Abbreviations: MP, matrix protein gene; NSP, non-structural protein gene; NP, nucleoprotein gene; HA, hemagglutinin gene; NA, neuraminidase gene; HN, hemagglutinin-neuraminidase gene; 5′ NCR, 5′ non-coding region; F, fusion protein gene; P, polymerase gene; H, hexon gene; NK, not known. Numbers of laboratories using the individual targets are shown in parentheses.

Results of tests for adenoviruses 4 and 7 and rhinovirus types 16, 72 and 90, respectively, are shown, separated by semicolon.

Discussion

The study described here is the first large-scale external quality assessment scheme for the molecular detection of respiratory viruses. Application of NATs for respiratory virus diagnosis is increasing as it is recognised as the best method for rapid, sensitive and specific diagnostic results (van Elden et al., 2002). Sixteen of the 17 participants in this study have developed their own ‘in-house’ NAT assays using a large variety of NAT protocols. In order for laboratories to assess the quality of these in-house assays an external QA programme has been developed. The performance in external QA panels is an integral part in accreditation and quality management of clinical diagnostic laboratories.

The preparation of a suitable panel that is reproducible and of a high quality to assess the performance of the NATs is demanding. In this study, samples were grown in cell culture and dilutions were made so sensitivity and limited specificity of assays for these viruses could be assessed. The panels were assessed as a small-scale dilution series prior to distribution in a second laboratory to determine the detection limit after a freeze-thaw step. Thereafter, the same stock viruses were used to prepare the final panels. These samples were not assessed by more than one reference centre prior to distribution so there have been some differences between the initial small-scale dilution series and the bulk panels sent to all participants. The lowest positive dilution used for OC43 was detected by none of the participants, so this dilution was too low for inclusion in further panels. However, for many of the panels, a number of the NATs used have detected all dilutions of the viruses in the panels, showing that the panels employed gave a good indication of NAT performance. In the future, the panel could be produced by freeze-drying the samples and include more pre-distribution testing to ensure more reproducible panels, as employed in other QCMD schemes (Schirm et al., 2002, Schloss et al., 2003).

The panels employed in this distribution used samples that reflected the limit of detection according to NATs used in two reference laboratories. The clinical relevance of these detection levels was not assessed; as it is not known what detection limit is required to detect clinically relevant respiratory virus infections. The limit of detection employed in these panels probably reflects a good guide to assess the sensitivity of NATs. At present, there is no international reference material for respiratory virus NAT's and these panels can provide the basis for new internationally recognised stocks.

One false positive was detected in the negative sample in the HRV panel. No false positives were detected in any other panels, giving a 1.1% false positive rate in the whole distribution. This shows a high level of technical skill in the participating laboratories. In other QA programmes in which many ‘in-house’ methods were used, the false positivity rates have been much higher (Schloss et al., 2003). The reduction of the false positivity rate may also be as a result of the use of real-time PCR, which reduces the chance of contamination.

In general, correct results were obtained with a mean of 93.8% of the strong positive samples, with participants failing to detect a mean of 23.2% and 53.0% of the positive and low positives, respectively. The false negative results observed were either results of samples below the detection limit of the assays, or of primer and probes not detecting the viruses at all. The selection of primers and probes for respiratory viruses is essential as a single NAT is employed for a specific virus that needs to detect many variants. Adenoviruses comprise of six different serotypes that show large sequence variation and RNA viruses show even greater sequence variation, as they lack proof reading capability. The virus included in the hMPV panel was a very well characterised (van den Hoogen et al., 2001), and therefore this information would have been used to generate NATs whereas the PIVs were clinical isolates with no publicly available sequence information. This knowledge of sequence information would affect the design of the NATs, and hence one of the reasons why the hMPV panel had good performance scores and the PIV poor.

There were some assays that did not detect a particular virus at all, e.g. PIV 1 and RSV while other NAT's by the same laboratory performed well. This is a result of poor primer and probe design to a non-conserved region of the virus. Most of the panels only contained one virus type; however, the panel for hRV contained three subtypes and the panel for ADV contained two subtypes. These two panels showed that some assays were unable to detect certain subtypes, even in the high positive sample, but were able to detect the weak positive with other subtypes. This is a result of primer and/or probe selection, which were unable to detect these viruses owing to the sequence variation seen in these strains. Therefore, participants who missed these strains would need to adapt their NAT in order to detect the possible variations. These results indicate the problem of using NAT methods for detection of respiratory viruses, in that they are not catch-all methods and careful evaluation of the primers and probes needs to be performed to ensure that no false negatives are obtained. All respiratory viruses detected in clinical samples will have sequence variation. So, in future panels more viruses to determine the ability of assays to detect clinical isolates might be included. In the case of influenza A and B, the strains included could be the most common circulating strains from the previous winter. In this way, the quality of the NATs can be monitored for detection of a wide variety of viral strains.

The other reason that assays failed to detect viruses in the weak positive was the lack of sensitivity of the NAT or procedures in the laboratory. There are a lot of factors that affect NAT sensitivity but this study gives some indications as to ways to improve sensitivity with PCR format, selection of gene target for the PCR and design of primers and probe. Firstly, all the individual components of the 83 in-house NAT could achieve maximal sensitivity in one or more panels, so the difference in sensitivity is probably due to inter-laboratory variation. Another factor that affects sensitivity is the methodology. However, all methodologies, except the one laboratory using the RV®Chip assay, could obtain maximum scores. So, again, the differences are probably due to inter-laboratory variations. A wide range in performance could be seen with the same methodology, e.g. real-time PCR.

The selection of the gene target was also shown to affect the sensitivity, in that some targets result in a more sensitive NAT. Comparisons of different targets have been shown with hMPV where either the polymerase or the nucleoproteins were shown to be more sensitive than matrix, fusion and phosphoprotein (Cote et al., 2003, Maertzdorf et al., 2004).

The results here suggest that for influenza A the M protein is a good target and the HN for PIV 3. The conservation of the target is also important as the primers and probe need to detect all the virus types, so in some cases, a primer with two or three mismatches may work for a particular virus although the sensitivity will be affected. Therefore, assays, which only detected the strong positive dilution using a good sensitive methodology, may need to check for a mismatch of the primer and probe sequence and potentially include further forward or reverse primers to achieve good sensitivity. Inclusion of additional primers or probes to improve the range of detection has been shown in the case of enteroviruses and rhinoviruses previously (Deffernez et al., 2004, Nijhuis et al., 2002).

Respiratory virus detection by culture has limitations owing to the sensitivities of cell lines to clinical isolates, this is particularly apparent with hRV and hCoV. Although reference strains can be used to check sensitivity of a particular cell line this does not necessarily mean that it will detect all clinical isolates. This problem can be equally true for poorly evaluated NATs. Nevertheless, NAT is becoming a valuable diagnostic tool, and hence the need for laboratories to perform quality control to evaluate sensitivity.

The production and distribution of this first respiratory virus panel showed that 92.4% of participants detected the strong positive and 47% of participants detected the low positive correctly. The panel also gives a very good indication to participant as to where to improve the assay's design as well as providing a good measure for the integrity of the NAT employed by a laboratory.