Diagnostic accuracy of urinary antigen tests for pneumococcal pneumonia among patients with acute respiratory failure suspected pneumonia: a systematic review and meta-analysis.

BACKGROUND/
OBJECTIVES: Urinary antigen tests have been used for the rapid identification of Streptococcus pneumoniae infection in patients with pneumonia, thereby leading to earlier targeted therapy than when using conventional diagnostic culture methods. This study aimed to update the knowledge on the diagnostic accuracy of urinary antigen tests for S. pneumoniae among patients with acute respiratory failure suspected of pneumonia based on a systematic review and meta-analysis.
METHODS: A systematic search was performed using MEDLINE and the Cochrane Central Register of Controlled Trials for studies published up to 3 June 2020. Prospective and retrospective cohort studies (in English) that reported on the diagnostic performance of urinary antigen tests versus culture or smear diagnostic methods in adult patients with clinically diagnosed pneumonia were selected and analysed. The QUADAS-2 tool was used to assess the risk of bias, and a bivariate random effects model was applied to perform a meta-analysis of the selected studies.
RESULTS: A total of 2179 studies were screened, of which 30 met the eligibility criteria for quality assessment and meta-analysis. Overall, data from 12 366 patients, including 1548 patients (12.5%) with the target condition and suspected pneumococcal pneumonia, were included in the analysis. The overall quality of the included studies was determined to be serious. The calculated pooled sensitivity and specificity were of 0.66 (95% CI 0.62 to 0.69) and 0.90 (95% CI 0.85 to 0.93), respectively.
CONCLUSIONS: The urinary antigen test is useful for achieving a definitive diagnosis of S. pneumoniae infection in patients with pneumonia. © Author(s) (or their employer(s)) 2022. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

The protocol was registered in advance and submitted according to Preferred Reporting Items for Systematic Reviews and Meta-analyses.

Statistical analysis methods and bias risk assessment tools recommended in the Cochrane Handbook were used.

Papers reported in languages other than English were not included, and the Embase database and Grey literature were not used as data sources, which may have resulted in missing relevant papers or publication bias.

Introduction

Streptococcus pneumoniae is the most common cause of community-acquired pneumonia in adults and is also the main cause of pneumonia-causing acute respiratory failure.1 The mortality rate of pneumococcal pneumonia varies depending on the severity of the disease at onset and host factors, but the 90-day mortality rate is very high, at approximately 25%–30%, when accompanied by bacteraemia.2

Pneumococcal pneumonia diagnosis is usually confirmed by the identification of S. pneumoniae in sputum samples by Gram staining, or its detection in cultures of blood samples or respiratory specimens, such as sputum or pleural fluid. Gram staining is rapid but not highly sensitive/specific.3 Culture tests take several days to provide results. Therefore, many cases of pneumonia are treated empirically. Empiric therapy is generally useful because it takes into account knowledge of the pathogen, including resistance/susceptibility, based on local high-quality surveillance system, the patient’s risk factors and comorbidities, and the severity of symptoms. However, not every country, region or hospital has high-quality surveillance system, and even if they do, if the urine antigen test can rapidly diagnose S. pneumoniae infection, a more specific antimicrobial agent may be selected. As a result, we believe that it may be helpful to reduce the individual risk of infections caused by resistant bacteria and antibiotic-associated Clostridium difficile infections.4 5

BinaxNOW Streptococcus pneumoniae Antigen Card (BinaxNOW-SP; Abbott, Abbott Park, Illinois, USA) is an immunochromatographic test that detects the C-polysaccharide coat protein of S. pneumoniae in urine. It is used worldwide as a rapid diagnostic method for S. pneumoniae infection in patients with pneumonia, allowing to obtain clinically valuable results within 15 min of urine collection. In 2013, two systematic reviews and meta-analyses were conducted using BinaxNOW-SP as the index test, along with established culture or smear methods as reference standard, in patients with community-acquired pneumonia.6 7 However, several studies were subsequently reported that consisted of very large sample sizes, which tended to be slightly less sensitive than the previous systematic reviews, therefore, we considered that it was necessary to update the evidence for the diagnostic accuracy.6–10 Therefore, to close this knowledge gap, we conducted a systematic review and meta-analysis to compare the sensitivity and specificity of urinary antigen tests (UATs) with established culture or smear methods in the diagnosis of patients with acute respiratory failure suspected of pneumonia.

Methods

We performed a systematic review and meta-analysis of previously published studies on diagnostic test accuracy (DTA). We adhered to the methodological standards outlined in the handbook for DTA reviews of Cochrane11 and used the Preferred Reporting Items for Systematic Reviews and Meta-analyses of DTA Studies12 to report our findings. The review protocol was prospectively registered in the University Hospital Medical Information Network Clinical Trials Registry (UMIN 000041081) and it is also available in online supplemental material. The need for ethical approval and consent was waived for this systematic review.

Patient and public involvement

There was no patient and public involvement in the whole process of conducting this research.

Population, index test and target condition

The target participants were adult patients with acute respiratory failure suspected of having pneumonia. If a study involved both children and adults, we extracted only the data related to the adult patients and excluded studies in which data related to adults could not be distinguished from those of children. The index test of interest was UATs, such as BinaxNow-SP. The target condition was pneumococcal pneumonia diagnosed using reference standard methods. For the reference standard, data were considered positive if at least one positive result was obtained by Gram stain of sputum, or culture of blood, pleural fluid or respiratory samples (ie, sputum, bronchoalveolar lavage fluid or other).

Study eligibility and selection

All studies written in English, including prospective, retrospective and observational (cohort or cross-sectional) studies, and secondary analyses of randomised controlled trials, were selected. We excluded diagnostic case-control studies (two-gate study) and case studies that lacked DTA data, namely true positive (TP), false positive (FP), true negative (TN) and false negative (FN) values. Two authors independently screened each study for eligibility and extracted the data. Disagreements among reviewers were resolved by discussion or a third reviewer.

Electronic searches

To identify all eligible studies, we searched the Medical Literature Analysis and Retrieval System Online (MEDLINE) via PubMed (accessed on 3 June 2020), and the Cochrane Central Register of Controlled Trials (CENTRAL) (accessed on 3 June 2020). The details of the search strategy are provided in online supplemental file 1. The developing search strategy was supported by medical librarians at the Kyoto Prefectural University of Medicine (Kyoto, Japan).

Data extraction and quality assessment

The following data were extracted using a predefined data extraction form that included the study characteristics (author, year of publication, country, design, sample size, clinical settings, conflict of interest and funding source), patient characteristics (inclusion/exclusion criteria, and clinical and demographic characteristics), index test (UATs), reference standard (Gram stain of sputum, blood culture, sputum culture, pleural fluid culture, or culture of any other respiratory sample) and diagnostic accuracy parameters (TP, FP, FN and TN). Two investigators evaluated the risk of bias using the QUADAS-2 tool,13 which includes four risk of bias domains and three domains of applicability. Any disagreements were resolved by discussion or a third reviewer. Given the absence of evidence for publication bias in DTA studies and the lack of reliable methods for its assessment, no statistical evaluation of publication bias was performed.11

Statistical analysis and data synthesis

The ‘Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy’ was used.11 Diagnostic sensitivity and specificity estimates with 95% CIs were captured in paired forest plots to evaluate study heterogeneity. We used a bivariate random effects model for the meta-analysis. Random-effects model attempted to generalise findings beyond the included studies by assuming that the selected studies are random samples from a larger population.14 All analyses were performed using Review Manager V.5.3 (Cochrane Collaboration, London, UK) and STATA/SE V.16.1 (StataCorp). All statistical analyses were conducted using a two-sided alpha error of 5%.

Heterogeneity assessment

A subgroup analysis, prespecified in the protocol, was performed to investigate whether sensitivity and specificity were different among the following subgroups: (1) differences in reference standard, (2) presence of immunocompromised patients, (3) antibiotics use prior to UAT was performed, (4) inpatients or outpatients and (5) types of UAT used.

Sensitivity analysis

We performed a sensitivity analysis restricted to studies with a low risk of bias in all domains of QUADAS-2.

Results

A total of 2179 studies were screened, among which 30 met the eligibility criteria and were included in the quality assessment and meta-analysis. The detailed selection protocol and reasons for exclusion are shown in figure 1.

Figure 1

Study selection protocol (PRISMA flow chart). PRISMA, Preferred Reporting Items for Systematic Reviews and Meta-analyses.

Study characteristics

Data from 12 366 patients were retrieved from the 30 studies, among who were 1548 patients (12.5%) with pneumococcal pneumonia. The median and mean age of the patients included ranged from 45 to 74 years, and most were male (51%–79%). The median prevalence of pneumococcal pneumonia was 16.7% (IQR: 9.5–22.9). Twenty-one studies were prospective, five studies were retrospective and four were of unknown nature. Most studies were conducted in inpatient settings, but three studies included outpatients. The patient characteristics and reference standards used in each study are summarised in table 1. The index test was BinaxNow-SP in all the studies. Reference standards were defined by either positive cultures of blood, respiratory samples (or smear) or pleural fluid in 18 studies, by either positive cultures of blood or respiratory samples (or smear) in 9 studies, by blood culture alone in 2 studies and by respiratory samples culture (or smear) alone in 1 study.

Table 1

Summary of the characteristics of the studies included in the analysis

(Ref.) Study		Country	Design	Cohort size(n)	Setting	Mean age(years)	Sex (%) (male/female)	Immunodeficiency (%)	Prior antibiotics (%)	Reference standard*
(18)	Abdeldaim et al., 2009	Denmark	Prospective	142	Admission	63	Unknown	0	66	B
(19)	Athlin et al., 2017	Sweden	Unknown	141	Unknown	74(median)	56/44	Unknown	Unknown	C
(20)	Briones et al., 2006	Spain	Prospective	911	Admission	66	64/36	0	Unknown	A
(21)	Burel et al., 2001	French	Unknown	91	Admission	Unknown	Unknown	Unknown	Unknown	A
(22)	Burgos et al., 2018	Spain	Prospective	219	Admission	67	63/37	Ratio unknown	Unknown	A
(23)	Butler et al., 2003	USA	Retrospective	147	Admission	45(median)	70/30	0	0	A
(24)	Farina et al.,2002	Italy	Unknown	104	Admission	Unknown	Unknown	Unknown	Unknown	B
(25)	Fukushima et al., 2015	Japan	Unknown	112	Unknown	72/67(male/female, median)	68/32	Unknown	0	D
(26)	Genné et al., 2006	Switzerland	Prospective	67	Admission	68	57/43	Unknown	0	B
(27)	Gutiérrez et al., 2003	Spain	Prospective	452	Mixed	57	63/37	Unknown	23	A
(28)	Sordé et al., 2011	Spain	Prospective	383	Admission	64	67/33	20	Unknown	A
(29)	Ikegame et al., 2017	Japan	Prospective	69	Admission	78	70	Unknown	Ratio unknown	A
(30)	Ishida et al., 2004	Japan	Prospective	349	Admission	65	65/35	Ratio unknown	55	A
(8)	Laijen et al., 2017	Netherlands	Retrospective	681	Admission	67	60/40	Ratio unknown	21	A
(31)	Lasocki et al., 2006	France	Retrospective	140	Admission	69(Pneumonia, median)	66/34	Unknown	66	A
(9)	Lee et al., 2020	Korea	Retrospective	1257	Admission	73(+UAT)	64/36(+UAT)	0	Unknown	A
(32)	Marcos et al., 2003	Spain	Prospective	398	Admission	50	79/21	21(HIV-1)	Ratio unknown	A
(10)	Molinos et al., 2015	Spain	Prospective	3135	Admission	66	65/35	0	Ratio unknown	A
(33)	Murdoch et al., 2001	New Zealand	Prospective	420	Admission	68(median)	51/49	Ratio unknown	76	B
(34)	Strålin et al., 2004	Sweden	Prospective	215	Admission	71(median)	53/47	Unknown	27	B
(35)	Tzeng et al., 2006	Taiwan	Retrospective	1243	Admission	Unknown	Unknown	Unknown	Unknown	A
(36)	Weatherall et al., 2008	Australia	Prospective	59	Mixed	79(median)	56/44	0(Neutropenia)	26	B
(37)	Zhou et al., 2018	China	Prospective	165	Mixed	48(median)	57/43	0(HIV)	67	B
(38)	Rosón et al., 2004	Spain	Prospective	220	Mixed	66	71/29	0	18	A
(39)	van Der Eerdenet al., 2005	Netherlands	Prospective	262	Admission	64	54/46	0	26	B
(40)	Lauderdale et al., 2005	Taiwan	Prospective	168	Admission	56	64/36	1.2	16	A
(41)	Kobashi et al., 2007	Japan	Prospective	156	Admission	62	71/29	Unknown	45	A
(42)	del Mar García-Suárezet al., 2007	Spain	Prospective	268	Admission	60	64/36	Unknown	Unknown	A
(43)	Hohenthal et al., 2008	Finland	Prospective	333	Admission	50	52/48	0	31	C
(44)	Ercis et al., 2006	Turkey	Prospective	59	Mixed	18–86(range)	64/36	6.8	15	B

*Reference standard test used: A, composite of blood culture, respiratory sample culture (or smear), and pleural culture; B, composite of blood culture, respiratory samples culture (or smear); C, blood culture alone; D, respiratory samples culture (or smear) alone.

Risk of bias assessment

For patient selection, we evaluated 10 studies as having a high risk of bias or high concern for applicability (figure 2) as the exclusion of immunocompromised patients was inappropriate and that the sample was not consecutive when the time frame for inclusion of patients was limited. For the index test, one study was rated as high risk of concern in applicability. The study used early morning urine for the diagnosis of pneumococcal pneumonia; however, it was not considered as common and thus the applicability of the results to other settings and actual clinical practice was considered as high concern in the applicability. For the reference standard, we evaluated 11 studies as having a high risk of bias or high concern for applicability because the reference standard was not a composite of blood culture, respiratory sample culture (or smear) and pleural fluid culture. In the patient flow assessment, we assessed 16 studies to have a high risk of bias because blood cultures as a reference standard were not performed for all patients. Taken together, the overall quality of the included studies was serious.

Figure 2

Summary of bias risk assessment. Bias risk was evaluated using the QUADAS-2 tool. The QUADAS-2 tool is designed to assess the quality of primary diagnostic accuracy studies. Green: low risk of bias or low concern in applicability. Red: high risk of bias or high concern in applicability.

Meta-analysis results

A summary of the diagnostic accuracy of UTAs for pneumococcal pneumonia in each study is presented in a forest plot in figure 3. The calculated pooled sensitivity and specificity were of 0.66 (95% CI 0.62 to 0.69) and 0.90 (95% CI 0.85 to 0.93), respectively. In a population of 1000 patients with a given target condition prevalence of 10%, the following was detected: 66 patients (95% CI 62 to 69) with TP, 34 patients (95% CI 31 to 38) with FN, 810 patients (95% CI 765 to 837) with TN and 90 patients (95% CI 63 to 135) with FP. The findings for the different prevalence estimates (5%–15%) are presented in online supplemental file 2.

Figure 3

Paired forest plot of the studies included in the meta-analysis. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Subgroup and sensitivity analyses

We conducted subgroup analysis among two reference standards: (1) either positive culture of blood, respiratory samples (or smear) or pleural fluid, (2) either positive culture of blood or respiratory samples (or smear), (3) positive blood culture only and (4) positive respiratory samples (or smear) (figure 4A). Based on the 17 studies using reference standard A (see table 1), the sensitivity and specificity were of 0.69 (95% CI 0.64 to 0.73) and 0.88 (95% CI 0.80 to 0.93), respectively. Based on the nine studies using the reference standard B, the sensitivity and specificity were of 0.59 (95% CI 0.52 to 0.65) and 0.93 (95% CI 0.87 to 0.96), respectively. We could not conduct a subgroup analysis of reference standards C and D as each of these diagnostic approaches was only used in a few studies.

Figure 4

Paired forest plot of the studies included in the subgroup analysis based on (A) difference in reference standard, (B) presence of immunocompromised patients, and inclusion of (C) only inpatients, or both inpatients and outpatients. FN, false negative; FP, false positive; TN, true negative; TP, true positive.

Regarding the presence of immunocompromised patients, since no study assessed exclusively this population, we conducted subgroup analysis between excluded immunocompromised participants and mixed, 10 and 8 studies, respectively (figure 4B). The sensitivity and the specificity of the included group of studies were of 0.64 (95% CI 0.57 to 0.70) and 0.93 (95% CI 0.78 to 0.97) and in the mixed group were of 0.65 (95% CI 0.60 to 0.70) and 0.83 (95% CI 0.79 to 0.87), respectively.

Regarding inpatients or outpatients, since there was no study comprising only outpatients, we conducted a subgroup analysis between studies with only inpatients and with both inpatients and outpatients (23 and 5 studies, respectively) (figure 4C). The sensitivity and specificity of the studies with only inpatients were of 0.66 (95% CI 0.62 to 0.71) and 0.89 (95% CI 0.82 to 0.93), and in studies with mixed populations were of 0.67 (95% CI 0.56 to 0.76) and 0.90 (95% CI 0.79 to 0.96), respectively.

We could not conduct subgroup analysis regarding presence of prior antibiotics at UAT nor on different types of UATs due to lack of data. Sensitivity analysis was also not performed because of the low number of studies with a low risk of bias in all the domains of QUADAS-2.

Discussion

This systematic review and meta-analysis included 30 studies, in which the pooled sensitivity and specificity were of 0.66 (95% CI 0.62 to 0.69) and 0.90 (95% CI 0.85 to 0.93). These results indicate moderate sensitivity and high specificity, which were in accordance with those of previous meta-analyses.5 6 However, a previous analysis comprising 10 studies reported a higher sensitivity (0.75, 95% CI 0.71 to 0.79) and specificity (0.95, 95% CI 0.92 to 0.98).5 This difference may be due to the exclusion of studies in which more than one-third of the target population was administered prior antibiotics and by excluding pneumonia of unknown causative organisms.

Here, the subgroup analysis revealed no significant variations in sensitivity and specificity owing to differences in the reference standards. Moreover, the specificity tended to be lower in studies including immunocompromised patients, suggesting that caution may be necessary in clinical practice when interpreting UAT results from immunocompromised patients. In addition, studies including outpatients showed similar results to those including only inpatients, suggesting that the UATs may be as useful in outpatients as in inpatients. However, no studies have included outpatients; thus, further research is required.

The strengths of our study are as follows. It has been more than 7 years since the last existing systematic review was published, and the results of two investigations showed some variability.5 6 In the present systematic review, we were able to report updated results by including eight studies published since 2013, including studies with large sample sizes. Second, we were able to evaluate the sensitivity and specificity by including immunocompromised patients, which may be helpful in clinical practice. Third, the results of this study suggest that the collected information may be applicable to both outpatients and inpatients.

Nonetheless, this study also has some limitations. First, we did not include the Embase database or the grey literature as data sources and only included articles described in English for inclusion. Therefore, there is a possibility of missing relevant studies or publication bias. However, we attempted to extract as many related studies as possible by manual and citation search. Second, acute respiratory failure was diagnosed based on the clinical symptoms used to diagnose pneumonia and the degree of severity that required hospitalisation. Therefore, some patients with pneumonia without acute respiratory failure might be included in this study population. Third, paediatric patients and patients with nosocomial pneumonia were not included in this systematic review. The diagnostic accuracy for these patients needs further study. Fourth, the overall quality of the study was judged to be serious, mainly because blood cultures were not collected from all participants, but it was unknown which participants in each study did not have blood cultures collected, which may have affected the results of this study. Fifth, although we set the reference standard as gram stain of sputum, or culture of blood, pleural fluid, or respiratory samples and it may be accepted in the previous literature, there is some opinion that it is difficult to define the gold standard for the diagnosis of pneumococcal pneumonia as well as other infectious disease. Recently, in a research question of diagnostic accuracy for infectious disease without widely accepted gold standard, the latent class analysis is suggested as an alternative method. We did not perform it because it was not planned in the analysis protocol; however, in further research, it may be valuable to consider the latent class analysis. Sixth, we were unable to evaluate the effect of antibiotic use, although some studies reported that 21%–70% of the participants received prior antibiotics. Fourth, all the included studies used BinaxNow-SP; thus, we could not evaluate the effect of different types of UATs. Seventh, we were not able to evaluate the effect of differences in disease severity. Nonetheless, we believe that this effect is small because the standard mechanism of the antigen test regardless of the symptoms. Eighth, previous reports have indicated that UATs inspection performance may vary from period to period, although the cause is not known.15 16 This study did not examine this point in the subgroup analysis, however, future analyses might need to consider heterogeneity due to differences in inspection performance by period. Ninth, UAT does not detect antimicrobial susceptibility/resistance and may have no direct impact on prescriptions of antimicrobials. Even if UAT is used, antimicrobials should be selected based on knowledge of the pathogen based on local surveillance system, patient risk factors, underlying disease and severity. Tenth, patients who have recently developed pneumococcal pneumonia may provide FP results for several weeks after onset.17 Therefore, when using UAT, the patient’s recent history of pneumonia should be ascertained and interpreted with this in mind. Finally, the diagnostic accuracy of UAT for pneumococcal pneumonia might have changed during the COVID-19 epidemic. At the time of the search for inclusion studies, there were no studies available after the COVID-19 outbreak, so the impact of the COVID-19 outbreak could not be considered; the diagnostic accuracy of UAT for pneumococcal pneumonia after the COVID-19 outbreak will be considered in future studies.

Taken together, the reported high pooled specificity and moderate pooled sensitivity indicate that UATs are useful to rule in pneumococcus pneumonia rather than to rule it out. Despite of the above-mentioned limitations, our study indicates that UATs may be useful for the diagnosis of patients suspected of having pneumonia.

Conclusion

We conducted a systematic review and meta-analysis, which included recent studies with a large sample size, to compare the sensitivity and specificity of UATs as diagnostic tools for the assessment of patients with acute respiratory failure suspected of pneumonia. Our results were similar to those of previous studies, demonstrating that UATs have high specificity and moderate sensitivity. Hence, UATs can be useful for rapidly achieving a definitive diagnosis in patients suspected of having pneumonia.