In patients presenting to the emergency department with skin and soft tissue infections what is the diagnostic accuracy of point-of-care ultrasonography for the diagnosis of abscess compared to the current standard of care? A systematic review and meta-analysis.

OBJECTIVES: The primary objective of this systematic review was to determine the accuracy of point-of-care ultrasonography (POCUS) in diagnosing abscess in emergency department (ED) patients with skin and soft tissue infections (SSTI). The secondary objective was the accuracy of POCUS in the paediatric population subgroup.
SETTING: Prospective studies set in emergency departments. PARTICIPANTS: Emergency department patients (adult and paediatric) presenting with SSTI and suspected abscess. PRIMARY AND SECONDARY OUTCOME MEASURES: This systematic review was conducted according to Cochrane Handbook guidelines, and the following databases were searched: PubMed, MEDLINE, EMBASE and the Cochrane database of systematic reviews (1946-2015). We included prospective cohort and case-control studies investigating ED patients with SSTI and abscess or cellulitis, a defined POCUS protocol, a clearly defined gold standard for abscess and a contingency table describing sensitivity and specificity. Two reviewers independently ascertained all potentially relevant citations for methodologic quality according to QUADAS-2 criteria. The primary outcome measure was the sensitivity and specificity of POCUS for abscess. A preplanned subgroup (secondary) analysis examined the effects in paediatric populations, and changes in management were explored post hoc.
RESULTS: Of 3028 articles, 8 were identified meeting inclusion criteria; all were rated as good to excellent according to QUADAS-2 criteria. Combined test characteristics of POCUS on the ED diagnosis of abscess for patients with SSTI were as follows: sensitivity 96.2% (95% CI 91.1% to 98.4%), specificity 82.9% (95% CI 60.4% to 93.9%), positive likelihood ratio 5.63 (95% CI 2.2 to 14.6) and negative likelihood ratio 0.05 (95% CI 0.01 to 0.11).
CONCLUSIONS: A total of 8 studies of good-to-excellent quality were included in this review. The use of POCUS helps differentiate abscess from cellulitis in ED patients with SSTI. TRIAL REGISTRATION NUMBER: CRD42015017115. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/.

Strengths of our study include the exhaustive search strategy, reproducible protocols and strict adherence to systematic review methodology. The use of standardised and validated data collection and extraction tools limited bias and increased inter-rater reliability.

Important limitations of our systematic review and meta-analysis include: (1) owing to the small number of included studies, assessment of publication bias is difficult and (2) a patient presenting with an SSTI may initially have cellulitis but develop an abscess; this is especially important if there was a time lag between the index test and the reference standard.

A strength of this study was the ability to conduct a subgroup analysis of the diagnostic accuracy in paediatric patients, who may not tolerate physical examination, blood testing and needle aspiration as readily as adults.

Introduction

Skin and soft tissue infections (SSTIs) are a common presenting symptom to the emergency department (ED).1 The two most frequently encountered clinical entities are cellulitis and abscesses. Substantial degrees of overlap between the clinical presentation of cellulitis and abscesses frequently create clinical uncertainty in differentiating the two conditions.2 3 This is notably true for specific populations including paediatrics, where physical examination may be unremarkable.4 Since abscesses require incision and drainage or needle aspiration, and cellulitis is treated with systemic antibiotics, distinguishing the two is essential.5 Blind needle aspiration for purulence can be undertaken, but this is a painful and unnecessary procedure in patients with cellulitis only. As a corollary, underappreciating an abscess can lead to inappropriate and ineffective treatment with antibiotics, leading to complications, additional ED visits and increased cost.4 5 As ED visits for SSTIs have doubled contemporaneously since the emergence of community acquired methicillin-resistant Staphylococcus aureus (MRSA) in the early 1990s,1 6 the availability of an objective tool to differentiate an abscess from cellulitis is necessary to optimise patient care.7

In patients presenting with SSTI, the treatment of cellulitis and abscess differs substantially. As a result, a high level of diagnostic accuracy is important to inform correct treatment for patients presenting with each condition. Point-of-care ultrasound (POCUS) has been integrated into the training of emergency physicians.8 POCUS has been hypothesised to help identify fluid collections suggestive of abscess to help guide appropriate therapy.9 To the best of our knowledge, there is one prior systematic review from 2015,10 and no prior meta-analysis on this topic completed to assess the diagnostic accuracy of bedside ultrasound for the diagnosis of abscess in patients presenting with SSTI in the ED.10

The primary objective of this systematic review was to determine the accuracy of POCUS in diagnosing abscess in ED patients with SSTI. The secondary objective was the accuracy of POCUS in the paediatric population subgroup.

Methods

Study design

The investigators developed a systematic review protocol according to PRISMA guidelines11 and the Cochrane Handbook,12 and this was recorded a priori with the Prospero registry (CRD42015017115) (see online data supplement S1). The Cochrane Handbook for Diagnostic Test Accuracy Reviews12 and accepted guidelines were adhered to.13

Search strategy

Investigators searched Ovid MEDLINE, Ovid EMBASE and Cochrane Library for journal articles and conference proceedings prior to 31 March 2016. An experienced health sciences librarian assisted with the development of the preliminary search strategy in Ovid MEDLINE based on the research question: What is the accuracy of bedside of ultrasound for diagnosing abscess in the emergency department? The search strategy was independently reviewed by two medical librarians and validated against a sample result set of 21 studies identified by the primary investigator.

The librarian adapted the search as minimally as possible before executing it in Ovid Embase and Cochrane Library. Duplicate citations were removed, and the final references were delivered to the primary investigator in a format compatible with EndNote citation management software. A search alert in Ovid MEDLINE was enabled to re-run the search on a monthly interval and send the investigator updates of any new publications (The search strategy is available as online supplementary data S2.).

We used Science Citation Index to retrieve reports citing the relevant articles identified from our search in MEDLINE and EMBASE and then entered relevant studies identified into PubMed. We then used the ‘Related articles’ feature as suggested by Sampson et al.14 We conducted online bibliographic searches of the table of contents for Critical Ultrasound Journal for each issue of the past 5 years. We manually searched the bibliographies of all potential articles (including review articles) to identify articles not identified by our primary search. Our grey literature search included scrutinising reference lists of potential articles and searches of abstracts of major emergency medicine conferences (Society of Academic Emergency Medicine, American College of Emergency Physicians and Canadian Association of Emergency Physicians). We contacted abstract authors for further information.

Study selection

We included prospective cohort and case–control studies evaluating the diagnostic accuracy of POCUS in the diagnosis of abscess in ED patients. Only studies involving patients with SSTI and clinical uncertainty regarding abscess or cellulitis were included. The index test was the use of POCUS for the detection of abscess in ED patients with SSTI. We used a combined reference standard of (1) purulent discharge from and incision and drainage, (2) abscess or cellulitis on CT according to radiologist opinion or (3) final diagnosis from clinical follow-up. No restriction was made on the protocol of ultrasonography used to diagnose abscess, and no restriction on the type of emergency physician was made. No restriction on the type of machine or transducer used was applied. We excluded case reports, retrospective studies and other types of case–control studies. In addition, we excluded studies that did not report sensitivity or specificity or if data could not be extracted to construct a 2×2 table. Finally, we excluded studies including patients in the primary care or inpatient setting.

Data collection and processing

Two review authors independently identified potential articles for inclusion by scanning the titles and abstracts of articles (DDC, TJ). Any disagreement was resolved by consensus. When this did not result in agreement, a third reviewer (JC) was involved to reach agreement. Two review authors (DB, FXS) independently extracted data from the selected articles using prepared data extraction sheets. Disagreement was resolved by consensus or by involvement of a third reviewer (JC). No attempt was made to mask the author's name or the journal's name. A data extraction form was developed and pilot-tested for validity and accuracy (see online supplementary data S3). We extracted information on: author, title, journal name, year of publication, study design (prospective cohort, case–control), setting in which the study was conducted, protocol of ultrasonography used, reference standard chosen, QUADAS-2 items15 and data on sensitivity and specificity or data for 2×2 table if possible.

Outcome measures

The primary outcome for this study was the sensitivity and specificity of POCUS for the diagnosis of abscess in the ED. Our secondary outcome was the sensitivity and specificity of POCUS in the paediatric population subgroup. A post hoc secondary outcome was the reported change in management due to POCUS reported in the different studies. This was felt to be a clinically important outcome to include in the final review, which we had not initially included in our systematic review protocol.

Validity assessments

Two review authors (DB, FXS) independently assessed the methodological quality of each selected article using the QUADAS-list.15 Disagreement was resolved by consensus or involvement of a third reviewer (JC). The QUADAS-2 assesses four potential areas for bias and applicability to the research question: (1) patient selection—the risk of bias was high if the study was a case–control design, enrolment was non-consecutive or the study had inappropriate exclusions; (2) index test—if the results from incision and draining were incorporated into the US results, the risk of bias was high; (3) references standard—risk of bias was high if the reference standard could misclassify the target condition, or the reference standard interpreted with knowledge of POCUS results and (4) flow and timing—the risk of bias was high if not all patients received the same POCUS protocol (index test), not all patients received the same reference standard, or not all patients were included in the analysis.

Primary data analysis

We presented individual study results graphically by plotting sensitivity and specificity estimates on a forest plot to visually assess for heterogeneity, and on the hierarchical summary receiver-operating characteristic (HSROC) space to visually assess for the presence of a threshold effect. The HSROC may control for the lack of an ideal reference standard and is recommended in the DTA guidelines.12

We explored possible sources of heterogeneity related to spectrum, design characteristics and method of ultrasound used. We combined data for meta-analysis using the HSROC model to obtain summary estimates of the pairs of sensitivity and specificity and a summary line. All data analyses were conducted using Stata (V.11.2, Stata Corp, College Station, Texas, USA), and results were managed in REVMAN (V.5.2, the Nordic Cochrane Centre, Copenhagen, Denmark).

Results

Characteristics of retrieved studies

Our search strategy returned a total of 3110 citations, which resulted in 3028 citations once duplicates were removed. After reviewing the abstracts of 70 articles and the full text of 25, we selected eight studies for inclusion in the final systematic review and meta-analysis (see figure 1 for the PRISMA diagram).

Figure 1

PRISMA flow diagram for the study.

The eight studies included in the final systematic review, and meta-analysis contained 747 eligible patients.4 16–22 This included three studies from the adult ED setting16–19 and five studies from the paediatric ED setting.4 20–22 All studies except one19 were conducted in the USA. More detailed characteristics of the included studies are available in table 1. Analysis of the data extraction process by two independent reviewers (DB, FXS) revealed a κ value of 0.80 (SE 0.25).

Table 1

Characteristics of studies included in final meta-analysis

	Quraishi	Squire	Tayal	Sivitz	Berger	Iverson	Marin	Adams
Year of publication	1997	2005	2006	2010	2012	2012	2013	2016
Journal	Clinical Otolaryngology	Academic Emergency Medicine	Academic Emergency Medicine	Journal of Emergency Medicine	American Journal of Emergency Medicine	American Journal of Emergency Medicine	Academic Emergency Medicine	Journal of Pediatrics
Country of study	Ireland	USA	USA	USA	USA	USA	USA	USA
Patient population	Paediatric ED	ED	ED	Paediatric ED	ED	Paediatric ED	Paediatric ED	Paediatric ED
Number of patients (lesions)	23 (23)	107 (107)	126 (126)	50 (50)	40 (40)	65 (65)	755 (873)	148 (151)
Number and type of operators	Unknown	Unknown number of emergency physicians and residents	5 emergency physicians	1 paediatric emergency physician and 1 fellow	Unknown number of emergency physicians and residents	Unknown number of paediatric emergency physicians and fellows	8 paediatric emergency physicians or fellows	8 paediatric emergency physicians, 2 paediatric emergency medicine fellows
Soft tissue ultrasound training or qualifications	Unknown	30 min of didactic and hands-on training	At least five supervised soft tissue scans	30 min didactic, at least 5 soft tissue scans	15 min didactic session	2 60 min didactic and hands-on training sessions repeated quarterly	6-hour training including lecture and hands-on practice	1–2 day course, plus at least 25 abscess scans reviewed by ultrasound director
Quality assurance of scans	Unknown	Unknown	Unknown	Images recorded and inter-rater agreement measured	None	Inter-rater reliability checked throughout study	75% of scans reviewed by blinded sonologist	10% of scans repeated by second operator
Blinding	No	Unclear	Treating clinicians blinded, data collection non-blinded	Intermittent	Yes	Intermittent	Yes	Treating clinicians non-blinded; ultrasonographers blinded
US machine and probe	Unclear	BK Hawk 2102 8 MHz linear; Sonosite Titan 10 MHz linear	Shimadzu model 400 and 450 7.5 MHz linear	Sonosite Micromaxx 8–13 MHz linear	Sonosite Turbo or Micromaxx 10 MHz linear	Siemen Sonoline G605 linear	Sonosite Micromaxx 6–13 MHz or 5–10 10 MHz linear, or 2–5 MHz curved array	Sonosite Edge 6–13 MHz linear
US protocol	Unclear	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Reference standard	Positive I&D	Positive I&D and follow-up	Unclear	Positive I&D	Positive I&D	Positive I&D	Positive I&D and follow-up	I&D or follow-up
Industry sponsored	Unclear	No	No	No	No	No	Unclear	Unclear
Time to complete US study	Unclear	Not recorded	Not recorded	Not recorded	Not recorded	Not recorded	Unclear	Not recorded
Prospective or retrospective	prospective	Prospective	Prospective	Prospective	Prospective	Prospective	Prospective	Prospective

Quality of included studies

Assessment of the methodologic quality of the eight included studies using the QUADAS tool15 revealed most of the studies to be of moderate-to-high quality (figures 2 and 3).

Figure 2

QUADAS-2 assessment of risk of bias for included studies.

Figure 3

QUADAS-2 summary graph.

Main results

The sensitivity of POCUS in the eight included studies ranged from 65.0% to 100% and the specificity from 30.0% to 100% (figure 4). Meta-analysis of the eight studies included in our final review demonstrated a point estimate of 96.2% (95% CI 91.1% to 98.4%) for the sensitivity of POCUS. The point estimate for specificity is 82.9% (95% CI 60.4% to 93.9%) (figure 5). The positive likelihood ratio (LR) was 5.6 (95% CI 2.2 to 14.6), and the negative LR was 0.05 (95% CI 0.02 to 0.11).

Figure 4

Forest plot of included studies.

Figure 5

HSROC curve of final meta-analysis. HSROC, hierarchical summary receiver-operating characteristic.

The preplanned, subgroup analysis of paediatric patients demonstrated similar point estimates for sensitivity 93.9% (95% CI 84.8% to 97.7%) and specificity 82.9% (95% CI 34.2% to 97.9%). The positive LR for paediatric patients was 5.5 (95% CI 0.9 to 33.9), and the negative LR was 0.07 (95% CI 0.03 to 0.15) (figure 6).

Figure 6

HSROC curve for paediatric subgroup analysis. HSROC, hierarchical summary receiver-operating characteristic.

Analysis of the Cooks' distance for potential influence on the final HSROC point estimates was conducted. Two studies, Marin et al17 18 and Tayal et al,21 demonstrated CooksD values >1.23

Data supporting changed management (to perform or not perform a drainage) after POCUS was provided in five of seven studies.4 15 16 20 22 In studies of paediatric patients, the rate of management change ranged from 14% to 27%.4 20 22 The proportion of patients who were initially determined to need drainage based on clinical examination and who subsequently ended up not receiving a drainage based on POCUS findings ranged from 12% to 20%.4 20 The proportion of patients who ended up receiving a drainage based on POCUS findings after initially being determined not to require drainage ranged from 13% to 18%.4 20 Sivitz et al20 found that management changes occurred most often in the quintiles representing equivocal pretest probabilities (ie, 2–3 out of 5) in 36% of cases. Similarly, Adams et al22 demonstrated that POCUS changed management most often in the context of an equivocal physical examination for the presence of abscess or when the pretest probability of abscess was not high (<90%).

Studies in adults demonstrated a slightly higher rate of change in management ranging from 17% to 56%.15 16 The proportion of patients who received unplanned drainage after POCUS ranged from 23% to 40%.15 16 The proportion of patients who did not receive drainage despite being determined to require it after clinical examination ranged from 12% to 36%. Separating the pretest probabilities of the presence of abscess into deciles, it was found that POCUS had an effect on management at every decile from 10% to 90%.15 Since the study by Marin et al21 blinded treating physicians to POCUS results, changes in management were unable to be determined.

Discussion

The primary objective was to assess the test accuracy of POCUS to diagnose abscess in ED patients with SSTI. Although the eight studies differed in terms of sensitivity and specificity, the pooled estimates of 96.2% (95% CI 91.1 to 98.4) sensitivity and 82.9% (95% CI 60.4 to 93.9) specificity are favourable. This assists clinicians by demonstrating that POCUS, a rapid, non-invasive, painless, easily repeatable test, can distinguish between abscess and cellulitis in the vast majority of cases. This could provide a greater degree of diagnostic certainty in SSTI patients presenting with equivocal signs and symptoms, thus leading to appropriate therapy more rapidly.

Our findings are particularly important in children, who may not tolerate physical examination, blood testing and needle aspiration as readily as adults. In our planned subgroup analysis, paediatric patients demonstrated similar point estimates for sensitivity 94.9% (95% CI 88.0 to 97.8), and specificity 83.1% (95% CI 46.6 to 96.5). This may provide paediatricians and emergency physicians caring for children with an additional valuable to tool to discern between cellulitis and abscess in children with equivocal signs and symptoms.

A recent review by Alsaawi et al10 examined this same topic, however we feel that our study is stronger for several important reasons. In our study, two independent reviewers screened all titles for inclusion, potentially minimising selection bias. In our study, we included the same five studies as Alsaawi et al, and two additional studies,19 22 one of which was unpublished at the time of the Alsaawi study. In addition, in our study we were able to conduct a quantitative synthesis (meta-analysis) of the available data to provide accurate point estimates of the sensitivity and specificity of POCUS in patients with SSTI in the ED.

POCUS resulted in management changes—to perform or not perform a drainage—in 14–56% of cases in the reviewed studies.4 15 16 20 22 The Infectious Disease Society of America defines abscesses as “painful, tender, and fluctuant red nodules, often surmounted by a pustule and surrounded by a rim of erythematous swelling.”24 This definition is challenged by the high rates of management change born out of these studies. This implies that the clinical examination is neither sensitive nor specific for detecting abscesses. In the Tayal et al study physicians had an error rate of 30–50% regardless of pretest probability of abscess based on clinical assessment.16 For instance, fluctuance is an imprecise indicator of abscess as only 6 of 17 patients who underwent drainage of neck abscesses had fluctuance on examination.19 We demonstrate that POCUS can accurately diagnose abscess in paediatric and adult populations and is likely superior to clinical examination.

Adams et al22 suggested that change in management occurred in one in four ultrasound studies performed. The issue of whether or not patient outcomes are impacted by identifying the presence or absence of an abscess has received little study. Three studies stated that small abscesses (eg, <0.3 mL volume) were deemed too small to drain and only received medical therapy.4 20–22 Only Sivitz et al20 investigated longer term outcomes and found that there were no return visits to the emergency department in these patients. It is unknown whether there is a size at which abscesses become clinically significant. Decisions to not drain small abscesses are based on clinical context and expert opinion. Cellulitis and abscess exist on a spectrum of disease in SSTIs and can evolve over time. Seven patients with an initial diagnosis of cellulitis without abscess remained febrile despite antibiotic treatment 72 hours after initial treatment. Six of seven patients ended up receiving an incision and drainage after a repeat ultrasound demonstrated an abscess.19 What remains unknown is what, if any, the clinical significance of these management changes are—it is possible that unrecognised abscesses treated medically with antibiotics will resolve with no sequelae. The utility of POCUS in preventing invasive procedures is more compelling, especially in paediatric populations where principles of reducing painful procedures and avoiding sedation and its associated risks are relevant.25 A study of adults demonstrated that invasive drainage was prevented most often in those with high pretest probabilities of abscess.16 Thus, a clinical approach of performing POCUS on patients before proceeding with a drainage attempt is justifiable. Further study on the impact of more accurate abscess diagnosis because of POCUS on patient-oriented outcomes is needed.

Strengths of our study include the comprehensive search strategy, reproducible protocols and adherence to systematic review methodology. The use of standardised and validated data collection and extraction tools limited bias and increased inter-rater reliability.

In summary, the evidence suggests that POCUS can accurately distinguish between cellulitis and abscess in the ED. The accuracy was similar between the adult and paediatric patient population. Further studies are needed to determine the impact of adding POCUS to the clinical assessment of patients presenting with SSTI.

Limitations

A number of issues warrant notice. Owing to the small number of included studies, assessment of publication bias is difficult. It is important to note that different protocols and different reference standards introduce heterogeneity.

A single study included in our meta-analysis, by Quraishi et al 1997, appears to be an outlier for sensitivity.19 Differences in patient populations, POCUS training or equipment may explain this variation from the other included studies. Multiple attempts to contact the authors for further information were unsuccessful.

A key element is timing: a patient presenting with an SSTI may initially have cellulitis but develop an abscess; this is especially important if there was a time lag between the index test and the reference standard. SSTI in different anatomic locations may predispose to abscess or cellulitis, as could pre-existing trauma or surgery, and there is no way to ascertain potential direction of bias.