Evaluation of the diagnostic performance of a decision tree model in suspected acute appendicitis with equivocal preoperative computed tomography findings compared with Alvarado, Eskelinen, and adult appendicitis scores: A STARD compliant article.

This study evaluated the diagnostic performance of a new clinical approach based on decision tree (DT) analysis in adult patients with equivocal computed tomography (CT) findings of acute appendicitis (AA) compared with previous scoring systems.This retrospective study of 244 adult patients with equivocal CT findings included appendicitis (AG, n = 80) and non-appendicitis (NAG, n = 164) groups. The chi-squared automatic interaction detection algorithm was for AA prediction. A receiver operating characteristic curve analysis and area under the curve (AUC) were used to compare the DT analysis with Alvarado, Eskelinen score, and adult appendicitis scores (AAS).The following factors were selected for AA prediction: rebound tenderness severity, migration, urinalysis, symptom duration, leukocytosis, neutrophil count, and C-reactive protein levels. The DT comprised 11 final nodes with the following AA probabilities: node 1, 100% (16/16); node 2, 90% (9/10); node 3, 80% (8/10); node 4, 60.9% (14/23); node 5, 50% (3/6); node 6, 43.8% (7/16); node 7, 22.6% (12/53); node 8, 13% (10/77); node 9, 5.6% (1/18); node 10, 0% (0/12); and node 11, 0% (0/3). The AUC of the DT was higher (0.850 [95% confidence interval {CI}; 0.799-0.893]) than the Alvarado score (0.695 [95% CI; 0.633-0.752]), AAS (0.749 [95% CI; 0.690-0.802]), and the Eskelinen score (0.715 [95% CI; 0.654-0.770]). The results were statistically significant when compared with the AUCs of the Alvarado score, Eskelinen score, and AAS (P < .001, P < .001, P = .003, respectively).The DT-based approach facilitated AA diagnosis and determination of clinical status in patients with equivocal preoperative CT findings and ambiguous results.

Introduction

Acute appendicitis (AA) is a medical condition that requires a prompt and accurate diagnosis to ensure that morbidity, mortality, and unnecessary surgical intervention rates are as low as possible.[ In recent years, most of the emergency physicians or general surgeons have used abdominal computed tomography (CT) scans to accurately establish AA diagnosis clinically, before surgical intervention.[ However, the equivocal findings of an abdominal CT scan complicate the diagnosis and surgical decision-making in AA.[ A study suggested that the incidences of equivocal CT findings of AA are approximately 5% to 13.1%.[ Among these patients with equivocal preoperative CT findings, AA is detected in up to 30%.[ Daly et al[ suggested that equivocal CT findings involving suspected appendicitis represent a third group. These patients with symptoms carry an approximately 30% chance of appendicitis.[ In these patients, re-evaluation and reassessment using ultrasound, CT images and magnetic resonance imaging (MRI) have been reported to improve diagnostic accuracy.[ However, in most institutions, these methods are traditionally not frequently used to aid in the diagnosis of AA. Therefore, clinicians may find it challenging to establish appendicitis based on clinical presentation alone.[

Clinical scoring systems have been used to support the diagnosis of AA and reduce the use of abdominal CT scans. The Alvarado score was introduced in 1986 for its simplicity and clinical advantage.[ A previous study reported the diagnostic performance of the clinical scoring systems (Alvarado score, Eskelinen score, appendicitis inflammation score [AIR], adult appendicitis score [AAS], and Raja Isteri Pengiran Anak Saleha score [RIPASA]) in adult patients with suspected AA with equivocal preoperative CT findings. In previous studies, the diagnostic power of this clinical scoring system was moderately accurate (Alvardo score; 0.698, AIR; 0.668, RIPASA; 0.653, Eskelinen; 0.710, AAS; 0.726). In the study that evaluated the diagnostic power of these scoring systems on equivocal AA, the scores were found to be less accurate than other studies.[ Therefore, if patients with suspected AA and equivocal CT findings are classified as a third group, the previous scoring systems may not be accurate for AA diagnosis. Therefore, a new scoring system or method is needed.

The decision tree (DT) analysis is a data mining framework resembling a flowchart for the extraction of predictive models.[ The DT model facilitates analysis of relationships between different covariates, or creation of algorithms that predict a specific target variable. Recently, this model was used in a clinical setting to predict the diagnosis of various diseases by exploring the relationships between different covariates.[

Therefore, we hypothesize that a clinical approach based on a DT facilitates the diagnosis of AA with equivocal preoperative CT findings compared with the previous scoring systems. To the best of our knowledge, studies investigating the diagnostic performance of clinical approaches based on DT in adult patients with suspected AA with equivocal preoperative CT findings have yet to be reported. The aims of this study were to evaluate the diagnostic performance of a new clinical approach based on DT and compare the diagnostic performance of DT-based approaches with previous scoring systems such as the Alvarado score, Eskelinen score, and AAS in adult patients with suspected AA and equivocal preoperative CT findings.

Methods

Study design

This study was designed as a single-center, observational, retrospective study conducted between April 2011 and November 2015 after obtaining the approval (IRB no. EMC16-03) of the Institutional Review Board of Bundang Jesaeng General Hospital, Seongnam, South Korea. Informed consent was waived for the current study.

Enrolled patients

During the study period, intravenously enhanced abdominal CT scans were performed on a total of 4690 patients (15 years or older) with suspected AA to evaluate the appendix. Preoperative diagnosis was confirmed in 4427 of these patients as follows: definitive appendicitis (n = 1577), probable appendicitis (n = 346), probably not appendicitis (n = 355), and normal appendix (n = 2149). Nineteen patients with incomplete medical records could not be compared with the previous clinical scoring systems, and therefore, were not included in the study. A total of 244 adult patients (≥15 years) with suspected AA and equivocal CT findings were finally enrolled (Fig. 1).

Figure 1

Study flowchart. AG = appendicitis group, CT = computed tomography, NAG = non-appendicitis group.

Clinical scores and data collection

The clinical scoring systems based on the Alvarado, Eskelinen, and AAS were compared with the new clinical approach based on DT. The scoring systems were selected based on their performance in predicting AA in patients with equivocal CT findings in a previous study.[ The parameters used to calculate the Alvarado score, Eskelinen score, and AAS were obtained from patient medical records and laboratory report systems. Patient demographics, history, and urinalysis results were also collected. All data were collected comprehensively by a single emergency medicine physician. We defined fever as a body temperature greater than 37.3 °C. The laboratory reference values were based on the high reference values at the hospital. Leukocytosis was defined as a white blood cell (WBC) count of >10.0 (×109/L). An elevated neutrophil count was defined as higher than 80% of the total WBC count, and an elevated C-reactive protein (CRP) was defined by levels >0.8 mg/L. The Alvarado score was calculated based on the cumulative scores for nausea/vomiting (score 1), anorexia (score 1), migratory pain to the right lower quadrant (RLQ) (score 1), RLQ direct tenderness (score 2), rebound tenderness (score 1), elevated body temperature (≥37.3 °C) (score 1), elevated WBC (≥10.0 × 109/L) (score 2), and elevated neutrophil count (>75%) (score 1).[ The Eskelinen score was calculated as a sum of the total scores in the following categories: RLQ pain (3.51), migratory pain to RLQ (3.51), duration of pain (2.13), RLQ direct tenderness (11.41), rebound tenderness (4.25), guarding (6.62), and elevated WBC count (≥10.0 × 109/L) (5.88).[ The AAS was also calculated using the same method but under the following categories: RLQ pain (2), migratory pain to the RLQ (2), RLQ direct tenderness (3; men and women older than 50 years, score 1; women aged 16–49 years), rebound tenderness (2, mild; and 4, moderate to strong), elevated WBC (score 1, 7.2 ≤ WBC < 10.9 [×109/L]; score 2, 10.9 ≤ WBC < 14.0 [×109/L]; score 3, ≥14.0 [×109/L]); elevated neutrophil count (score 2, 62% ≤ neutrophil < 75%; score 3, 75% ≤ neutrophil < 83%; and score 4, ≥83%), and CRP (mg/L) (<24-h symptoms [score 2, 4 ≤ CRP < 11; score 3, 11 ≤ CRP < 25; score 5, 25 ≤ CRP < 83; score 1, CRP ≥ 83], and ≥24-h symptoms [score 2, 12 ≤ CRP < 152; score 1, CRP ≥ 152]).[

The final diagnoses were established as follows. In patients who underwent surgery, the diagnosis of AA was based on histological findings showing transmural infiltration of neutrophils in the appendix. In the case of patients who did not undergo surgery, the 2-week follow-up medical records were reviewed. Patients who were not followed up at our medical center were contacted directly via telephone, and for patients who were transferred to another medical center, the medical center was contacted directly via telephone. Enrolled patients were classified into 2 groups according to the final diagnosis: appendicitis group (AG, n = 80) and non-appendicitis group (NAG, n = 164).

Imaging methods and interpretation

A 16-slice multidetector CT scanner (Brilliance 16, Philips Healthcare, Eindhoven, the Netherlands) with intravenous administration of contrast material was used for all CT scans. Oral contrast medium was not administered in this study. A whole abdominal scan was performed from the diaphragm to the symphysis pubis. Other parameters included collimation, 1.5 mm; rotation time, 0.75 seconds; and pitch, 1.188. The reconstructed images consisted of axial and coronal sections measuring 3 to 5 mm. The tube voltage and tube current was 120 kVp and 150 to 300 mA, respectively. All CT scans were contrast-enhanced, with a 60-second delay after a starting dose of 2 mL/kg body weight of iohexol (Omnipaque 350, GE Healthcare, Princeton, NJ), and iopamidol (Pamiray 370, Dongkook Pharmaceutical, Seoul, Korea). The contrast medium was administered as an infusion through an antecubital vein at 4 mL/s.

A single abdominal radiologist with >10 years of experience in interpreting abdominopelvic CT scans retrospectively reviewed equivocal cases to identify the following CT signs of appendicitis: outer-to-outer diameter ≥6 mm (without intraluminal air), peri-appendiceal fat stranding or fluid, enhancement of the appendiceal wall, and concentric wall thickening. The presence of any one of these findings was defined as equivocal appendicitis. The diameter of the appendix was measured based on the axially enhanced sections and the largest portion of the appendix that was visible. If the appendix was not visualized or if the radiologist failed to trace the entire appendix, the patient was considered negative for appendicitis.[

Statistical analysis

All data analysis was performed using SPSS 23 statistics software (IBM, Inc., Armonk, NY) and MedCalc ver. 7.4 (MedCalc Software, Mariakerke, Belgium). Continuous data were expressed as means ± standard deviation or median (interquartile range) based on the results of normal distribution analysis. Categorical data are presented as absolute values, together with frequency distribution where appropriate.

The Mann–Whitney U test and independent t test were used to compare age and each of the clinical scores between AG and NAG according to the normal distribution analysis. The Chi-square test was used to compare the sexes of the 2 groups. A P value of <.05 was considered statistically significant.

The DT analysis was conducted using SPSS Decision Trees Version 23 (IBM, Inc., Armonk, NY). The variables used in this study were established in previous studies of AA diagnosis. The variables included sex, age, symptoms (anorexia, nausea or vomiting, migrated pain to RLQ), signs (RLQ direct tenderness, guarding, rebound tenderness, guarding, elevated body temperature), laboratory findings (elevated neutrophils and leukocystosis), and symptom duration (defined as the time between symptom onset and the CT scan). The chi-squared automatic interaction detection (CHAID) algorithm was used to create the tree models for AA prediction. The analysis was run automatically with the nodes defined as follows: parent node, 10 subjects, child nodes, 1 subject and the depth of tree in 3. The significance of the merged nodes and node splitting was set at a P value of <.05 and <.2, respectively.

The diagnostic accuracy of the new clinical approach based on DT and the previous scoring system was compared using the receiver operating characteristic (ROC) curves and the area under the curve (AUC) values plotted and calculated with MedCalc ver. 7.4 (MedCalc Software, Mariakerke, Belgium). A P value of <.05 was considered statistically significant.

The sample size based on our previous study yielded a sensitivity of 82.0%, a specificity of 53.9%, and a 24.8% incidence of appendicitis with a cut-off value of AAS 8. Based on the sample size calculation, 230 subjects were needed to achieve a precision of .10 for sensitivity, or specificity with an alpha error of .05, and a power of 80%, according to a previous study.[ Considering the study duration, a total sample size of 244 subjects was included in this study.

Results

General characteristics of enrolled patients

The patient demographics, different scoring systems, and the clinical symptoms and signs are summarized in Table 1. No significant difference was observed in sex ratio, age, nausea or vomiting, anorexia, elevated body temperature, or elevated CRP between the groups (P = .388, P = .429, P = .463, P = .880, P = .936, P = .398, respectively). Among the 102 operations performed, 21.6% (n = 22 in NAG) tested negative with appendectomy. Alternative diagnoses in patients who underwent surgery but had negative appendectomy results were as follows: acute gastroenteritis (n = 6), pelvic inflammatory disease (n = 2), diverticulitis (n = 2), appendiceal mucocele (n = 2), mesenteric lymphadenopathy (n = 2), and non-specific findings (n = 8). The diagnosis of 142 patients who did not undergo surgery was as follows: acute gastroenteritis (n = 55, 38.7%), non-specific findings (n = 32, 26.0%), pelvic inflammatory disease (n = 27, 19.0%), mesenteric lymphadenopathy (n = 8, 5.6%), diverticulitis (n = 6, 4.2%), ovarian cystic rupture (n = 4, 2.8%), acute pyelonephritis (n = 3, 2.1%), and ureteric stones (n = 2, 1.4%).

Table 1

Patient and clinical demographics under each scoring system in different groups.

DT analysis

The factors selected for DT analysis to predict AA were as follows: rebound tenderness severity, migration, urinalysis, symptom duration, leukocytosis, elevated/normal neutrophil count, and elevated/normal CRP levels. The DT comprises 11 final nodes, as shown in Fig. 2. The severity of rebound tenderness was selected in the parent node. Node 11 (n = 3, 1.2% of the total number of patients) showed absence of appendicitis in patients with grade 2 rebound tenderness severity and symptom duration >3288 minutes (54.8 hours). Patients were diagnosed with appendicitis if the node 1 showed grade 2 rebound tenderness severity and the symptom duration was <3288 minutes (54.8 hours), and normal CRP levels (n = 16, 6.6% of the total number of patients). Node 10 (n = 12, 4.9% of the total number of patients) showed that none of the patients manifesting rebound tenderness severity (grade 1), normal urinalysis results, and normal neutrophil counts had appendicitis. Node 2 (n = 10, 4.1% of the total number of patients) showed that 90% of patients with grade 1 severity of rebound tenderness, abnormal urinalysis results, and migratory pain had appendicitis. In node 9 (n = 18, 7.4% of the total number of patients), 94.4% of the patients without rebound tenderness but exhibiting migratory pain, and a lapse of >1638 minutes (27.3 hours) from the symptom onset until CT scans (17/18), showed no appendicitis. Finally, in nodes 8, 7, 6, 5, 4, and 3, the likelihood of testing negative for appendicitis was 87.0% (67/77), 77.4% (41/53), 56.2% (9/16), 50% (3/6), 39.1% (9/23), and 20% (2/10), respectively. The results for each final node are summarized in Table 2.

Figure 2

The new clinical approach using decision tree analysis for predicting acute appendicitis in patients with equivocal CT findings. AG = appendicitis group, CRP = C-reactive protein, CT = computed tomography, NAG = non-appendicitis group, RT = rebound tenderness, WBC = white blood cell count.

Table 2

Summary of each final node, in the descending order of probability of appendicitis.

ROC analysis comparing the diagnostic performance of DT-based clinical approach and clinical scores

The results of the calculated ROC curve and AUC are presented in Fig. 3. Compared with the AUC in the previous scoring systems, the AUC of the DT in this study scored higher (0.850 [95% confidence interval {CI}; 0.799–0.893]) than the Alvarado score (0.695 [95% CI; 0.633–0.752]), AAS (0.749 [95% CI; 0.690–0.802]), and Eskelinen score (0.715 [95% CI; 0.654–0.770]). The DT-based model showed a significant statistical difference compared with the AUCs of the Alvarado score, Eskelinen score, and AAS (P < .001, P < .001, P = .003, respectively).

Figure 3

The ROC curve and the AUC of each clinical score, and the new clinical approach based on decision tree analysis. The solid line represents the new clinical approach using the decision tree analysis, with an AUC of 0.850 (95% confidence interval; 0.799–0.893). The Alvarado score is represented by a dashed line, with an AUC of 0.695 (95% confidence interval; 0.633–0.752). The Eskelinen score is represented by the dash-dot-dot line, with an AUC of 0.715 (95% confidence interval; 0.654–0.770). The adult appendicitis score is represented by the dotted line, with an AUC of 0.749 (95% confidence interval; 0.690–0.802). The AUC in the decision tree analysis was statistically significant compared with other scoring systems (P > .05). AUC = area under the curve, ROC = receiver operating characteristic.

Discussion

In this study, we utilized a DT analysis combining different parameters for the accurate diagnosis of AA in adult patients with equivocal preoperative CT findings. The new clinical approach based on DT yielded high power (AUC 0.850) for the diagnosis of adult patients with suspected AA and equivocal preoperative CT findings. The DT model is simple to use in a clinical setting. Physicians can use the model by allocating patients to specific subgroups based on the probability of AA by simply following a flowchart-like tree framework.

In a previous study published by Daly et al,[ it was suggested that patients suspected of AA with equivocal CT results should be classified as a third group, and radiologists should inform the referring physician that symptomatic patients with equivocal CT scans carry an approximately 30% risk of testing positive for appendicitis.[ Under these conditions, a clinical diagnosis of appendicitis is a challenge.[ Unfortunately, the diagnostic power of the previous clinical scoring systems were less accurate (0.5 < AUC ≤ 0.7, Alvardo score; 0.698, AIR; 0.668, RIPASA; 0.653) to moderately accurate (0.7 < AUC ≤ 0.9, Eskelinen; 0.710, AAS; 0.726) clinically.[ Furthermore, 74.1% to 94.1% of patients with suspected AA were categorized under an intermediate probability group and therefore, the scoring systems were of limited use in establishing accurate diagnosis.[ However, under the new model, the results provided diagnostic and treatment decisions (surgery or discharge) in 59 (24.2%) out of 244 patients. Seventy-seven patients (31.6%) may be recommended a discharge plan with short-term follow-up in the outpatient department. While a definitive diagnosis could not be made in another 108 patients (44.3%) using the new model, it may still facilitate prognostic evaluation of these patients based on admission for observation. These results show that DT model is a useful clinical tool in deciding the disposition of patients with equivocal CT findings.

Nodes 11, 10, 9, 2, and 1 were useful in diagnosing AA among 59 patients (24.2% of the total number of patients) in our study. As seen in nodes 11, 10, and 9, 32 out of 33 patients did not manifest AA, and nodes 2 and 1, 25 out of 26 patients showed AA. The results can be categorized according to the following variables: in the case of patients without AA, the rebound tenderness severity was grade 2; however, the symptom duration was >3288 minutes (54.8 hours) (node 11 [n = 3, 1.2% of the total number of patients]), the rebound tenderness severity was grade 1, and the urinalysis and neutrophil counts were normal (node 10 [n = 12, 4.9% of the total number of patients]), and presence of rebound tenderness along with migratory pain, and a symptom duration >1638 minutes (27.3 hours) (node 9 [94.4%, 17/18]). On the other hand, patients with AA showed a rebound tenderness severity of grade 2, the symptom duration was <3288 minutes, without CRP elevation (node 1 [n = 16, 6.6% of the total number of patients]), and rebound tenderness severity grade 1, with a normal urinalysis results, and migratory pain (node 2 [90%, 9/10]). Studies show a negative AA rate of 14% to 20% in the purely clinical setting.[ As our study is based on equivocal CT findings to establish a definitive clinical diagnosis, we assumed a negative appendectomy rate similar to previous studies in purely clinical setting. Therefore, based on this negative rate in purely clinical setting, we suggest discussing surgical intervention with the patients that fall under the NAG group in node 2. The results in node 8 (n = 77, 31.6% of the total number of patients) showed a P-value of .149, which suggested limited statistical significance. However, it may still be useful in the clinical setting as it suggested that only 13% of patients without rebound tenderness and normal leukocytes had a final diagnosis of appendicitis. Therefore, patients in this group may be recommended a discharge plan with short-term follow-up in the outpatient department. On the other hand, 108 (44.3%) patients were categorized into nodes 7, 6, 5, 4, and 3 with a negative probability for appendicitis calculated as 77.4% (41/53), 56.2% (9/16), 50% (3/6), 39.1% (9/23), and 20% (2/10), respectively. Therefore, although none of these cases may require immediate surgery, it may be safe to admit patients for further observation and studies.

Interestingly, symptom duration was a useful factor in evaluation of patients with suspected AA, as seen in nodes 11, 9, and 4. The probability of appendicitis was low if the symptom duration was greater than the time selected by the individual nodes. The cut-off values in the nodes were 54.8 hours in node 11, and 27.3 hours in nodes 9 and 4. Other studies suggest that the duration of symptoms may aid in AA diagnosis.[ The criteria for the CRP levels in AAS varied according to symptom duration and the cut-off value was 24 hours after the onset of symptoms.[ Another study suggested that appendicitis may be difficult to diagnose within 12 hours from the onset of symptoms, and after 36 hours the likelihood of perforation was high, and therefore, if the symptoms lasted longer than 36 hours, the chances of appendicitis decreased.[ These results are similar to our study results, wherein a symptom duration of <27 hours and >56 hours yielded a low probability of appendicitis in adult patients with equivocal preoperative CT findings. Therefore, we suggest that symptom duration should be considered when diagnosing AA, without any generalizations based on these few studies. Further studies with larger samples are needed to support this claim.

The present study has several limitations. First, the study was designed retrospectively. Therefore, the diagnostic performance of the new clinical approach based on DT was restricted compared with the previous clinical scoring systems. Second, the study is limited by the small size of the patient population in some final nodes because the DT model was created by setting the child node to one to maximize the probability of AA diagnosis. A further large, randomized controlled study may be required to confirm the results of this study. Third, the CT findings were interpreted by only a single radiologist and therefore, had no interobserver variation, which may have led to greater selection bias in the diagnosis based on equivocal CT findings.

In conclusion, the new clinical approach based on DT presented in this study facilitated the diagnosis of AA in adult patients with suspected AA and equivocal preoperative CT findings. It is also useful in deciding the disposition of patients with equivocal results. The parameters used in the new clinical approach based on DT include rebound tenderness severity, migration, urinalysis, symptom duration, leukocytosis, elevated or normal neutrophil levels, and elevated or normal CRP levels.

Author contributions

Conceptualization: Hyo Jung Kang, Hyuncheol Kang, Seong Beom Oh, Jung Hwan Ahn.

Data curation: Hyo Jung Kang, Hyuncheol Kang, Bohyun Kim, Min Seok Chae, Young Rock Ha, Jung Hwan Ahn.

Formal analysis: Hyo Jung Kang, Hyuncheol Kang, Young Rock Ha, Jung Hwan Ahn.

Investigation: Bohyun Kim, Min Seok Chae, Jung Hwan Ahn.

Methodology: Hyo Jung Kang, Hyuncheol Kang, Bohyun Kim, Min Seok Chae, Seong Beom Oh, Jung Hwan Ahn.

Supervision: Hyuncheol Kang, Bohyun Kim, Young Rock Ha, Seong Beom Oh, Jung Hwan Ahn.

Validation: Hyuncheol Kang.

Writing – original draft: Hyo Jung Kang, Jung Hwan Ahn.