Interobserver variability in the classification of appendicitis during laparoscopy.

BACKGROUND: The intraoperative classification of appendicitis dictates the patient's postoperative management. Prolonged antibiotic prophylaxis is recommended for complex appendicitis (gangrenous, perforated, abscess), whereas preoperative prophylaxis suffices for simple appendicitis. Distinguishing these two conditions can be challenging. The aim of this study was to assess interobserver variability in the classification of appendicitis during laparoscopy.
METHODS: Short video recordings taken during laparoscopy for suspected appendicitis were shown to surgeons and surgical residents. They were asked to: classify the appendix as indicative of no, simple or complex appendicitis; categorize the appendix as normal, phlegmonous, gangrenous, perforated and/or abscess; and decide whether they would prescribe postoperative antibiotics. Inter-rater reliability was evaluated using Fleiss' κ score and the S* statistic.
RESULTS: Some 80 assessors participated in the study. Video recordings of 20 patients were used. Interobserver agreement was minimal for both the classification of appendicitis (κ score 0·398, 95 per cent c.i. 0·385 to 0·410) and the decision to prescribe postoperative antibiotic treatment (κ score 0·378, 0·362 to 0·393). Agreement was slightly higher when published criteria were applied (κ score 0·552, 0·537 to 0·568).
CONCLUSION: There is considerable variability in the intraoperative classification of appendicitis and the decision to prescribe postoperative antibiotic treatment.

Introduction

Acute appendicitis is a highly prevalent gastrointestinal disorder among both children and adults. It is the most common abdominal surgical emergency worldwide1, 2, 3, 4. Its severity can be classified into two distinct types based on operative findings: simple and complex appendicitis5, 6. Around 25–30 percent of all acute appendicitis is considered complex7, 8, 9, 10, 11, 12.

A classification to distinguish simple and complex appendicitis was recently provided by Bhangu and colleagues5. In this classification, a phlegmonous appendix is considered simple appendicitis, whereas gangrenous appendicitis, perforated appendicitis and periappendiceal abscess formation are regarded as complex appendicitis. Previous studies13, 14, 15, 16 have shown that the intraoperative assessment of the appendix frequently does not concur with the histopathological assessment. Intraoperative findings were found to be more predictive of the postoperative course (complications) than the histopathological classification13. Hence, postoperative management should probably be guided by the intraoperative classification of appendicitis.

Perioperative antibiotic prophylaxis should suffice for simple appendicitis17, whereas postoperative antibiotic treatment (or prolonged prophylaxis) is recommended for complex appendicitis18, 19. However, distinguishing simple from complex appendicitis during laparoscopy can be challenging. To date, only one study20 has evaluated interobserver variation in the intraoperative classification of acute appendicitis. The authors concluded that agreement on perforated versus non‐perforated appendicitis was poor. Such significant interobserver variation may account for variation in perioperative management and postoperative outcomes reported in the literature. An important shortcoming of that study is that static images were used. No study has yet been performed using video footage from laparoscopic appendicectomies to evaluate interobserver reliability in the classification of appendicitis.

The aim of the present study was to obtain further insight into the interobserver variability among surgeons in the intraoperative classification of appendicitis, using video fragments of laparoscopic procedures.

Methods

A cross‐sectional inter‐rater reliability study was performed to assess variation in classification of appendicitis. Short video fragments of the appendix, recorded during laparoscopy for suspected appendicitis, were constructed from patient files, and shown to surgeons and surgical residents in a survey. As very little evidence was available in the published literature to provide information for calculation of a target sample size, a pilot study was undertaken. The objectives of this pilot study were to obtain preliminary data on inter‐rater reliability and to test the face validity of the video survey system. Twenty surgeons and residents from the surgical departments of one university hospital and three teaching hospitals in the Rotterdam area participated in this pilot. Fifteen video fragments were used. From the pilot study results, it was calculated (via simulation) that a target sample size of 20 videos, each assessed by 40 different participants, should yield an adequate level of precision for a Fleiss κ estimate. To enhance participation, two video surveys were constructed, each containing ten videos. Surgeons and surgical residents in training from all regions of the Netherlands were invited to participate in this survey during a 2‐day national surgical congress that took place in May 2017 (Chirurgendagen 2017).

Video assessments

Each video assessment consisted of one or two short fragments (10–20 s) followed by three multiple choice questions. The video fragments came from patients who underwent laparoscopy for suspected appendicitis in one of the aforementioned teaching hospitals between May 2016 and May 2017. A diverse selection of appendices was shown in the videos, varying in size, colour and degree of peritonitis in their surroundings.

Outcomes

Participants were first asked to classify the appendix in the video as indicative of no appendicitis, simple appendicitis or complex appendicitis (classification 1). No definition of simple and complex appendicitis was given beforehand. Participants were also asked to rank the appendix as normal, phlegmonous, gangrenous, perforated and/or abscess (classification 2). These answers were subsequently categorized following the definition of Bhangu and colleagues (classification 3)5. Where gangrenous appendix, perforated appendix and/or abscess was identified, the answer was categorized as indicating complex appendicitis. If only normal or phlegmonous was selected, the answer was categorized as not indicating complex appendicitis (Table 1). Finally, participants were asked to choose whether they would prescribe postoperative antibiotic treatment (yes or no). After assessing the videos, participants were asked to answer some questions about local hospital protocols and their personal opinion on the indications for postoperative antibiotic treatment, and on the duration and route of administration of such treatment.

Table 1

Classification of acute appendicitis*

	Features
Simple appendicitis	Phlegmonous appendix
Complex appendicitis	Gangrenous appendix
	Perforated appendix
	Abscess (pelvic/abdominal)

Simplified from the classification system of Bhangu et al.5.

Statistical analysis

Inter‐rater reliability was evaluated using Fleiss' κ coefficient and the S* statistic21, 22, 23, 24, 25 for classification 1, classification 3 and the decision on whether to prescribe postoperative antibiotics. Because multiple answers were allowed for classification 2, these could not be considered strictly independent and no direct interobserver correlation could be calculated; only percentage agreement results are reported for this classification. κ statistics are useful for assessing reproducibility, and grossly estimating the degree of agreement between observers beyond that expected by chance alone. Fleiss' κ score is related to Cohen's κ score, and is intended for measuring reliability among more than two observers25, 26. The level of agreement is classified into six categories based on the κ score: none (0·01–0·20); minimal (0·21–0·39); weak (0·40–0·59); moderate (0·60–0·79); strong (0·80–0·90); and almost perfect (over 0·90)24. P < 0·050 indicates that the estimated κ score itself is not due to chance27.

κ statistics were calculated for the following subsets of participants: all participants, surgeons and surgical residents. The S* statistic (a weighted S index for ordinal variables21, 22) was calculated for participants specialized in abdominal/oncological surgery and those who performed appendicectomy at least once per month, owing to the varying numbers of participants per survey for these two groups.

In addition, simple descriptives were used to evaluate intraobserver concordance for each video assessment. A video classified as complex appendicitis in classification 1 should positively concur with a complex appendicitis in classification 3 (based on classification 2) and the prescription of postoperative antibiotics. Likewise, a simple appendicitis in classification 1 should concur with no complex appendicitis in classification 3 and a decision not to prescribe postoperative antibiotics.

Results

Eighty surgeons and residents from 35 different hospitals participated in the study. Twenty‐nine participants (36 per cent) worked in the Rotterdam area, 48 (60 per cent) in hospitals in other regions of the Netherlands, and the remaining three worked abroad (in Curacao, Norway and Belgium) (Table 2).

Table 2

Basic demographics of the study participants

	No. of participants (n = 80)
Experience
Surgeons	46 (58)
Operating on adults	39 (49)
Operating on children	1 (1)
Operating on both	6 (8)
Surgical trainees	34 (43)
4th to 6th year of training	12 (15)
1st to 3rd year of training	22 (28)
Differentiation
Differentiated into specialty	57 (71)
Abdominal/oncological surgery	35 (44)
Trauma surgery	9 (11)
Vascular surgery	9 (11)
Other	4 (5)
Not yet differentiated	23 (29)
Frequency of appendicectomies (per month)
Often (> 3)	33 (41)
Regularly (≥ 1)	54 (68)
Rarely (< 1)	26 (33)

Values in parentheses are percentages.

Interobserver agreement

For classification 1, the percentage agreement ranged from 53 to 98 per cent across the videos (Table S1, supporting information). A Fleiss' κ score of 0·398 reflected minimal agreement among the participants (Table 3). For classification 2, the percentage agreement ranged from 53 to 100 per cent, 50 to 100 per cent, 50 to 100 per cent, 60 to 100 per cent and 63 to 100 per cent for a normal appendix, phlegmonous appendix, gangrenous appendix, perforated appendix and appendicular abscess respectively (Table S1, supporting information). For classification 3, the percentage agreement ranged from 53 to 100 per cent (Table S1, supporting information). Interobserver agreement was weak, with a κ score of 0·552. For decision to prescribe postoperative antibiotics, the percentage agreement ranged from 55 to 100 per cent. Interobserver agreement was also minimal, with a κ score of 0·378.

Table 3

Interobserver agreement: Fleiss' κ analysis

	No. of participants per video	κ score	P
Classification 1: no, simple or complex appendicitis
All participants	40	0·398 (0·385, 0·410)	< 0·001
Surgeons	23	0·361 (0·338, 0·383)	< 0·001
Surgical trainees	17	0·459 (0·429, 0·489)	< 0·001
Classification 3: complex appendicitis or not *
All participants	40	0·552 (0·537, 0·568)	< 0·001
Surgeons	23	0·521 (0·493, 0·548)	< 0·001
Surgical trainees	17	0·608 (0·571, 0·646)	< 0·001
Decision on postoperative antibiotics: yes or no
All participants	40	0·378 (0·362, 0·393)	< 0·001
Surgeons	23	0·352 (0·324, 0·379)	< 0·001
Surgical trainees	17	0·444 (0·406, 0·481)	< 0·001

Values in parentheses are 95% confidence intervals for κ.

According to system of Bhangu and colleagues5 (Table 1).

κ scores for reliability were higher for the residents than for the surgeons (Table 3). Those for abdominal/oncological surgeons and participants who performed appendicectomy at least monthly were similar, reflecting minimal to weak interobserver agreement (Table S2, supporting information).

Intraobserver concordance

In 119 (14·9 per cent) of all 800 assessments (20 videos each assessed by 40 participants) classification 1 did not match classification 3. In 75 of the 119 (63·0 per cent), participants assessed the video as showing simple appendicitis while also ranking it as a gangrenous appendicitis. In 99 of 800 assessments (12·4 per cent) classification 1 did not match the decision whether to prescribe postoperative antibiotics. In about half of these instances, antibiotics were not prescribed, even though the rater assessed the video as showing complex appendicitis. In the other half, antibiotics were prescribed, even though the video was assessed as showing simple appendicitis.

Postoperative antibiotic treatment

Some 39 and 63 per cent of participants felt that prolonged antibiotic prophylaxis was not indicated for appendicitis with localized pus and for gangrenous appendicitis respectively (Table 4). Prolonged prophylaxis for less than 3 days was uncommon in hospital protocols (3 per cent), whereas 31 per cent of participants indicated this to be their personal preference (Table 5). The majority of participants preferred a combination of intravenous and oral administration, whereas only 35 per cent indicated this was the route of administration defined by the protocol in their hospital.

Table 4

Indications for postoperative antibiotics after appendicectomy (80 participants)

	Local hospital protocol			Personal preference
	Indicated	Not indicated	Uncertain*	Indicated	Not indicated
Appendicitis with localized pus	39 (49)	21 (26)	20 (25)	49 (61)	31 (39)
Gangrenous appendicitis	23 (29)	34 (43)	23 (29)	30 (37)	50 (63)
Perforated appendicitis	77 (96)	2 (3)	1 (1)	76 (95)	4 (5)
Appendicitis in presence of abscess	70 (88)	1 (1)	9 (11)	72 (90)	8 (10)
Appendicitis with purulent peritonitis	74 (93)	2 (3)	4 (5)	76 (95)	4 (5)

Values in parentheses are percentages.

Participants responded they were uncertain whether it was indicated in the local protocol.

Table 5

Preferred (minimum) duration of treatment and route of administration of antibiotics (80 particpants)

	Local hospital protocol	Personal preference
Duration (days)
5	32 (40)	16 (20)
3	46 (58)	39 (49)
< 3	2 (3)	25 (31)
Route
Completely intravenous	51 (64)	26 (33)
Intravenous and oral*	28 (35)	50 (63)
Missing answer	1 (1)	4 (5)

Values in parentheses are percentages.

Intravenous administration initially, switched to oral if the patient's condition allows.

Discussion

The present study demonstrated minimal interobserver agreement in the intraoperative classification of appendicitis. There was also minimal agreement on the choice whether or not to prescribe postoperative antibiotics. These results suggest that the current classification of appendicitis is highly unreliable, and that the indications for the administration of postoperative antibiotic treatment vary greatly among surgeons and surgical trainees.

In some part, a varying definition of complex appendicitis may account for the variation in classification. As confirmed in the survey results, some surgeons do not classify a gangrenous appendicitis as complex but others do. Likewise, differences in protocols and opinions may partly account for variability in the decision whether or not to prescribe postoperative antibiotic treatment. This is especially true for appendicitis with localized pus and for gangrenous appendicitis, as indicated by the participants. κ scores for reliability remained weak even after categorizing the participants' assessments according to the definition of Bhangu and colleagues5. This implies that the terms in this classification system might still be too vague. Inter‐rater reliability was slightly better for surgical residents. κ scores were consistently higher among the residents compared with all participants or surgeons only. This may be attributed to the fact that residents, although still in training, are perhaps more focused on adhering to definitions. Reliability was similarly poor for the subgroups of abdominal/oncological surgeons and participants who performed appendicectomy at least monthly, compared with all study participants. This implies that there is considerable variability in the classification even among more experienced surgeons.

If a variable simply has two clearly defined outcomes, rater reliability is likely to be high24. As soon as multiple outcome measures are in play and the distinction between them is more challenging, reliability can be affected negatively24. An accurate intraoperative classification of appendicitis requires the assessors to make fine distinctions and many factors may affect their judgement. Some smaller perforations are not easily detected, but may well be clinically relevant. Signs of necrosis in gangrenous appendicitis may be difficult to distinguish from colour changes due to vascular obstruction. The appearance of the appendix and its surroundings may change during surgery. Furthermore, the level of detail perceived by the surgeon is also dependent on the quality of the laparoscopic equipment. In most studies of appendicitis, a specific type of appendicitis is being investigated or outcome is being compared between different types. The validity of these studies, however, may be questionable owing to inaccurate classification of the appendicitis, as indicated by the present findings. For example, a previous study28 reported an increased risk of infectious complications after appendicectomy for complex compared with simple appendicitis. If postoperative management depends on the surgeon's intraoperative classification of appendicitis, which seems to be arbitrary, these results may not be valid. This was also stated by Ponsky and co‐workers20 in 2009. They reported considerable variability, comparable to the present results (interclass coefficient 0·27–0·36 (interpreted in the same way as κ score) for distinguishing perforated from non‐perforated appendicitis). Their conclusion was that the available ICD classification was too limited and more objective assessment points should be defined.

In the present study, agreement improved slightly after converting the participants' answers according to Bhangu and colleagues' more detailed definition of complex appendicitis. Inter‐rater reliability remained weak, however, suggesting that the diagnosis of complex appendicitis would still be unreliable even if surgeons adhered strictly to this definition. An intraoperative classification tool consisting of more clear‐cut objective factors could perhaps improve interobserver agreement in classification and postoperative management. However, it is questionable whether any intraoperative assessment will be reliable enough. Perhaps the emphasis should not rely (solely) on intraoperative findings, but on more quantifiable variables, such as serum C‐reactive protein and white blood cell count29. Several radiological and laboratory factors have been associated with adverse outcomes after appendicectomy30, 31, 32, 33, 34, 35. A combination of these and intraoperative findings may result in more consistent postoperative management36.

Interestingly, one‐third of the participants in this study would prefer to restrict postoperative antibiotics to fewer than 3 days after appendicectomy, whereas only 3 per cent indicated that this was standard practice at their hospital. Moreover, 40 per cent of the participants indicated that the standard duration of treatment at their hospital was 5 days. This implies that prolonged antibiotic prophylaxis could be reduced substantially if the decision on duration was left to the surgeons themselves. This is an interesting thought, taking into account hospital costs and the alarming emergence of antimicrobial resistance worldwide that warrants optimization of antibiotic use.

The present study was limited by showing the participants only 10–20 s of video footage on which to base their classification. In reality, the surgeon has the entire operation to decide on the type of appendicitis and postoperative treatment. This may have resulted in underestimation of rater reliability. Agreement on the classification and postoperative treatment may have been better if it were tested under circumstances better resembling the real situation. A follow‐up study incorporating more and/or longer videos, according to a standardized format that specifies the required content of the video fragments, could be interesting.

Disclosure

The authors declare no conflict of interest.

Table S1 Percent agreement analysis

Table S2 Kappa coefficient analysis per subgroup

Click here for additional data file.