Inter-rater reliability of the American Society of Anesthesiologists physical status rating for emergency gastrointestinal surgery.

Aim: The American Society of Anesthesiologists Physical Status (ASA-PS) classification system is used worldwide and has also been incorporated into various prediction rules. However, concerns have been raised regarding inter-rater agreement in various surgical fields. Although emergency gastrointestinal surgery is relatively common and associated with high postoperative mortality, a reliability study has not yet been undertaken in this field. The aim of the present study was to investigate the inter-rater reliability of ASA-PS for emergency gastrointestinal surgery.
Methods: Three sets of scenarios were generated for each ASA-PS class (2E, 3E, and 4E) in emergency gastrointestinal surgery, resulting in nine scenarios. These scenarios described the preoperative profiles of patients in one hospital. Two or three anesthesiologists from 18 other hospitals provided scores for ASA-PS for each scenario.
Results: Fifty anesthesiologists scored the ASA-PS class. Between 66% and 90% of these anesthesiologists assigned the same ratings as the reference ratings for the individual scenarios. Inter-rater reliability was assessed using Fleiss' kappa (95% confidence interval) of 0.55 (0.54-0.56, P < 0.001) and an intraclass correlation coefficient (95% confidence interval) of 0.79 (0.63-0.93, P < 0.001).
Conclusion: The results of the present study revealed the consistency of ASA-PS ratings between anesthesiologists for emergency gastrointestinal surgery. The ASA-PS may serve as a reliable variable in the prediction rules for this field.

Introduction

The original objective of the American Society of Anesthesiologists Physical Status (ASA‐PS) classification was to create a uniform system for statistical analyses.1 However, many modern prediction models include ASA‐PS as an independent variable.2, 3, 4, 5, 6, 7 Although the reliability of ASA‐PS is important for prediction models, previous studies reported fair to moderate inter‐rater agreement between different anesthesiologists.8, 9, 10, 11, 12, 13

Emergency gastrointestinal surgery is relatively common. According to national statistics in Japan, the number of procedures for intestinal obstruction accounts for 44% of cases of colectomy for malignant tumors. In addition to the volume, emergency gastrointestinal surgery is associated with high risks of postoperative morbidity and mortality, leading to extensive healthcare costs.14 The preoperative prediction of a patient's prognosis is of clinical importance in treatment planning and informed consent. We previously constructed a prediction rule for postoperative mortality, designated as Calculation of Post‐operative Risk in Emergency Surgery (CORES).15 The National Surgical Quality Improvement Program (NSQIP) is also used as a prediction rule in a wide variety of emergency and elective surgeries.6 Both CORES and NSQIP include ASA‐PS as an independent variable.

As patients undergoing emergency gastrointestinal surgery present with various degrees of physiological decompensation, ASA‐PS ratings in these patients may vary among anesthesiologists. Therefore, the inter‐rater reliability of ASA‐PS in this field urgently needs to be assessed, but has not yet been investigated. The present study was undertaken to evaluate the inter‐rater reliability of ASA‐PS for emergency gastrointestinal surgery.

Methods

This study was carried out in the surgery/anesthesiology network in the National Hospital Organization (NHO), Japan. Nineteen NHO hospitals participated in this study. The study protocol was approved by the Central Institutional Review Board of the NHO (Tokyo, Japan) on October 3, 2014 (H26‐1003001). Anesthesiologists in these hospitals had graduated from different universities and did not share any common background. Each hospital had between 2 and 17 anesthesiologists and managed 800 to 5500 surgical cases per year.

Three scenarios were constructed for ASA‐PS: 2E, 3E, and 4E, based on actual cases in the first author's hospital, resulting in nine scenarios for emergency gastrointestinal surgical cases (see Appendix). The first author selected scenarios from the latest three consecutive clinical cases in each category of the ASA‐PS class in the hospital in order to avoid selection bias, the surgeries for which were carried out between September 2013 and October 2014. These scenarios consisted of history, vital signs, and laboratory data and were reviewed for their compatibility by two members of the protocol committee. These scenarios were written in a Word file (Microsoft, Redmond, WA, USA) in the order of Patient 1 to Patient 9 and were delivered by e‐mail to the anesthesiologists of the 18 hospitals of the NHO, except for the hospital at which the scenarios were made. Prior to their delivery, no discussion, arrangement, or training was given on ASA‐PS ratings between the hospitals. Two or three anesthesiologists in each hospital provided ASA‐PS scores for each scenario. The ASA‐PS scores were anonymously entered into an Excel file (Microsoft, Redmond, WA, USA) and submitted to the principal investigator of the participating hospitals.

Statistical analyses were carried out using Stata 12.0 (StataCorp LP, College Station, TX, USA). A two‐tailed P‐value of 0.05 was used to define significance. We measured agreement between ASA‐PS ratings assigned by the anesthesiologists using a two‐way random intraclass correlation coefficient (ICC), as reported by Sankar and colleagues.13 Sample size was determined as 50 based on the assumption that ICC was 0.7 with a half width of 95% confidence interval of 0.2. We also analyzed the agreement between ASA‐PS ratings using Fleiss’ kappa.16 These indices ranged between 0 and 1.0, with a higher value indicating greater consistency. Landis and Koch characterized reliability statistical values of 0.00–0.20 as slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial, and values exceeding 0.80 as almost perfect.17

Results

The two members of the protocol committee independently qualified the case‐based scenarios based on the availability of the clinical data as well as information about history. Fifty anesthesiologists from 18 hospitals scored ASA‐PS ratings for the nine scenarios. The response rate was 100%. Figure 1 shows histograms of the ASA ratings obtained. Between 66% and 90% of the anesthesiologists scored the same rating for each scenario as the reference scores. The ICC (95% confidence interval) among these anesthesiologists was 0.79 (0.63–0.93, P < 0.001); Fleiss’ kappa was 0.55 (0.54–0.56, P < 0.001). There was no special tendency of ASA ratings amongst the hospitals.

Figure 1

Distribution of American Society of Anesthesiologists Physical Status (ASA‐PS) ratings assigned to nine case‐based scenarios by 50 anesthesiologists in the surgery/anesthesiology network of the National Hospital Organization, Japan. The study investigated the inter‐rater reliability of ASA‐PS for emergency gastrointestinal surgery.

Discussion

Postoperative morbidity and mortality rates for emergency gastrointestinal surgery are high. The ASA‐PS score is used as an independent variable in prediction rules in this setting.6, 15 Therefore, we determined whether the ASA‐PS rating was reliable between anesthesiologists. The results obtained showed that agreement between anesthesiologists was relatively high and, thus, reliability may be regarded as satisfactory. To the best of our knowledge, this is the first study to investigate the inter‐rater reliability of ASA‐PS in the field of emergency gastrointestinal surgery.

When ASA‐PS was originally devised in 1941,18 Saklad emphasized that its aim was not to estimate “operative risk”, but to facilitate the “tabulation of statistics”. However, many studies have been undertaken in an attempt to determine the relationship between ASA‐PS and the risk of perioperative morbidity and/or mortality.2, 3, 4, 5, 7, 14, 15, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28 Apart from a few studies,25, 26, 28 most found a strong correlation and suggested the usefulness of ASA‐PS as an independent risk factor in certain prediction models.4, 5, 14, 15, 19, 23, 24 If inter‐rater reliability is guaranteed, ASA‐PS has the potential to be a very useful factor in prediction models.

Several studies have reported the reliability or validity of ASA‐PS ratings since Owens et al. investigated the consistency of ratings in 1978.8 Most of these studies used hypothetical case scenarios for the different anesthesiologists’ ratings, and revealed only fair inter‐rater agreement. Although the case scenarios may have been made deliberately in these studies, the purpose of verifying consistency was not achieved. For example, a wide range of replies were supplied for the fourth case scenario by Hynes and Lawler (subarachnoid hemorrhage),9 with all five possible ASA grades. Their scenarios were only descriptive and included few of the objective data needed to accurately determine ASA‐PS. A prospective cross‐over study designed to determine the degree of agreement for ASA grades between two different anesthesiologists showed a moderate agreement (κ = 0.53),29 with a retrospective cohort study comparing ASA‐PS scores assigned at a preoperative assessment clinic versus the operating theatre showing higher reliability (κ = 0.61).13 These two studies were based on “real world” clinical practices.

In the present study, most anesthesiologists graded a case into the same class in all scenarios. We tested inter‐rater agreement using two measurements, Fleiss’ kappa and ICC. Fleiss’ kappa is a conservative measure of inter‐rater reliability for categorical ratings that removes the portion of agreement due to chance. An interpretation of Fleiss’ kappa has not yet been established because the number of categories and subjects affects the magnitude of the value. Ihejirika and colleagues16 analyzed the inter‐rater reliability of ASA‐PS using nine scenarios, similar to the present study, and obtained moderate agreement (κ = 0.51). They concluded that substantial agreement strength for reliability was achieved.16 The present study achieved a higher value for F‐ κ (0.55) than that reported by Ihejirika and colleagues.16 The ICC is a measure of inter‐rater reliability for quantitative ratings. However, it has sometimes been used to assess ASA‐PS.13 The value in the current study of 0.79 means “substantial”. Based on these results, we concluded that the inter‐rater agreement of ASA‐PS for emergency gastrointestinal surgery may be substantial.

There were some limitations in the present study. First, we did not use actual clinical patients, but case‐based scenarios. The scenarios lacked radiographic images and electrocardiogram data. These limited data may influence the decisions by anesthesiologists. Second, this was not a large volume cohort study, using only nine case scenarios. Furthermore, each scenario may be relatively easy to judge ASA‐PS ratings. We selected scenarios from the latest three consecutive clinical cases in each category of the ASA‐PS class in one hospital in order to prevent selection bias. However, the consistency of this study suggests that anesthesiologists are more likely to grade in a similar manner, at least for typical cases. Finally, we did not use the ASA IE scenario, which narrows the range of potential options and, thus, biases reliability upwards.

In conclusion, the ASA‐PS rating was substantial between anesthesiologists for emergency gastrointestinal surgery. Thus, ASA‐PS may serve as a reliable index in prediction rules for this field.

Conflict of Interest

None.

Appendix. Case‐based scenarios to assess the inter‐rater reliability of the American Society of Anesthesiologists Physical Status rating for emergency gastrointestinal surgery.

Click here for additional data file.