Reliability of the Modified Clavien-Dindo-Sink Complication Classification System in Pediatric Orthopaedic Surgery.

BACKGROUND: There is no standardized complication classification system that has been evaluated for use in pediatric or general orthopaedic surgery. Instead, subjective terms such as major and minor are commonly used. The Clavien-Dindo-Sink complication classification system has demonstrated high interrater and intrarater reliability for hip-preservation surgery and has increasingly been used within other orthopaedic subspecialties. This classification system is based on the magnitude of treatment required and the potential for each complication to result in long-term morbidity. The purpose of the current study was to modify the Clavien-Dindo-Sink system for application to all orthopaedic procedures (including those involving the spine and the upper and lower extremity) and to determine interrater and intrarater reliability of this modified system in pediatric orthopaedic surgery cases.
METHODS: The Clavien-Dindo-Sink complication classification system was modified for use with general orthopaedic procedures. Forty-five pediatric orthopaedic surgical scenarios were presented to 7 local fellowship-trained pediatric orthopaedic surgeons at 1 center to test internal reliability, and 48 scenarios were then presented to 15 pediatric orthopaedic surgeons across the United States and Canada to test external reliability. Surgeons were trained to use the system and graded the scenarios in a random order on 2 occasions. Fleiss and Cohen kappa (κ) statistics were used to determine interrater and intrarater reliabilities, respectively.
RESULTS: The Fleiss κ value for interrater reliability (and standard error) was 0.76 ± 0.01 (p < 0.0001) and 0.74 ± 0.01 (p < 0.0001) for the internal and external groups, respectively. For each grade, interrater reliability was good to excellent for both groups, with an overall range of 0.53 for Grade I to 1 for Grade V. The Cohen κ value for intrarater reliability was excellent for both groups, ranging from 0.83 (95% confidence interval [CI], 0.71 to 0.95) to 0.98 (95% CI, 0.94 to 1.00) for the internal test group and from 0.83 (95% CI, 0.73 to 0.93) to 0.99 (95% CI, 0.97 to 1.00) for the external test group.
CONCLUSIONS: The modified Clavien-Dindo-Sink classification system has good interrater and excellent intrarater reliability for the evaluation of complications following pediatric orthopaedic upper extremity, lower extremity, and spine surgery. Adoption of this reproducible, reliable system as a standard of reporting complications in pediatric orthopaedic surgery, and other orthopaedic subspecialties, could be a valuable tool for improving surgical practices and patient outcomes.

The evaluation of outcomes following surgery, including complications, is critical to improving the quality of patient care. Patients, health-care providers, and health-care payers ultimately benefit from a simple, objective, and reliable tool to assess surgical complications.

Clavien et al. developed a classification system for use in general surgery, consisting of 4 grades of surgical complications[1]. To reduce subjectivity, Clavien et al. focused on grading the severity of complications on the basis of the therapy required to treat the complication[1]. In 2004, Dindo et al. built on this system by adding a fifth grade, including the categorization of “disability,” removing “hospital stay,” making “organ failure” a more severe complication, and establishing the validity and reliability of this system[2]. This modified system, known as the Clavien-Dindo system, is widely used in general surgery for quantifying patient outcomes[2,3]. In 2012, Sink et al. adapted the Clavien-Dindo system for use in hip-preservation surgery[4]. Sink et al. used 5 grades, based on the treatment that the complication required and its long-term morbidity, and included specific examples of hip-related complications for each grade (Table I). The Clavien-Dindo-Sink system demonstrated high interrater and intrarater reliability for grading complications following hip-preservation surgery. That system has routinely been used in studies evaluating complications following hip-preservation surgery[5-15] and has been applied to total knee arthroplasty[16].

Although the Clavien-Dindo-Sink[4] classification system is increasingly being used across orthopaedics, to our knowledge its performance has not been evaluated in general orthopaedics or, more specifically, in pediatric orthopaedic surgery. The purposes of the current study were (1) to modify the Clavien-Dindo-Sink complication classification system such that it could be applied to all types of orthopaedic surgery (including spine surgery as well as upper and lower extremity surgery) and (2) to determine the accuracy and interrater and intrarater reliability of this modified system for grading complications related to pediatric orthopaedic surgery.

Materials and Methods

The Clavien-Dindo-Sink classification system was minimally modified to include examples from various orthopaedic conditions and anatomical locations. Grade-I complications are those that do not result in deviation from routine follow-up, are of minimal clinical relevance, and can be treated with simple therapy such as antiemetics, antipyretics, analgesics, diuretics, electrolytes, oral antibiotics, and physical therapy. Examples of such complications include postoperative fever, nausea, constipation, minor urinary tract infection, asymptomatic Grade-I or II heterotopic ossification, and wound issues not requiring a change in postoperative care.

Grade-II complications are those that result in deviations from the normal postoperative course, including unplanned or additional clinic or office visits, that can be treated on an outpatient basis without additional invasive, radiographic, or surgical procedures. Examples of such complications include superficial wound infection necessitating additional clinic visits; transient neurapraxia, resulting from positioning or surgical retraction, that resolves under close observation; nerve palsy requiring bracing and closer observation with complete resolution; delayed union; and deep-vein thrombosis necessitating anticoagulation.

Grade-III complications are those that require surgical, endoscopic, or radiographic interventions, or an unplanned hospital admission. Examples of such complications include nonunion, postoperative fracture, deep infection, surgical hematoma, clinically important heterotopic ossification requiring surgical excision, and admission for intravenous antibiotics or anticoagulation.

Grade-IV complications were previously described as those that were life-threatening, that required admission to the intensive care unit, that were not treatable (with the potential for long-term disability), or that warranted an organ resection, joint resection, or joint replacement. Grade-IV complications were redefined as those that were life or limb-threatening (not just life-threatening) and were separated into 2 subcategories: Grade IVa (indicating no long-term disability) and Grade IVb (indicating a long-term disability). Examples of such complications include osteonecrosis, permanent nerve injury, major vascular injury, pulmonary embolus, central nervous system complications, and permanent organ dysfunction. Grade V indicates death of the patient (Table II).

A survey consisting of clinical pediatric orthopaedic scenarios was prepared on the basis of the known complication profiles of various conditions and case scenarios from the literature[17-25]. A grade was assigned to each complication scenario with use of the modified Clavien-Dindo-Sink scale. All grades were approximately equally represented in the sample scenarios. Each scenario included the age and sex of the patient, type of surgery, complication presentation and course, treatment, and long-term outcome (see Appendix).

Surveys were presented to 2 different surgeon populations to test the internal and external reliability of the modified Clavien-Dindo-Sink classification system. To test internal reliability, 45 case scenarios were presented to 7 local fellowship-trained pediatric orthopaedic surgeons at 1 center. These raters reviewed the case scenarios in paper format and were permitted to reference the classification guidelines, including examples, while they evaluated each scenario. A minimum of 2 weeks later, these 7 raters were asked to rate the same 45 scenarios in a different random order. The answers were then transferred to an Excel database (Microsoft).

To test external reliability, 48 case scenarios were presented to 15 pediatric orthopaedic surgeons across the United States and Canada. The raters had a variety of subspecialty interests, including pediatric spine, sports, upper extremity, hip, foot and ankle, and neuromuscular disorders. The 15 raters had the option of reviewing the scenarios either in paper format or through an electronic survey (SurveyMonkey.com) and were again permitted to reference the classification guidelines, including examples, while evaluating each scenario. A minimum of 2 weeks after the first round of responses had been collected, a second survey presenting the same 48 case scenarios in a different random order was administered. The answers were exported into an Excel database.

All raters were trained to use the modified complication classification system by reading a description of the system and reviewing 6 sample scenarios with an answer key. The raters were provided with the opportunity to ask questions.

The proportion of responses that matched the known correct classification for each scenario, as assigned by the author who originally modified the Clavien-Dindo system for use in orthopaedic surgery (E.L.S.), was assessed. Interrater and intrarater reliability were calculated for both the internal and external reliability groups. Statistical analysis was performed in a blinded manner such that the names of the surgeon-raters were blinded. Sample sizes of 45 and 48 scenarios were evaluated by the 7 internal and 15 external raters, respectively. The sample size was constructed on the basis of the similar study by Sink et al. assessing the Clavien-Dindo-Sink classification system in hip-preservation surgery[4]. The Fleiss kappa (κ) method for multiple raters was used to determine interrater reliability, and weighted Cohen kappa (κ) analysis was used for intrarater reliability (with 95% confidence intervals [CIs]), with use of SAS software (version 9.4; SAS Institute). According to the system described by Altman, κ values of 0.60 to 0.80 indicate good agreement and κ values of >0.80 indicate excellent agreement[26].

Results

Internal Reliability

The overall κ for interrater reliability (and standard error) was 0.76 ± 0.01 (p < 0.0001). The κ by grade was good to excellent, ranging from 0.61 for Grade IVa to 1.00 for Grade V (Table III). The weighted κ value for intrarater reliability was excellent for all raters, ranging from 0.83 (95% CI, 0.71 to 0.95) to 0.98 (95% CI, 0.94 to 1.00) (Table IV).

External Reliability

The overall κ value for interrater reliability (and standard error) was 0.74 ± 0.01 (p < 0.0001). The κ by grade ranged from 0.53 for Grade I to 1.00 for Grade V (Table III). The weighted κ value for intrarater reliability for the external raters was also excellent, ranging from 0.83 (95% CI, 0.73 to 0.93) to 0.99 (95% CI, 0.97 to 1.00) (Table IV). The percentage of correct ratings by internal and external raters varied from 65.3% for Grade I to 100.0% for Grade V (Table V).

Discussion

The purpose of the present study was to determine the interrater and intrarater reliability of the Clavien-Dindo-Sink classification system when applied to a variety of pediatric orthopaedic complications. Overall intrarater reliability was excellent in both the internal and external pediatric orthopaedic surgeon test groups. Interrater reliability was good for both the internal and external test groups.

The definition and classification of the severity of a complication following surgery should be objective and reproducible. In defining a complication, Clavien et al. stated that a complication is an unexpected occurrence in the course of surgery and recovery that induces change in the management of the patient[1]. Clavien et al. considered complications to be separate entities from expected sequelae of surgery, which are the inherent results of surgery (such as a scar) or the failure to achieve the intended goals of surgery[1]. Both Dindo et al.[2] and Clavien et al.[3] discouraged the use of the terms minor and major to describe complications as these terms are inherently subjective and do not describe how the complication affects the management of the patient or the potential for long-term morbidity. For example, a postoperative sciatic neurapraxia that completely resolves in 6 weeks with no long-term morbidity or effect on patient outcomes would be called a major complication in many studies. In the Clavien-Dindo-Sink system, it would be classified as a Grade-II complication as it does not require any invasive procedures or a return to surgery and, with complete recovery, does not affect long-term function. Therefore, in this example, a complication that may subjectively be termed as major requires only observation. If the nerve does not recover and requires exploration, it would be classified as Grade IV, with the potential for permanent disability. If the nerve deficit can be treated such that there is minimal appreciable functional deficit, then it would be classified as Grade IVa, without serious disability. If not correctable with tendon transfers, nerve-grafting, or other techniques, it would be classified Grade IVb, with permanent disability. The grading is reflective of the impact the complication has on the patient’s life as well as the burden on the medical system to manage the complication.

With the current emphasis on increasing quality objectives (including the improvement of patient outcomes and satisfaction) as well as decreasing costs, the classification of complications is of great interest. In a review of the general surgery literature, Rosenthal et al. examined the reporting of complications in 43 surgical randomized controlled trials that were published in 2010 in 3 surgical journals[27]. Exact definitions of intraoperative complications were provided in 13% of studies, and exact definitions of postoperative complications were provided in 50% of studies[27]. A variety of classification systems were used, most of which classified complications according to severity. The authors found that a classification system was used for intraoperative and postoperative complications in 9% and 54% of studies, respectively[27].

Specific to the orthopaedic surgery literature, Goldhahn et al. performed a systematic review of 112 randomized controlled orthopaedic surgical trials published in 5 major orthopaedic journals in an 18-month period[28]. Although two-thirds of the trials included complications as trial outcomes, clear definitions of anticipated complications were provided in only 8 trials. Among the trials in which complications were reported, impaired function was not explicitly considered and reported as a complication 93% of the time, and an unexpected reoperation was considered a complication in only 50% of fracture-treatment trials.

Both of those reviews[27,28] highlight the importance of a standard, objective system to categorize complications arising as a result of surgical intervention. Thus far, such a classification system is not used consistently in the pediatric orthopaedic surgical literature. Sink et al. modified the Clavien-Dindo complication classification system that originally had been validated for use in general surgery and applied it to hip-preservation surgery[4]. We further modified his system by adding a distinction between serious complications with and without long-term disability (Grades IVa and IVb), and the case examples were expanded for use in spine as well as upper and lower extremity orthopaedic surgery. In the present study, the interrater and intrarater reliability of this modified classification system was investigated with use of pediatric orthopaedic case scenarios. Good to excellent interrater and excellent intrarater reliability were demonstrated with this modified system when tested both internally and externally. Interrater and intrarater reliability were slightly higher in the internal test group than the external test group. It is possible that the lower reliability values in the external group may be attributed to the longer lag time in obtaining responses. Data were obtained from raters in the internal reliability group within a month after they had been trained on the use of the modified classification system, whereas some raters in the external reliability group responded almost 6 months after initially receiving training and survey administration. As the raters were fellowship-trained, board-certified surgeons at multiple, diverse orthopaedic centers, the modified Clavien-Dindo-Sink classification system should be reproducible among surgeons with varying training and work environments. This modified classification system is a clinically relevant improvement on the Clavien-Dindo-Sink system as it differentiates between limb-threatening complications (Grade IV) that result in long-term disability (Grade IVb) and those that do not (Grade IVa). The modified system can be applied to any orthopaedic anatomical location and age group, making it universally applicable in the field of orthopaedic surgery.

There is great interest in the orthopaedic community in having a reliable and validated complication classification system. The Clavien-Dindo-Sink classification system already has been used to classify complications in multiple peer-reviewed publications[5-14,29-34]. At some centers, the classification system has been built into electronic medical records and is becoming part of standard documentation (unpublished personal communication).

The present study had some limitations. First, although the classification system should be applicable to spine and upper and lower extremity cases regardless of patient demographics, the system was only tested by having pediatric orthopaedic surgeons assess pediatric orthopaedic case scenarios. Second, the study aimed to present a varied list of complications by body part, severity, and type (nerve injury, infection, technical error, and intraoperative fracture), but it was not possible to provide examples for every anatomical location or complication that exists. Additionally, this classification system does not capture the number of unplanned interventions required to treat the complication or the time period over which management of the complication occurred. Although we have been able to report on reliability, we have not evaluated the validity of the modified Clavien-Dindo-Sink system. The original Clavien-Dindo classification system has been validated[2]; future work could focus on validation of the modified Clavien-Dindo-Sink classification system. Finally, this classification system does not capture differences in surgeon preferences in managing a complication. Despite these stated limitations, this system shows sufficient reliability for use as a standard research outcome tool. This system should be easily applicable to all orthopaedic age groups and subspecialties. At this time, we are not advocating for the use of this system for the purposes of billing or compensation.

Given the good intrarater and excellent interrater reliability and simplicity of this modified classification system, adoption of this grading scheme as a standard of reporting complications in orthopaedic surgery should be considered. This classification system allows for a more meaningful rating of the complications that may occur after surgery. It provides information on how the complication impacted the patient’s life by assessing the level of further treatment required and whether or not there was a persistent disability. A reproducible, reliable system to assess orthopaedic surgical complications—rather than simply grading complications as major or minor—is of value to patients, providers, and researchers. Each of the scenarios included in the present study described 1 complication to test the application of this modified classification system on adverse events of differing severity. However, in reality, a given procedure can have a number of complications, which may change during the follow-up period. We recommend that each complication should be graded with the Clavien-Dindo-Sink system; thus, any given patient may have >1 rating, and these ratings can change over time, depending on the duration of follow-up. This system would be useful in reports of research evaluating treatment, where in addition to patient-reported outcomes, authors should list individual complications as well as their Clavien-Dindo-Sink grade. The use of this classification system would inform the reader about how frequently the treatment of interest results in long-term morbidity and what additional treatments may be required. This classification system may be a valuable tool for comparing surgical outcomes, and through research and quality-improvement initiatives, may contribute to improvements in surgical practices, ultimately improving patient care.

Appendix

A table showing the adapted grading system and examples of the clinical scenarios are available with the online version of this article as a data supplement at jbjs.org (http://links.lww.com/JBJSOA/A71).

TABLE I

Clavien-Dindo-Sink Complication Classification System*

Grade	Definition	Specific Complications
I	A complication that requires no treatment and has no clinical relevance; there is no deviation from routine follow-up during the postoperative period; allowed therapeutic regimens include antiemetics, antipyretics, analgesics, diuretics, electrolytes, antibiotics, and physiotherapy	Asymptomatic Grade-I or II heterotopic ossification, postoperative fever, nausea, constipation, minor UTI; wound problems not requiring a change in postoperative care
II	A deviation from the normal postoperative course (including unplanned clinic visits) that requires outpatient treatment: either pharmacological or close monitoring as an outpatient	Superficial wound infection (additional clinic visits), transient neurapraxia from positions or surgical retraction that resolves under observation, nerve palsy requiring bracing and close observation (complete resolution), trochanteric delayed union
III	A complication that is treatable but requires surgical, endoscopic, or radiographic procedure(s), or an unplanned hospital readmission	Trochanteric nonunion, fracture, deep infection, surgical hematoma, heterotopic ossification requiring surgical excision, DVT (requiring admission and anticoagulation)
IV	A complication that is life-threatening, requires ICU admission, or is not treatable with potential for permanent disability; a complication that requires organ resection (THA)	Osteonecrosis, permanent nerve injury, major vascular injury, pulmonary embolism, CNS complications, organ dysfunction
V	Death

UTI = urinary tract infection, DVT = deep-vein thrombosis, ICU = intensive care unit, THA = total hip replacement, and CNS = central nervous system. (Reproduced, with modification, from: Sink EL, Leunig M, Zaltz I, Gilbert JC, Clohisy J; Academic Network for Conservational Hip Outcomes Research Group. Reliability of a complication system for orthopaedic surgery. Clin Orthop Rel Res. 2012 Aug;470[8]:2220-6. Reproduced with permission.)

TABLE II

Modification of the Clavien-Dindo-Sink Complication Classification System*

Grade	Definition	Examples
I	A complication that does not result in deviation from routine follow-up in the postoperative period and has minimal clinical relevance and requires minimal treatment (e.g., antiemetics, antipyretics, analgesics, diuretics, electrolytes, antibiotics, and physiotherapy) or no treatment	Postoperative fever, nausea, constipation, uncomplicated urinary tract infection, asymptomatic Grade-I or II heterotopic ossification, wound issue not requiring a change in postoperative care
II	A deviation from the normal postoperative course (including unplanned clinic/office visits) that requires outpatient treatment, either pharmacological or close monitoring as an outpatient	Superficial wound infection (additional clinic visits), transient neurapraxia from positioning or surgical retraction that resolves under observation, delayed union, nerve palsy requiring bracing and close observation, deep-vein thrombosis requiring outpatient anticoagulation that resolves entirely
III	A complication that is treatable but requires surgical, endoscopic, or interventional radiology procedure(s), or an unplanned hospital readmission	Nonunion, postoperative fracture treatable with surgery, deep infection treatable with IV antibiotics, surgical hematoma requiring drainage or debridement, heterotopic ossification requiring surgical excision
IVa	A complication that is life or limb-threatening, and/or requires ICU admission, a complication with potential for permanent disability but treatable, a complication that may require organ/joint resection/replacement. No long-term disability	Permanent nerve injury treatable with tendon transfers, pulmonary embolus requiring ICU admission with full recovery, compartment syndrome treatable with fasciotomy, brain hemorrhage with temporary hemiplegia, renal failure requiring temporary dialysis, ICU admission for fat embolism with no long-term disability. No long-term disability
IVb	A complication that is life or limb-threatening, and/or requires ICU admission, a complication that is not treatable, a complication that requires organ/joint resection/replacement or salvage surgery. With long-term disability	Osteonecrosis requiring hip replacement, permanent nerve injury with foot drop requiring long-term bracing, major vascular injury requiring amputation, brain hemorrhage with resultant permanent hemiplegia, kidney failure requiring permanent dialysis. With long-term disability
V	Death

IV = intravenous, and ICU = intensive care unit.

TABLE III

Interrater Reliability

Grade	Kappa	Standard Error	P Value
Internal test group
I	0.76	0.03	<0.0001
II	0.69	0.03	<0.0001
III	0.73	0.03	<0.0001
IVa	0.61	0.03	<0.0001
IVb	0.77	0.03	<0.0001
V	1.00	0.03	<0.0001
Overall	0.76	0.01	<0.0001
External test group
I	0.53	0.01	<0.0001
II	0.63	0.01	<0.0001
III	0.73	0.01	<0.0001
IVa	0.62	0.01	<0.0001
IVb	0.92	0.01	<0.0001
V	1.00	0.01	<0.0001
Overall	0.74	0.01	<0.0001

TABLE IV

Intrarater Agreement

		95% CI
Rater	Cohen Kappa (κ)	Lower Bound	Upper Bound
Internal test group
1	0.93	0.86	1.00
2	0.91	0.81	1.00
3	0.85	0.76	0.94
4	0.94	0.88	1.00
5	0.98	0.94	1.00
6	0.98	0.93	1.00
7	0.83	0.71	0.95
External test group
1	0.83	0.73	0.93
2	0.90	0.84	0.96
3	0.84	0.75	0.93
4	0.88	0.81	0.95
5	0.83	0.74	0.91
6	0.98	0.95	1.00
7	0.88	0.80	0.96
8	0.90	0.83	0.98
9	0.88	0.79	0.97
10	0.99	0.97	1.00
11	0.93	0.87	0.98
12	0.88	0.80	0.97
13	0.89	0.81	0.97
14	0.91	0.84	0.98
15	0.89	0.82	0.96

TABLE V

Percentage of Correct Ratings by Internal and External Raters

Grade	Responses	Number Correct	Percent Correct
I	176	115	65.3%
II	176	146	83.0%
III	176	161	91.5%
IVa	162	122	75.3%
IVb	176	158	89.8%
V	169	169	100.0%