Accuracy of surgical complication rate estimation using ICD-10 codes.

BACKGROUND: The ICD-10 codes are used globally for comparison of diagnoses and complications, and are an important tool for the development of patient safety, healthcare policies and the health economy. The aim of this study was to investigate the accuracy of verified complication rates in surgical admissions identified by ICD-10 codes and to validate these estimates against complications identified using the established Global Trigger Tool (GTT) methodology.
METHODS: This was a prospective observational study of a sample of surgical admissions in two Norwegian hospitals. Complications were identified and classified by two expert GTT teams who reviewed patients' medical records. Three trained reviewers verified ICD-10 codes indicating a complication present on admission or emerging in hospital.
RESULTS: A total of 700 admissions were drawn randomly from 12 966 procedures. Some 519 possible complications were identified in 332 of 700 admissions (47·4 per cent) from ICD-10 codes. Verification of the ICD-10 codes against information from patients' medical records confirmed 298 as in-hospital complications in 141 of 700 admissions (20·1 per cent). Using GTT methodology, 331 complications were found in 212 of 700 admissions (30·3 per cent). Agreement between the two methods reached 83·3 per cent after verification of ICD-10 codes. The odds ratio for identifying complications using the GTT increased from 5·85 (95 per cent c.i. 4·06 to 8·44) to 25·38 (15·41 to 41·79) when ICD-10 complication codes were verified against patients' medical records.
CONCLUSION: Verified ICD-10 codes strengthen the accuracy of complication rates. Use of non-verified complication codes from administrative systems significantly overestimates in-hospital surgical complication rates.

Introduction

The Institute of Medicine's seminal report1 on medical errors initiated safety awareness and implementation of preventive patient safety strategies. Patient harm remains a challenge in healthcare and up to 35 per cent of patients are exposed to complications during their hospital stay2. A majority of identified complications (over 65 per cent) are attributed to surgical care3, 4, 5.

A number of methods have been used to detect adverse events, patient harm or complications. These include prospective observation of unfolding care processes6, the Clavien–Dindo classification of complications7, incident reporting8, and retrospective review of patient records, such as the Harvard method9 and the Global Trigger Tool (GTT) developed by the Institute for Healthcare Improvement (IHI)10. Under‐reporting of complications in incident reporting systems remains a challenge11. Full record review is thought to identify most complications, with the GTT method revealing ten times more complications than other methods12. The GTT involves searching for ‘trigger’ words that can indicate a complication (such as decubitus, intubation, naloxone), tracking changes over time13, and studying the effect of new interventions to improve patient safety14. The GTT is labour‐intensive, and therefore mostly recommended for internal use. A less resource‐demanding alternative is to use electronically extracted disease and complication codes from hospital administrative data that have already been entered into hospital databases15, 16.

ICD‐9 and ICD‐10 have been used by more than 100 countries, and contributed to more than 20 000 scientific publications17. In Norway, it has been mandatory to use the ICD‐10 system since 1999. Discharging physicians have to code diseases and complications that are detected in patient records and hospital administrative systems. The codes are frequently also used for reimbursement. Comparing data on complications across nations based on ICD‐10 codes is common, but, owing to variation in coding practices and poor quality of registered data, caution in interpreting patterns and comparisons is advised18.

Surgical complications often have a significant personal, family, economic and thus wider societal impact. Reliable knowledge of codes indicating complications, and methods to apply them, are warranted. Concerns have been raised regarding the reliability and validity of different diagnostic codes, such as those for venous thromboembolism19, stroke20, 21, sepsis22, infections23 and myocardial infarction24.

Consistent knowledge of surgical complications may inform and could influence healthcare policies and facilitate future safety targets. The aim of the present study was to investigate the accuracy of using ICD‐10‐coded surgical complications compared with the GTT as a reference standard, by conducting a concurrent validation study of ICD‐10‐coded complications. The ICD‐10 classification system and the GTT method were chosen as they are well established nationally and globally. The hypothesis was that ICD‐10 codes identifying complications, as currently used, overestimate actual procedure‐related complications, especially as those present on admission are not distinguished from complications that arise during the hospital stay.

Methods

This observational study with prospective data collection investigated perioperative complications in two Norwegian hospitals: one tertiary teaching hospital (referral for 1·1 million inhabitants) and one community hospital (referral for 110 000 inhabitants). A sample of surgical admissions was drawn randomly from a larger group comprising various surgical procedures. Adult surgical patients (aged at least 18 years) admitted for hospital care (lasting at least 24 h) between November 2012 and March 2015 were included from the two hospitals. Exclusion criteria were: rehabilitation admissions, ambulatory patients, donor surgery and patients who declined to participate in the study. The study was approved by the Western Norway Regional Ethical Research Committee (2012/560/REK West) and the data privacy unit at the central community hospital (Ref: 2012/3060). The study protocol was registered in ClinicalTrials.gov (NCT01872195).

Global Trigger Tool

The GTT was used to identify complications in patients' medical records. GTT‐identified complications are covered by the IHI's definition of an adverse event: ‘an unintended physical injury resulting from or contributed to by medical care that requires additional monitoring, treatment or hospitalization, or has a fatal outcome’13. The GTT method involves a two‐stage review process performed by nurses and physicians. Reviewers searched for ‘trigger’ words that may or may not indicate patient harm. The Norwegian GTT protocol based on the IHI guidelines was followed13. Two GTT teams investigated patient records to identify any word from 55 predefined trigger words that could indicate patient harm. A positive trigger word led the two teams to classify the occurrence of complications from a list of 23 categories. Both teams consisted of registered nurses with clinical experience ranging from 7 to 35 years, and experience with use of the GTT ranging from beginner to 5 years. One team included a senior anaesthetist and the other a surgeon. The members of the two teams received a joint 2‐h educational session delivered by two doctors experienced in use of the GTT. According to the GTT protocol, the teams reviewed medical summaries, medication logs, laboratory results, prescriptions, surgical procedural records, anaesthesia records, nursing registrations, discharge records, ICD‐10 codes and other relevant documentation.

Severity of complications identified by the GTT was classified according to the international GTT template that is used routinely by Norwegian hospitals (not only as part of the present study): E, temporary harm – additional monitoring or treatment needed; F, temporary harm – initial or extended hospital stay; G, permanent harm; H, life‐supporting treatment needed; and I, death25. In admissions with several GTT‐identified complications describing the same injury, the complication contributing to the injury was allocated a severity level. An example is postoperative bleeding resulting in reoperation: this was analysed as one complication (bleeding) with one severity level (F).

ICD‐10 complication codes

Primary outcomes were complications during in‐hospital care. A complication was defined as an adverse outcome: ‘an unintended and undesired occurrence in the healthcare process, which causes harm to the patient’26. The ICD‐10 codes indicating complications were identified by using complications as classified by the American College of Surgeons' National Surgical Quality Improvement Program27 and studies investigating surgical complications28, 29, 30. Based on previous research publications on checklists and surgical complications, 154 ICD‐10 complication codes were included in this study (Table S1, supporting information).

The codes investigated were extracted electronically from patient medical records using the hospital administrative data systems for routinely collected data. All patient records with any identified ICD‐10 complication code were reviewed to verify whether the ICD‐10 complication code was already linked to the patient's condition at the time of admission or arose during the hospital stay. A complication resulting from a previous admission rather than the present one was not included as a complication in the admission analysed in the present study. Three clinical researchers (an intensive care nurse, a nurse anaesthetist and a senior intensivist), different from the GTT teams, independently reviewed the patient's medical records and verified the codes as indicative of a complication already being present on admission, or one that emerged during the hospital stay and/or at discharge. Admissions with one or two complications were classified by a single reviewer. All admissions with three or more complications were discussed between all three reviewers, and consensus was obtained to ensure agreement in number and types of complications. The ICD‐10 complication code reviewers and the GTT record review teams were blinded to each other's reviews.

Reliability and validity

Reliability was assessed for both teams classifying complications using the GTT method in the same 20 random medical records. After classification, agreement on the presence of a complication, numbers of complications and levels of severity was tested. In addition, three clinical researchers, with no involvement in the GTT classification, reviewed the same discharge ICD‐10 codes in 30 new random medical records. The agreement on patients having a complication or not during the hospital stay and number of complications was tested.

In the second phase, concurrent validity31 was studied, comparing complications using the two different methods: GTT (reference standard) and ICD‐10 complication codes. Validation here refers to agreement in identifying complications in the same admissions using the two different methods32.

Statistical analysis

Sample size calculations were based on the assumption that 14 per cent of the study population would acquire a complication in hospital according to ICD‐10 codes, based on available evidence28, 30. Because patient record review is expected to reveal more complications12, it was further assumed that, if an ICD‐10 complication code were attributed to an admission, the risk of identifying a complication according to the GTT (patient harm of category E, F, G, H, I) would be twice the risk had no such code been present. Based on these assumptions, to obtain 90 per cent power and a significance level of 5 per cent, inclusion of at least 636 patient admissions was required.

A Venn diagram was used to illustrate associations between surgical complications identified by ICD‐10 codes and GTT reviews. Cohen's κ and weighted κ statistics were used to test reliability, with assessment of the strength of agreement among the ICD‐10 code reviewers and between the GTT teams by means of inter‐rater reliability tests33. Standard classification of κ coefficient values was used: less than 0·20, poor agreement; 0·21–0·40, fair; 0·41–0·60, moderate; 0·61–0·80, good; and 0·81–1·00, very good33.

Logistic regression was used to analyse the relationship between complications identified using a verified ICD‐10 code compared with complications identified by the GTT review of patients' records; the results are reported as odds ratios (ORs) with 95 per cent confidence intervals. P ≤ 0·050 was considered statistically significant. Data were analysed using SPSS® version 24 for Windows® (IBM, Armonk, New York, USA). Weighted κ analysis was performed using Stata® version 14.0, and Venn diagrams were drawn using the Stata procedure pvenn (StataCorp, College Station, Texas, USA).

Results

A study sample of 700 surgical admissions in 695 patients was drawn randomly from a larger group of 12 966 surgical procedures. Some 87·4 per cent were from the tertiary hospital and 12·6 per cent from the community hospital. Surgical procedures in the community hospital included gastrointestinal surgery (such as appendicectomy and colonic resection) and urology (for example prostatectomy and ureteric stent). Those in the tertiary hospital included neurosurgery (such as disc herniation surgery, excision of intracranial lesion, evacuation of haematoma, external drainage), gynaecology (hysterectomy, oophorectomy, vaginal fistula repair, perineorrhaphy), orthopaedics (osteosynthesis or reposition of fractured limbs, hip or knee replacements, external fixation, malleolus surgery) and thoracic surgery (ascending aorta vascular prosthesis, cardiopulmonary bypass, aortic valve replacement, circulatory anastomosis). Patient characteristics are shown in Table 1. Mean(s.d.) age was 58·3(18·1) (range 18–99) years. In total, the data set represented 5350 days of admission, with a median of 5·8 (i.q.r. 3·1–8·8) and mean(s.d.) of 7·6(8·3) days per stay.

Table 1

Characteristics of 700 surgical patient admissions in two hospitals in western Norway from November 2012 to March 2015

	No. of patients (n = 700)
Age (years)
18–64	417 (59·6)
≥ 65	283 (40·4)
Sex
M	309 (44·1)
F	391 (55·9)
Duration of hospital stay (days)
1	72 (10·3)
2–7	350 (50·0)
8–14	199 (28·4)
≥ 15	79 (11·3)
Incision time (min)
≤ 30	83 (11·9)
31–60	125 (17·9)
61–180	392 (56·0)
≥ 181	100 (14·3)
ASA fitness grade
I	115 (16·4)
II	305 (43·6)
III	249 (35·6)
IV	30 (4·3)
V	1 (0·1)
Urgency of surgery
Elective	395 (56·4)
Emergency	305 (43·6)
Surgical specialty
Neurosurgery	129 (18·4)
Orthopaedics	223 (31·9)
Gynaecology	111 (15·9)
Thoracic	149 (21·3)
General	88 (12·6)
Hospital type
Tertiary	612 (87·4)
Central	88 (12·6)

Values in parentheses are percentages.

Complications detected by the Global Trigger Tool method

Using the GTT method, a total of 331 (range 1–7) complications were identified in 212 of 700 admissions (30·3 per cent). Seventy‐seven admissions were identified with more than one complication describing an injury. The distribution of the GTT complications is shown in Table 2. A majority were classified as temporary: E in 111 of 331 (33·5 per cent) and F in 200 (60·4 per cent). Thirteen (4·0 per cent) were regarded as representing permanent harm and classified as G. None were classified as H (life‐supporting treatment needed) and complications in seven patients (2·1 per cent) were classified as I (death). Infection‐related complications constituted 41·1 per cent and 26·0 per cent were classified as other surgical complications.

Table 2

Complications classified according to the Global Trigger Tool in 23 categories for the 212 of 700 patient admissions with patient harm in two hospitals in western Norway from November 2012 to March 2015

	One or more GTT complications*
Other surgical complications†	86 (26·0)
Surgical‐site infection	35 (10·6)
Urinary tract infection	34 (10·3)
Low respiratory infection	30 (9·1)
Other infection	26 (7·9)
Postoperative bleeding/haematoma	24 (7·3)
Postoperative respiratory complication	23 (6·9)
Reoperation	20 (6·0)
Ventilator‐associated pneumonia	10 (3·0)
Organ failure	10 (3·0)
Medication‐related (including blood and fluid therapy)	9 (2·7)
Deteriorating chronic condition	6 (1·8)
Bleeding	5 (1·5)
Thrombosis/emboli	3 (0·9)
Decubitus	2 (0·6)
Other	2 (0·6)
Allergy	1 (0·3)
Fracture	1 (0·3)
Central venous line infection	1 (0·3)
Medical technical equipment failure	1 (0·3)
Postpartum/obstetric complication	1 (0·3)
Wrong surgical site	1 (0·3)
Fall	0 (0)
Total no. of complications	331 (100)

Values in parentheses are percentage of total number of complications.

Among 212 patient admissions.

Drop foot, rupture of dura, pleural fluid, necrosis, vision disturbances, infarction, atrial fibrillation, other. GTT, Global Trigger Tool.

ICD‐10 complication code classification

Electronic extraction of ICD‐10 codes identified 519 complication codes in 332 patient records of the 700 admissions (complication rate 47·4 per cent). After excluding codes representing complications already present on admission, 141 of 700 admissions (20·1 per cent) with a total of 298 complications were found to occur in hospital. The number of complications per hospital stay ranged from one to six. The distribution of the ICD‐10 complication codes is summarized in Table 3. After verifying the complications, the order of frequency of complication types changed from cardiac, fall, respiratory and infections to cardiac, respiratory, infections and other. Of note, all 96 codes for patient falls were found to represent falls occurring before, and not during, the hospital stay.

Table 3

Distribution of complications in 332 surgical admissions identified using ICD‐10 complication codes, and distribution of verified complications in 141 surgical admissions from patients' records in two western Norwegian hospitals from November 2012 to March 2015

	Extracted ICD‐10 codes(n = 332 admissions)	Verified ICD‐10 codes(n = 141 admissions)
Respiratory	79 (15·2)	55 (18·5)
Pneumonia	21 (4·0)	20 (6·7)
Respiratory, other	58 (11·2)	35 (11·7)
Cardiac	151 (29·1)	95 (31·9)
Cardiac arrhythmia	65 (12·5)	49 (16·4)
Congestive heart failure	17 (3·3)	11 (3·7)
Cardiac, other	69 (13·3)	35 (11·7)
Infections	65 (12·5)	47 (15·8)
Sepsis	13 (2·5)	9 (3·0)
Surgical site	20 (3·9)	13 (4·4)
Urinary tract	24 (4·6)	20 (6·7)
Infections, other	8 (1·5)	5 (1·7)
Surgical wound rupture	5 (1·0)	4 (1·3)
Nervous system	13 (2·5)	11 (3·7)
Delirium, somnolence, other	3 (0·6)	2 (0·7)
Cerebral infarction	10 (1·9)	9 (3·0)
Bleeding	17 (3·3)	15 (5·0)
Embolism	5 (1·0)	2 (0·7)
Nutrition	28 (5·4)	23 (7·7)
Malnutrition, other nutritional deficiencies	12 (2·3)	11 (3·7)
Other disorders of fluid, electrolyte and acid–base balance	16 (3·1)	12 (4·0)
Anaesthesia	3 (0·6)	3 (1·0)
Mechanical implantation	16 (3·1)	7 (2·3)
Fall	96 (18·5)	0 (0)
Other complications	41 (7·9)	36 (12·1)
Total no. of complications	519 (100)	298 (100)

Values in parentheses are percentage of total number of complications. Detailed list of included ICD‐10 complication codes can be found in Table S1 (supporting information).

Reliability analysis

Analysis of agreement in classifying complications in 20 random medical records using the GTT method revealed that the two teams reached 85 per cent agreement in terms of the presence of a complication, 65 per cent regarding numbers of complications and 75 per cent on the levels of severity. The κ values for inter‐rating agreement between the teams were 0·700, 0·504 (weighted) and 0·688 (weighted) respectively. Three clinical researchers reviewed the same discharge ICD‐10 codes in 30 random medical records. Agreement was 91 per cent in terms of patients having a complication or not during the hospital stay, and 77 per cent for agreement on actual number of complications. Accordingly, the κ values for inter‐rater reliability were 0·816 and 0·731 respectively.

Validating complications by ICD‐10 versus Global Trigger Tool

To investigate concurrent validity, it was determined whether admissions with ICD‐10 complications were the same admissions as those identified as having one or more complications by the GTT methodology. The similarity between the two classification methods increased from 68·3 per cent before clinical verification of the ICD‐10 complication codes to 83·3 per cent after excluding ICD‐10 codes representing complications already present on admission (Fig. 1).

Figure 1

Agreement between methods of identifying admissions with complications versus no complications: a using ICD‐10 codes extracted from administrative data and b using ICD‐10 codes verified from patients' records. GTT, Global Trigger Tool

Logistic regression was used to quantify the importance of clinically verifying ICD‐10 complication codes rather than using them without verification. Admissions with unverified ICD‐10 codes (332) were at increased odds of also having a GTT‐identified complication (OR 5·85, 95 per cent confidence interval 4·06 to 8·44), whereas admissions with verified ICD‐10 codes (141) increased the odds substantially (OR 25·38, 15·41 to 41·79). Ninety‐four admissions with complications according to GTT methodology did not have an ICD‐10 code reflecting a complication (Fig. 1).

Discussion

This study found that complications during the hospital stay were overestimated when crude ICD‐10 codes were used in surgical admissions. By excluding codes representing conditions already present on admission, the complication rate decreased from 47·4 to 20·1 per cent. This provides quantifiable evidence of the detrimental impact of coding practices on the ability of ICD‐10 codes to indicate a true complication in patient care. Based on the present findings, it does not appear feasible to detect and disclose all complications and level of severity using a single method. A substantial decrease in complications was found with accurate ICD‐10‐verified complication codes compared with ICD‐10 codes present on admission. These findings support the hypothesis of the study. The GTT method is designed to inform about local complications and patient safety initiatives over longer periods of time13, whereas the ICD‐10 (if used accurately) may be used both locally and in large epidemiological studies to inform on larger patient safety interventions.

The complication rate obtained using the GTT in the present study was 30·3 per cent of all admissions. This is at the upper end of the range reported in studies included in a recent systematic review2. That review, however, included studies across both medical and surgical specialties. Focusing solely on surgical patient populations, as in the present study, would be expected to result in higher rates than in mixed patient populations5. Regarding level of severity, the majority of complications identified by the GTT (93·9 per cent) were found to be associated with temporary harm. Similar findings regarding severity have been documented elsewhere34, 35.

In the present study, the agreement between the ICD‐10 and GTT methods increased from 68·3 to 83·3 per cent following clinical researchers' verification of the ICD coding. Other studies7, 15, 36 have investigated complications using different detection methods. The high rates of agreement here might be explained by avoidance of use of complications reported voluntarily by healthcare personnel as a comparator. There is evidence for under‐reporting of complications in voluntary reporting systems12, which would likely lead to lower agreement between methods. The present analysis included a large number of complication codes (154 in total), which might have increased the number of complications identified, thus offering a broader perspective on surgical complication analyses. Moreover, a large number of clinically reviewed patient records were included, which is likely to have increased the number of complications found and analysed by this methodology compared with smaller studies35.

A total of 94 admissions with GTT‐identified complications were not identified by ICD‐10 codes. There may be several reasons for this discrepancy. In a busy clinical practice, physicians may fail to use correct ICD‐10 codes owing to lack of training in the use of such codes and/or time constraints, as pointed out in a national report37. The finding also demonstrates differences in methodology between the two systems for identifying complications. The GTT method may include complications before admission if they are linked to medical treatment13, whereas the ICD‐10 codes should consider only complications that emerge in hospital to be ‘true’ complications. The present findings have significant practical implications. If hospitals are to work on preventing or addressing patient safety risks, reliable knowledge of risk factors will be needed. Deriving such knowledge and developing patient safety programmes based solely on administratively collected complication data does not represent an effective strategy, based on the present findings. More accurate evidence concerning in‐hospital complications is needed to tailor surgical patient safety interventions. Examples from this study suggest that a focus on respiratory and cardiac complications, infections and nutrition is needed. It was also shown here that all patient falls occurred before admission. These findings are important as ICD‐10 coding is widely used to report on complications, carry out research, and to inform healthcare policies and hospital funding17. Yet few studies have reported similar procedures for clinical verification of ICD‐10‐coded patient‐level data30. Such studies are urgently required to inform decision‐making and funding. On a practical level, an electronic ‘flag’ built into ICD‐10 classification systems can be recommended, so that the coder can identify a ‘complication’ already present on admission. Such a flagging option is available in the USA, Canada and Australia38. This improves coding accuracy without the requirement for significant financial investment or training, thereby enhancing the value of inexpensive complication reports based on routinely collected data.

Prospective recording of complications on a severity scale, using a validated system such as the Clavien–Dindo classification7, would be ideal. This would probably lead to the availability of more accurate and clinician‐reported data in prospective databases of postoperative morbidity, which could offer a better picture of surgical care quality. However, this would have training and resource implications if introduced as standard practice, and this is not currently done routinely in Norwegian hospitals.

The present study has limitations. Only surgical patients were included, so the results cannot be extrapolated directly to the larger cohort of medical admissions. Second, a standard Norwegian version of the GTT protocol was used and not a trigger protocol especially designed for surgical patients, known as the Surgical Trigger Toolkit. This was because the expert GTT teams had already been trained to use the standard version; in addition, there is no validated Norwegian version of the Surgical Trigger Toolkit available. However, the GTT actually covers all but two of the trigger words available in the Surgical Trigger Toolkit and hence the coverage is very similar. Third, the preventability of the identified complications was not investigated. Classifying preventability is not included as part of national GTT team training in Norway, nor is it recommended as a part of the GTT protocol13. Further research should analyse preventability in a similarly structured manner2, 39. Furthermore, when studying in‐hospital complications, those related to previous admissions had to be excluded. This may have led to under‐reporting of complications, mainly owing to coding practices being related to each hospital admission and not to each patient throughout the healthcare pathway. Finally, as a result of natural differences between the ICD‐10 and GTT systems, it may be questioned whether admissions identified by both methods actually had the same (type of) complications. Simply put, although an admission might have been identified as complicated by both tools, the type of complication identified by one of the two systems may have differed from that identified by the other. This would not affect overall complication rates, but could affect the types of complication found and consequently the hospital's targets for improvement.

The study also has strengths, including: bringing together two methods for assessing surgical safety; the overall high level of expertise among the reviewers; the inclusion of two separate hospitals; and the good reliability of the analyses. Regarding reliability, the inter‐rater reliability analysis is a methodological strength. The GTT teams showed good agreement for detection and severity of complications, and moderate agreement regarding the number of complications present. The two GTT teams had expert members from both hospitals (with knowledge of local reporting practices). The inter‐rater agreement among the ICD‐10 reviewers was even stronger. This is a prerequisite for studies reporting data that require clinical judgement and the seniority of the reviewers ensured this.

The accuracy of ICD‐10 complication codes is improved when in‐hospital complications are verified with record reviews. Crude data with unverified ICD‐10 codes significantly overestimate surgical complications within hospitals because complications present on admission are included. This can represent a severe bias for national and international comparisons of quality and safety of surgical care.

Table S1 International Classification of Diseases‐10 codes indicating a complication (called complication code)

Click here for additional data file.