Accuracy and reliability of injury coding in the national Dutch Trauma Registry.

OBJECTIVE: Injury coding is well known for lack of completeness and accuracy. The objective of this study was to perform a nationwide assessment of accuracy and reliability on Abbreviated Injury Scale (AIS) coding by Dutch Trauma Registry (DTR) coders and to determine the effect on Injury Severity Score (ISS). Additionally, the coders' characteristics were surveyed.
METHODS: Three fictional trauma cases were presented to all Dutch trauma coders in a nationwide survey (response rate 69%). The coders were asked to extract and code the cases' injuries according to the AIS manual (version 2005, update 2008). Reference standard was set by three highly experienced coders. Summary statistics were used to describe the registered AIS codes and ISS distribution. The primary outcome measures were accuracy of injury coding and inter-rater agreement on AIS codes. Secondary outcome measures were characteristics of coders: profession, work setting, experience in injury coding and training level in injury coding.
RESULTS: The total number of different AIS codes used to describe 14 separate injuries in the three cases was 89. Mean accuracy per AIS code was 42.2% (range 2.4-92.7%). Mean accuracy on number of AIS codes was 23%. Overall inter-rater agreement per AIS code was 49.1% (range 2.4-92.7%). The number of assigned AIS codes varied between 0 and 18 per injury. Twenty-seven percentage of injuries were overlooked. ISS was correctly scored in 42.4%. In 31.7%, the AIS coding of the two more complex cases led to incorrect classification of the patient as ISS < 16 or ISS ≥ 16. Half (47%) of the coders had no (para)medical degree, 26% were working in level I trauma centers, 37% had less than 2 years of experience and 40% had no training in AIS coding.
CONCLUSIONS: Accuracy of and inter-rater agreement on AIS injury scoring by DTR coders is limited. This may in part be due to the heterogeneous backgrounds and training levels of the coders. As a result of the inconsistent coding, the number of major trauma patients in the DTR may be over- or underestimated. Conclusions based on DTR data should therefore be drawn with caution.

Introduction

All inclusive trauma systems intend to address all aspects of care for the injured throughout the chain of acute care. Their implementation has led to reduced mortality and improved outcome of care for severely injured patients worldwide [1, 2]. Trauma registries are considered essential in mature trauma systems, for the evaluation of outcome of care, for regional and international benchmarking purposes and for trauma quality improvement programs [1, 3]. However, the use of trauma registry data for these purposes relies heavily on accurate data collection. A major limitation associated with trauma registries is data quality and data completeness [3-5]. Data in most trauma registries are notoriously incomplete [6]. Also, ill inclusion of patients and incorrect data extraction may lead to erroneous data and biased conclusions. Non-validated registry data may affect quality assurance and performance improvement [4] and in the end may harm optimal care [7].

The WHO-originated International Classification of Diseases [8] and the Abbreviated Injury Scale (AIS) [9] are universally used for injury coding. Currently, these classifications are fundamental components of trauma outcome research and quality improvement programs. In addition, the AIS is the basis of injury severity measures [10], such as the Injury Severity Score (ISS) [11], the New Injury Severity Score (NISS) [12] and related prediction models such as the Trauma and Injury Severity Score (TRISS) [13]. Although used all over the world, inter-rater agreement in AIS coding is moderate [14-19].

In the Dutch Trauma Registry (DTR), trauma patients are coded according to the AIS. The registry was developed to collect and record data on a national scale for policy making, quality surveillance and advancement of trauma care and to conduct scientific research. Annual DTR reports are published by the Dutch Network for Emergency Care. The number of annually treated severely injured trauma patients (ISS ≥16) is a dominant parameter in profiling trauma centers, which demonstrates the importance of adequate selection and subsequent coding of trauma patients for the registry.

The aim of this study was to assess the accuracy and reliability on AIS injury coding by DTR coders in the Netherlands.

Methods

Study design

Three realistic but fictional trauma cases were presented in a web-based environment. The optimal number of cases was reckoned to be three, as more cases were expected to have a negative impact on the response rate. Approval of the Medical Ethics Committee was not needed since no patient data were used in this study.

Participants

The managing directors of the 11 trauma regions in the Netherlands approved the study protocol and declared their support. Once approval was obtained, the online survey was sent to all 62 DTR coders working in the 11 trauma regions.

Response rate

Participation by the DTR coders was voluntary. To improve the response rate, non-responding coders received a reminder after 3 and 6 months by mail. Reasons for non-response were not recorded. Forty-one coders completed the entire survey and provided AIS codes for all cases, while 43 coders provided AIS codes only for Case 1. Thus, the response rate for Case 1 was 69% (43/62) and for Cases 2 and 3 was 66% (41/62).

Cases

The case descriptions included crucial information regarding trauma mechanism, prehospital data, findings on physical examination and diagnostics to facilitate injury coding. To resemble daily practice, terminology in the fictional cases was not always explicit. The cases were ordered by progressive complexity in order to reveal differences in data extraction and injury coding. As only Dutch coders were included, the case descriptions were composed in the Dutch language. The translated case descriptions are presented in Table 1.

Table 1

Presented cases (translated from Dutch)

Case 1 A 21-year-old male falls from a tree on his left knee. He complains of soreness of his left knee and ankle. He did not hurt his head. At physical examination, normal vital signs are noted. He is conscious and his Glasgow Coma Scale is maximal. Particularly, he complains about a painful left ankle. Examination of the head, thorax, abdomen, pelvis and spine is unremarkable. Swelling of the left ankle is noted. The sensibility of his lower leg is globally intact. Dorsiflexion and plantar flexion are forceful. Normal pulsations of the anterior and posterior tibial arteries are noted. Examination of the right leg shows no abnormalities. Anteroposterior X-ray examination shows a dislocated fracture of the medial malleolus and subluxation of the tibiotalar joint.

Case 2 A 28-year-old female is brought to the emergency department after she staggered over a bicycle earlier that night. Apparently, she was intoxicated during the event. Afterward, she went home and went straight to bed. When she woke up, she was short of breath and had chest pains. In addition, she noticed marked swelling of the right ankle and weightbearing was barely capable. She was sent in by the general practitioner. At presentation, patient is communicative and complains about chest pains on the left side. She is nauseous but did not have to vomit. She has a clear upper airway and has no neck pain. Blood pressure measures 112/64 mmHg, heart frequency is 94 bpm and body temperature is 37.4°C. Respiratory rate is 18/min. Trachea is in midline. At auscultation of the chest, normal breath sounds are heard. The left side of the chest is painful at palpation. No crepitus or subcutaneous emphysema is noted. There is moderate tenderness in the left epigastric area. However, no signs of peritonitis are found. Examination of the pelvis and spine is unremarkable. Swelling of the lateral side of the right ankle is noted, and patient is not able to bear weight. There are no skin abnormalities of her right foot. On a chest X-ray, fractures of the left ninth and tenth ribs are noted. There are no radiological abnormalities of the heart, diaphragm and right hemithorax. Ultrasound examination of the abdomen reveals a trace of free intra-abdominal fluid between the left kidney and spleen, as well as in Douglas space. Additionally, a contrast-enhanced CT scan was performed, revealing a subcapsular hematoma of the spleen. The hematoma encompasses approximately 40% surface area and surrounding free fluid is noted. In addition, a parenchymal laceration of 2 cm is seen, as well as a contrast blush near one of the trabecular vessels. In Douglas space, a small amount of fluid is noted. Roentgenography of the right foot shows a non-displaced intra-articular fracture of the cuboid. The fracture line extends into the calcaneocuboidal joint.

Case 3 A 34-year-old male lost control driving his motor scooter. Velocity is unknown. He was not wearing a helmet and presumably hit his head against a lamppost. Spectators declared he was immediately unconscious. The length of unconsciousness is not clear, perhaps a few minutes. On arrival of the emergency medical services, patient was alert and complaining about headache and a sore neck. During examination, he seems restless, with BP 118/72 and pulse 92 per min. Respiratory rate is 15 per min. Pupils are equal and reacting to light. He has an abrasion of the forehead and a laceration over the bridge of the nose, extending over the right cheek. The airway is clear, normal breath sounds are heard bilaterally and the chest is not painful on compression. The abdomen is not tender and limp. The pelvis is stable. There are no abnormalities of the extremities, and neurovascular examination is unremarkable. On presentation, patient is immobilized on a long spine board, using a rigid collar, head blocks and straps.In the trauma bay, patient is lethargic, but the airway is clear in spite of some blood in the oropharynx. The nose has swollen. The spine is sufficiently immobilized. Normal breath sounds are heard over the ventral and posterolateral chest. The trachea is in midline; the neck veins are not distended. Respiratory rate is 12 per min. Oxygen saturation is 97%. Chest X-ray shows no abnormal findings. Examination of the abdomen and pelvis is unremarkable. Focused assessment sonography of the abdomen does not show intraperitoneal free fluid, no fluid in the pericardium and no suspicion of ventral pneumothorax. Pupils are equal and reactive to light. There are no signs of lateralization. GCS is E2M5V3. Cerebrospinal fluid is leaking from his right ear. After undressing, no additional external injuries are found. At rectal examination, normal sphincter tone is noted.The patient is transported to the CT scanner and CT scans of the cerebrum and cervical spine are obtained. On CT, soft tissue swelling of the forehead is noted, as well as a fracture of the anterior wall of the frontal sinus, and the posterior wall is not involved. Additionally, a fracture of the nasal bone extending into the right orbital floor, the pterygoid plate and the maxillary sinus on the right is noted. Furthermore, fractures of the right maxilla and base of the skull extending into the mastoid are seen. Finally, a few small contusions in the right frontal area of the brain and a contrecoup lesion in the left occipital area are noted. There is no swelling of the brain or obscuration of in the gray–white matter junctions. Also, gyri and sulci are appearing normal. No abnormalities of the C-spine are noted.

Reference standard

All injuries were extracted and coded independently according to the AIS by an expert panel. This panel comprised two trauma surgeons who were trained in AIS coding. One has more than 5 years and the other over 2 years of experience in injury coding. Both members are daily involved in injury coding for a level I trauma center and responsible for regional auditing services on AIS coding. On recommendation of the Dutch Network for Emergency Care, a third and highly experienced trauma coder from a different trauma region was involved. This coder was involved in AIS training on a national level and involved in preparing the annual reports of data from the DTR. All members coded the cases independently and specified their arguments for the selected codes in separate documents. Disagreement in coding by the panel members was solved in a consensus meeting. Reference codes were agreed upon unanimously, and these were defined as the reference standard. According to the expert panel’s reference standard, Case 1 had three injuries, Case 2 had four injuries and Case 3 had seven injuries (14 injuries in total; Table 2). As in clinical practice, some injuries were described in such a way that, according to the expert panel, more than one AIS code was applicable. For the 14 injuries, 18 different AIS codes were judged correct (Table 2).

Table 2

Distribution of different AIS codes assigned by coders

Case	Injury	Reference injury codea	Not coded	Number of different codes	Accuracy (%)	Agreement (%)
1	Swelling of the left ankle	810 402.1	41	1	2/43 (4.7)	2/43 (4.7)
	Dislocated fracture of the medial malleolus	854 331.2 or 854 351.2 or 854 361.2	1	12	25/43 (58)	17/43 (40)
	Subluxation of the tibiotalar joint	877 120.1	2	8	27/43 (63)	27/43 (63)
2	Swelling of the lateral side of the right ankle	810 402.1	40	1	1/41 (2.4)	1/41 (2.4)
	Fractures of the left ninth and tenth rib	450 202.2	2	2	38/41 (93)	38/41 (93)
	Subcapsular hematoma of the spleenb	544 224.3	0	8	16/41 (39)	30/41 (73)
	Intra-articular fracture of the cuboid	857 661.2	0	6	33/41 (80)	33/41 (80)
3	Abrasion of the forehead	110 202.1 or 210 202.1	8	2	33/41 (80)	30/41 (73)
	Laceration over the bridge of the nose	210 600.1	1	3	9/41 (22)	29/41 (71)
	Swollen nose	110 402.1 or 210 402.1	34	2	7/41 (17)	6/41 (15)
	Cerebrospinal fluid is leaking	150 204.3	0	4	31/4 1(76)	31/41 (76)
	Fracture of the anterior wall of the frontal sinus	150 400.2	25	2	3/41 (7.3)	13/41 (32)
	Fracture of the nasal bone extending into the right orbital floor, the pterygoid plate and the maxillary sinus	250 806.2	0	18	16/41 (39)	18/41 (44)
	A few small contusions in the right frontal area of the brain and a contrecoup lesion in the left occipital area	140 622.3	0	14	4/41 (9.8)	10/41 (24)

Applicable AIS codes according to the expert panel;

22 responders coded >1 AIS code.

Survey

Coders were asked to extract and code injuries of the three presented cases according to the AIS manual (version 2005, Update 2008). Additionally, background information was obtained about (i) the coders’ profession (physician, nurse, data manager, secretary or student); (ii) years of experience in AIS coding (<1 year, 1–2 years, 3–5 years and >5 years); (iii) training level in AIS coding (local training, AIS course or no training) and (iv) work setting (level I, level II or level III trauma hospital, or more than one hospital). In the Netherlands, the highest level of trauma care is provided by level I trauma centers, which are predominantly academic medical centers. Every region has one coordinating level I trauma center and several level II and III hospitals, irrespective of the rural or urban setting.

Data analysis

Data on accuracy and inter-rater agreement on AIS injury coding were analyzed per case.

Frequencies of registered AIS codes per injury were documented. If an injury was not coded by the coder, this was considered ‘incorrectly coded’. The overall number of different codes used per injury was determined. Frequencies of AIS codes and ISS were documented per case as well [11].

Accuracy of AIS coding

Accuracy in AIS coding was expressed per injury as the percentage of correct AIS codes for a specific injury and per ISS body region as the percentage of correct AIS codes for all injuries in that specific body region.

Inter-rater agreement on AIS injury coding

Inter-rater agreement on AIS injury coding was defined as the maximal percentage of coders documenting the same AIS code for a specific injury, irrespective of whether that code was correct. Inter-rater agreement on AIS injury coding per ISS body region was defined as the maximal percentage of coders documenting the same AIS code for all injuries in that specific body region.

Results

Characteristics of coders

The overall response rate was 69.4% (43/62). Eight of the 43 (18.6%) responding coders were physicians, 11 (25.6%) were nurses and four (9.3%) were medical students. twenty (46.5%) coders without a medical or paramedical background worked predominantly as data manager (n = 18; 41.8%) or secretary (n = 2; 4.7%). Fifteen (34.9%) coders were responsible for injury coding in more than one hospital in their trauma region. Sixteen (37.2%) coders had less than 2 years of experience in injury coding according to the AIS classification. Thirteen (30.2%) coders had taken a nationally or internationally organized AIS injury coding course. Thirteen (30.2%) had received on-the-job training in injury coding, whereas 17 (39.5%) coders had never received any kind of structured education in injury coding.

AIS codes

In total, 580 AIS codes were to be assigned by the coders. For Case 1, 129 AIS codes were supposed to be coded by the 43 coders (three AIS codes per coder). Likewise, 164 and 287 AIS codes were to be coded for Cases 2 and 3 by the 41 coders, respectively (four and seven AIS codes per coder, respectively). Overall, 426 injuries were registered using 514 AIS codes (86 codes by 43 coders for Case 1, 155 codes by 41 coders for Case 2 and 273 codes by 41 coders for Case 3). Thus, some injuries were given more than one code. For all three cases with 14 injuries altogether, the coders reported 89 different AIS codes including six (1.2%) non-existing codes. Five of the 14 injuries (35.7%) were coded by all coders. Twenty-seven percentage of the required AIS codes (154/580) were not reported. Most of these injuries (n = 124; 21.4%) were in the external body region (for instance, in Cases 1 and 2, swelling of the overlying skin was erroneously not coded by most of the coders (41/43 and 40/41, respectively), Table 2). Eleven (2.1%) non-existing injuries were registered (e.g. a renal injury in Case 2). The number of AIS codes varied between 0 and 18 per injury (Table 2). The number of AIS codes for Case 1 varied between 1 and 4 (Figure 1a), 3–5 injuries were coded for Case 2 (Figure 1b) and 2–11 injuries for Case 3 (Figure 1c). In 29 of 125 coded cases (23.2%), the coders registered the same number of AIS codes, as did the expert group: 7/43 (16.3%) for Case 1, 18/41 (43.9%) for Case 2 and 4/41 (9.8%) for Case 3.

Figure 1

(a) Number of AIS codes coded by coders (Case 1; n = 43). (b) Number of AIS codes coded by coders (Case 2; n = 41). (c) Number of AIS codes coded by coders (Case 3; n = 41).

Injury severity

The ISSs per case assigned by the coders are presented in Table 3. According to the reference codes, Case 1 had an ISS of 5, Case 2 of 17 and Case 3 of 14. Overall, the correct ISS was scored in 42.4% of the coded cases. For Case 1, the median ISS was 4 (range 4–9) and was scored correctly by 7% of the coders. For Case 2, the median ISS was 14 (range 8–24). The ISS was scored correctly by 17 coders (41.4%). Twenty-one (51.2%) coders failed to identify Case 2 as a severely injured patient (ISS ≥ 16). For Case 3, the median ISS was 14 (range 10–21). The ISS was scored correctly by 80.4% (n = 33 coders). Five (12.2%) coders incorrectly scored the patient as severely injured (ISS ≥ 16).

Table 3

Distribution of ISSs computed from AIS codes assigned by coders

	N a	ISS	N	Median ISS	ISS ≥ 16b
Case 1	43	4	38
		5c	3
		9	2
				4	0
Case 2	41	8	1
		12	18
		13	1
		14	1
		17c	17
		20	1
		24	2
				14	20
Case 3	41	10	1
		13	2
		14c	33
		17	2
		19	2
		21	1
				14	5

Number of coders;

number of coders coding ISS ≥ 16;

ISS according to the expert panel.

Accuracy

Accuracy of AIS coding per injury varied between 2.4 and 93% (Table 2). In total, 42.2% of all registered injury codes (245/580) were accurate. The mean accuracy of injury coding per case was 41.9%, 53.7% and 35.9%, for Cases 1–3, respectively. When excluding AIS grade 1 skin injuries in our study, mean accuracy of injury coding increased to 60.5% and to 70.7% for Cases 1 and 2, respectively. Serious injuries (i.e. AIS severity score ≥ 3) were correctly coded in 9.8–76% (mean 41.5%). For Case 1, one coder coded all injuries correctly. For Cases 2 and 3, no coder coded all injuries correctly. Accuracy per ISS body region varied between 25% for external injuries and 93% for chest injuries (Table 4).

Table 4

Accuracy and inter-rater agreement in injury coding per ISS body region

ISS body region	Accuracy (%)	Agreement (%)
Head or neck	38/123 (31)	54/123 (44)
Face	16/41 (39)	18/41 (44)
Chest	38/41 (93)	38/41 (93)
Abdominal or pelvic contents	16/41 (39)	30/41 (73)
Extremities or pelvic girdle	85/127 (67)	77/127 (61)
External	52/207 (25)	68/207 (33)

Inter-rater agreement

The inter-rater agreement for all injuries together was 49.1% and varied between 2.4 and 92.7% per injury (Table 2). For Cases 1–3, mean inter-rater agreement on injury coding was 35.7%, 62.2% and 47.7%, respectively. Inter-rater agreement on injury coding varied per body region between 43.9% for ‘head or neck’ and ‘face’ and 92.7% for ‘chest’ (Table 4).

Discussion

Principal findings

Our study demonstrates limited accuracy and reliability of injury coding according to the AIS manual (version 2005, Update 2008) by DTR coders. Mean accuracy of injury coding per injury was 42.2%, and overall inter-rater agreement on injury coding was 49.1%. This demonstrates that adequate and consistent AIS injury coding in trauma registries is challenging.

Interpretation within the context of the wider literature

The current findings are in concordance with previously published work on variability of trauma registry data and reflect the ongoing discussion about the need for improved data accuracy [14, 17, 18, 20, 21].

Arabian et al. found a mean accuracy in injury coding of 71% [21]. As only one case was used and only two injuries were to be coded, comparability with our study is limited. Ringdal et al. found 61.5% agreement of AIS codes with a reference standard. They demonstrated a significant relationship between experience in injury coding and correctly coded injuries [21]. Similar to our results, a recent national validation study of the Norwegian Trauma Registry [14] showed 43% concordance of AIS codes, and 90% of non-registered AIS codes were AIS 1. In another study recoding 120 cases in the Queensland Trauma Registry, on average 39% of the codes used by any two coders for each of the injured persons were identical [20]. In the Netherlands, two independent studies demonstrated substantial accuracy on the number of AIS codes in regional trauma registries [17, 18]. In a subgroup analysis of severely injured patients (ISS ≥ 16), however, accuracy was fair (intraclass correlation coefficient = 0.33) [17], and the authors concluded that discrepancies in injury coding are more likely to occur in this population. These studies focused on reliability on the number of AIS codes, whereas the present study aimed on accuracy of injury coding as well.

In the present study, many coders documented identical incorrect injury codes, indicating the presence of recurrent coding errors. Many skin injuries in conjunction with deeper structure lesions were not coded, reflecting the unfamiliarity with the generic coding rule that both should be coded as separate injuries within the specific AIS chapter. As most skin injuries reflect only minor injuries (AIS severity code 1), the effect on the ISS is modest. Discordant AIS codes may impair prediction of survival models like TRISS and NISS [22]. However, multiple studies showed no effect on population median ISS and NISS [10, 14, 17, 19, 20].

In concordance with other studies, we found only fair inter-rater agreement on the number of AIS codes [14, 15]. A substantial variation in injury coding was noted for some injuries (Table 2). As 66% of all Dutch coders completed the cases, we consider this finding as representative for injury coding in the DTR. This raises questions about whether coders are sufficiently trained on medical terminologies such as those used in the medical charts that are the source of data extraction. Knowledge of medical terminology and familiarity with reading medical charts are indispensable for correct data extraction and injury coding, as well as knowledge of the generic coding rules. Alternatively, injury descriptions in the AIS dictionary might not always be applicable or sufficiently explicit, leading to variations in coding. In our study, we did not expect to find this high amount of variation, as nearly two-thirds of the coders had more than 3 years of experience in AIS coding.

The percentage of overlooked injuries in our study is in accordance with the study of Ringdal et al., who found that 31% overlooked injuries in 50 recoded cases—63% of which were AIS grade 1 or 2 [10]. In another study, 12% grade 1 injuries were overlooked [14]. Our study shows substantial increase in accuracy in AIS coding after exclusion of grade 1 skin injuries, indicating the obvious need for adequate extraction and documentation of soft tissue injuries.

Dedicated educational programs seem to improve injury coding quality [23]. In another study, trauma surgeon involvement in the registration process led to identification of additional injuries and ISS increase to ≥16 in 1.2% of recoded cases [19]. Another study showed physicians and nurses to produce more reliable results in AIS injury coding than both emergency medical technicians and nonclinical technicians [15]. Therefore, medical background and sound knowledge of pathoanatomy seem to be important factors for adequate injury coding.

Strengths and limitations

In many studies on validation of AIS injury coding of actual trauma registry data, ratings are typically compared to re-ratings by a single second coder [15, 17, 18]. To omit biased results, we established an expert panel of three highly experienced coders to set a reference standard [10, 14]. Despite their experience, the general amount of variation in coding suggests that there may not be just one ultimate true code, but rather several codes that can be applicable to describe similar injuries. As fictional cases were used instead of a sample of actual DTR data, our conclusions are limited to variation in coding between Dutch coders, and no conclusions regarding the quality of actual DTR data can be drawn. No subgroup analyses based on professional background, work setting, experience in injury coding or level of training were done due to the small number of coders per subgroup. Lastly, coders’ arguments for electing AIS codes were not evaluated. As no studies regarding this issue were found in the literature, further work on this might benefit accuracy of injury coding.

Implications for policy, practice and research

Reliability of injury coding affects the quality and usefulness of DTR data for benchmarking and quality improvement efforts. DTR data quality needs to be improved by uniform (inter)national training of coders [21]. Coders’ certification and continuous educational programs coordinated and supervised by the regional coordinating trauma center can further attribute to sound registry data. Further studies on the effect of implementation of dedicated training programs are warranted. Furthermore, more insight into the actual process of injury coding is needed. The quality of AIS-derived measurements in the DTR may be augmented by standardized interregional data validation and the implementation of a clinician-coder double-reading team [21].

Conclusions

In this study, the accuracy and inter-rater agreement in AIS injury scoring in Dutch DTR coders is limited. AIS-derived measurements like ISS and TRISS may hence not be considered reliable enough for the quality assurance and policy making that these data are currently used for. Lack of reliability in AIS coding could in part be explained by the heterogenic backgrounds and training levels of the coders. Conclusions based on DTR data should therefore be drawn with caution. Additional training for registry coders and studies to improve the reliability of AIS injury coding are needed.

Click here for additional data file.