Discrepancies in interpretation of night-time emergency computed tomography scans by radiology residents.

BACKGROUND: In many emergency radiology units, most of the night-time work is performed by radiology residents. Residents' preliminary reports are typically reviewed by an attending radiologist. Accordingly, it is known that discrepancies in these preliminary reports exist.
PURPOSE: To evaluate the quality of night-time computed tomography (CT) interpretations made by radiology residents in the emergency department.
MATERIAL AND METHODS: Retrospectively, 1463 initial night-time CT interpretations given by a radiology resident were compared to the subspecialist's re-interpretation given the following weekday. All discrepancies were recorded and classified into different groups regarding their possible adverse effect for the emergency treatment. The rate of discrepancies was compared between more and less experienced residents and between different anatomical regions.
RESULTS: The overall rate of misinterpretations was low. In 2.3% (33/1463) of all night-time CT interpretations, an important and clinically relevant diagnosis was missed. No fatalities occurred due to CT misinterpretations during the study. The total rate of discrepancies including clinically irrelevant findings such as anatomical variations was 12.2% (179/1463). Less experienced residents were more likely to miss the correct diagnosis than more experienced residents (18.3% vs. 10.9%, odds ratio [OR] = 1.82, P = 0.001). Discrepancies were more common in body CT interpretations than in neurological CTs (18.1% vs. 9.1%, OR = 2.30, P < 0.0001).
CONCLUSION: The rate of clinically important misinterpretations in CT examinations by radiology residents was found to be low. Experience helps in lowering the rate of misinterpretations.

Introduction

In modern day emergency medicine, imaging studies are an important part of diagnostics, both in traumatic and non-traumatic cases (1–4). Computed tomography (CT) is often the modality of choice due to its speed, availability, and diagnostic accuracy (5, 6). In many emergency radiology units, night-time work is done to some extent by radiology residents. Residents’ autonomous work is considered a valuable part of training and also eases the workload of attending radiologists (7). Radiology residents’ preliminary reports are typically reviewed by an attending radiologist in a reasonable time-frame, e.g. the next morning or next weekday (8).

Rather extensive research has been done on the accuracy and discrepancies of preliminary CT reports given by radiology residents (9–14). Most of the earlier studies have been carried out in large centers with busy emergency radiology units. Our study focused on discrepancies in radiology residents’ CT interpretations in a Scandinavian university hospital with approximately a dozen CT scans performed per night. Ultimately, the purpose of this study was to assess the quality of night-time CT interpretations made by radiology residents in the emergency department.

Material and Methods

Settings

This study was conducted in a medium-sized university hospital, which provides 24-h emergency services to approximately 410,000 people living in the area. There were approximately 86,000 visits to the hospital’s emergency department in 2016, including traumatic and non-traumatic cases. The emergency radiology unit is located right next to the emergency department and is equipped with CT, magnetic resonance imaging (MRI), ultrasound, and conventional radiography capabilities. A total of 18,751 CT scans were performed by the emergency radiology unit in the year 2016.

During normal office hours and in the evenings, the radiology emergency unit always has at least one attending radiologist present. On night shifts, from 21:00 to 08:00, the emergency radiology unit has only one person interpreting the examinations: either an attending radiologist or a radiology resident. If a resident is working on the night shift they have an assigned attending radiologist to whom they can call if extra help is needed. All night-time CT and MRI examinations interpreted by a radiology resident are reviewed by an attending radiologist the following weekday.

In our hospital, radiology residents must have at least one year of work experience before they are allowed to work the night shift in the emergency radiology unit. After two years of work experience, night shifts become mandatory. One resident has typically 2–4 night shifts every month.

Data

Research was conducted retrospectively and therefore no informed consent was obtained. All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. All the CT scans interpreted by a radiology resident between 21:00 and 08:00 from 1 January to 31 December 2016 were included in this study. Data were acquired from the hospital’s picture archiving and communication systems (PACS). In total, 10 residents worked on 165 night shifts during this period; 1463 CT scans were performed and interpreted in these night shifts independently by a radiology resident, i.e. no phone consultations were made. All examinations were later reviewed by an attending radiologist. Cases were categorized into four groups: important and acute discrepancies; important but non-acute discrepancies; non-important discrepancies; and no discrepancies. Discrepancies were also ranked as perceptual or cognitive.

Discrepancy was considered important and acute if the missed diagnosis did alter or could have altered the course of emergency treatment. This group included the most serious interpretation errors, some of them potentially fatal. Discrepancy was considered important but non-acute if the missed diagnosis had nothing to do with the current emergency but would require further examination or follow-up in the future. Non-specific tumors and incidental aneurysms without bleeding were the two biggest groups of important non-acute discrepancies. Discrepancy was considered non-important if the missed diagnosis had nothing to do with the current emergency and would not require further examination or follow-up. These discrepancies included mainly anatomical variations, congenital anomalies, postoperative findings, and lesions that had already been diagnosed.

Discrepancy was ranked perceptual if the pathological finding was not detected at all. If the pathology was detected but incorrectly interpreted, the discrepancy was ranked cognitive. CT examinations were divided into two groups based on the anatomic area in question. Examinations of the head, neck, and cervical spine were considered neurological CT examinations. Examinations of the thorax, abdomen, and extremities were categorized as body CT examinations. Both groups included traumatic and non-traumatic emergencies.

CT interpretations were divided into two groups based on the work experience of the resident giving the preliminary report. The cut-off point was set at 30 months. Radiology residency in Finland lasts for 72 months, including 12 months of elective services in primary healthcare. The ratio of neurological and body CT examinations was similar in the two groups. The median experience of radiology residents working in night shifts was 37 months. The least and most experienced residents had 13 and 47 months of working experience, respectively.

Statistical analysis

The frequency of all discrepancies in different severities was calculated with comparisons made between groups of less and more experienced radiology residents, between neurological and body examinations, and between different time-slots of the night shift. A Chi-squared statistical test was performed to assess the differences between the groups. SPSS 23.0 was used in analyzing the data.

Results

Out of all 1463 reviewed CT interpretations, 1284 (87.8%) were considered to have no discrepancies. The remaining 179 (12.2%) cases had some level of discrepancy. Of these discrepancies, 33 (2.3%) were classified as important and acute, meaning that missed diagnosis had or could potentially have altered the course of treatment during the night shift before the mistake was noticed in the reviewing process. In 48 (3.3%) cases, the missed diagnosis was considered not to have altered the course of emergency treatment, but the finding needed to be followed up later on. The largest group of discrepancies, 98 (6.7%), was classified to be unimportant regarding the emergency treatment and also did not need to be followed up. All discrepancies are presented in Table 1. Discrepancies did not have a recognizable pattern regarding the anatomical areas or underlining pathologies. A random sample of all three classes of discrepancies is listed in Table 2.

Table 1.

Resident experience and discrepancies.

	<30 months of experience	≥30 months of experience	Total
Median experience (months)	28	38
Total number of reviewed examinations	263 (18.0)	1200 (82.0)	1463 (100)
Body CT examinations	83 (31.6)	385 (32.1)	468 (32.0)
Neurological CT examinations	180 (68.4)	815 (67.9)	995 (68.0)
Discrepancies	48 (18.3)	131 (10.9)	179 (12.2)
− Important and acute	14 (5.3)	19 (1.6)	33 (2.3)
− Important but not acute	10 (3.8)	38 (3.2)	48 (3.3)
− Unimportant	24 (9.1)	74 (6.2)	98 (6.7)
Perceptual discrepancies	36 (13.7)	79 (6.6)	115 (7.9)
Cognitive discrepancies	12 (4.6)	52 (4.3)	64 (4.4)
Discrepancies in body CT	26 (31.3)	62 (16.1)	88 (18.8)
Discrepancies in neurological CT	22 (12.2)	69 (8.5)	91 (9.1)

Values are presented as n (%) unless otherwise stated.

Table 2.

Random sample of discrepancies in different groups.

Acute and important missed diagnosis

Ovarian torsion

Ureteral calculi

Intestinal perforation by a large gallstone

Pulmonary embolism

Acute subdural hematoma

Non-acute missed diagnosis that requires follow-up

Meningioma

Liver cirrhosis

Cerebral artery aneurysm

Unspecified liver lesion

Splenic artery aneurysm

Missed diagnosis with no further clinical impact

Anatomical variations in intracranial arteries

Vertebral anomalies

Osteoma

Duodenal diverticulum

Sliding hiatus hernia

All 33 discrepancies that had importance to emergency treatment were re-evaluated and 16 (1.1% of all initial scans) of these were considered to be potentially life-threatening mistakes. However, no fatalities occurred in these 16 patients. Short descriptions of these 16 cases are listed in Table 3 and three example cases are pictured in Fig. 1.

Table 3.

Potentially life-threatening missed diagnoses (all 16 cases).

Pulmonary embolism missed (4 cases)

Intracranial hemorrhage missed (2 cases)

Acute cerebral infarct missed

Skull fracture missed

Mesenteric vein thrombosis missed

Portal vein thrombosis missed

Active gastric bleeding missed

Kidney laceration missed on a trauma patient

Spleen injury missed on a trauma patient

Intestinal perforation by a large gallstone mistaken for diverticulum

Ovarian torsion mistaken for an ovarian tumor

Acute cerebral infarct mistaken for an artifact

Fig. 1.

Example case of missed CT diagnoses. (a) An axial CT image showing a small subdural effusion (white arrowheads) adjacent to the falx. (b) A coronal CT image describing a laceration (white arrow) of the left kidney. (c) An axial CT image depicting a large gallstone (black arrow) perforating to the lumen of the duodenum.

Radiology residents with < 30 months of work experience were more likely to have discrepancies in their reports compared to residents with ≥30 months of experience. In total residents, with < 30 months of experience had interpreted 263 CT examinations and in 48 (18.3%) reports there was some sort of discrepancy. Residents with ≥30 months of experience had interpreted 1200 CT examinations and in 131 (10.9%) reports a discrepancy was detected. The difference between these two groups was statistically significant (odds ratio [OR] = 1.82, P = 0.001).

Perceptual discrepancies were more common than cognitive discrepancies (115 vs. 64 errors). In total, 4.4% of all examinations had a cognitive discrepancy. Cognitive discrepancies were equally common in the groups of less and more experienced radiology residents. Radiology residents with < 30 months of working experience were more likely to have a perceptual discrepancy in their report. They had a total of 36/263 (13.7%) perceptual discrepancies in their examinations, whereas residents with ≥30 months of experience had 79/1200 (6.6%) cognitive discrepancies in their examinations (OR = 2.25, P < 0.0001).

Of all 1463 CT examinations, 995 (68.0%) were classified as neurological CTs and 468 (32.0%) as body CTs. Discrepancies were less common in neurological CTs than in body CTs regardless of the resident’s experience. The rate of discrepancies in body CT examination reports was 88/468 (18.1%) compared to 91/995 (9.1%) in neurological CTs (OR = 2.30, P < 0.0001). Eventually, the time of night (e.g. midnight vs. the early hours of the morning just before the night shift ends) did not have a significant effect on the rate of missed diagnosis in the preliminary reports (Fig. 2).

Fig. 2.

Percentage of preliminary CT interpretations with discrepancies, divided into four groups based on the time of interpretation. There is no statistically significant difference between the groups.

Discussion

In this study, a 2.3% rate of clinically important discrepancies was found, which is well in line with previous studies. In a recent study from 2016, Platon et al. found that the rate of potentially life-threatening mistakes in radiology residents’ initial CT interpretations was 0.8% and the total rate of serious discrepancies was 2.3% (13). In a large study by Ruchman et al. in 2007, the rate of major discrepancies in radiology residents’ night-time reports was 2.6%. Ruchman et al.’s study included interpretations of conventional radiographs and CT, MRI, and sonography examinations (8). In our study, the rate of potentially life-threatening discrepancies was 1.1%. We consider this to be low, especially since no actual fatalities were recorded due to mistakes in the preliminary CT reports.

Furthermore, 3.3% of all reviewed CT examinations had an incidental follow-up requiring a finding that was missed in the first reading but noticed in the review. In these cases, the mistake in the preliminary report was harmless, as the report was corrected in the reviewing process and the information was passed on to clinicians. However, this demonstrates the importance of the reviewing process. In our study, we also recorded the discrepancies that had no clinical importance whatsoever. These discrepancies included minor findings such as congenital anomalies or anatomical variations and had no importance to patient’s treatment in emergency situations or in the future. The rate of this kind of discrepancies was quite high at 6.7%. In many earlier studies, such insignificant discrepancies were not recorded so comparing this rate to earlier studies is problematic.

As noted in earlier studies, most discrepancies in CT interpretations are perceptual rather than cognitive. Our study showed the same tendency. The rate of perceptual discrepancies was higher among the less experienced radiology residents. The rate of cognitive discrepancies did not change regarding the experience but was nevertheless rather low at 4.4%. Discrepancies were more common in body CT examinations than in neurological CT examinations. This finding has been reported previously and is probably due to multiple factors. Neurological CT scans cover a smaller anatomical area and pathologies may be easier to spot than in body CTs. In addition, in our hospital about two-thirds of all emergency CTs are neurological, giving residents more experience with neurological CTs than body CTs.

In some previous studies, it has been noted that mistakes in CT interpretations are most likely to happen in the last few hours of the night shift. However, in our study we did not find any correlation between the time of the night and the rate of mistakes. This may have to do with our hospital’s relatively small amount of night-time examinations: on average only about 10 CT examinations are performed in a night shift. Usually the emergency radiology unit quiets down at around 03:00 and the radiologist is allowed to rest in the sleeping quarters located in the proximity of the emergency radiology unit. Therefore, the radiologist gets to rest during the shift and this might explain why the rate of mistakes does not increase, even in the early morning just before the end of the shift.

Our study is not without limitations. The data consist of reports given by only 10 radiology residents. The individual differences among the low number of residents could have caused bias. Second, the attending radiologist’s interpretation was chosen for the reference standard without taking their potential mistakes into consideration.

In conclusion, a very small portion of CT interpretations given by radiology residents contains clinically relevant mistakes. The more experience the radiology resident has, the fewer mistakes they make. Interpretations of neurological CT examinations have fewer discrepancies than those of body CT examinations. Our radiology residents did not make any more mistakes in the last hours of the night shift than in the beginning of the shift, which might be explained by the relatively low number of CT scans during night shifts.