A computer-assisted systematic quality monitoring method for cervical hip fracture radiography.

BACKGROUND: A thorough quality analysis of radiologic performance is cumbersome. Instead, the prevalence of missed cervical hip fractures might be used as a quality indicator.
PURPOSE: To validate a computer-based quality study of cervical hip fracture radiography.
MATERIAL AND METHODS: True and false negative and positive hip trauma radiography during 6 years was assessed manually. Patients with two or more radiologic hip examinations before surgery were selected by computer analysis of the databases. The first of two preoperative examinations might constitute a missed fracture. These cases were reviewed.
RESULTS: Out of 1621 cervical hip fractures, manual perusal found 51 (3.1%) false negative radiographic diagnoses. Among approximately 14,000 radiographic hip examinations, there were 27 (0.2%) false positive diagnoses. Fifty-seven percent of false negative reports were occult fractures, the other diagnostic mistakes. There were no significant differences over the years. Diagnostic sensitivity was 96.9% and specificity 99.8%. Computer-assisted analysis with a time interval of at least 120 days between the first and the second radiographic examination discovered 39 of the 51 false negative reports.
CONCLUSION: Cervical hip trauma radiography has high sensitivity and specificity. With computer-assisted analysis, 76% of false negative reports were found.

Introduction

Systematic quality monitoring programs that evaluate the outcome of the entire chain from imaging to reporting are rarely undertaken in radiology since manual evaluations are time-consuming and expensive. One report (1) described a semi-automated way to evaluate several imaging chains in radiology, which were much easier to perform than purely manual analyses.

Radiographic diagnosis of cervical hip fracture is suitable for systematic quality monitoring (1) for several reasons: Cervical hip fracture is an important and common fracture with high co-morbidity (2) and increased mortality (3), with severe implications on healthcare economy and efficiency (4–6). A missed fracture means unnecessary patient suffering, with the risk of a non-displaced fracture progressing into a displaced fracture. Usually, the radiographic diagnosis of hip fracture is straightforward (5), but in a small percentage of examinations radiography is false negative (7–10).

To facilitate systematic quality monitoring of cervical hip fracture radiography, a computer-assisted analysis method was developed, called the rule of two (11), based on the fact that patients with hip fractures who are not operated on usually have symptoms for an extended period of time. Consequently, almost all patients with a missed fracture will undergo repeat imaging with radiography, magnetic resonance imaging (MRI) or computed tomography (CT) until the correct diagnosis is determined.

The aim of the current study was to evaluate the performance of a computer-assisted quality monitoring tool in cervical hip fracture.

Material and Methods

The current report is a validation of a computerized study. This, in turn, is based on the retrospective analysis of the outcome of all hip radiography performed for suspicion of cervical hip fracture in 2003 and during the years 2007–2011; a quality control project without the need for informed consent. The setting was a university hospital with a catchment area for primary hip fracture surgery for about 320,000 people.

Imaging

All hip trauma radiography was done with an anterior-posterior (AP) and a cross-table lateral radiograph, and an AP pelvic radiograph. In cases without obvious hip fracture, the examination was often supplemented by two anterior oblique pelvic radiographs, with a rotation of about 20–30°. Supplemental CT imaging was done on various scanners, with multiplanar reformations in the axial, coronal, and sagittal planes. Supplemental MRI was mostly done on 1.5 T scanners, using a hip fracture protocol consisting of coronal T1-weighted (T1W) and short tau inversion recovery (STIR) sequences, often combined with axial T1W and STIR or fat-suppressed T2-weighted (T2W) sequences. All imaging was archived in the regional digital radiology archive, where all imaging for this and the surrounding hospitals within a 100 km radius was available.

Data

Data were retrieved from both the surgery and the radiology databases to find all patients with a cervical hip fracture. The orthopedic surgery database provided a list of all patients operated on for cervical hip fracture. From the radiology information system (RIS), data on all hip radiography examinations were retrieved. All reports on radiography, MRI, and CT for suspicion of hip trauma or fracture, as well as those lacking a clearly stated indication for the examination, were selected for study.

Manual analysis

All dates for radiologic examinations were manually perused in the RIS to find true and false negative and positive hip radiography reports. For each hip radiography the RIS was perused for at least 6 months backwards to find potential prior trauma episodes with radiographic examinations, and followed for up to 1 year to find missed fractures. Hip radiography with a negative report and an uneventful follow-up in the RIS without evidence of a later fracture was scored as true negative. Cases without follow-up imaging were also scored as true negative since no further complications from the index trauma required repeat imaging, i.e. no hip fracture. Cases where a cervical hip fracture became apparent on subsequent imaging with radiography, MRI, or CT within 6 months without interval trauma were scored as false negative. Cases where a cervical hip fracture could be detected on the radiographs, and the report was definite or suspicious for cervical hip fracture, were scored as true positive. Cases where a fracture or a suspicious fracture was reported and was not surgically treated, could not be seen at review or on follow-up imaging, and was without follow-up complications were scored as false positive.

Scoring was based on the archived original reports on the studies and not on a re-evaluation of the radiographs. A re-evaluation of the radiographs was only done to classify undiagnosed fractures into occult (invisible) (12,13) and missed (in retrospect visible) fractures. To reduce subjectivity two observers re-evaluated equivocal cases.

Computer-assisted analysis (rule of two)

Surgery of cervical hip fracture requires prior radiologic diagnosis. If more than one radiologic examination is performed before surgery, the first of these examinations might be false negative. Among the patients who had been operated on for cervical hip fracture a subgroup was identified with more than one preoperative radiologic study. This subgroup was thought to contain most of the patients with false negative reports and was reviewed.

For the computer-assisted analysis using the rule of two, all radiologic hip examinations (radiography, CT, and MRI), whether trauma-related or not, were merged into one list which was then, by computer, compared with the list of patients operated on for cervical hip fracture. Thus, by applying this rule of two on the RIS data, all patients operated on for cervical hip fracture with more than one preoperative radiologic hip examination were identified. All these cases were manually re-evaluated in the RIS and PACS. Four different maximum time intervals between the two preoperative radiologic examinations (14, 30, 60, and 120 days) were evaluated.

Statistical analysis

Descriptive statistics were used to characterize the data. Sensitivity and specificity were calculated based on frequencies for true and false positive and negative radiography reports. A χ2 analysis was performed to test for significance of differences of false negative reports between years, with the level of significance set to P < 0.05.

Results

Patients

For 13,996 patients referred to radiography for suspicion of cervical hip fracture, the female/male ratio was 1.94. The mean age was 73 years (age range, 0–106 years); for girls/women, the mean age was 76 years (age range, 0–106 years); for boys/men, the mean age was 68 years (age range, 0–103 years).

Manual review

Out of more than 31,000 hip radiographs, including CT and MRI, about 14,000 were performed because of trauma or suspected fracture, and among these, 1570 were true positive for cervical hip fracture. There were 51 false negative reports (3.1% of all fractures) and 27 false positive reports (0.2% of all non-fractured hips) (Table 1). Among 47 patients with cervical hip fracture, who for various reasons were not operated, there were five false negative radiography reports. Twelve cases with a preliminary report on call of “no fracture” were corrected within 24 h in the final report. These were not counted as diagnostic mistakes.

Table 1.

Diagnostic accuracy of radiography of cervical hip fracture. Results of manual perusal from 2003 and 2007–2011.

	2003	2007	2008	2009	2010	2011	Total	Annual average
Radiography, CT, and MRI of the hip	4816	6485	5104	4832	5824	4348	31,409	5235
Hip radiography with trauma referral	2096	2385	2379	2172	2399	2565	13,996	2333
No cervical hip fracture (true negative)	1834	2098	2075	1913	2125	2303	12,348	2058
Cervical hip fracture (true positive)	253	266	291	249	259	252	1570	262
Fracture not diagnosed (false negative)	7 (2.7%)	12 (4.3%)	10 (3.3%)	4 (1.6%)	9 (3.4%)	9 (3.4%)	51 (3.1%)	9 (3.1%)
Fracture diagnosis wrong (false positive)	2 (0.1%)	9 (0.4%)	3 (0.1%)	6 (0.3%)	6 (0.3%)	1 (0.0%)	27 (0.2%)	5 (0.2%)

In 29 of 51 false negative studies, there was an occult (invisible) fracture, even at review. In 22 false negative studies, the fracture could be seen at review, constituting diagnostic mistakes (Table 2). In 10 of these 22 cases, the fracture was identified solely on the pelvic radiographs.

Table 2.

False negative radiographic diagnosis of cervical hip fracture with occult and non-occult fractures at review. Results of manual perusal from 2003 and 2007–2011.

	2003	2007	2008	2009	2010	2011	Total
All cervical hip fractures	260	278	301	253	268	261	1621
False negative	7 (2.7%)	12 (4.3%)	10 (3.3%)	4 (1.6%)	9 (3.4%)	9 (3.4%)	51 (3.1%)
Occult fractures	4 (1.5%)	6 (2.2%)	6 (2.0%)	2 (0.8%)	6 (2.2%)	5 (1.9%)	29 (1.8%)
Non-occult fractures at review	3 (1.2%)	6 (2.2%)	4 (1.3%)	2 (0.8%)	3 (1.1%)	4 (1.5%)	22 (1.4%)
Subgroup of non-occult fractures where the fracture was visible only on pelvis radiograph	0	2	3	2	2	1	10

There were no significant differences in diagnostic accuracy between years. Overall diagnostic sensitivity of radiography of cervical hip fracture was 96.9% and specificity was 99.8%.

Computer-assisted analysis using the rule of two

Four different maximum time intervals between the two preoperative hip radiographs were tested. The interval of 14 days identified 126 cases where two radiologic studies had been done before surgery. Twenty-four cases were false negative. The interval of 30 days identified 30 false negative reports among 147 cases. The interval of 60 days found 38 false negative reports among 169 cases. Increasing the interval to 120 days identified 39 false negative reports among 203 cases (Table 3). Further increase of the interval did not disclose additional false negative cases. The rule of two failed to identify 12 false negative cases. The reasons were: no surgery (n = 5); surgery at another hospital (n = 1); different surgical registration code (n = 3); and wrong radiologic examination code (n = 3).

Table 3.

Manual perusal compared with computer-assisted analysis using the rule of two for radiographic diagnosis of cervical hip fracture during 2003 and 2007–2011. Number of false negative results for four different time intervals between the first and the second radiologic study, and number of reports to evaluate at each interval.

	2003	2007	2008	2009	2010	2011	Total	Reports toprocess manually (n)
False negative reports found by manual perusal	7	12	10	4	9	9	51
False negative reports found by computer-assisted analysis using the rule of two
14-day interval	1	5	6	1	4	7	24	126
30-day interval	6	9	8	2	4	7	36	147
60-day interval	6	9	8	3	5	7	38	169
120-day interval	6	10	8	3	5	7	39	203

Discussion

The current study confirmed that radiography has high sensitivity and specificity for cervical hip fracture. It showed that a computer-assisted analysis of cervical hip fracture radiography can be used for quality monitoring, especially if the digital coding is optimal. Of all missed cervical hip fractures, about half were radiographically occult and the other half were diagnostic mistakes.

The current report on manual perusal of all hip and pelvis radiography, CT, and MRI with trauma indication during 6 years identified all false negative and false positive reports, enabling the calculation of sensitivity and specificity for hip radiography in suspect cervical hip fracture.

The findings in the current study are in agreement with previous reports which showed similar high sensitivity for radiography of cervical hip fracture (Table 4) with a rate of false negative radiographs in the range of 0.2–4.9%. The rate of disagreement in cervical hip fracture diagnosis is thus similar to but somewhat lower than published data on general discrepancy rates between radiologists, which are in the range of 3–6% (14–16).

Table 4.

Studies reporting false negative and false positive hip radiography in cervical hip fracture.

Author, year	Patients (n)	Type of patient	Patients with cervical hip fracture (n)	Further evaluated patients (n)	Outcome reference	FN (%)	Occult FN (%)	Missed FN (%)	FP (%)
Pathak, 1997 (7)	1108	Cervical hip fracture	1108		Surgery	2.7	0.8	1.9
Pandey, 1998 (22)	770	Suspect hip fracture	N/A	33	MRI	0.2
Lim, 2002 (37)	422	Definite or suspect hip fracture	N/A	57	MRI	0.2
Dominguez, 2005 (9)	764		301	62	MRI	4.9
Current study	13,996	Suspect hip fracture	1621		Surgery, radiography, CT, MRI	3.1	1.8	1.4	0.2

FN, false negative; FP, false positive.

A little more than half of the false negative reports in the current study were caused by truly occult fractures, not visible even in retrospect. A truly occult fracture is defined as clinical symptoms or signs of fracture without radiographic evidence (12,13). Occult and suspect hip fractures may be further investigated by MRI (17–23) or by CT (5,10,24–29). Modern CT, especially when viewed with a soft tissue window width/level setting of about 400/40 (26), has the ability to evaluate bone marrow edema and lipohemarthrosis and has a diagnostic capability approaching that of MRI (26,28,29). CT is a valuable alternative when MRI is unavailable or contraindicated (25). Bone scintigraphy is not recommended in acute trauma. MRI is quicker and more specific (30,31) and there is a risk of false negative outcome when bone scanning is performed too early, especially in elderly patients and in patients with metabolic diseases (32–34). However, in stress fracture bone scintigraphy is a highly sensitive imaging modality.

In ten of the false negative reports the only finding suggesting a fracture was a minimal cortical break in the femoral neck, visible on one or more of the pelvic radiographs but not on the hip radiographs. No other systematic cause was found for false negative reports due to diagnostic mistakes. Several factors seemed to add to the diagnostic difficulties, e.g. low bone mineral content, underexposed images, obesity, subcapital location of the fracture, and presence of only minute cortical disruptions. Another reason was failure to note an asymmetric projection of the hips where the fracture was suggested by unilateral shortening and external rotation of the fractured hip. In some cases, calcifications of the femoral artery masked the fracture and in some cases severe osteoarthritis with large osteophytes interfered with diagnosis (35). The false positive diagnoses were caused by similar difficulties. Further sources of error were image artefacts mimicking fractures and physeal vestiges which were misinterpreted as fractures.

A quality analysis by a manual perusal of hip radiography (11) is extremely time-consuming. At the authors’ institution between 2000 and 3000 radiography reports annually would have to be reviewed. The computer-assisted evaluation requires much less manual labor. After creation of patient lists and computerized selection of the study group, far fewer reports need to be checked. During the 6 years of the study, more than 31,000 hip radiographs were performed. In 1621 cervical hip fractures, 1570 radiographic examinations were true positive (Table 1). In total, 1574 of 1621 fractures were operated. Of these, at most 203 cases (12.9% of the operated fractures; Table 3) were flagged in the computerized analysis as having two or more radiographic examinations before surgery and would require a manual evaluation. This means that 87% of the operated fractures would not have to be evaluated in the RIS or PACS if the computerized analysis were used.

A disadvantage of the computerized method was that for several reasons only about 76% of the false negative reports were identified (Table 3). Another disadvantage was that the rule of two cannot identify false positive reports, and thus it is not possible to calculate both the sensitivity and specificity of hip fracture radiography. However, for a year-by-year comparison of diagnostic quality, the ratio between false negative reports and total number of fractures might well be sufficient. A computer-assisted analysis using the rule of two is a practical and quick way to systematically evaluate the diagnostic quality of cervical hip fracture radiography. The method is not exact, but should be good enough to show systematic changes or trends in the diagnostic quality, especially if the false negative cases are reviewed to decide if the misses are due to occult fractures or to diagnostic mistakes.

In the current study, only cervical hip fracture radiography was used as a quality assurance model. Cervical hip fracture has the advantage of being binary in diagnosis (there are almost no incomplete fractures) as well as in treatment (almost all fractures are treated surgically) (36) with very few dropouts from the inclusion process. Other types of fractures may be less well suited for computerized quality analysis, lacking a clear surgical outcome reference.

Quality control programs, similar to the current study, have been undertaken elsewhere where other aspects of radiology have been analyzed. In a report by Liu et al. (1), eight subspecialties were requested to select a specific component of its imaging reports or procedures that could be measured and compared with a readily available reference standard database such as in the current report. Also, reference accuracy levels had to be available from, for example, published meta-analyses or multi-center studies. The manual analysis was mostly performed by staff, minimizing additional workload for radiologists. Compared with more commonly instituted peer-review programs, reviewer bias and erroneous second reviewer interpretations are avoided by such database analysis.

Among the limitations of the study are the fact that study cases might be missed due to radiography or surgery at different hospitals. One such case was identified, where surgery was performed at another location and the patient thus did not appear in the local surgery database. Regarding radiologic studies performed at other locations, the risk of missing one in the current study was much less since the regional database contains all imaging from surrounding hospitals within a 100 km radius. Further studies are needed to demonstrate whether the fact that only 76% of false negative reports were identified by this computerized method is an acceptable level for the purpose of rapid and economic quality evaluation of hip fracture radiography.

In conclusion, the computerized analysis detected 76% of the false negative radiography reports and could be used for diagnostic quality evaluation of cervical hip fracture. About 3% of all cervical hip fractures were missed at initial radiography.