Interobserver and Intraobserver Reliability in the Salter Classification of Avascular Necrosis of the Femoral Head in Developmental Dysplasia of the Hip.

BACKGROUND: Avascular necrosis (AVN) of the femoral head is a concerning complication that can result from treatments for developmental dysplasia of the hip (DDH). AVN can lead to degenerative osteoarthritis, persistent acetabular dysplasia, reduced function, and continuing hip pain. The incidence of AVN reported in the DDH literature is widely varied (0% to 73%). This variability may arise from lack of consensus on what constitutes true AVN in this patient population, and lack of clear criteria provided in studies reporting incidence rates.
METHODS: A multicentre, prospective database of infants diagnosed with DDH between 2010 and 2014 from 0 to 18 months of age was analyzed for patients treated by closed reduction (CR). Twelve pediatric orthopaedic surgeons completed 2 rounds of AVN assessments. Deidentified anteroposterior radiographs at most recent follow-up were provided to surgeons along with patient age at radiographic assessment, length of follow-up, ands affected hip. Ten of 12 surgeons completed a third round of assessments where they were provided with 1 to 2 additional radiographs within the follow-up period. Radiographic criteria for total AVN described by Salter and colleagues were used. Surgeons rated the presence of AVN as "yes" or "no" and kappa values were calculated within and between rounds.
RESULTS: A total of 69 hips in 60 patients were assessed for AVN a median of 22 months (range: 12 to 36) post-CR. Interobserver kappa values for rounds 1, 2, and 3 were 0.52 (range: 0.11 to 0.90), 0.61 (range: 0.21 to 0.90), and 0.53 (range: 0.10 to 0.79), respectively. Intraobserver agreement for AVN diagnosis was an average of 0.72 (range: 0.31 to 0.96).
CONCLUSIONS: Despite using the most commonly referenced diagnostic criteria, radiographic diagnosis of AVN following CR in DDH patients demonstrated only moderate agreement across surgeons. The addition of sequential radiographs did not improve cross-observer reliability, and while substantial agreement was seen within observers, the range of intraobserver kappa values was large. LEVEL OF EVIDENCE: Level I-diagnostic study.

The term developmental dysplasia of the hip (DDH) includes a spectrum of hip abnormalities which has an incidence of 1 to 34 per 1000 live births. It is the leading cause of hip osteoarthritis in young adults.1,2 Treatment for DDH depends both on the age of the patient as well as severity of dysplasia. Typically, abduction brace is first line treatment before 6 months of age, with closed reduction (CR) and spica casting or open reduction as more frequently used options in older or more complex patients.1,3 Avascular necrosis (AVN) of the femoral head is a concerning complication that can result from treatment for DDH treatment.

Though the exact etiology of AVN is currently unknown, direct vascular compression and/or excessive pressure on the femoral head are thought to be the root causes,4 which lead to lack of blood flow and cellular death. This can lead to proximal femoral growth disturbance and deformity, degenerative osteoarthritis, persistent acetabular dysplasia, reduced function, continuing hip pain, and potential need for an early hip replacement. At this point, it has not yet been elucidated why certain patients develop AVN and others do not. Many variables have been identified as risk or protective factors, such as age, prereduction traction, and CR versus open reduction; however, some of these have been found to increase risk in some studies and decrease risk in others.5–12 Furthermore, some studies suggest that none of these factors are related to the development of AVN secondary to DDH treatment.4,11,13–17 There is consensus, however, that patients who develop AVN are more likely to require subsequent surgeries and report poorer clinical outcomes.11

The incidence of AVN in the DDH literature is widely varied (0% to 73%).4–6,11,13–16,18–20 While some of this variability is inherent to the different treatment methods used, much of it may also arise from lack of consensus on what constitutes true AVN in this patient population, and the lack of clear criteria provided in studies reporting incidence rates. Different AVN classification systems used include the Gage and Winter, Kalamchi and McEwen, Salter and colleagues, and Bucholz and Ogden.7,9,21,22 The lack of clarity in the DDH literature has made cross-study comparison of this important treatment complication difficult and prevented the definitive identification of risk factors and prognostic factors for the development of AVN following treatment for DDH. There has yet to be a classification system for AVN validated in pediatric patients with DDH. The purpose of this study was to determine the interobserver and intraobserver reliability of the radiographic diagnosis (Salter criteria) of AVN following CR for DDH. The Salter classification was chosen as this method aims for early identification of ischemia that can lead to prolonged growth disturbance and deformity long term.

METHODS

This was a multicentre, international prospective study of patients diagnosed with DDH from 0 to 18 months of age and treated by CR between 2010 and 2014 from 7 centers in the United States, Canada, Australia, and the United Kingdom. Patients with a minimum 1-year follow-up post-CR were included for assessment of AVN at the most recent follow-up radiograph. This study was reviewed and approved by the Research Ethics Board at the University of British Columbia (H11-02191).

Pediatric orthopaedic surgeons with a practice focus on the pediatric hip and contributing to the multicentre prospective study were invited to participate in three rounds of AVN assessment. For round 1, they were all provided with deidentified anteroposterior radiographs from the most recent follow-up along with patient age at radiographic assessment, length of follow-up, and the affected hip. Two weeks after the first round, the radiographs were shuffled and sent back to the surgeons for the second round. Twelve pediatric orthopaedic surgeons—practicing at 2 Canadian centers, 5 US centers, 1 Australian center, and 1 UK center—completed the first 2 rounds of AVN assessments. Ten of the 12 surgeons completed a third round of assessment where they were provided with 1 to 2 additional radiographs within the follow-up period. All surgeons were sent the Salter and colleagues paper, outlining the AVN criteria (Table 1), with the file containing the hip radiographs. An example of the sequential radiograph series is provided in Figures 1A–C.

TABLE 1

Criteria for the Diagnosis of AVN21

Salter Criteria
1. Failure of appearance of the ossific nucleus of the femoral head during 1 y or longer after reduction
2. Failure of growth of an existing ossific nucleus during 1 year or longer after reduction
3. Broadening of the femoral neck within 1 year after reduction
4. Increased radiographic density of the femoral head followed by the radiographic appearance of fragmentation
5. Residual deformity of the femoral head and neck when reossification is complete. These deformities include coxa magna, coxa plana, coxa vara, and a short, broad femoral neck

FIGURE 1

Example cases—radiograph series provided for rating. In the file that was provided, the surgeons were given the sequential radiographs, the affected hip, the patient age in months, and the number of months post-closed reduction surgery for each radiograph in the series. They were asked to provide a yes or no answer with respect to determining the presence of AVN. AVN indicates avascular necrosis; CR, closed reduction.

In their assessments, surgeons were instructed to use the radiographic criteria for AVN described by Salter. They rated the presence of AVN as “yes” or “no” and Cohen’s Kappa values were calculated within and between rounds to determine interobserver and intraobserver reliability, respectively (Fig. 2). The kappa values were interpreted using the Landis and Koch guidelines, indicating that agreement with κ <0.00 is poor, 0.00 to 0.20 is slight, 0.21 to 0.40 is fair, 0.41 to 0.60 is moderate, 0.61 to 0.80 is substantial, and 0.81 to 1.00 is almost perfect.23 Kappa values are reported along with 95% confidence interval (CI).

FIGURE 2

Flowchart of methods. A file with 69 radiographs from 60 patients, and their pertaining information, outlined in Figure 1, were sent to surgeons. Two weeks after the first round, they were sent the same file with the radiographs shuffled and asked to repeat the task. Finally, 10 of the 12 surgeons completed the third round 2 weeks later, where they were given 1 to 2 additional radiographs. In each round, interobserver kappa values were calculated, and intraobserver kappa values were calculated between rounds 1 and 2. AVN indicates avascular necrosis.

RESULTS

A total of 69 hips in 60 patients were assessed for AVN, a median 22 months (range: 12 to 36) post-CR. Interobserver kappa values for round 1, 2, and 3 were 0.52 (95% CI: 0.48-0.57; range: 0.11 to 0.90), 0.61 (95% CI: 0.57-0.65; range: 0.21 to 0.90), and 0.53 (95% CI: 0.48-0.59; range: 0.10 to 0.79), respectively, where values of 0.41 to 0.60 are considered moderate agreement, and 0.61 to 0.80 are considered substantial agreement. Interobserver agreement did not remain constant between pairs over the 3 rounds. For example, surgeon 2 and 7 had the lowest agreement of all pairs in round 1 (kappa=0.11; 61% agreement) and the highest in round 3 (kappa=0.79; 93% agreement). All pairwise surgeon inter-rater kappa values are provided in Supplemental Table 1 (Supplemental Digital Content 1, http://links.lww.com/BPO/A415). Intraobserver agreement for AVN diagnosis was an average of 0.72 (95% CI: 0.61-0.83; range: 0.31 to 0.96). All surgeon intrarater reliability values are provided in Supplemental Table 2 (Supplemental Digital Content 1, http://links.lww.com/BPO/A415).

Examples of almost perfect agreement of no AVN, almost perfect agreement of yes AVN, and minimal agreement (50% split) are provided in Figures 3A–C, respectively.

FIGURE 3

Example cases—levels of agreement between surgeons. In these examples, there was near perfect agreement between surgeons to the presence (A) or absence (B) of AVN in these children. In some radiographs, there was minimal agreement between surgeons, with 50% of surgeons answering yes and 50% answering no (C). AVN indicates avascular necrosis; CR, closed reduction.

DISCUSSION

The results of this study and the variability seen in the data demonstrate the need for the development of a comprehensive, standardized classification system for AVN specific to the DDH patient population. As of yet, no classification system has been validated for this population. We showed that, despite using the most commonly referenced diagnostic criteria, only moderate agreement was seen across specialized pediatric orthopaedic surgeons for DDH, and the addition of sequential radiographs during follow-up did not improve interobserver reliability. Substantial intraobserver agreement was seen; however, the range of kappa values across surgeons was large, suggesting that surgeon experience and/or practice location may be a contributing factor. Discrepant agreement levels across cases demonstrate that diagnostic variability largely arises from borderline AVN cases.

Few studies have been conducted in the DDH population to determine which classification system is best for diagnosing AVN; however, 1 study looked at the reliability of the Bucholz and Ogden criteria, and found an inter-rater reliability of 0.34 for all raters, which indicates poorer agreement when compared with the Salter criteria, used in this study.24 Important methodological differences do exist between studies to consider: the former study included surgeons, a musculoskeletal radiologist, and orthopaedic trainees, and we included exclusively pediatric orthopaedic surgeons as participants. In addition, the Bucholz and Ogden criteria are graded, and most of the disagreement was between grades I and II, whereas the Salter criteria is a yes or no diagnosis. Because of these differences, it is difficult to make direct comparisons, but the Salter criteria appears either to be more reliable in the DDH population, or simply provides a more general diagnosis. However, the agreement in the Salter criteria is still only moderate, leaving much room for improvement. Some of the existing variability may be reduced in the future through the development and integration of artificial intelligence technology that can automate and standardize the process. However, artificial intelligence still relies on human input and requires a “perfect case” comparison; consequently, there is an evident need for the development of a better objective classification system for AVN.

In addition, there is ongoing debate about potentially important differences between proximal femoral growth disturbance that may be transient, and true AVN. For example, O’Brien et al25 suggest that AVN is not an appropriate term to describe the growth disturbances resulting from treatment for DDH. Instead, the authors propose that characterization of growth disturbance lines and the contour of the physis may be a more accurate means of identifying and describing this phenomenon. Keret and MacEwen26 also suggest a modified approach to the Salter criteria that involves classifying growth disturbances into 3 groups: (1) mild vascular insufficiency; (2) partial necrosis or moderate vascular insufficiency; or (3) total necrosis of the femoral head or severe vascular insufficiency. These proposed modified or alternate classifications will be important to consider in the development of a standardized system for diagnosis.

There are a few limitations of this study. First, we did not control for respondents’ years of experience or practice location, and this likely explains some of the observed variation. However, we included only specialized pediatric orthopaedic surgeons, all of whom had been practicing for >5 years with a specific practice focus on pediatric hip. Second, for the x-rays, we included only anteroposterior radiographs in this study because this view was available for all patients, but other views such as frog leg lateral were not consistently available. Third, we do not know the ultimate impact of a mild growth disturbance early after surgery. Some of these early, mild cases may not be clinically impactful to the patient. Finally, hips that are in the early stages of developing AVN may appear radiographically normal, despite the presence of hip pain, and therefore diagnosis may be delayed until discernable on plain radiograph, or through functional exploration of the bone, which involves a bone scan and investigation into medullary circulation hemodynamics.27 Functional exploration is not routinely carried out, however, and follow-up radiographs are more clinically relevant for the DDH population. Ultimately, to confirm the accuracy of the Salter criteria in correctly diagnosing AVN, large long-term follow-up studies need to be conducted to determine both the accuracy of diagnosis and the clinical and functional impact. In addition, alternate diagnostic methods such as those proposed in O’Brien and colleagues and Karat and colleagues will be valuable to consider.

In conclusion, we found that the inter-rater reliability for the radiographic diagnosis of AVN secondary to DDH treatment “moderate,” and while intrarater reliability was “substantial,” there was a large range of kappa values. It is imperative that an accurate and reliable classification system for this patient population be developed so that we can begin to quantify the incidence of AVN as a treatment complication and identify risk factors and prognostic factors. Further studies are needed to differentiate between a temporary growth disturbance and true AVN. These will help direct diagnosis and further research to resolve the current controversies regarding protective and risk factors, ultimately improving clinical outcomes.

Supplemental Digital Content is available for this article. Direct URL citations appear in the printed text and are provided in the HTML and PDF versions of this article on the journal's website, www.pedorthopaedics.com.