INTER- AND INTRA-OBSERVER AGREEMENT IN THE MILCH AND WEISS SYSTEMS.

OBJECTIVE: The aim of this study is to estimate the inter- and intra-observer agreement of the Weiss and Milch classification systems in radiological studies of fractures of the lateral humeral condyle in pediatric patients.
METHODS: An agreement study was performed with non-probability sampling of consecutive cases with a sample size of a hundred radiological studies, which were evaluated by three experienced orthopedic surgeons and three resident physicians; following a thorough comparison of both inter- and intra-observer agreements over a six-week period based on the Fleiss' kappa, which was used to determine the inter- and intra-observer agreement rates of both classifications.
RESULTS: The overall reliability of the entire group of evaluators for the Milch classification in the inter-observer evaluation was κ = 0.13, 95% CI, 0.08-0.18, and the intra-observer evaluation was κ = 0.08, 95% CI, 0.06-0.11. For the Weiss classification, the overall evaluation had an inter-observer agreement of κ = 0.53, 95% CI, 0.50-0.57, and an intra-observer agreement of κ = 0.22, 95% CI, 0.20 -0.24.
CONCLUSION: In the present study, the Weiss classification system demonstrated greater agreement than the Milch classification; however, the latter may require complementary studies, such as an arthrography to enhance classification accuracy. Level of Evidence II, Diagnostic Studies - Investigating Diagnostic Examination.

INTRODUCTION

The ideal system for classifying fractures is one that enhances communication between doctors, standardizes research terminology, and guides the decision-making process on the best treatment for each patient. , An increase in the incidence of pediatric fractures of the lateral humeral condyle has recently been reported. Some authors report an increase of 9.3%, , while others report a range between 12 and 13.5%. , Needless to say, the accurate classification of this type of fracture displacement and joint stability is extremely important to assess the best treatment for these patients, which could be surgical or conservative. In addition, open reduction and internal fixation could be an unnecessary treatment that could potentially increase the risk of avascular necrosis due to extensive dissection. Nevertheless, conservative treatment, or a percutaneous fixation on an unstable or displaced fracture, could result in consolidation failure and residual deformities. Despite being one of the most widely used classification systems in pediatric orthopedic hospital services, the original Milch classification has not yet demonstrated its usefulness in defining treatment and prognosis, and has been reported to have an inter-observer agreement with a Cohen kappa κ = 0.12. The recently-introduced Weiss classification system, which is also based on joint displacement and congruence, may be able to predict fracture complications, and could indicate the best treatment to follow. The use of an adequate classification could reduce indications for surgery, especially in fractures with minimal displacement, provided the displacement and follow-up can be ensured, hence the importance of using a proper classification system.

There have been no repeatability and reproducibility tests among different observers with different levels of experience and training in the use of these classification systems. Therefore, the main objective of this study was to determine the repeatability and reproducibility of the Weiss and Milch classification systems for pediatric fractures of the lateral humeral condyle in pediatric patients by evaluating the intra- and inter-observer agreements in radiological studies with multiple observers and outcomes.

MATERIAL AND METHODS

In 2017, an agreement study was performed at the Pediatric Orthopedics Service of the High-specialty Medical Unit of the Trauma and Orthopedics Hospital (Hospital de Traumatología y Ortopedia No. 21) of the Mexican Social Security Institute in Monterrey, Mexico. Prior approval from the Research Ethics Committee (R-2017-1903-11) was gained before conducting the study; all the parents signed an informed consent form, and radiological studies from the daily admission sheets of patients between 2015 and 2016 were taken from our hospital's imaging database, for use in the study. Patients aged between 3 and 14 years of age with anteroposterior and lateral radiological studies of the affected elbow were included. Patients with both fracture and elbow luxation, ipsilateral fracture in another area of the elbow, or with repeat fractures, were excluded. In order to measure agreement, the studies were blindly sorted and individually evaluated by three experienced pediatric orthopedists, and by three fourth-year orthopedic surgical residents. The evaluators were summoned on different days and were given the definitions and diagrams of the Milch and Weiss classifications. Figure 1 describes the Milch classification for lateral humeral condyle fractures, Type I having a fracture line that is lateral to the trochlear groove, and Type II having a fracture line into the trochlear groove. Figure 2 describes the Weiss classification system for lateral humeral condyle fractures; Type I fractures have a displacement of less than 2 mm, while Type II fractures have a displacement greater than or equal to 2mm with an intact articular cartilage, and Type III fractures have a displacement greater than or equal to 2 mm with an affected articular surface. The radiographs were shown to the evaluators. Each evaluator was given a test-like form and asked to classify the radiological studies, without receiving any information about the treatment that was previously prescribed to each patient. The amount of time considered adequate for measurement of intra-observer agreement was six weeks, during which time the images from the same cohort of patients were randomly shown to the evaluators. To estimate the sample size, previously reported data of Cohen's kappa from the Pennington et al. study was taken into account. An estimation of the confidence interval in studies seeking agreement among multiple observers was used, as reported by Rotondi et al. and Donner et al. In the present study, which had three observers in each group and an α = 0.05, with a lower limit of 0.70 and an upper limit of 0.90, the results for the sample size was 65 patients, but the decision to evaluate 100 radiographs was made in order to increase the variability. The data were both categorically and mutually exclusive for each category. For the analysis, the general kappa index was calculated as described by Fleiss. The agreement levels for the kappa coefficient were in accordance with those proposed by Landis and Koch, who classified them in categories ranging from: < 0.00, poor agreement; to 0.00-0.20 slight agreement; 0.21 −0.40 fair agreement; 0.41 −0.60 moderate agreement; 0.61-0.80 substantial agreement, and 0.81 −1.00 almost perfect agreement. All the statistical analyses were performed with SPSS statistical software version 22 (SPSS, Chicago, IL, USA).

Figure 1

Milch classification for lateral humeral condyle fractures.

Figure 2

Weiss classification for lateral humeral condyle fractures.

RESULTS

One hundred radiological tests were analyzed by the evaluators, from which the intra- and inter-observer agreements were determined. Among the three experienced orthopedic surgeons, the overall inter-observer reproducibility for the Milch classification system resulted in moderate agreement (κ = 0.41; 95% CI, 0.30-0.53). On the other hand, the same group of observers obtained substantial agreement (κ = 0.61; 95% CI, 0.51-0.69) for the Weiss classification in overall inter-observer reproducibility. For the evaluations performed by the three orthopedic surgical residents, the Milch classification gained very poor, null and negative agreement (κ = −0.04; 95% CI, −0.07-0.15) in the overall inter-observer reproducibility. In the same group of residents, the results of the overall inter-observer reproducibility for the Weiss classification indicated moderate agreement (κ = 0.47; 95% CI, 0.39-0.55). The details of each assessment, as well as the types of classification and fracture patterns, are shown in Tables 1 and 2. The estimation of the intra-observer agreement performed by the group of experienced observers, specialists in pediatric orthopedics, using the Milch classification, showed agreement of κ = 0.20; 95% CI, 0.15-0.25. In the same group of observers, the results obtained for the Weiss classification were κ = 0.22; 95% CI, 0.19-0.28, which translated into fair agreement in both evaluations. The intra-observer agreement in the orthopedic residents group showed poor agreement (κ = −0.01; 95% CI, −0.06-0.04) when using the Milch classification. Finally, the evaluation of intra-observer agreement among the same group of residents using the Weiss classification showed slight agreement (κ = 0.17; 95% CI, 0.13-0.20). The details of the intra-observer evaluation are shown in Tables 3 and 4; Table 5 shows the agreement of the entire group of evaluators. Overall, the Weiss classification obtained moderate inter-observer agreement (κ = 0.53; 95% CI, 0.50-0.57) and fair intra-observer agreement (κ = 0.22; 95% CI, 0.20-0.24). Using the Milch classification, the inter-observer agreement was κ = 0.13; 95% CI, 0.08-0.18 and the intra-observer agreement was κ = 0.08; 95% CI, 0.06-0.11, which translates into slight agreement for both results.

Table 1

Inter-observer agreement for the Milch and Weiss classification (Group of expert evaluators).

Classification	Kappa	95% CI
Milch
I	0.41	0.30 – 0.53
II	0.41	0.30 – 0.53
Overall	0.41	0.30 – 0.53
Weiss
I	0.72	0.61 – 0.83
II	0.37	0.26 – 0.49
III	0.68	0.57 – 0.80
Overall	0.61	0.51 – 0.69

Table 2

Inter-observer agreement for the Milch and Weiss classification (Medical residents group).

Classification	Kappa	95% CI
Milch
I	0.04	- 0.08 – 0.15
II	0.04	- 0.08 – 0.15
Overall	0.04	- 0.08 – 0.15
Weiss
I	0.47	0.38 – 0.60
II	0.27	0.15 – 0.38
III	0.62	0.51 – 0.73
Overall	0.47	0.39 – 0.55

Table 3

Intra-observer agreement for the Milch and Weiss classification (Group of expert evaluators).

Classification	Kappa	95% CI
Milch
I	0.20	0.15 – 0.25
II	0.20	0.15 – 0.25
Overall	0.20	0.15 – 0.25
Weiss
I	0.27	0.21 – 0.32
II	0.12	0.07 – 0.17
III	0.26	0.21 – 0.32
Overall	0.22	0.19 – 0.28

Table 4

Intra-observer agreement for the Milch and Weiss classification (Medical residents group).

Classification	Kappa	95% CI
Milch
I	- 0.01	- 0.06 – 0.04
II	- 0.01	- 0.06 – 0.04
Overall	- 0.01	- 0.06 – 0.04
Weiss
I	0.17	0.12 – 0.22
II	0.05	0.001 – 0.10
III	0.25	0.20 – 0.30
Overall	0.17	0.13 – 0.20

Table 5

Inter-observer and Intra-observer agreement for the Milch and Weiss classification (Entire group).

Intra-observer agreement			Inter-observer agreement
Classification	Kappa	95% CI	Kappa	95% CI
Milch
I	0.08	0.06 – 0.11	0.13	0.08 – 0.18
II	0.08	0.06 – 0.11	0.13	0.08 – 0.18
Overall	0.08	0.06 – 0.11	0.13	0.08 – 0.18
Weiss
I	0.25	0.23 – 0.27	0.59	0.54 – 0.64
II	0.10	0.07 – 0.12	0.30	0.25 – 0.35
III	0.30	0.27 – 0.32	0.67	0.62 – 0.72
Overall	0.22	0.20 – 0.24	0.53	0.50 – 0.57

DISCUSSION

Multiple attempts have been made to design a classification system for lateral humeral condyle in pediatric patients that is able to answer questions regarding the mechanism of injury, anatomical difference in the displacement, and the best treatment for each fracture. To assess such questions, the concordance of both classification systems was compared. In the present study, the Milch classification, when used by medical residents, obtained intra-observer agreement of κ = −0.01 and inter-observer agreement of κ = 0.04, indicating poor and fair agreements, respectively. This corroborates with the previous study performed by Pennington et al. which reported fair inter-observer agreement (κ = 0.12); however, that same study obtained moderate intra-observer agreement (κ = 0.45) when comparing the original and modified Milch classifications. The evaluations performed by the expert physicians with the Milch classification showed a higher degree of inter-observer agreement, achieving moderate agreement (κ = 0.41) along with slight intra-observer agreement (κ = 0.20), which could possibly reflect repeatability with variable consistency and stability. Moreover, the Weiss classification in the current study demonstrated higher degrees of agreement than those demonstrated with the Milch classification in the group of expert physicians, and the inter-observer estimation was substantial (κ = 0.61). On the contrary, the intra-observer agreement fell to fair levels (κ = 0.22), which may be attributed to external reliability in the interpretations of the classification, since the evaluations were performed under similar conditions.

On the other hand, attempts to enhance evaluation of these fractures, as performed by Song et al. , using the Jakob classification conclude that in order to properly classify the degree of displacement and prescribe the most adequate treatment, an internal oblique radiograph in type I, and an arthrography in types II and III of said classification, must be performed. A report evaluating fractures that showed minimal displacement in a radiograph, therefore being treated conservatively, recommended that the routine use of magnetic resonance imaging in follow-up is consistent with a stable fracture, thus preventing the need for further imaging or clinical studies. Another recent report concluded that apart from assessing joint stability, fractures with a displacement greater than 2 mm should be assessed for surgery and that the majority of them should be treated with an open reduction and internal fixation.

One of the main limitations of our study is that in the classification developed by Weiss et al. for patients whose fracture caused uncertainty regarding joint congruence and had a displacement greater than 2 mm, arthrography was used to determine the best treatment: closed reduction with nails, or open reduction with internal fixation. Arthrography was not used in the present study, which may have affected the correct classification of the joint congruence of the fracture. This limitation is possibly reflected in the difficulty of classifying fractures as a type II, which obtained fair reliability (κ = 0.27). However, it was reported in these evaluations that in fractures classified as type III with displacement greater than 4 mm, arthrography was not performed to corroborate the joint congruence. In view of this, arthrography that have be used in fractures with displacement greater than 4 mm that could have an intact articular surface and may be treated with closed reduction and nails. Another limitation of this study is that no internal oblique radiographs were taken; these have proven to be a good method to determine the greatest displacement of the fracture, and this knowledge is useful for classifying and treating fractures.

CONCLUSION

In hospitals where resources are limited, there is a need for an easy-to-interpret classification system that does not require the costly studies. Based on our results, conservative treatment for lateral humeral condyle pediatric fractures in these environments should only be indicated when follow-up is guaranteed. In the present study, the Weiss classification showed greater agreement than the Milch classification; nonetheless, complementary studies, such as arthrography, may be needed to classify fractures adequately.