Does the META score evaluating osteoporotic and metastatic vertebral fractures have enough agreement to be used by orthopaedic surgeons with different levels of training?

PURPOSE: Differentiating osteoporotic vertebral fractures (OVF) from metastatic vertebral fractures (MVF) is difficult. A magnetic resonance imaging (MRI)-based score (META score) aiming to differentiate OVF and MVF was recently published; however, an independent agreement assessment is required before the score is used. We performed such independent agreement evaluation, including raters with different levels of training.
METHODS: Sixty-four patients with confirmed OVF or MVF were evaluated by six raters (three spine surgeons and three orthopaedic residents) using the META score. We used the intra-class correlation coefficient (ICC) to evaluate inter- and intra-observer agreement and the kappa statistic (κ) to determine the agreement for individual score criteria. We calculated the area under the receiver-operating characteristic curve (AUC) to establish the score accuracy.
RESULTS: The inter-observer agreement was poor [ICC = 0.22 (0.12-0.33)]; spine surgeons [ICC = 0.75 (0.66-0.83)] had better agreement than that of residents [ICC = 0.06 (- 0.07 to 0.23)]. The intra-observer agreement was poor [ICC = 0.15 (- 0.04 to 0.30)]; both spine surgeons [ICC = 0.21 (0.05-0.41)] and residents exhibited poor agreement [ICC = - 0.06 (- 0.40 to 0.20)]. The agreement for each specific criterion varied from κ = 0.24 to κ = 0.38. The AUC was 0.57 (0.64 for spine surgeons and 0.51 for residents, p < 0.01).
CONCLUSION: The inter-observer agreement using the META score was adequate for spine surgeons but not for residents; the intra-observer agreement was poor. These results do not support the standard use of the META score to differentiate OVF and MVF. These slides can be retrieved under Electronic Supplementary Material.