BACKGROUND: The emerging "pay for performance" national initiative mandates the development of valid metrics for risk stratification and performance assessment. The International Classification Injury Severity Score (ICISS) predicts survival from injury and is calculated as the product of survival risk ratios (SRRs) for a patient's 3 worst injuries. Survival risk ratios are derived as the proportion of fatalities for every International Classification of Diseases, Ninth Edition, Clinical Modification, diagnosis in a "benchmark" population. We hypothesized that the ICISS prediction model derived from the National Pediatric Trauma Registry (NPTR) would accurately predict mortality in an independent sample from a single pediatric trauma center (PTC) and could be applied to the NSQIP methodology to analyze performance. METHODS: The ICISS survival probabilities (Ps) were calculated for PTC patients using SRRs computed from 102,608 NPTR records. Records with a single diagnosis and Ps of 1 were excluded from the analysis. Receiver operator characteristics analysis (ROC) was used to evaluate the accuracy of Ps to predict mortality. The Hosmer-Lemeshow statistic was used to determine the degree that the NPTR-derived expected probabilities matched the observed mortality profile at the PTC. Program performance from 2000 to 2004 was then evaluated using Ps adjusted by logit transformation to predict expected mortality (E) for each year cohort. Observed mortality divided by expected mortality (O/E) was calculated for each year group to compare PTC performance to the NSQIP standard of one. The influence of injury severity on these results was determined by evaluating the correlation between O/E and mean Ps of each year cohort. RESULTS: A total of 1523 records were analyzed. The ROC area under the curve (AUC ) for Ps was .947 (confidence interval, .934-.957). The Hosmer-Lemeshow statistic (chi(2) = 5.102; df = 8; P = .747, not significant) indicated the model fit the data well. Adjusted O/E ratio after logit transformation of Ps for the PTC demonstrated initial performance slightly below standard (1.000778) followed by performance better than expected for the subsequent 4 years (range, .6466-.9784). The ratio of observed (O) to expected (E) demonstrated no correlation to mean Ps (r(2) = .378; P = .208). CONCLUSION: These data validate the application of injury diagnosis derived survival probabilities as objective metrics for determining performance using the NSQIP methodology. Incorporation of these objective predictors of expected outcome to calculation of the risk adjusted O/E ratio enables trend analysis of program performance over time. The lack of significant correlation between O/E and mean Ps demonstrates that NSQIP does indeed reflect process of care while adjusting for severity of patient pathologic condition.
BACKGROUND: The emerging "pay for performance" national initiative mandates the development of valid metrics for risk stratification and performance assessment. The International Classification Injury Severity Score (ICISS) predicts survival from injury and is calculated as the product of survival risk ratios (SRRs) for a patient's 3 worst injuries. Survival risk ratios are derived as the proportion of fatalities for every International Classification of Diseases, Ninth Edition, Clinical Modification, diagnosis in a "benchmark" population. We hypothesized that the ICISS prediction model derived from the National Pediatric Trauma Registry (NPTR) would accurately predict mortality in an independent sample from a single pediatric trauma center (PTC) and could be applied to the NSQIP methodology to analyze performance. METHODS: The ICISS survival probabilities (Ps) were calculated for PTC patients using SRRs computed from 102,608 NPTR records. Records with a single diagnosis and Ps of 1 were excluded from the analysis. Receiver operator characteristics analysis (ROC) was used to evaluate the accuracy of Ps to predict mortality. The Hosmer-Lemeshow statistic was used to determine the degree that the NPTR-derived expected probabilities matched the observed mortality profile at the PTC. Program performance from 2000 to 2004 was then evaluated using Ps adjusted by logit transformation to predict expected mortality (E) for each year cohort. Observed mortality divided by expected mortality (O/E) was calculated for each year group to compare PTC performance to the NSQIP standard of one. The influence of injury severity on these results was determined by evaluating the correlation between O/E and mean Ps of each year cohort. RESULTS: A total of 1523 records were analyzed. The ROC area under the curve (AUC ) for Ps was .947 (confidence interval, .934-.957). The Hosmer-Lemeshow statistic (chi(2) = 5.102; df = 8; P = .747, not significant) indicated the model fit the data well. Adjusted O/E ratio after logit transformation of Ps for the PTC demonstrated initial performance slightly below standard (1.000778) followed by performance better than expected for the subsequent 4 years (range, .6466-.9784). The ratio of observed (O) to expected (E) demonstrated no correlation to mean Ps (r(2) = .378; P = .208). CONCLUSION: These data validate the application of injury diagnosis derived survival probabilities as objective metrics for determining performance using the NSQIP methodology. Incorporation of these objective predictors of expected outcome to calculation of the risk adjusted O/E ratio enables trend analysis of program performance over time. The lack of significant correlation between O/E and mean Ps demonstrates that NSQIP does indeed reflect process of care while adjusting for severity of patient pathologic condition.
Authors: Casey J Allen; Amy E Wagenaar; Davis B Horkan; Daniel J Baldor; William M Hannay; Jun Tashiro; Nicholas Namias; Juan E Sola Journal: Pediatr Surg Int Date: 2016-06-02 Impact factor: 1.827