Dan Zeltzer1, Ran D Balicer2,3, Tzvi Shir2, Natalie Flaks-Manov2, Liran Einav4,5, Efrat Shadmi2,6. 1. School of Economics, Tel Aviv University. 2. Clalit Health Services, Clalit Research Institute, Tel Aviv. 3. Department of Epidemiology, Faculty of Health Sciences, Ben Gurion University, Beersheba, Israel. 4. Department of Economics, Stanford University, Stanford, CA. 5. National Bureau of Economic Research, Cambridge, MA. 6. Faculty of Social Welfare and Health Sciences, University of Haifa, Haifa, Israel.
Abstract
OBJECTIVE: The objective of this study was to evaluate the incremental predictive power of electronic medical record (EMR) data, relative to the information available in more easily accessible and standardized insurance claims data. DATA AND METHODS: Using both EMR and Claims data, we predicted outcomes for 118,510 patients with 144,966 hospitalizations in 8 hospitals, using widely used prediction models. We use cross-validation to prevent overfitting and tested predictive performance on separate data that were not used for model training. MAIN OUTCOMES: We predict 4 binary outcomes: length of stay (≥7 d), death during the index admission, 30-day readmission, and 1-year mortality. RESULTS: We achieve nearly the same prediction accuracy using both EMR and claims data relative to using claims data alone in predicting 30-day readmissions [area under the receiver operating characteristic curve (AUC): 0.698 vs. 0.711; positive predictive value (PPV) at top 10% of predicted risk: 37.2% vs. 35.7%], and 1-year mortality (AUC: 0.902 vs. 0.912; PPV: 64.6% vs. 57.6%). EMR data, especially from the first 2 days of the index admission, substantially improved prediction of length of stay (AUC: 0.786 vs. 0.837; PPV: 58.9% vs. 55.5%) and inpatient mortality (AUC: 0.897 vs. 0.950; PPV: 24.3% vs. 14.0%). Results were similar for sensitivity, specificity, and negative predictive value across alternative cutoffs and for using alternative types of predictive models. CONCLUSION: EMR data are useful in predicting short-term outcomes. However, their incremental value for predicting longer-term outcomes is smaller. Therefore, for interventions that are based on long-term predictions, using more broadly available claims data is equally effective.
OBJECTIVE: The objective of this study was to evaluate the incremental predictive power of electronic medical record (EMR) data, relative to the information available in more easily accessible and standardized insurance claims data. DATA AND METHODS: Using both EMR and Claims data, we predicted outcomes for 118,510 patients with 144,966 hospitalizations in 8 hospitals, using widely used prediction models. We use cross-validation to prevent overfitting and tested predictive performance on separate data that were not used for model training. MAIN OUTCOMES: We predict 4 binary outcomes: length of stay (≥7 d), death during the index admission, 30-day readmission, and 1-year mortality. RESULTS: We achieve nearly the same prediction accuracy using both EMR and claims data relative to using claims data alone in predicting 30-day readmissions [area under the receiver operating characteristic curve (AUC): 0.698 vs. 0.711; positive predictive value (PPV) at top 10% of predicted risk: 37.2% vs. 35.7%], and 1-year mortality (AUC: 0.902 vs. 0.912; PPV: 64.6% vs. 57.6%). EMR data, especially from the first 2 days of the index admission, substantially improved prediction of length of stay (AUC: 0.786 vs. 0.837; PPV: 58.9% vs. 55.5%) and inpatient mortality (AUC: 0.897 vs. 0.950; PPV: 24.3% vs. 14.0%). Results were similar for sensitivity, specificity, and negative predictive value across alternative cutoffs and for using alternative types of predictive models. CONCLUSION: EMR data are useful in predicting short-term outcomes. However, their incremental value for predicting longer-term outcomes is smaller. Therefore, for interventions that are based on long-term predictions, using more broadly available claims data is equally effective.