Rod L Walker1, Susan M Shortreed1, Rebecca A Ziebell1, Eric Johnson1, Jennifer M Boggs2, Frances L Lynch3, Yihe G Daida4, Brian K Ahmedani5, Rebecca Rossom6, Karen J Coleman7, Gregory E Simon1. 1. Kaiser Permanente Washington Health Research Institute, Seattle, Washington, United States. 2. Kaiser Permanente Colorado, Institute for Health Research, Aurora, Colorado, United States. 3. Kaiser Permanente Northwest, Center for Health Research, Portland, Oregon, United States. 4. Kaiser Permanente Hawaii, Center for Integrated Health Care Research, Honolulu, Hawaii, United States. 5. Henry Ford Health System, Center for Health Policy & Health Services Research, Detroit, Michigan, United States. 6. Department of Research, HealthPartners Institute, Minneapolis, Minnesota, United States. 7. Department of Research and Evaluation, Kaiser Permanente Southern California, Pasadena, California, United States.
Abstract
BACKGROUND: Suicide risk prediction models have been developed by using information from patients' electronic health records (EHR), but the time elapsed between model development and health system implementation is often substantial. Temporal changes in health systems and EHR coding practices necessitate the evaluation of such models in more contemporary data. OBJECTIVES: A set of published suicide risk prediction models developed by using EHR data from 2009 to 2015 across seven health systems reported c-statistics of 0.85 for suicide attempt and 0.83 to 0.86 for suicide death. Our objective was to evaluate these models' performance with contemporary data (2014-2017) from these systems. METHODS: We evaluated performance using mental health visits (6,832,439 to mental health specialty providers and 3,987,078 to general medical providers) from 2014 to 2017 made by 1,799,765 patients aged 13+ across the health systems. No visits in our evaluation were used in the previous model development. Outcomes were suicide attempt (health system records) and suicide death (state death certificates) within 90 days following a visit. We assessed calibration and computed c-statistics with 95% confidence intervals (CI) and cut-point specific estimates of sensitivity, specificity, and positive/negative predictive value. RESULTS: Models were well calibrated; 46% of suicide attempts and 35% of suicide deaths in the mental health specialty sample were preceded by a visit (within 90 days) with a risk score in the top 5%. In the general medical sample, 53% of attempts and 35% of deaths were preceded by such a visit. Among these two samples, respectively, c-statistics were 0.862 (95% CI: 0.860-0.864) and 0.864 (95% CI: 0.860-0.869) for suicide attempt, and 0.806 (95% CI: 0.790-0.822) and 0.804 (95% CI: 0.782-0.829) for suicide death. CONCLUSION: Performance of the risk prediction models in this contemporary sample was similar to historical estimates for suicide attempt but modestly lower for suicide death. These published models can inform clinical practice and patient care today. Thieme. All rights reserved.
BACKGROUND: Suicide risk prediction models have been developed by using information from patients' electronic health records (EHR), but the time elapsed between model development and health system implementation is often substantial. Temporal changes in health systems and EHR coding practices necessitate the evaluation of such models in more contemporary data. OBJECTIVES: A set of published suicide risk prediction models developed by using EHR data from 2009 to 2015 across seven health systems reported c-statistics of 0.85 for suicide attempt and 0.83 to 0.86 for suicide death. Our objective was to evaluate these models' performance with contemporary data (2014-2017) from these systems. METHODS: We evaluated performance using mental health visits (6,832,439 to mental health specialty providers and 3,987,078 to general medical providers) from 2014 to 2017 made by 1,799,765 patients aged 13+ across the health systems. No visits in our evaluation were used in the previous model development. Outcomes were suicide attempt (health system records) and suicide death (state death certificates) within 90 days following a visit. We assessed calibration and computed c-statistics with 95% confidence intervals (CI) and cut-point specific estimates of sensitivity, specificity, and positive/negative predictive value. RESULTS: Models were well calibrated; 46% of suicide attempts and 35% of suicide deaths in the mental health specialty sample were preceded by a visit (within 90 days) with a risk score in the top 5%. In the general medical sample, 53% of attempts and 35% of deaths were preceded by such a visit. Among these two samples, respectively, c-statistics were 0.862 (95% CI: 0.860-0.864) and 0.864 (95% CI: 0.860-0.869) for suicide attempt, and 0.806 (95% CI: 0.790-0.822) and 0.804 (95% CI: 0.782-0.829) for suicide death. CONCLUSION: Performance of the risk prediction models in this contemporary sample was similar to historical estimates for suicide attempt but modestly lower for suicide death. These published models can inform clinical practice and patient care today. Thieme. All rights reserved.
Authors: Gregory E Simon; Eric Johnson; Jean M Lawrence; Rebecca C Rossom; Brian Ahmedani; Frances L Lynch; Arne Beck; Beth Waitzfelder; Rebecca Ziebell; Robert B Penfold; Susan M Shortreed Journal: Am J Psychiatry Date: 2018-05-24 Impact factor: 18.112
Authors: Yuval Barak-Corren; Victor M Castro; Solomon Javitt; Alison G Hoffnagle; Yael Dai; Roy H Perlis; Matthew K Nock; Jordan W Smoller; Ben Y Reis Journal: Am J Psychiatry Date: 2016-09-09 Impact factor: 18.112
Authors: Rebecca C Rossom; Gregory E Simon; Arne Beck; Brian K Ahmedani; Bradley Steinfeld; Michael Trangle; Leif Solberg Journal: Psychiatr Serv Date: 2016-04-01 Impact factor: 3.084
Authors: Ronald C Kessler; Irving Hwang; Claire A Hoffmire; John F McCarthy; Maria V Petukhova; Anthony J Rosellini; Nancy A Sampson; Alexandra L Schneider; Paul A Bradley; Ira R Katz; Caitlin Thompson; Robert M Bossarte Journal: Int J Methods Psychiatr Res Date: 2017-07-04 Impact factor: 4.035
Authors: Gregory E Simon; Susan M Shortreed; Eric Johnson; Rebecca C Rossom; Frances L Lynch; Rebecca Ziebell; And Robert B Penfold Journal: J Am Med Inform Assoc Date: 2019-12-01 Impact factor: 4.497
Authors: Ben Van Calster; Laure Wynants; Dirk Timmerman; Ewout W Steyerberg; Gary S Collins Journal: J Am Med Inform Assoc Date: 2019-12-01 Impact factor: 4.497
Authors: Patrick Rockenschaub; Vincent Nguyen; Robert W Aldridge; Dionisio Acosta; Juan Miguel García-Gómez; Carlos Sáez Journal: BMJ Open Date: 2020-02-13 Impact factor: 2.692
Authors: Rebecca B Morse; Abigail C Bretzin; Silvia P Canelón; Bernadette A D'Alonzo; Andrea L C Schneider; Mary R Boland Journal: Appl Clin Inform Date: 2022-03-09 Impact factor: 2.342