Hideki Endo1,2, Shigehiko Uchino3, Satoru Hashimoto4, Yoshitaka Aoki5, Eiji Hashiba6, Junji Hatakeyama7, Katsura Hayakawa8, Nao Ichihara9, Hiromasa Irie10, Tatsuya Kawasaki11, Junji Kumasawa12, Hiroshi Kurosawa13, Tomoyuki Nakamura14, Hiroyuki Ohbe15, Hiroshi Okamoto16, Hidenobu Shigemitsu17, Takashi Tagami18, Shunsuke Takaki19, Kohei Takimoto20, Masatoshi Uchida21, Hiroaki Miyata9,22. 1. Department of Healthcare Quality Assessment, Graduate School of Medicine, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan. hidendo-thk@umin.ac.jp. 2. Department of Health Policy and Management, School of Medicine, Keio University, 35 Shinanomachi, Shinjuku-ku, Tokyo, 160-8582, Japan. hidendo-thk@umin.ac.jp. 3. Intensive Care Unit, The Jikei University School of Medicine, 3-19-18 Nishi-Shinbashi, Minato-ku, Tokyo, 105-8471, Japan. 4. Department of Anesthesiology and Intensive Care Medicine, Kyoto Prefectural University of Medicine, 465 Kajii-cho, Kawaramachi-Hirokoji, Kamigyo-ku, Kyoto, 602-8566, Japan. 5. Department of Anesthesiology and Intensive Care Medicine, Hamamatsu University School of Medicine, 1-20-1 Handayama, Higashi-ku, Hamamatsu, Shizuoka, 431-3192, Japan. 6. Division of Intensive Care, Hirosaki University Hospital, 53 Honcho, Hirosaki, Aomori, 036-8203, Japan. 7. Department of Emergency and Critical Care Medicine, National Hospital Organization Tokyo Medical Center, 2-5-1, Higashigaoka, Meguro-ku, Tokyo, 152-8902, Japan. 8. Department of Emergency and Critical Care Medicine, Saitama Red Cross Hospital, 1-5 Shintoshin, Chuo-ku, Saitama, 330-8553, Japan. 9. Department of Healthcare Quality Assessment, Graduate School of Medicine, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan. 10. Department of Anesthesiology, Kurashiki Central Hospital, 1-1-1 Miwa, Kurashiki, Okayama, 710-8602, Japan. 11. Department of Pediatric Critical Care, Shizuoka Children's Hospital, 860 Urushiyama, Aoi-ku, Shizuoka, Shizuoka, 420-8660, Japan. 12. Department of Critical Care Medicine, Sakai City Medical Center, 1-1-1 Ebaraji-cho, Nishi-ku, Sakai, Osaka, 593-8304, Japan. 13. Department of Pediatric Critical Care Medicine, Hyogo Prefectural Kobe Children's Hospital, 1-6-7 Minatojima Minamimachi, Chuo-ku, Kobe, Hyogo, 650-0047, Japan. 14. Department of Anesthesiology and Critical Care Medicine, Fujita Health University School of Medicine, 1-98 Dengakugakubo, Kutsukake-cho, Toyoake, Aichi, 470-1192, Japan. 15. Department of Clinical Epidemiology and Health Economics, School of Public Health, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-0033, Japan. 16. Department of Critical Care Medicine, St. Luke's International Hospital, 9-1 Akashi-cho, Chuo-ku, Tokyo, 104-8560, Japan. 17. Department of Intensive Care Medicine, Graduate School of Medicine, Tokyo Medical and Dental University, 1-5-45 Yushima, Bunkyo-ku, Tokyo, 113-8519, Japan. 18. Department of Emergency and Critical Care Medicine, Nippon Medical School Musashikosugi Hospital, 1-396 Kosugimachi, Nakahara-ku, Kawasaki, Kanagawa, 211-8533, Japan. 19. Department of Anesthesiology and Critical Care Medicine, Yokohama City University, 3-9 Fukuura, Kanazawa-ku, Yokohama, Kanagawa, 236-0004, Japan. 20. Department of Intensive Care Medicine, Kameda Medical Center, 929 Higashi-cho, Kamogawa, Chiba, 296-8602, Japan. 21. Department of Emergency and Critical Care Medicine, Dokkyo Medical University, 880 Kitakobayashi, Mibu-machi, Shimotsuga-gun, Tochigi, 321-0293, Japan. 22. Department of Health Policy and Management, School of Medicine, Keio University, 35 Shinanomachi, Shinjuku-ku, Tokyo, 160-8582, Japan.
Abstract
BACKGROUND: The Acute Physiology and Chronic Health Evaluation (APACHE) III-j model is widely used to predict mortality in Japanese intensive care units (ICUs). Although the model's discrimination is excellent, its calibration is poor. APACHE III-j overestimates the risk of death, making its evaluation of healthcare quality inaccurate. This study aimed to improve the calibration of the model and develop a Japan Risk of Death (JROD) model for benchmarking purposes. METHODS: A retrospective analysis was conducted using a national clinical registry of ICU patients in Japan. Adult patients admitted to an ICU between April 1, 2018, and March 31, 2019, were included. The APACHE III-j model was recalibrated with the following models: Model 1, predicting mortality with an offset variable for the linear predictor of the APACHE III-j model using a generalized linear model; model 2, predicting mortality with the linear predictor of the APACHE III-j model using a generalized linear model; and model 3, predicting mortality with the linear predictor of the APACHE III-j model using a hierarchical generalized additive model. Model performance was assessed with the area under the receiver operating characteristic curve (AUROC), the Brier score, and the modified Hosmer-Lemeshow test. To confirm model applicability to evaluating quality of care, funnel plots of the standardized mortality ratio and exponentially weighted moving average (EWMA) charts for mortality were drawn. RESULTS: In total, 33,557 patients from 44 ICUs were included in the study population. ICU mortality was 3.8%, and hospital mortality was 8.1%. The AUROC, Brier score, and modified Hosmer-Lemeshow p value of the original model and models 1, 2, and 3 were 0.915, 0.062, and < .001; 0.915, 0.047, and < .001; 0.915, 0.047, and .002; and 0.917, 0.047, and .84, respectively. Except for model 3, the funnel plots showed overdispersion. The validity of the EWMA charts for the recalibrated models was determined by visual inspection. CONCLUSIONS: Model 3 showed good performance and can be adopted as the JROD model for monitoring quality of care in an ICU, although further investigation of the clinical validity of outlier detection is required. This update method may also be useful in other settings.
BACKGROUND: The Acute Physiology and Chronic Health Evaluation (APACHE) III-j model is widely used to predict mortality in Japanese intensive care units (ICUs). Although the model's discrimination is excellent, its calibration is poor. APACHE III-j overestimates the risk of death, making its evaluation of healthcare quality inaccurate. This study aimed to improve the calibration of the model and develop a Japan Risk of Death (JROD) model for benchmarking purposes. METHODS: A retrospective analysis was conducted using a national clinical registry of ICU patients in Japan. Adult patients admitted to an ICU between April 1, 2018, and March 31, 2019, were included. The APACHE III-j model was recalibrated with the following models: Model 1, predicting mortality with an offset variable for the linear predictor of the APACHE III-j model using a generalized linear model; model 2, predicting mortality with the linear predictor of the APACHE III-j model using a generalized linear model; and model 3, predicting mortality with the linear predictor of the APACHE III-j model using a hierarchical generalized additive model. Model performance was assessed with the area under the receiver operating characteristic curve (AUROC), the Brier score, and the modified Hosmer-Lemeshow test. To confirm model applicability to evaluating quality of care, funnel plots of the standardized mortality ratio and exponentially weighted moving average (EWMA) charts for mortality were drawn. RESULTS: In total, 33,557 patients from 44 ICUs were included in the study population. ICU mortality was 3.8%, and hospital mortality was 8.1%. The AUROC, Brier score, and modified Hosmer-Lemeshow p value of the original model and models 1, 2, and 3 were 0.915, 0.062, and < .001; 0.915, 0.047, and < .001; 0.915, 0.047, and .002; and 0.917, 0.047, and .84, respectively. Except for model 3, the funnel plots showed overdispersion. The validity of the EWMA charts for the recalibrated models was determined by visual inspection. CONCLUSIONS: Model 3 showed good performance and can be adopted as the JROD model for monitoring quality of care in an ICU, although further investigation of the clinical validity of outlier detection is required. This update method may also be useful in other settings.
Entities:
Keywords:
Benchmarking; Quality improvement; Quality indicator; Recalibration; Risk of death; Risk prediction model