Polina V Kukhareva1, Charlene Weir2, Guilherme Del Fiol3, Gregory A Aarons4, Teresa Y Taft5, Chelsey R Schlechter6, Thomas J Reese7, Rebecca L Curran8, Claude Nanjo9, Damian Borbolla10, Catherine J Staes11, Keaton L Morgan12, Heidi S Kramer13, Carole H Stipelman14, Julie H Shakib15, Michael C Flynn16, Kensaku Kawamoto17. 1. Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA. Electronic address: polina.kukhareva@utah.edu. 2. Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA. Electronic address: charlene.weir@utah.edu. 3. Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA. Electronic address: guilherme.delfiol@utah.edu. 4. Department of Psychiatry, UC San Diego ACTRI Dissemination and Implementation Science Center, UC San Diego, La Jolla, CA, USA; Dissemination and Implementation Science Center, Altman Clinical and Translational Research Institute, UC San Diego, La Jolla, CA, USA. Electronic address: gaarons@health.ucsd.edu. 5. Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA. Electronic address: teresa.taft@utah.edu. 6. Department of Population Health Sciences, Center for Health Outcomes and Population Equity, Huntsman Cancer Institute, University of Utah, Salt Lake City, UT, USA; Center for Health Outcomes and Population Equity, Huntsman Cancer Institute, University of Utah, Salt Lake City, UT, USA. Electronic address: chelsey.schlechter@hci.utah.edu. 7. Department of Biomedical Informatics, Vanderbilt University, Nashville, TN, USA. Electronic address: thomas.reese@vumc.org. 8. Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA. Electronic address: rebecca.curran@hsc.utah.edu. 9. Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA. Electronic address: claude.nanjo@utah.edu. 10. Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA. Electronic address: damian.borbolla@utah.edu. 11. College of Nursing, University of Utah, Salt Lake City, UT, USA. Electronic address: catherine.staes@hsc.utah.edu. 12. Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA. Electronic address: keaton.morgan@utah.edu. 13. Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA. Electronic address: heidi.kramer@hsc.utah.edu. 14. Department of Pediatrics, University of Utah, Salt Lake City, UT, USA. Electronic address: carole.stipelman@hsc.utah.edu. 15. Department of Pediatrics, University of Utah, Salt Lake City, UT, USA. Electronic address: julie.shakib@hsc.utah.edu. 16. Department of Pediatrics, University of Utah, Salt Lake City, UT, USA; Department of Internal Medicine, University of Utah, Salt Lake City, UT, USA; Community Physicians Group, University of Utah Health, Salt Lake City, UT, USA. Electronic address: michael.flynn@hsc.utah.edu. 17. Department of Biomedical Informatics, University of Utah, Salt Lake City, UT, USA. Electronic address: kensaku.kawamoto@utah.edu.
Abstract
OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.
OBJECTIVE: Our objective was to develop an evaluation framework for electronic health record (EHR)-integrated innovations to support evaluation activities at each of four information technology (IT) life cycle phases: planning, development, implementation, and operation. METHODS: The evaluation framework was developed based on a review of existing evaluation frameworks from health informatics and other domains (human factors engineering, software engineering, and social sciences); expert consensus; and real-world testing in multiple EHR-integrated innovation studies. RESULTS: The resulting Evaluation in Life Cycle of IT (ELICIT) framework covers four IT life cycle phases and three measure levels (society, user, and IT). The ELICIT framework recommends 12 evaluation steps: (1) business case assessment; (2) stakeholder requirements gathering; (3) technical requirements gathering; (4) technical acceptability assessment; (5) user acceptability assessment; (6) social acceptability assessment; (7) social implementation assessment; (8) initial user satisfaction assessment; (9) technical implementation assessment; (10) technical portability assessment; (11) long-term user satisfaction assessment; and (12) social outcomes assessment. DISCUSSION: Effective evaluation requires a shared understanding and collaboration across disciplines throughout the entire IT life cycle. In contrast with previous evaluation frameworks, the ELICIT framework focuses on all phases of the IT life cycle across the society, user, and IT levels. Institutions seeking to establish evaluation programs for EHR-integrated innovations could use our framework to create such shared understanding and justify the need to invest in evaluation. CONCLUSION: As health care undergoes a digital transformation, it will be critical for EHR-integrated innovations to be systematically evaluated. The ELICIT framework can facilitate these evaluations.
Keywords:
Clinical decision support; Evaluation framework; Health information technology; Health technology assessment; Human factors engineering; Implementation science
Authors: Rebecca L Curran; Polina V Kukhareva; Teresa Taft; Charlene R Weir; Thomas J Reese; Claude Nanjo; Salvador Rodriguez-Loya; Douglas K Martin; Phillip B Warner; David E Shields; Michael C Flynn; Jonathan P Boltax; Kensaku Kawamoto Journal: J Am Med Inform Assoc Date: 2020-08-01 Impact factor: 4.497
Authors: Elizabeth Murray; Eric B Hekler; Gerhard Andersson; Linda M Collins; Aiden Doherty; Chris Hollis; Daniel E Rivera; Robert West; Jeremy C Wyatt Journal: Am J Prev Med Date: 2016-11 Impact factor: 5.043
Authors: Matthew T Neame; Gerri Sefton; Matthew Roberts; David Harkness; Ian P Sinha; Daniel B Hawcutt Journal: Int J Med Inform Date: 2020-08-14 Impact factor: 4.046
Authors: Pallavi Ranade-Kharkar; Scott P Narus; Gary L Anderson; Teresa Conway; Guilherme Del Fiol Journal: J Biomed Inform Date: 2018-07-12 Impact factor: 6.317
Authors: Shinji Tarumi; Wataru Takeuchi; George Chalkidis; Salvador Rodriguez-Loya; Junichi Kuwata; Michael Flynn; Kyle M Turner; Farrant H Sakaguchi; Charlene Weir; Heidi Kramer; David E Shields; Phillip B Warner; Polina Kukhareva; Hideyuki Ban; Kensaku Kawamoto Journal: Methods Inf Med Date: 2021-05-11 Impact factor: 2.176