Jill Boylston Herndon1, Krishna Aravamudhan2, Ronald L Stephenson3, Ryan Brandon4, Jesley Ruff5, Frank Catalanotto6, Huong Le7. 1. Key Analytics and Consulting, LLC, Gainesville, Florida, USA (Dr Herndon was with the Department of Health Outcomes and Policy, University of Florida College of Medicine, Gainesville, Florida, USA, when this study was conducted). 2. Dental Quality Alliance, American Dental Association, Chicago, Illinois, USA. 3. Stephenson Developers Group, Clearwater, Florida, USA. 4. Exan Group, Vancouver, Canada. 5. American Dental Partners, Inc., Wakefield, Massachusetts, USA. 6. Department of Community Dentistry and Behavioral Science, University of Florida College of Dentistry, Gainesville, Florida, USA. 7. Asian Health Services, Oakland, California and National Network for Oral Health Access, Denver, Colorado, USA.
Abstract
OBJECTIVE: To describe the stakeholder-engaged processes used to develop, specify, and validate 2 oral health care electronic clinical quality measures. MATERIALS AND METHODS: A broad range of stakeholders were engaged from conception through testing to develop measures and test feasibility, reliability, and validity following National Quality Forum guidance. We assessed data element feasibility through semistructured interviews with key stakeholders using a National Quality Forum-recommended scorecard. We created test datasets of synthetic patients to test measure implementation feasibility and reliability within and across electronic health record (EHR) systems. We validated implementation with automated reporting of EHR clinical data against manual record reviews, using the kappa statistic. RESULTS: A stakeholder workgroup was formed and guided all development and testing processes. All critical data elements passed feasibility testing. Four test datasets, representing 577 synthetic patients, were developed and implemented within EHR vendors' software, demonstrating measure implementation feasibility. Measure reliability and validity were established through implementation at clinical practice sites, with kappa statistic values in the "almost perfect" agreement range of 0.80-0.99 for all but 1 measure component, which demonstrated "substantial" agreement. The 2 validated measures were published in the United States Health Information Knowledgebase. CONCLUSION: The stakeholder-engaged processes used in this study facilitated a successful measure development and testing cycle. Engaging stakeholders early and throughout development and testing promotes early identification of and attention to potential threats to feasibility, reliability, and validity, thereby averting significant resource investments that are unlikely to be fruitful.
OBJECTIVE: To describe the stakeholder-engaged processes used to develop, specify, and validate 2 oral health care electronic clinical quality measures. MATERIALS AND METHODS: A broad range of stakeholders were engaged from conception through testing to develop measures and test feasibility, reliability, and validity following National Quality Forum guidance. We assessed data element feasibility through semistructured interviews with key stakeholders using a National Quality Forum-recommended scorecard. We created test datasets of synthetic patients to test measure implementation feasibility and reliability within and across electronic health record (EHR) systems. We validated implementation with automated reporting of EHR clinical data against manual record reviews, using the kappa statistic. RESULTS: A stakeholder workgroup was formed and guided all development and testing processes. All critical data elements passed feasibility testing. Four test datasets, representing 577 synthetic patients, were developed and implemented within EHR vendors' software, demonstrating measure implementation feasibility. Measure reliability and validity were established through implementation at clinical practice sites, with kappa statistic values in the "almost perfect" agreement range of 0.80-0.99 for all but 1 measure component, which demonstrated "substantial" agreement. The 2 validated measures were published in the United States Health Information Knowledgebase. CONCLUSION: The stakeholder-engaged processes used in this study facilitated a successful measure development and testing cycle. Engaging stakeholders early and throughout development and testing promotes early identification of and attention to potential threats to feasibility, reliability, and validity, thereby averting significant resource investments that are unlikely to be fruitful.
Authors: Jill Boylston Herndon; James J Crall; Krishna Aravamudhan; Frank A Catalanotto; I-Chan Huang; Nancy Rudner; Scott L Tomar; Elizabeth A Shenkman Journal: J Public Health Dent Date: 2015-02-13 Impact factor: 1.821
Authors: Sarah Hudson Scholle; Joachin Roski; Daniel L Dunn; John L Adams; Donna Pillitterre Dugan; L Gregory Pawlson; Eve A Kerr Journal: Am J Manag Care Date: 2009-01 Impact factor: 2.229
Authors: Stewart B Harris; Richard H Glazier; Jordan W Tompkins; Andrew S Wilton; Vijaya Chevendra; Moira A Stewart; Amardeep Thind Journal: BMC Health Serv Res Date: 2010-12-23 Impact factor: 2.655
Authors: Jill Boylston Herndon; Scott L Tomar; Frank A Catalanotto; Nancy Rudner; I-Chan Huang; Krishna Aravamudhan; Elizabeth A Shenkman; James J Crall Journal: J Am Dent Assoc Date: 2015-08 Impact factor: 3.634
Authors: Michael F Furukawa; Jennifer King; Vaishali Patel; Chun-Ju Hsiao; Julia Adler-Milstein; Ashish K Jha Journal: Health Aff (Millwood) Date: 2014-08-07 Impact factor: 6.301
Authors: Jean Beauchamp; Page W Caufield; James J Crall; Kevin Donly; Robert Feigal; Barbara Gooch; Amid Ismail; William Kohn; Mark Siegal; Richard Simonsen Journal: J Am Dent Assoc Date: 2008-03 Impact factor: 3.634
Authors: Patricia C Dykes; Mica Curtin-Bowen; Troy Li; Avery Pullman; Alexandra Businger; Stuart Lipsitz; Ania Syrowatka; Michael Sainlaire; Tien Thai; David W Bates Journal: AMIA Annu Symp Proc Date: 2022-02-21
Authors: Luci K Leykum; Erin P Finley; Lauren S Penney; Julie Parish Johnson; Jacqueline A Pugh; Polly H Noel Journal: Health Serv Res Date: 2022-03-04 Impact factor: 3.734