Jung G Kim1, Hector P Rodriguez2, Eric S Holmboe3, Kathryn M McDonald4, Lindsay Mazotti5, Diane R Rittenhouse6, Stephen M Shortell7, Michael H Kanter8. 1. is Assistant Professor, Kaiser Permanente Bernard J. Tyson School of Medicine, Department of Health Systems Science. 2. is the Kaiser Permanente Professor of Health Policy and Management, University of California, Berkeley School of Public Health. 3. is Chief Research, Milestone Development, and Evaluation Officer, Accreditation Council for Graduate Medical Education. 4. is the Bloomberg Distinguished Professor of Health Systems, Quality, and Safety, Johns Hopkins Schools of Medicine and Nursing. 5. is Assistant Physician-in-Chief, Kaiser Permanente East Bay and Director, Clinical Experience/Associate Professor of Clinical Science, Kaiser Permanente School of Medicine. 6. is Senior Fellow, Mathematica, and Professor, University of California, San Francisco. 7. is Blue Cross of California Distinguished Professor of Health Policy and Management Emeritus, Dean Emeritus, and Professor, Graduate School, University of California, Berkeley School of Public Health. 8. is Chair and Professor of Clinical Science, Kaiser Permanente School of Medicine.
Abstract
Background: Graduate medical education (GME) program leaders struggle to incorporate quality measures in the ambulatory care setting, leading to knowledge gaps on how to provide feedback to residents and programs. While nationally collected quality of care data are available, their reliability for individual resident learning and for GME program improvement is understudied. Objective: To examine the reliability of the Healthcare Effectiveness Data and Information Set (HEDIS) clinical performance measures in family medicine and internal medicine GME programs and to determine whether HEDIS measures can inform residents and their programs with their quality of care. Methods: From 2014 to 2017, we collected HEDIS measures from 566 residents in 8 family medicine and internal medicine programs under one sponsoring institution. Intraclass correlation was performed to establish patient sample sizes required for 0.70 and 0.80 reliability levels at the resident and program levels. Differences between the patient sample sizes required for reliable measurement and the actual patients cared for by residents were calculated. Results: The highest reliability levels for residents (0.88) and programs (0.98) were found for the most frequently available HEDIS measure, colorectal cancer screening. At the GME program level, 87.5% of HEDIS measures had sufficient sample sizes for reliable measurement at alpha 0.7 and 75.0% at alpha 0.8. Most resident level measurements were found to be less reliable. Conclusions: GME programs may reliably evaluate HEDIS performance pooled at the program level, but less so at the resident level due to patient volume.
Background: Graduate medical education (GME) program leaders struggle to incorporate quality measures in the ambulatory care setting, leading to knowledge gaps on how to provide feedback to residents and programs. While nationally collected quality of care data are available, their reliability for individual resident learning and for GME program improvement is understudied. Objective: To examine the reliability of the Healthcare Effectiveness Data and Information Set (HEDIS) clinical performance measures in family medicine and internal medicine GME programs and to determine whether HEDIS measures can inform residents and their programs with their quality of care. Methods: From 2014 to 2017, we collected HEDIS measures from 566 residents in 8 family medicine and internal medicine programs under one sponsoring institution. Intraclass correlation was performed to establish patient sample sizes required for 0.70 and 0.80 reliability levels at the resident and program levels. Differences between the patient sample sizes required for reliable measurement and the actual patients cared for by residents were calculated. Results: The highest reliability levels for residents (0.88) and programs (0.98) were found for the most frequently available HEDIS measure, colorectal cancer screening. At the GME program level, 87.5% of HEDIS measures had sufficient sample sizes for reliable measurement at alpha 0.7 and 75.0% at alpha 0.8. Most resident level measurements were found to be less reliable. Conclusions: GME programs may reliably evaluate HEDIS performance pooled at the program level, but less so at the resident level due to patient volume.
Authors: Alina Smirnova; Stefanie S Sebok-Syer; Saad Chahine; Adina L Kalet; Robyn Tamblyn; Kiki M J M H Lombarts; Cees P M van der Vleuten; Daniel J Schumacher Journal: Acad Med Date: 2019-05 Impact factor: 6.893
Authors: Jung G Kim; Hector P Rodriguez; Stephen M Shortell; Bruce Fuller; Eric S Holmboe; Diane R Rittenhouse Journal: Acad Med Date: 2021-03-01 Impact factor: 6.893
Authors: Brian M Wong; Karyn D Baum; Linda A Headrick; Eric S Holmboe; Fiona Moss; Greg Ogrinc; Kaveh G Shojania; Emma Vaux; Eric J Warm; Jason R Frank Journal: Acad Med Date: 2020-01 Impact factor: 6.893
Authors: Sarah Hudson Scholle; Joachin Roski; Daniel L Dunn; John L Adams; Donna Pillitterre Dugan; L Gregory Pawlson; Eve A Kerr Journal: Am J Manag Care Date: 2009-01 Impact factor: 2.229
Authors: Hector P Rodriguez; Ted von Glahn; Hong Chang; William H Rogers; Dana Gelb Safran Journal: Am J Med Qual Date: 2009 Jan-Feb Impact factor: 1.852