Patricia J Hicks1, Melissa Margolis, Sue E Poynter, Christa Chaffinch, Rebecca Tenney-Soeiro, Teri L Turner, Linda Waggoner-Fountain, Robin Lockridge, Stephen G Clyman, Alan Schwartz. 1. P.J. Hicks is professor of clinical pediatrics, Department of Pediatrics, Perelman School of Medicine at the University of Pennsylvania and Children's Hospital of Philadelphia, Philadelphia, Pennsylvania. M. Margolis is senior measurement scientist, National Board of Medical Examiners, Philadelphia, Pennsylvania. S.E. Poynter is associate professor of pediatrics, Department of Pediatrics, University of Cincinnati and Cincinnati Children's Hospital, Cincinnati, Ohio. C. Chaffinch is manager, Product Discovery, National Board of Medical Examiners, Philadelphia, Pennsylvania. R. Tenney-Soeiro is assistant professor of clinical pediatrics, Department of Pediatrics, Perelman School of Medicine at the University of Pennsylvania and Children's Hospital of Philadelphia, Philadelphia, Pennsylvania. T.L. Turner is associate professor, Department of Pediatrics, Baylor College of Medicine, and director, Center for Research, Innovation, and Scholarship in Medical Education, Texas Children's Hospital, Houston, Texas. L. Waggoner-Fountain is associate professor of pediatrics, Department of Pediatrics, University of Virginia School of Medicine, Charlottesville, Virginia. R. Lockridge is a PhD candidate, Howard University, Washington, DC, and manager, Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network (APPD LEARN), McLean, Virginia. S.G. Clyman is senior vice president, Research and Discovery, National Board of Medical Examiners, Philadelphia, Pennsylvania. A. Schwartz is Michael Reese Endowed Professor of Medical Education, associate head of medical education, and research professor of pediatrics, University of Illinois at Chicago, Chicago, Illinois, and director, Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network (APPD LEARN), McLean, Virginia.
Abstract
PURPOSE: To report on the development of content and user feedback regarding the assessment process and utility of the workplace-based assessment instruments of the Pediatrics Milestones Assessment Pilot (PMAP). METHOD: One multisource feedback instrument and two structured clinical observation instruments were developed and refined by experts in pediatrics and assessment to provide evidence for nine competencies based on the Pediatrics Milestones (PMs) and chosen to inform residency program faculty decisions about learners' readiness to serve as pediatric interns in the inpatient setting. During the 2012-2013 PMAP study, 18 U.S. pediatric residency programs enrolled interns and subinterns. Faculty, residents, nurses, and other observers used the instruments to assess learner performance through direct observation during a one-month rotation. At the end of the rotation, data were aggregated for each learner, milestone levels were assigned using a milestone classification form, and feedback was provided to learners. Learners and site leads were surveyed and/or interviewed about their experience as participants. RESULTS: Across the sites, 2,338 instruments assessing 239 learners were completed by 630 unique observers. Regarding end-of-rotation feedback, 93% of learners (128/137) agreed the assessments and feedback "helped me understand how those with whom I work perceive my performance," and 85% (117/137) agreed they were "useful for constructing future goals or identifying a developmental path." Site leads identified several benefits and challenges to the assessment process. CONCLUSIONS: PM-based instruments used in workplace-based assessment provide a meaningful and acceptable approach to collecting evidence of learner competency development. Learners valued feedback provided by PM-based assessment.
PURPOSE: To report on the development of content and user feedback regarding the assessment process and utility of the workplace-based assessment instruments of the Pediatrics Milestones Assessment Pilot (PMAP). METHOD: One multisource feedback instrument and two structured clinical observation instruments were developed and refined by experts in pediatrics and assessment to provide evidence for nine competencies based on the Pediatrics Milestones (PMs) and chosen to inform residency program faculty decisions about learners' readiness to serve as pediatric interns in the inpatient setting. During the 2012-2013 PMAP study, 18 U.S. pediatric residency programs enrolled interns and subinterns. Faculty, residents, nurses, and other observers used the instruments to assess learner performance through direct observation during a one-month rotation. At the end of the rotation, data were aggregated for each learner, milestone levels were assigned using a milestone classification form, and feedback was provided to learners. Learners and site leads were surveyed and/or interviewed about their experience as participants. RESULTS: Across the sites, 2,338 instruments assessing 239 learners were completed by 630 unique observers. Regarding end-of-rotation feedback, 93% of learners (128/137) agreed the assessments and feedback "helped me understand how those with whom I work perceive my performance," and 85% (117/137) agreed they were "useful for constructing future goals or identifying a developmental path." Site leads identified several benefits and challenges to the assessment process. CONCLUSIONS: PM-based instruments used in workplace-based assessment provide a meaningful and acceptable approach to collecting evidence of learner competency development. Learners valued feedback provided by PM-based assessment.
Authors: Daniel J Schumacher; Beth King; Michelle M Barnes; Sean P Elliott; Kathleen Gibbs; Jon F McGreevy; Javier Gonzalez Del Rey; Tanvi Sharma; Catherine Michelson; Alan Schwartz Journal: J Grad Med Educ Date: 2018-08