Jessica V Rich1, Sue Fostaty Young2, Catherine Donnelly3, Andrew K Hall4,5, J Damon Dagnone4, Kristen Weersink4, Jaelyn Caudle4, Elaine Van Melle5, Don A Klinger6. 1. Faculty of Education, Queen's University, Kingston, Ontario, Canada. 2. Centre for Teaching and Learning, Queen's University, Kingston, Ontario, Canada. 3. School of Rehabilitation Therapy, Queen's University, Kingston, Ontario, Canada. 4. Department of Emergency Medicine, Queen's University, Kingston, Ontario, Canada. 5. Royal College of Physicians and Surgeons of Canada, Ottawa, Ontario, Canada. 6. Te Kura Toi Tangata, University of Waikato, Hamilton, New Zealand.
Abstract
RATIONALE, AIMS, AND OBJECTIVES: Programmatic assessment has been identified as a system-oriented approach to achieving the multiple purposes for assessment within Competency-Based Medical Education (CBME, i.e., formative, summative, and program improvement). While there are well-established principles for designing and evaluating programs of assessment, few studies illustrate and critically interpret, what a system of programmatic assessment looks like in practice. This study aims to use systems thinking and the 'two communities' metaphor to interpret a model of programmatic assessment and to identify challenges and opportunities with operationalization. METHOD: An interpretive case study was used to investigate how programmatic assessment is being operationalized within one competency-based residency program at a Canadian university. Qualitative data were collected from residents, faculty, and program leadership via semi-structured group and individual interviews conducted at nine months post-CBME implementation. Data were analyzed using a combination of data-based inductive analysis and theory-derived deductive analysis. RESULTS: In this model, Academic Advisors had a central role in brokering assessment data between communities responsible for producing and using residents' performance information for decision making (i.e., formative, summative/evaluative, and program improvement). As system intermediaries, Academic Advisors were in a privileged position to see how the parts of the assessment system contributed to the functioning of the whole and could identify which system components were not functioning as intended. Challenges were identified with the documentation of residents' performance information (i.e., system inputs); use of low-stakes formative assessments to inform high-stakes evaluative judgments about the achievement of competence standards; and gaps in feedback mechanisms for closing learning loops. CONCLUSIONS: The findings of this research suggest that program stakeholders can benefit from a systems perspective regarding how their assessment practices contribute to the efficacy of the system as a whole. Academic Advisors are well positioned to support educational development efforts focused on overcoming challenges with operationalizing programmatic assessment.
RATIONALE, AIMS, AND OBJECTIVES: Programmatic assessment has been identified as a system-oriented approach to achieving the multiple purposes for assessment within Competency-Based Medical Education (CBME, i.e., formative, summative, and program improvement). While there are well-established principles for designing and evaluating programs of assessment, few studies illustrate and critically interpret, what a system of programmatic assessment looks like in practice. This study aims to use systems thinking and the 'two communities' metaphor to interpret a model of programmatic assessment and to identify challenges and opportunities with operationalization. METHOD: An interpretive case study was used to investigate how programmatic assessment is being operationalized within one competency-based residency program at a Canadian university. Qualitative data were collected from residents, faculty, and program leadership via semi-structured group and individual interviews conducted at nine months post-CBME implementation. Data were analyzed using a combination of data-based inductive analysis and theory-derived deductive analysis. RESULTS: In this model, Academic Advisors had a central role in brokering assessment data between communities responsible for producing and using residents' performance information for decision making (i.e., formative, summative/evaluative, and program improvement). As system intermediaries, Academic Advisors were in a privileged position to see how the parts of the assessment system contributed to the functioning of the whole and could identify which system components were not functioning as intended. Challenges were identified with the documentation of residents' performance information (i.e., system inputs); use of low-stakes formative assessments to inform high-stakes evaluative judgments about the achievement of competence standards; and gaps in feedback mechanisms for closing learning loops. CONCLUSIONS: The findings of this research suggest that program stakeholders can benefit from a systems perspective regarding how their assessment practices contribute to the efficacy of the system as a whole. Academic Advisors are well positioned to support educational development efforts focused on overcoming challenges with operationalizing programmatic assessment.
Authors: Andrew S Parsons; Kelley Mark; James R Martindale; Megan J Bray; Ryan P Smith; Elizabeth Bradley; Maryellen Gusic Journal: J Gen Intern Med Date: 2022-06-16 Impact factor: 6.473
Authors: Suzanne Schut; Lauren A Maggio; Sylvia Heeneman; Jan van Tartwijk; Cees van der Vleuten; Erik Driessen Journal: Perspect Med Educ Date: 2020-10-21
Authors: Lawrence Fulton; Cristian Lieneck; Zo Ramamonjiarivelo; Clemens Scott Kruse; Matthew S Brooks Journal: BMC Med Educ Date: 2021-01-06 Impact factor: 2.463