Tracy Moniz1, Shannon Arntfield2, Kristina Miller3, Lorelei Lingard4, Chris Watling4, Glenn Regehr5. 1. Department of Communication Studies, Mount Saint Vincent University, Halifax, Nova Scotia, Canada. 2. Department of Obstetrics and Gynaecology, Western University, London, Ontario, Canada. 3. Department of Health and Rehabilitation Sciences, Western University, London, Ontario, Canada. 4. Centre for Education Research and Innovation, Western University, London, Ontario, Canada. 5. Centre for Health Education Scholarship, University of British Columbia, Vancouver, British Columbia, Canada.
Abstract
CONTEXT: Reflective writing is a popular tool to support the growth of reflective capacity in undergraduate medical learners. Its popularity stems from research suggesting that reflective capacity may lead to improvements in skills such as empathy, communication, collaboration and professionalism. This has led to assumptions that reflective writing can also serve as a tool for student assessment. However, evidence to support the reliability and validity of reflective writing as a meaningful assessment strategy is lacking. METHODS: Using a published instrument for measuring 'reflective capacity' (the Reflection Evaluation for Learners' Enhanced Competencies Tool [REFLECT]), four trained raters independently scored four samples of writing from each of 107 undergraduate medical students to determine the reliability of reflective writing scores. REFLECT scores were then correlated with scores on a Year 4 objective structured clinical examination (OSCE) and Year 2 multiple-choice question (MCQ) examinations to examine, respectively, convergent and divergent validity. RESULTS: Across four writing samples, four-rater Cronbach's α-values ranged from 0.72 to 0.82, demonstrating reasonable inter-rater reliability with four raters using the REFLECT rubric. However, inter-sample reliability was fairly low (four-sample Cronbach's α = 0.54, single-sample intraclass correlation coefficient: 0.23), which suggests that performance on one reflective writing sample was not strongly indicative of performance on the next. Approximately 14 writing samples are required to achieve reasonable inter-sample reliability. The study found weak, non-significant correlations between reflective writing scores and both OSCE global scores (r = 0.13) and MCQ examination scores (r = 0.10), demonstrating a lack of relationship between reflective writing and these measures of performance. CONCLUSIONS: Our findings suggest that to draw meaningful conclusions about reflective capacity as a stable construct in individuals requires 14 writing samples per student, each assessed by four or five raters. This calls into question the feasibility and utility of using reflective writing rigorously as an assessment tool in undergraduate medical education.
CONTEXT: Reflective writing is a popular tool to support the growth of reflective capacity in undergraduate medical learners. Its popularity stems from research suggesting that reflective capacity may lead to improvements in skills such as empathy, communication, collaboration and professionalism. This has led to assumptions that reflective writing can also serve as a tool for student assessment. However, evidence to support the reliability and validity of reflective writing as a meaningful assessment strategy is lacking. METHODS: Using a published instrument for measuring 'reflective capacity' (the Reflection Evaluation for Learners' Enhanced Competencies Tool [REFLECT]), four trained raters independently scored four samples of writing from each of 107 undergraduate medical students to determine the reliability of reflective writing scores. REFLECT scores were then correlated with scores on a Year 4 objective structured clinical examination (OSCE) and Year 2 multiple-choice question (MCQ) examinations to examine, respectively, convergent and divergent validity. RESULTS: Across four writing samples, four-rater Cronbach's α-values ranged from 0.72 to 0.82, demonstrating reasonable inter-rater reliability with four raters using the REFLECT rubric. However, inter-sample reliability was fairly low (four-sample Cronbach's α = 0.54, single-sample intraclass correlation coefficient: 0.23), which suggests that performance on one reflective writing sample was not strongly indicative of performance on the next. Approximately 14 writing samples are required to achieve reasonable inter-sample reliability. The study found weak, non-significant correlations between reflective writing scores and both OSCE global scores (r = 0.13) and MCQ examination scores (r = 0.10), demonstrating a lack of relationship between reflective writing and these measures of performance. CONCLUSIONS: Our findings suggest that to draw meaningful conclusions about reflective capacity as a stable construct in individuals requires 14 writing samples per student, each assessed by four or five raters. This calls into question the feasibility and utility of using reflective writing rigorously as an assessment tool in undergraduate medical education.
Authors: Stella L Ng; Jeff Crukley; Ryan Brydges; Victoria Boyd; Adam Gavarkovs; Emilia Kangasjarvi; Sarah Wright; Kulamakan Kulasegaram; Farah Friesen; Nicole N Woods Journal: Adv Health Sci Educ Theory Pract Date: 2022-01-01 Impact factor: 3.629
Authors: Alisha Brown; Joshua Jauregui; Jonathan S Ilgen; Jeff Riddell; Douglas Schaad; Jared Strote; Jamie Shandro Journal: West J Emerg Med Date: 2019-12-19
Authors: Renato Franco; Camila Ament Giuliani Franco; Marco Antonio de Carvalho Filho; Milton Severo; Maria Amelia Ferreira Journal: Int J Med Educ Date: 2020-02-14