David J Turbow1, Thomas P Werner, Ellen Lowe, Huy Q Vu. 1. University of St. Augustine for Health Sciences, 700 Windy Point Dr., San Marcos, CA 92069, USA. Tel 760-591-3012, fax 7605913053. dturbow@usa.edu.
Abstract
BACKGROUND: There is growing interest in the use of rubrics to assess written work. This study aimed to determine whether or not the norming of a written communication rubric improved scoring consistency among clinical faculty in a critical thinking course. The benefits of a formalized norming process are described. METHODS: Faculty-raters were trained to apply the rubric to a signature assignment while participating in calibration workshops. For each rubric criterion, faculty examined whether or not heightened congruence in scoring resulted from the training. Inter-rater reliability was determined after raters independently scored de-identified essays. RESULTS: Pre-workshop intra-class correlations (ICCs) were acceptable (i.e., >0.7) for three of five rubric criteria. Post-workshop ICCs for only two criteria were acceptable: disciplinary conventions, and sources and evidence. Rater attrition and lag-time between calibration and post-workshop activities likely contributed to reduced consistency. DISCUSSION: The rubric was useful for discriminating writing proficiency. Norming led to revision of the signature assignment, the rubric design, and a need for writing workshops. These changes will result in better student preparation for composing evidence-informed essays. Less-rigid approaches are worthy of future exploration.
BACKGROUND: There is growing interest in the use of rubrics to assess written work. This study aimed to determine whether or not the norming of a written communication rubric improved scoring consistency among clinical faculty in a critical thinking course. The benefits of a formalized norming process are described. METHODS: Faculty-raters were trained to apply the rubric to a signature assignment while participating in calibration workshops. For each rubric criterion, faculty examined whether or not heightened congruence in scoring resulted from the training. Inter-rater reliability was determined after raters independently scored de-identified essays. RESULTS: Pre-workshop intra-class correlations (ICCs) were acceptable (i.e., >0.7) for three of five rubric criteria. Post-workshop ICCs for only two criteria were acceptable: disciplinary conventions, and sources and evidence. Rater attrition and lag-time between calibration and post-workshop activities likely contributed to reduced consistency. DISCUSSION: The rubric was useful for discriminating writing proficiency. Norming led to revision of the signature assignment, the rubric design, and a need for writing workshops. These changes will result in better student preparation for composing evidence-informed essays. Less-rigid approaches are worthy of future exploration.
Authors: Rafael García-Ros; Maria-Arantzazu Ruescas-Nicolau; Natalia Cezón-Serrano; Juan J Carrasco; Sofía Pérez-Alenda; Clara Sastre-Arbona; Constanza San Martín-Valenzuela; Cristina Flor-Rufino; Maria Luz Sánchez-Sánchez Journal: Int J Environ Res Public Health Date: 2021-05-06 Impact factor: 3.390