OBJECTIVE: Reflection enables learners to analyze their experiences and capture the wisdom that lies within. Effective teaching requires reliable methods of assessment. Several methods of assessing reflective writing have been described; however, they often require significant training, and reliability has seldom been assessed. This study was designed to determine the interrater reliability of a method of assessing reflective writing by using a modified Bloom's Taxonomy. METHODS: Twenty-one third-year medical students maintained reflective journals throughout their pediatric clerkship. A coding schema based on Bloom's Taxonomy was developed to assess the level of cognitive processing evident in the journals. Journals were independently assessed by 3 raters. Percent agreement, kappa statistics, and intraclass correlation coefficients (ICC [2,1]) were used to assess interrater reliability. RESULTS: Three hundred eight entries from 21 journals were assessed. Percent agreement ranged from 78.2% to 100%. Kappa statistic for each level ranged from 0.57 +/- 0.04 to 0.73 +/- 0.04, and for the highest level of processing evident it ranged from 0.52 +/- .04 to 0.58 +/- 0.04. ICC (2,1) for each level of cognitive processing ranged from 0.62 (F = 6.20; P = .000) to 1.00, and for the highest level of cognitive processing evident, it was 0.79 (F = 12.42; P = .000). Substantial to almost perfect agreement was attained. CONCLUSIONS: Reflective journals allow learners to revisit their experiences for critical analysis and deeper learning. This study describes a reliable method, based on Bloom's Taxonomy, of determining whether learners have achieved higher order thinking through reflective journal writing. This method can provide a baseline for facilitating higher order processing, critical thinking, and reflective practice.
OBJECTIVE: Reflection enables learners to analyze their experiences and capture the wisdom that lies within. Effective teaching requires reliable methods of assessment. Several methods of assessing reflective writing have been described; however, they often require significant training, and reliability has seldom been assessed. This study was designed to determine the interrater reliability of a method of assessing reflective writing by using a modified Bloom's Taxonomy. METHODS: Twenty-one third-year medical students maintained reflective journals throughout their pediatric clerkship. A coding schema based on Bloom's Taxonomy was developed to assess the level of cognitive processing evident in the journals. Journals were independently assessed by 3 raters. Percent agreement, kappa statistics, and intraclass correlation coefficients (ICC [2,1]) were used to assess interrater reliability. RESULTS: Three hundred eight entries from 21 journals were assessed. Percent agreement ranged from 78.2% to 100%. Kappa statistic for each level ranged from 0.57 +/- 0.04 to 0.73 +/- 0.04, and for the highest level of processing evident it ranged from 0.52 +/- .04 to 0.58 +/- 0.04. ICC (2,1) for each level of cognitive processing ranged from 0.62 (F = 6.20; P = .000) to 1.00, and for the highest level of cognitive processing evident, it was 0.79 (F = 12.42; P = .000). Substantial to almost perfect agreement was attained. CONCLUSIONS: Reflective journals allow learners to revisit their experiences for critical analysis and deeper learning. This study describes a reliable method, based on Bloom's Taxonomy, of determining whether learners have achieved higher order thinking through reflective journal writing. This method can provide a baseline for facilitating higher order processing, critical thinking, and reflective practice.