BACKGROUND: A requirement for all Accreditation Council for Graduate Medical Education (ACGME) approved residencies is the provision of "an opportunity for residents to participate in research." To comply with this requirement, most training programs encourage their residents to conduct research and to report their results. Few guidelines exist, however, for assessing the efficacy of the presentations. The goal of this pilot study was to develop a valid, one-page scoring rubric to be used during oral resident research presentations. Such a scoring rubric will facilitate acceptable agreement among faculty raters. METHODS: Content validity was addressed by adhering to the Standards for Educational and Psychological Testing. A one-page, five-domain, behaviorally worded scoring rubric was developed. Inter-rater reliability was derived and three ACGME General Competencies were also addressed within the rubric. RESULTS: The initial scoring rubric was tested with 11 resident oral presentations. The inter-rater reliability was 0.56 using Cronbach's alpha. The rubric was modified and the scale restricted to a 3-point scale. It was then tested with 17 additional presentations, which were independently rated by two general surgery faculty members. Cronbach's Alpha increased to 0.61. CONCLUSIONS: An objective method to evaluate a resident's oral research presentation has been successfully piloted. This content valid rubric possesses good inter-rater reliability according to established guidelines. Clearly defined behaviors have been outlined within the rubric. Program directors will have psychometrically sound evidence for the ACGME. Future research will address generalizability and concurrent validity using other types of resident assessment data.
BACKGROUND: A requirement for all Accreditation Council for Graduate Medical Education (ACGME) approved residencies is the provision of "an opportunity for residents to participate in research." To comply with this requirement, most training programs encourage their residents to conduct research and to report their results. Few guidelines exist, however, for assessing the efficacy of the presentations. The goal of this pilot study was to develop a valid, one-page scoring rubric to be used during oral resident research presentations. Such a scoring rubric will facilitate acceptable agreement among faculty raters. METHODS: Content validity was addressed by adhering to the Standards for Educational and Psychological Testing. A one-page, five-domain, behaviorally worded scoring rubric was developed. Inter-rater reliability was derived and three ACGME General Competencies were also addressed within the rubric. RESULTS: The initial scoring rubric was tested with 11 resident oral presentations. The inter-rater reliability was 0.56 using Cronbach's alpha. The rubric was modified and the scale restricted to a 3-point scale. It was then tested with 17 additional presentations, which were independently rated by two general surgery faculty members. Cronbach's Alpha increased to 0.61. CONCLUSIONS: An objective method to evaluate a resident's oral research presentation has been successfully piloted. This content valid rubric possesses good inter-rater reliability according to established guidelines. Clearly defined behaviors have been outlined within the rubric. Program directors will have psychometrically sound evidence for the ACGME. Future research will address generalizability and concurrent validity using other types of resident assessment data.
Authors: des Anges Cruser; Sarah K Brown; Jessica R Ingram; Alan L Podawiltz; Bruce D Dubin; John S Colston; Robert J Bulik Journal: Osteopath Med Prim Care Date: 2010-07-08