Annie McCluskey1, Bianca Bishop. 1. Discipline of Occupational Therapy, Faculty of Health Sciences, The University of Sydney, Lidcombe, New South Wales, Australia. a.mccluskey@usyd.edu.au
Abstract
INTRODUCTION: Health educators who teach professionals about evidence-based practice (EBP) need instruments to measure change in skills and knowledge. This study aimed to develop and evaluate the interrater reliability, internal consistency, and responsiveness of the Adapted Fresno Test (AFT) of competence in EBP. METHODS: Reliability testing involved 2 raters. The AFT was completed by 114 occupational therapists before, and 106 therapists after a 2-day workshop on EBP. A random sample of 20 completed tests (10 pre- and 10 postworkshop = Versions 1 and 2) were scored by 2 raters. Interrater reliability was calculated with the use of intraclass correlation coefficients (ICC, 1,2) for 7 summed subtest scores and the total AFT score. Internal consistency was calculated with the use of Cronbach's alpha. Responsiveness was calculated by comparing mean pre-post change scores, change in low scorers with high scorers. RESULTS: Interrater reliability ranged from good to excellent for individual AFT subtests (Version 1: ICC, 0.80-0.96; Version 2: 0.68-0.94), and excellent for total AFT scores (Version 1: ICC, 0.96; Version 2: ICC, 0.91). Internal consistency was acceptable (Cronbach's alpha, 0.74). Overall mean change on the 156-point test was 20.6 points (95% confidence interval [CI], 15.6-25.5), 26.8 points (95% CI, 21.6-1.9) for low scorers and -1.8 points (95% CI, -6.4 to 10.1) for high scorers. DISCUSSION: The AFT has acceptable psychometric properties, and measures change in knowledge and skills of rehabilitation professionals following EBP training. The test is most useful for evaluating change in novice learners.
INTRODUCTION: Health educators who teach professionals about evidence-based practice (EBP) need instruments to measure change in skills and knowledge. This study aimed to develop and evaluate the interrater reliability, internal consistency, and responsiveness of the Adapted Fresno Test (AFT) of competence in EBP. METHODS: Reliability testing involved 2 raters. The AFT was completed by 114 occupational therapists before, and 106 therapists after a 2-day workshop on EBP. A random sample of 20 completed tests (10 pre- and 10 postworkshop = Versions 1 and 2) were scored by 2 raters. Interrater reliability was calculated with the use of intraclass correlation coefficients (ICC, 1,2) for 7 summed subtest scores and the total AFT score. Internal consistency was calculated with the use of Cronbach's alpha. Responsiveness was calculated by comparing mean pre-post change scores, change in low scorers with high scorers. RESULTS: Interrater reliability ranged from good to excellent for individual AFT subtests (Version 1: ICC, 0.80-0.96; Version 2: 0.68-0.94), and excellent for total AFT scores (Version 1: ICC, 0.96; Version 2: ICC, 0.91). Internal consistency was acceptable (Cronbach's alpha, 0.74). Overall mean change on the 156-point test was 20.6 points (95% confidence interval [CI], 15.6-25.5), 26.8 points (95% CI, 21.6-1.9) for low scorers and -1.8 points (95% CI, -6.4 to 10.1) for high scorers. DISCUSSION: The AFT has acceptable psychometric properties, and measures change in knowledge and skills of rehabilitation professionals following EBP training. The test is most useful for evaluating change in novice learners.