J F Nitsche1, S Conrad2, S Hoopes2, M Carrel2, K Bebeau2, B C Brost2. 1. Division of Maternal Fetal Medicine, Department of OB-GYN, Wake Forest School of Medicine, Winston-Salem, NC 27157. Electronic address: jnitsche@wakehealth.edu. 2. Division of Maternal Fetal Medicine, Department of OB-GYN, Wake Forest School of Medicine, Winston-Salem, NC 27157.
Abstract
RATIONALE AND OBJECTIVES: We previously demonstrated validity evidence for our novel ultrasound-guided invasive procedure targeting tasks in the content, response process, relations with other variables, and consequences validity domains. Here, we investigate their internal structure by assessing their interrater, intrarater, and test-retest reliability. METHODS: In this Institutional Review Board approved nonrandomized interventional trial first year medical students performed our previously described dowel and straw ultrasound guidance targeting tasks as a pretest. Afterward, the training group had four weekly 1-hour training sessions. The control group had no further training. Both groups then had a posttest for both tasks. The training group was re-evaluated 2 and 5 months later. Completion time in seconds, errors, and error adjusted time (5 seconds penalty/error) were recorded. Pretest and posttest performance was compared within groups, and the amount of improvement from pretest to posttest was compared between groups. Interrater, intrarater, and test-retest interclass correlation coefficients (ICC) were calculated. RESULTS: Although some improvements from pretest to posttest were seen in both groups, greater improvements were seen in the training group. This skill was retained for at least several months. The interrater and intrarater ICCs were excellent (range 0.83-0.93). The test-retest ICCs were good to excellent in all but one performance measure (0.50-0.78). CONCLUSION: Student performance on the targeting tasks improved markedly after training and persisted for several months. The interrater and intrarater reliability were excellent, while the test-retest reliability was good. This provides additional validity evidence for our novel ultrasound-guided invasive procedure targeting curriculum.
RATIONALE AND OBJECTIVES: We previously demonstrated validity evidence for our novel ultrasound-guided invasive procedure targeting tasks in the content, response process, relations with other variables, and consequences validity domains. Here, we investigate their internal structure by assessing their interrater, intrarater, and test-retest reliability. METHODS: In this Institutional Review Board approved nonrandomized interventional trial first year medical students performed our previously described dowel and straw ultrasound guidance targeting tasks as a pretest. Afterward, the training group had four weekly 1-hour training sessions. The control group had no further training. Both groups then had a posttest for both tasks. The training group was re-evaluated 2 and 5 months later. Completion time in seconds, errors, and error adjusted time (5 seconds penalty/error) were recorded. Pretest and posttest performance was compared within groups, and the amount of improvement from pretest to posttest was compared between groups. Interrater, intrarater, and test-retest interclass correlation coefficients (ICC) were calculated. RESULTS: Although some improvements from pretest to posttest were seen in both groups, greater improvements were seen in the training group. This skill was retained for at least several months. The interrater and intrarater ICCs were excellent (range 0.83-0.93). The test-retest ICCs were good to excellent in all but one performance measure (0.50-0.78). CONCLUSION: Student performance on the targeting tasks improved markedly after training and persisted for several months. The interrater and intrarater reliability were excellent, while the test-retest reliability was good. This provides additional validity evidence for our novel ultrasound-guided invasive procedure targeting curriculum.