A comparison of the effectiveness of the team-based learning readiness assessments completed at home to those completed in class.

PURPOSE: The readiness assurance process (RAP) of team-based learning (TBL) is an important element that ensures that students come prepared to learn. However, the RAP can use a significant amount of class time which could otherwise be used for application exercises. The authors administered the TBL-associated RAP in class or individual readiness assurance tests (iRATs) at home to compare medical student performance and learning preference for physiology content.
METHODS: Using cross-over study design, the first year medical student TBL teams were divided into two groups. One group was administered iRATs and group readiness assurance tests (gRATs) consisting of physiology questions during scheduled class time. The other group was administered the same iRAT questions at home, and did not complete a gRAT. To compare effectiveness of the two administration methods, both groups completed the same 12-question physiology assessment during dedicated class time. Four weeks later, the entire process was repeated, with each group administered the RAP using the opposite method.
RESULTS: The performance on the physiology assessment after at-home administration of the iRAT was equivalent to performance after traditional in-class administration of the RAP. In addition, a majority of students preferred the at-home method of administration and reported that the at-home method was more effective in helping them learn course content.
CONCLUSION: The at-home administration of the iRAT proved effective. The at-home administration method is a promising alternative to conventional iRATs and gRATs with the goal of preserving valuable in-class time for TBL application exercises.

RCT Entities:   Population Interventions Outcomes

INTRODUCTION

Team-based learning (TBL) has been shown to be effective in undergraduate medical education as a part of basic science curricula [1]. The use of TBL in an anatomy course increased scores on a National Board of Medical Examiners Subject Examination and increased the rating of the course by recent graduates [2]. In another anatomy course, exam scores did not change significantly with TBL, however the course failure rate dropped [3]. Also using TBL in a physiology course improved performance on exams and improved the attitudes toward the course [4]. The TBL process begins with students studying the material and preparing for the session at home. In class, TBL sessions begin with the students answering multiple choice questions as individuals (individual readiness assurance tests, iRATs) and then answering the same questions as teams (group readiness assurance tests, gRATs). The iRAT and gRAT are followed by a discussion of the answers. This readiness assurance process (RAP) is followed by a group application exercise which requires the students to apply the knowledge they have just gained to a problem that they solve as a team. Since the RAP takes a significant amount of class time, we designed a strategy for having the students complete that portion of TBL at home or away from class. The advantage of the students completing readiness assurance at home is that it allows for more time to be spent on the group application exercise, which involves more rigorous thought about the material. In addition, it allows for shorter TBL sessions which may permit more of them to be used throughout the curriculum.

This study aims to compare the effectiveness of the iRAT completed at home to the RAP completed in class during a physiology course for first year medical students. It also examines student satisfaction with each process. Specifically, answers to the following questions are sought: Is there a difference in student performance on an assessment after they have completed an iRAT at home vs. the RAP in class?; Do students report alignment of iRAT questions to course content?; And do students have a learning preference for the method of readiness assurance administration?

METHODS

Subjects

At the beginning of the first year of medical school, the 102 medical students were divided into TBL teams of 5 to 6 students by evenly distributing them based on gender, ethnic background, and pre-medical school academic performance. During the first integrated basic science course, the students participated in three TBL sessions where the RAP was completed in class. Following this course, during the physiology portion of the second integrated basic science course (normal body), the current study was undertaken from September 2012 to December 2012. It was approved for exemption by the institutional review board of Duke University School of Medicine.

Study design

Using cross-over study design, the first-year medical student TBL teams were divided into two groups. In group 1 (teams 1-9), 54 students completed an iRAT at home (or away from class) for the first TBL session and 50 students completed the RAP in class for the second TBL session. In group 2 (teams 10-17) 48 students completed the RAP in class for the first TBL session and 46 students completed an iRAT at home for the second TBL session. The TBL sessions were four weeks apart.

At least a week prior to the TBL sessions, the students were informed which lectures the TBL session would cover. The components of the RAP or iRATs were not graded. The students completed graded iRATs and gRATs in the previous course. We did not grade the RAP or iRATs during this study since the assessments were administered in two different ways and we felt that this would not be fair to the students. Furthermore, since the physiology assessment was graded, the students were still held accountable for how well they were prepared for the session.

The in-class RAP consisted of an iRAT (12-13 multiple choice physiology questions in 15-17 minutes, administered online) that the students answered as individuals and a gRAT which consisted of the same questions that the students answered in their teams (both without the aid of outside resources). The students used immediate feedback assessment technique scratch-off forms during the gRAT. The last step of the in-class RAP was a large group discussion about any confusing aspects specific to the iRAT/gRAT questions or the material. This step usually lasted about 15 minutes.

The at-home iRAT consisted of the students answering the same 12-13 multiple choice physiology questions that the opposite group was answering in class. The students had several days to work on the questions, which were also administered online. In addition, the students answering the questions at home had unlimited time to answer the questions, and could use course materials to help them answer the questions. They received immediate feedback about whether or not they selected the correct answer and were instructed to continue answering the questions until they chose the correct answer. Once the correct answer was chosen, very often a justification of the answer would appear. In addition, some wrong answers also included explanations. Students were instructed to answer the at-home iRAT as individuals. In addition, when the students opened the iRAT, a statement appeared at the top of the screen in red that reminded them that they were to answer the questions on their own. There was no gRAT or discussion component for the students completing the iRAT at home.

Once the in-class RAP was finished, the students from groups 1 and 2 completed the physiology assessment. The physiology assessment consisted of 12 multiple-choice questions that covered the same material as the at-home iRAT and in-class RAP, but utilized unique questions. Students were not allowed to use outside resources during the 15-minute physiology assessment. The physiology assessment was graded as well as the group application exercise that followed. In addition, the students assessed themselves and their teammates using peer assessment at the end of the four TBL sessions of the course. Each student’s TBL grade was based 15% on their score from two physiology assessments, 75% on their score from the group application exercises, and 10% on their peer assessment score. The students’ TBL score was 15% of their physiology score in the normal body course.

Analysis

Statistical analysis was performed using SAS ver. 9.1.3 (SAS Institute Inc., Cary, NC, USA). For all continuous variables included in this study, descriptive statistics such as means and variances were ascertained. The variables were assessed for distributional normality using the Shapiro-Wilk test.

Kruskall-Wallis tests were used to compare student performance on physiology assessments after the at-home and in-class method of administration. Student performance was measured using mean physiology assessment scores. Mann-Whitney tests were used to compare physiology assessment question effectiveness between the at-home and in-class method of administration. Physiology assessment question effectiveness was measured using mean item difficulty and mean point biserial coefficient for all questions. Two-tailed significance was set at P=0.05 level. Student perceptions of their TBL experience were measured by their responses on course evaluation survey items.

RESULTS

Shapiro-Wilk tests were conducted and showed the variables to be not normally distributed. We therefore used non-parametric statistical analyses throughout this study.

Student performance

Kruskall-Wallis tests were conducted to compare student performance on the physiology assessments based upon method of iRAT or RAP administration. For the first TBL session, mean scores on the physiology assessment were not statistically different between students who completed the iRAT at home (9.1±1.8) and those that completed the RAP in class (8.7±1.7, H=1.21, 1 d.f., P=0.84). Likewise, for the second TBL session, mean scores on the physiology assessment by students who completed the iRAT at home (9.5±1.6) were not statistically different than mean scores for the students who completed the RAP in class (10.0±1.9, H=3.55, 1 d.f., P=0.05) (Fig. 1). These data suggest that method of administration had little impact on student performance on the physiology assessment.

Fig. 1.

Mean physiology assessment scores based on administration method (at-home vs. in-class) (maximum score 12).

Table 1 shows values for physiology assessment item analysis variables for each method of administration. Results of Mann-Whitney tests showed no significant difference in the effectiveness of physiology assessment questions, as measured by mean item difficulty and mean point biserial coefficient.

Table 1.

Statistical analysis of physiology assessments

Variable	Physiology assessment 1		Physiology assessment 2
	At-home (group 1)	In-class (group 2)	At-home (group 2)	In-class (group 1)
No. of items	12	12	12	12
No. of examinees	54	48	46	50
Mean raw score	9.1	8.7	9.5	10.0
Standard deviation	1.8	1.7	1.6	1.9
Reliability (KR-20)	0.42	0.31	0.46	0.62
Mean item difficulty	0.75	0.72	0.79	0.84
Biserial item discrimination	0.33	0.31	0.34	0.47

Team performance on the group application exercise after the at-home iRAT was not statistically different from performance after the in-class RAP. During session 1, teams who used the at-home iRAT had a group application exercise mean score (14.8±0.44 out of 15 points) similar to the mean score for teams who completed the RAP in class (15.0±0.0 out of 15 points). During session 2, teams who used the at-home iRAT had a mean score (12.8±0.89 out of 15 points) comparable to teams who completed the RAP in class (12.4±1.0 out of 15 points).

Student experience

At the conclusion of the course, students completed a course evaluation that included items about their TBL experience using a 5-point Likert scale (where 1=never, 2=seldom, 3=sometimes, 4=regularly, 5=almost always). Fig. 2 shows that students reported iRAT questions to be aligned with course content, whether they were administered at home (4.3±0.9) or in class (4.1±0.9). Fig. 2 also shows that students reported the RAP helped them learn course content, whether administered at-home (4.0±1.2) or in-class (3.7±1.2). Seventy out of 108 (65%) students preferred the at-home method of administration (Fig. 3); while 59 (55%) reported that the at-home method was more effective in helping them learn course content (Fig. 4).

Fig. 2.

Analysis of course evaluation data pertaining to the effectiveness of the at-home iRAT and in-class RAP. A five-point Likert-type scale was used for reported experience (1=never, 2=seldom, 3=sometimes, 4= regularly, 5=almost always) (n=108). iRAT, individual readiness assurance tests; gRAT, group readiness assurance tests.

Fig. 3.

Analysis of course evaluation data pertaining to preference for the at-home iRAT vs. the in-class RAP. The majority of students (65%) prefer the at-home iRAT method of administration compared to the in-class method (21%). 14% of students had no specific preference for the method of administration (n=108). iRAT, individual readiness assurance tests; gRAT, group readiness assurance tests.

Fig. 4.

Analysis of course evaluation data pertaining to the effectiveness of the at-home iRAT vs. the in-class RAP in helping the students learn course content. The majority of students (55%) reported that the at-home iRAT method was more effective in helping them learn course content compared to the in-class method (23%) (n=108). 22% of students reported that both formats were equally effective (n=108). iRAT, individual readiness assurance tests; gRAT, group readiness assurance tests.

DISCUSSION

The performance on the physiology assessment after at-home administration of an iRAT was equivalent to performance after traditional in-class administration of the RAP. During the at-home iRAT and in-class RAP, the students answered the same iRAT questions. However, the at-home iRAT did not include a gRAT or a discussion of any questions about the iRAT. The students working at home did receive explanations and justifications for the answers to the iRAT questions. This may have acted to replace the effect of the gRAT and the discussion after the gRAT on physiology assessment performance. In addition, more time spent on the at-home iRAT may have compensated for the lack of a gRAT when the students worked at home.

During the at-home iRAT the students were allowed to access course materials while during the in-class RAP students could not use outside resources. Despite the fact that the at-home iRAT was open-book, the students still performed equivalently on the physiology assessment. This supports research showing little difference in retention after open-book vs. closed-book assessments [5,6].

The medical students reported a learning preference for the at-home iRAT. The response was similar when the students were asked how well the at-home or in-class iRATs/gRATs were aligned with course content and how well they helped the students learn course content. However, when students were asked which method they preferred, almost two-thirds of students preferred at-home iRATs. When asked to compare which method was more effective in helping them learn course content, more than half of the students reported that the at-home iRAT was more effective. The benefit to the students is that they can go through the iRAT questions at their own pace at home. This allows some students more time to work through and think through the questions while allowing students already familiar with the material to complete the iRAT quickly.

Another issue to consider is how the at-home iRAT vs. the in-class RAP affects long-term retention of the course material. For both TBL sessions, we examined the mean scores on 3 or 4 physiology exam questions that were the most similar to iRAT questions. There was no significant difference in mean exam scores on those questions based upon method of administration. However, both groups of students were given access to the iRAT questions after the TBL sessions for exam study. As a result, we are not able to effectively examine effects on long-term retention, which is a limitation of our study.

From our study, it is not clear if elimination of the gRAT from the RAP affects the development of teamwork skills. Although students would still interact as a team during the group application exercise, it is possible that the gRAT plays a role in teamwork skill development that is not replaced during the group application exercise. Further studies are needed to examine this issue.

After a search of the literature, we believe this to be one of only a few published studies comparing the effectiveness of TBL to a modified version of TBL. One study compared the effectiveness of the RAP with and without the iRAT. Students who completed the iRAT scored better on the gRAT and completed it more quickly compared to students who only took the gRAT. However, there was no difference in performance on course exams [7]. In another study, the group application exercise was not graded. Based on student evaluations, the majority of students preferred this method [8]. As the use of TBL continues, hopefully further studies will present other effective means of tailoring TBL to a specific curriculum or situation.

In conclusion, this study suggests that at-home administration of the iRAT is a promising option for reducing the class time spent in the readiness assurance process. The at-home iRAT is an efficient method for preparing students for group application exercises and it is preferred by our students. Having students complete the iRAT at home allows more class time for TBL application exercises, when higher order thinking is required.