Effectiveness of resident as teacher curriculum in preparing emergency medicine residents for their teaching role.

INTRODUCTION: Over the past 30 years, recognizing the need and importance of training residents in teaching skills has resulted in several resident-as-teacher programs. The purpose of this study was to explore the impact of this teaching initiative and investigate the improvement in residents' teaching skills through evaluating their satisfaction and perceived effectiveness as well as assessing medical students' perception of the residents' teaching quality.
METHODS: This research is a quasi-experimental study with pre- and post-tests, continuing from Dec 2010 to May 2011 in Imam Hospital, Tehran University of Medical Sciences. In this survey, Emergency Medicine Residents (n=32) participated in an 8-hour workshop. The program evaluation was performed based on Kirkpatrick's model by evaluation of residents in two aspects: self-assessment and evaluation by interns who were trained by these residents. Content validity of the questionnaires was judged by experts and reliability was carried out by test re-test. The questionnaires were completed before and after the intervention. Paired sample t-test was applied to analyze the effect of RAT curriculum and workshop on the improvement of residents' teaching skills based on their self-evaluation and Mann-Whitney U test was used to identify significant differences between the two evaluator groups before and after the workshop.
RESULTS: The results indicated that residents' attitude towards their teaching ability was improved significantly after participating in the workshop (p<0.001). The result of residents' evaluation by interns showed no significant difference before and after the workshop (p=0.07).
CONCLUSION: On the whole, the educational workshop for Residents as Teacher for emergency medicine residents resulted in favorable outcomes in the second evaluated level of Kirkpatrick's model, i.e. it showed measurable positive changes in the self-assessments of medical residents about different aspects of teaching ability and performance. However, implementing training sessions for resident physicians, although effective in improving their confidence and self-assessment of their teaching skills, seems to cause no positive change in the third evaluated level of Kirkpatrick's model, i.e. the residents' behaviors, and it does not seem to raise students' satisfaction or meet their expectations.

Introduction

Facilitating the medical students and fellow residents’ learning has always been considered one of the essential elements of the medical curriculum by the Accreditation Council of Graduate Medical Education (ACGME) regardless of the students or residents’ career path or specialty (1). The Liaison Committee on Medical Education (LCME) emphasizes the need for programs to develop residents’ skills in teaching and residents should be trained in, and evaluated on their teaching skills (2). Furthermore, a shortage of dedicated faculty staff has increasingly resulted in senior residents being called upon to contribute to the process of teaching undergraduate medical students and junior residents (3).

It has been argued that residents are eager to teach (3). Surveys demonstrate that most residents enjoy teaching, value their roles as educators, and consider it an important component of their training (4,5). On the other hand, they do not feel fully equipped for this pivotal role (6). Without training in effective teaching methods, as well as specific reinforcement courses, they lose confidence in their teaching skills over time (7). In fact, many of them teach ineffectively (5).

Over the past 30 years, recognizing the need and the importance of training residents in teaching skills has resulted in several resident-as-teacher (RAT) programs (8,9). Many courses with a wide variety of clinician-educator curricula have been designed and implemented, and are becoming more common. These programs have widely variable formats, and hence produce variable results (10,11). Furthermore, program directors still express concerns that residents’ teaching skills can, and need to be improved (5,8,12). On the other hand, specific teaching strategies that enhance the teaching behavior of residents remain actually unclear (13). and although there are many published studies describing resident teaching courses, most have not been effectively evaluated (10).

Also the Emergency department compared to other departments is high-acuity and less structured to the workday. This might alter the expectations of the students and residents regarding their educational goals or make the role of educator more challenging (14). In 2005, the Society for Academic Emergency Medicine (SAEM) Undergraduate Education Committee produced a six-module curriculum according to “current literature recommendations for resident teaching skills” (6). These modules have not been validated yet. Although the suggested curriculum can provide a wealth of opportunities for improvement in residents’ ability to teach, objective outcome measures should be defined and assessed. Therefore to the best of our knowledge, there is no published literature addressing the effectiveness of this curriculum.

The most widely used framework for evaluation of a training program is Kirkpatrick’s model. In this framework there are four levels: 1) participants’ reactions to the training program, 2) learning and changes in knowledge, 3) skills or attitude related to the training program and behavior depicted as application of knowledge, and 4) organizational changes related to the program (15). Not all training programmers should be evaluated at all levels but the evaluation is performed at a higher level is more valuable. We plan to explore the impact of this teaching initiative (second level in Kirkpatrick model) and investigate the improvement in residents’ teaching skills through evaluating their satisfaction and perceived effectiveness as well as assessing medical students’ perception of the residents’ teaching quality (third level in Kirkpatrick model).

Methods

The study used a quasi-experimental (pre-test--post-test) design and was done from Dec 2010 to May 2011 in Emam Hospital, Tehran University of Medical Sciences. In this survey, all 1st and 2nd year residents of emergency medicine (EM) who were training in the Emergency Department were included. The residents (n=32) participated in an 8-hour workshop called "Resident as Teacher workshop," being conducted by medical educators. The workshop was designed based on the curriculum suggested by the Society for Academic Emergency Medicine (SAEM). This proposed curriculum addresses six aspects of resident teaching that are applicable to emergency medicine trainees, including “general principles of clinical teaching”, “bedside teaching”, “giving effective feedback”, “teaching procedures”, “teaching with high-fidelity patient simulation” and “teaching effective discussion-leading and lecturing” (6).

The residents’ evaluation was performed in two aspects, including self-assessment and evaluation by learners. The learners were interns who were trained by these residents.

The questionnaire developed to evaluate residents’ self-assessment of teaching in the clinical setting contained three sections. The first section comprised residents’ attitude towards their teaching ability and the second section included residents’ attitudes towards their teaching performance and the third part was on their self-assessment of different aspects of their teaching ability and performance. The content validity of the questionnaire was judged by 5 experts in emergency medicine who were involved in medical education. The reliability was carried out by test re-test. Cronbach’s alpha was used to examine the internal consistency of the questionnaire. Mean scores of the three parts of the questionnaire were calculated 0.88 that were respectively 0.85, 0.85 and 0.96.

The questionnaire used by the students to assess the teaching abilities of the medical residents was a valid and reliable instrument for assessing the quality of teaching among clinical faculty members that had been designed earlier by the department of education center in Tehran University of Medical Sciences. To assess the reliability of this questionnaire, 15 interns from another hospital filled in the questionnaires twice with an interval of 10 days and then the reliability was confirmed by Cronbach’s alpha. Both of the questionnaires were rated on a five-point Likert scale.

Interns as learners evaluated their residents’ theoretical and practical teaching skills by the questionnaire before and after the intervention. In this study, 84 and 67 interns were enrolled before and after the intervention, respectively. These interns spent at least one month of their internship period in the emergency department under the supervision of residents along their rotations. Another questionnaire assessing residents’ self-perceptions of teaching skills was completed by the residents. Paired sample t-test was used to analyze the effect of RAT workshop on the improvement of residents’ teaching skills based on their self-evaluation.

In order to examine the residents’ theoretical and practical teaching skills, means and standard deviations were calculated. Also Mann-Whitney U test was used to identify significant differences between the two evaluator groups before and after the workshop.

Results

All of the 32 emergency medicine residents, who enrolled in the study, completed the study and none dropped out. The distribution of the participating residents were 20 in 1st year (62.5%) and 12 in 2nd year (37.5%). The results indicated that the mean scores of participants’ self-assessments before and after the workshop were 3.5±.54 and 4.4±.54 out of 5, respectively. The mean differences was 0.88 and t=-19.5 with P<0.001. The results support the conclusion that residents’ attitude towards their all domains of teaching, including ability and performance, was improved significantly after participating in the workshop (Table 1). The residents’ self-assessments on different aspects of teaching ability and performance, such as theoretical teaching ability, the ability of clinical training, the ability to conduct lectures and discussions, and the ability to teach these procedures showed that such abilities had been improved after attending the workshop (p<0.001).

Table 1

Residents’ self-assessments on different aspects of teaching ability and performance

Sections	Before workshop Mean (1-5)	After workshop Mean (1-5)	t	Mean differences	p
Self-assessment of attitude towards teaching ability(6 Items)	3.68±0.29	4.5±0.14	-10.3	0.81	p<0.001
Self-assessment of attitude towards teaching performance (5 Items)	2.76±0.86	3.62±1.39	-3.19	0.86	p<0.04
Self-assessment on different aspects of their teaching ability and performance (38 Items)	3.59±0.45	4.49±0.26	-19.05	0.9	p<0.001
Total	3.51±0.54	4.4±0.54	-19.5	0.88	p<0.001

In the next section of workshop's evaluation all of the interns completed their questionnaires. The demographic distribution of the participating interns, before and after the intervention was fairly balanced. The result of residents’ evaluation by students was different. There was no significant difference between learners’ evaluation of the residents before and after the workshop (p=0.07) (Table 2).

Table 2

Resident’s evaluation by Interns before and after workshop

Measurement	Interns	Mean	p
Pre-workshop	84	20.9±3.70	0.07
Post-workshop	67	19.5±4.70

Discussion

Medical education is built on the basis of structured clinical skills (10). Although formal education to the medical students is carried out by attending specialists, medical students spend lots of hours with residents as informal teachers (10,16,17). As a result, the development of resident-as-teacher programs dates back to the early 1960s. Since then, the programs have become both more specialized and more common in western developed countries (18).

In this study, we investigated the effectiveness of a suggested 6-module resident-as-teacher curriculum by SAEM in preparing emergency medicine residents for their teaching role. The question we intended to answer was whether an 8-hour teacher-training workshop based on SAEM-suggested module would improve the teaching abilities of the medical residents.

In our study, the improvement in residents’ teaching skills was evaluated based on their perceptions of their skills and also through evaluating their medical students’ satisfaction with their teaching quality before and after the workshop that was conducted by attending physicians presenting the curricula of SAEM.

The results showed that the resident participants perceived improvement in their self-assessed teaching skills but students’ perceptions of the teaching abilities of the medical residents had no significant difference in teaching abilities of residents who underwent the training.

The efficacy of workshop, based on Kirkpatrick’s model, is interpretable. The training course was evaluated concerning the second and third levels of Kirkpatrick’s model, including improvement in learning and behavior, respectively. On the whole, educational workshop of Resident as Teacher for emergency medicine residents resulted in favorable outcomes in the second evaluated level of Kirkpatrick’s model, so it had measurable positive changes in the self-assessments of medical residents. However, we did not have any positive change in residents' behaviors. Their behaviors seemed not to raise the students’ satisfaction or meet their expectations.

In Julianne et al.’s study (2007) on the impact of a longitudinal cur­riculum to improve teaching skills, they reported similar results. The participants’ self-rated teaching skills significantly increased in all categories of the self-assessment inventory for teaching. Moreover, self-efficacy survey results revealed statistically significant increased participant confidence in all teaching skills (19).

Frattarelli and Kasuya (2003) supported the efficacy of the established program in the improvement of residents’ self-rating in their study (20).

In another study by Bing-You (1990), the internal medicine residents in a resident-managed workshop-type program of eight instructional hours, showed a lower level of organizational skills and the use of audiovisual materials and no significant change was observed in other skills (21). Similarly, Dunnington and DaRosa (1998) reported that few teaching micro-skills improved after residents had been assigned to retreating teaching sessions (16). However, Edward et al. (1998) reported improvements in teaching skills of different groups of residents after implementing training sessions to facilitate the translation of their clinical knowledge to practical teaching techniques (22). These findings showed that workshops aiming to improve teaching skills may have varying impacts on residents in different situation.

The absence of improvement in the evaluation of residents can have different causes. Lack of knowledge and experience of residents in the field of emergency medicine could make them unable to respond to the educational needs of learners. Also the emergency department compared to other departments is too busy and stressful. Therefore, the residents might lack time to apply their educational competencies.

Another reason could be lack of alignment between educational tasks with modules which are trained. For instance, one of the modules in this workshop was teaching with simulators, while residents do not have opportunities to apply this taught skill. Therefore, adjusting educational modules according to residents’ need is important and modifications in the curriculum consistent with aims and educational needs should be considered for each clinical setting separately.

The final cause might be lack of motivators. Although educational skills are included in the residency curriculum to be mastered by the residents, such skills are not evaluated in the assessment systems. Periodical reinforcement by assessing teaching skills is expected due to the fact that it can improve and maintain residents’ competency in this field (21).

Limitations

The limitations of this study include those variables which can have an impact on the long-term outcomes of adult learning and the solution to this limitation is typically to collect and analyze data from as many sources or during several times. Additional limitations were with respect to the methodology of the research. Since different student populations assess residents’ training skills, mismatch of evaluators’ knowledge and attitude towards educational issues can be a confabulating factor. Also our study was performed at a single institution, and the results may not be generalizable to other institutions.

Conclusion

In general, the workshop resulted in favorable outcomes in the second evaluated level of Kirkpatrick’s model and it showed measurable positive changes in the self-assessments of medical residents who took part in the workshop. Consistent with the literature, implementing training sessions for resident physicians, although effective in improving their confidence and self-assessment of their teaching skills, apparently does not cause any positive changes in residents' behaviors and does not fulfill the students’ satisfaction or meet their expectations. Adjusting the curriculum based on the clinical settings and creating opportunities to use educational skills could be effective methods to improve the quality of teaching skill and consequently use the gained knowledge and so improve medical students’ satisfaction. Although the training program improved residents’ self-ratings of teaching skills, this was not reflected in the students’ ratings of resident teaching skills and the possible explanations for this difference can be more explored.