Clinical instructors' perception of a faculty development programme promoting postgraduate year-1 (PGY1) residents' ACGME six core competencies: a 2-year study.

Objective The six core competencies designated by Accreditation Council for Graduate Medical Education (ACGME) are essential for establishing a patient centre holistic medical system. The authors developed a faculty programme to promote the postgraduate year 1 (PGY(1)) resident, ACGME six core competencies. The study aims to assess the clinical instructors' perception, attitudes and subjective impression towards the various sessions of the 'faculty development programme for teaching ACGME competencies.' Methods During 2009 and 2010, 134 clinical instructors participated in the programme to establish their ability to teach and assess PGY(1) residents about ACGME competencies. Results The participants in the faculty development programme reported that the skills most often used while teaching were learnt during circuit and itinerant bedside, physical examination teaching, mini-clinical evaluation exercise (mini-CEX) evaluation demonstration, training workshop and videotapes of 'how to teach ACGME competencies.' Participants reported that circuit bedside teaching and mini-CEX evaluation demonstrations helped them in the interpersonal and communication skills domain, and that the itinerant teaching demonstrations helped them in the professionalism domain, while physical examination teaching and mini-CEX evaluation demonstrations helped them in the patients' care domain. Both the training workshop and videotape session increase familiarity with teaching and assessing skills. Participants who applied the skills learnt from the faculty development programme the most in their teaching and assessment came from internal medicine departments, were young attending physician and had experience as PGY(1) clinical instructors. Conclusions According to the clinical instructors' response, our faculty development programme effectively increased their familiarity with various teaching and assessment skills needed to teach PGY(1) residents and ACGME competencies, and these clinical instructors also then subsequently apply these skills.

Introduction

The outbreak of the severe acute respiratory syndrome (SARS) epidemic in 2003 exposed serious deficiencies in Taiwan's healthcare and medical education systems.1 A project entitled the ‘Postgraduate General Medical Training Programme’ was announced by the Department of Health (DOH), Executive Yuan in August 2003. In this project, each PGY1 resident is required to complete a 6-month holistic healthcare-centred training programme before entering their subspecialist courses.2

In 2006, Taiwan Association of Medical Education (TAME) helped many teaching hospitals to establish ‘General Medical Training Demonstration Centres’ as means of improving medical education of clinical instructors, residents and medical students. Additionally, the DOH arranged to extend the current 6-month postgraduate general medicine training programme to a 1-year programme.3 After 2009, the DOH added a ‘Training Programme for Clinical Instructors’ to the ‘General Medical Training’ programme to help prepare the clinical instructors needed to train the more than 1300 PGY1 residents every year during the 1-year PGY1 programmes.4 5 The goal of the programme is to help clinical instructors such that they have the ability to teach to PGY1 residents the Accreditation Council for Graduate Medical Education (ACGME) core competencies, including medical knowledge, interpersonal and communication skills, system-based practice, practice-based learning and improvement, professionalism and patient care (http://www.acgme.org).6

Taipei Veterans General Hospital (VGH) is a regional medical centre that provides primary and tertiary care to active-duty and retired military personnel and their dependents, and is the primary teaching hospital for its general medicine residency programme.

Our hospital has continuously received economic support from the DOH for the ‘General Medicine Training Demonstration Center Programme’ since 2006. Over the past few years, we have deployed teaching resources and faculty development programmes at Taipei VGH to help establish a patient-centred healthcare system.7 Since 2009, Taipei VGH has aggressively implemented a ‘Clinical Instructor Competencies’ faculty development programme and sought to boost the skills of clinical instructors teaching competencies that target PGY1 residents.

This study uses data collected from the self-reported questionnaires given to all participants in this faculty development programme, and in it we have sought to evaluate the participants' feelings about the value of the various training sessions. Moreover, we have explored the contributions of the different activities of this programme to instilling ACGME competency teaching abilities in the participants. Finally, we have assessed the degree to which the abilities learnt within this programme by participants are applied.

Materials and methods

Background of postgraduate training of doctor in Taiwan

The system for postgraduate training of doctor is similar to the American model, which was developed by the ACGME. The postgraduate training of doctor is regulated by DOH, Executive Yuan of Taiwan ROC. The expectations of the postgraduate training of doctor are that residents are competent in the six core competencies.

Participants

This study involved 134 clinical instructors (physicians) in 10 specialties at Taipei VGH and cooperating hospitals between January 2009 and January 2011. All clinical instructors participated in the training course voluntarily.

Content of the ‘faculty development programme of ACGME competencies for clinical instructors’

Overall, the design of the content of faculty development programme by TAME was based on the coherent educational theory proposed by Hewson.8 It has been suggested that the programme should include videotapes, mini-lectures, group discussions, demonstrations, role play and simulated teaching experiences in order to promote a change in the attitudes, values, beliefs and assumptions about teaching of clinical instructors.8

Accordingly, TAME announced that the faculty development programme consisted of two parts, namely the ACGME competencies-based 40 h basic clinical-practice training course and a postcourse training workshop together with a videotape session. In general, the second part of training course is designed to re-enforce the efficacy of the faculty development programme. The postcourse training workshop for clinical instructors was held with the assistance of TAME. The lectures in the training workshop emphasised teaching skills related to the competencies and lasted at least 7 h. The aims of the training workshop and videotapes were to teach and discuss the ACGME competence-related teaching and assessment skills. At the same time, we design portfolios for all participants to allow them to record and certify their training courses. Finally, TAME certificated their teaching ability according to the records within the portfolios.9 The course can be implemented intermittently or continuously over a period of 3 months. All participants had chances to demonstrate and practice the teaching and evaluation skills learnt in the course.

First part: 40 h basal training course

Outpatient department (OPD) teaching demonstration

The application of competencies in OPD teaching was demonstrated once every week by a senior professor who is a member of the VGH educational committee. In the first hour, a senior professor demonstrated the teaching skills of seeing the patients, talking to families, interviewing and physical examination (PE). In the second hour, PGY1 residents visited a real patient independently, made a short case presentation and completed the writing of the charts. In the last hour, a senior professor demonstrated the teaching skills needed to create a care plan, how to carry out appropriate administrative activities and needs, and how to review the signs, orders and written notes placed on a chart by a PGY1 resident.

Itinerant bedside teaching demonstration

Itinerant bedside teaching was provided three times per week by different senior professors whose are not taking care of patients directly. In the first 0.5 h, PGY1 residents gave a case presentation in the consulting room and discussed clinical reasoning, communication and problem-solving skills with the senior professor and other clinical instructors. With the agreement of the patient, the bedside teaching team visited the patients to confirm the physical signs, make a provisional diagnosis and decide on the best diagnostic and therapeutic options in the following 1 h. Finally, in the last hour, the bedside teaching team returned to the consulting room and discussed how to apply the knowledge gained, including diagnostic test results together with the ability to interpret medical literature as well as the synthesis of information skills in the last hour.

Circuit bedside teaching demonstration

Circuit bedside teaching was demonstrated five times every week. Circuit bedside teaching involved a medical team including medical students, interns, PGY1 residents and an in-charge physician whose taking care of the patients that discussed. Circuit bedside teaching was carried out in the routine ward rounds. Before visiting the patients, in-charge physicians demonstrated the teaching skills of history taking, PE and communication abilities in the first 0.5 h. The medical team and participants demonstrated and practised the teaching skills of evidence-based medicine and self-directed learning ability in the following 1 h of ward runs. Finally, the teaching skills of administrative time management and record-keeping abilities were demonstrated and practised by all participants.

PE teaching demonstration

PE teaching was demonstrated three times every week. The lecture content consisted of instructional skills concerning PE. In the first hour, a trained lecturer demonstrated and practised teaching systemic PE skills to PGY1 residents and clinical instructors. Over the next half hour, trained lecturers, clinical instructors and PGY1 residents visited real patients to verify the specific physical signs reported by the PGY1 residents. In the last half hour, all participants discussed the meaning and significance of physical findings found by members of the PE teaching team.

Case-based discussion teaching demonstration

Senior professors and trained lectures also hosted case-based discussion (CbD) meetings five times each week. CbD mainly assesses the clinical reasoning and decision-making abilities of PGY1 residents. The detailed aspects of evaluation include medical record keeping, clinical assessment, investigation and referrals, treatment, follow-up and future planning, professionalism and overall clinical judgement abilities of PGY1 residents. In the first half hour, the PGY1 residents gave a case presentation to show their abilities. Next, the senior professors, trained lectures and clinical instructors discussed and interacted directly with the PGY1 residents about their performance over the last half hour.

Evidence-based medical teaching demonstration

One-hour evidence-based medical (EBM) meetings to train core competency were hosted by trained lecturers once every week. During the first week of each month, a lecture was given to PGY1 residents and participants. In the second and third week, PGY1 residents presented the application of EBM skills in solving difficult clinical problems presented by their patients in weekly EBM meeting. In the EBM meeting of the last week, participants were invited to comment and demonstrate their instructional skills with respect to EBM meetings.

Objective structural clinical examination teaching demonstration

The objective structural clinical examination (OSCE) consisted of 15 min at each of 12 competency-based stations, and was held once a month by the hospital's OSCE committee as in our previous report.10 All participants observed and practised serving as monthly PGY1 OSCE raters during the first 2 h. Finally, participants reviewed the PGY1 OSCE videos and discussed problems concerning OSCE application with members of the OSCE committee in the third hour of OSCE instruction.

Mini-clinical evaluation exercise teaching demonstration

Mini-clinical evaluation exercise (mini-CEX) demonstrations were provided four times every month by trained lecturers as in our previous report.11 In the first half hour, there were brief lectures on core competency-based mini-CEX evaluation. Over the next half hour, trained lecturers and participants then practised and discussed evaluating PGY1 residents' mini-CEX by watching videos.

Second part

Postcourse training workshop on ‘how to teach ACGME competencies’

This training workshop was held once every 2 months. Lecturers gave 45 min of instruction on teaching skills in each of six competencies and discussed teaching problems with all participants during the last 15 min of each session. After seven lecture/discussion sessions, participants were invited to share their teaching and competency application experiences during the last hour. The organisers also tried to establish a consensus concerning an adequate methodology for evaluation of the competencies and teaching of PGY1 residents. Finally, all participants were asked to fill out questionnaires in order to evaluate their perception of the training course as a whole.

Postcourse training videotapes on ‘how to teach ACGME competencies’

The Taipei VGH educational committee has produced videos of actual and simulated patients for use in ‘competency instructional skills’ tutorials. The video-based tutorials were provided by General Medicine Training Demonstration Center. Patients consented to the filming, which was carried out by a professional audio visual team in actual clinical settings, including outpatient clinics and hospital wards. Three to four short video clips were produced for each patient and demonstrated the application of competencies. The average length of these clips was 2 min. The first clip usually consisted of competency-based medical consultations demonstrated by a trained physician. Subsequent clips usually consisted of the patient's physical examination and follow-up medical consultations focusing on the discussion of investigation results and treatment options. When videos were used in the ‘competencies teaching skills’ tutorials, no written information was given to the participants. In other words, the only materials presented to the clinical instructors were the video clips.

Questionnaires

Evaluation approaches included objectives, expertise, management and participant-oriented aspects of the training programme.9 In our study, an anonymous 38-item questionnaire Kirkpatrick theory-based participant-oriented questionnaire was designed to evaluate the clinical instructors' perception of our training programme (tables 1, 2). Kirkpatrick has described four levels of training-programme outcomes that may be assessed.12 This first level is a measure of the participants' initial reaction to the programme. The second level is to assess the amount of knowledge and skill that participants learnt, while the third level evaluates the amount of knowledge and skills learnt that participants actually use in everyday work. The fourth level is an evaluation of the impact of the programme on the institution and society. It has been suggested that educational institutions should develop an institution-specific evaluation model to meet their particular needs including educational processes and outcomes. Therefore, although we chose Kirkpatrick's four level models as a guide for the evaluation, we adapted it to suit our needs. We interpreted the levels to be:

Table 1

Items from questionnaires for participants (Part 1)

Table 2

Items of questionnaires (Part 2); always: 100%; frequently: 75–100%; often: 50–75%; occasionally: 25–50%; rarely: 0–25% times

Level 1—reaction, an evaluation of participants' initial attitude and familiarity with the ACGME competencies, and ability to train ACGME competencies, which is the target of our programme. These questions are included in table 1.

Level 2—learning, an evaluation of participants' acquisition of each domain of ACGME competencies by different teaching and training activities. These questions were included in point 5 of table 2.

Level 3—transfer, an evaluation of the degree of participants' application of skills learnt from different teaching and training activities. These questions are included as point 6 of table 1. In the questionnaires, participants were requested to rate the frequency of their application of competencies using a five-point scale (1: always (100% of their teaching time); 2: frequently (75–100% of their teaching time); 3: often (50–75% of their teaching time); 4: occasionally (25–50% of their teaching time); 5: rarely (0–25% of their teaching time)).

Preliminary studies with senior physicians recruited from the Department of Internal Medicine permitted us to identify and eliminate unreliable and ambiguous items within our questionnaires in assessing levels 1–3 of Kirkpatrick theory. After a first validation of the questionnaires, six points and 13 items were chosen based on experts' comments and validated again using a group of experts who confirmed the quality of the selected six points and 13 items for evaluation purposes. The final questionnaire was adjusted to reflect feedback from the pilot session. To estimate the reliability of our questionnaires, the Cronbach α coefficient was calculated and our questionnaires found to have a reliability of 0.81. Additionally, the re-evaluation reliability of our questionnaires was around 0.85.

Before respondents answered each questionnaire, we provided written definitions of the six competencies to participants. In the first part of questionnaire, a five-item Likert scale was used to rate the degree of respondents' agreement with the teaching skills provided by different activities in the competency training programme for clinical instructors.13 In the second part, respondents were asked to indicate which of the corresponding six areas were learnt in each of the structured activities of the training courses. Recognising that the language of the competencies is rather general, we encouraged participants to use their judgement when deciding whether a particular teaching activity provided training in one or more area (tables 1, 2). The questionnaires were filled out after participants completed the basic 40 h training course, after the training workshop and after the videotape session. To assess the degree of application of skills learnt from different activities, participants were asked to fill out a follow-up questionnaire 3 months after completion the training course. All the activities in our programme were divided into three parts, namely teaching activities, evaluation activities and postcourse training workshop/video session (table 2).

Analysis

The average application of and attitudes towards competency instructional skills were analysed using paired t tests. Differences in the application of activity-orientated instructional skills, attitudes, familiarity and teaching ability between different groups were analysed by ANOVA. The degree of changes in the participants' attitude and familiarity to teach and train ACGME competence after 40 h basic training course, postcourse training workshop and videotapes were analysed using paired t tests and the p values of trends. Furthermore, the effect of previous training (years as an attending physician and whether the participant had taken the TAME course) and teaching experiences (being a PGY1 clinical instructors or mentor) on the average degree of application of competencies in teaching was analysed using the χ2 test.

Ethics statement

This study was approved by the Ethics Committee of Taipei Veteran General Hospital and complied with the principles of the Declaration of Helsinki Guidelines. In agreement with these standards, written informed consent was obtained from each participant.

Results

Characteristics of participants

Among the original 134 clinical instructors, 17 clinical instructors were not included in the study because they did not complete all aspects of the training, and additional seven clinical instructors did not complete the questionnaire; this yielded a final total of 110 study subjects. Among the 17 clinical instructors who did not complete the course, there were eight males (two from Gynaecology; three from Neurology; three from Emergency Medicine) and nine females (two from Surgery; three from Rehabilitation; two from Family Medicine; two from Psychiatrics). Additionally, only one male and one female clinical teacher among the above 17 clinical instructors had previous experience as a PGY1 mentor and clinical instructor (2/17, 12%). On the other hand, two females (one from Surgery; one from Paediatrics) and five males (two from Chest Medicine; one from Rehabilitation; two from Family Medicine) did not complete the questionnaire, and only one female among them (1/7, 14%) had experience as a PGY1 mentor. Moreover, most of the above clinical instructors (19/24, 79%) came from other cooperating hospitals rather than Taipei VGH. In a retrospective interview by email, most stated that their main reason for failing to complete the course was the time limit of 3 months. Thus it is probable that the complete rate of the programme would be increased if the programme did not insist that the participants complete all course within 3 months.

It can be seen from table 3 that 90 male and 20 female clinical instructors completed the programme. Participants came from different specialties including Internal Medicine (40/110, 37%), Surgery (20/110, 19%), Gynaecology (8/110, 8%), Paediatrics (10/110, 9%), Emergency Medicine (10/110, 9%) and others (Neurology, Psychiatrics, Chest Medicine, Rehabilitation and Family Medicine; 12/110, 18%). Table 3 shows that 42% and 66% of the clinical instructors in our study had experience of being a PGY1 mentor or a clinical instructor, respectively. The average degree of application of competency instructional skills was markedly higher for participants from internal medicine than for participants from other specialties. Interestingly, the degree of application of skills learnt from training showed a significant decrease with the increase in the individual's years as an attending physician (table 3, p value for trend: 0.0028). In other words, the application of competency teaching skills was negatively correlated with how many years the person had been an attending physician.

Table 3

Basal characteristics of participants (clinical instructors) (n=110)

	Case no: cases/total (%)	Degree of application
Gender
Male	90 (82)	3.5±0.8
Female	20 (18)	4.1±0.3
Specialty
Internal Medicine	40 (37)	4.2±0.9*
Surgery	20 (19)	2.3±0.5
Gynaecology	8 (8)	1.9±0.2
Paediatrics	10 (9)	3.1±0.6
Emergency Medicine	10 (9)	2.2±0.9
Other (Neurology, Psychiatrics, Chest Medicine, Rehabilitation, Family Medicine)	12 (18)	1.9±1.3
No of years as an attending physician
<8 years	34 (31)	4.1±0.7†
9–10 years	20 (18)	3.2±0.9†
17–24 years	14 (12)	3.1±1.4†
>24 years	42 (39)	2.9±1.3†
Postgraduate year-1 resident's mentor experience
Yes	48 (44)	4.3±1.1
No	62 (56)	3.0±0.7
Postgraduate year-1 resident's clinical instructor experience
Yes	66 (60)	3.9±1.5*
No	44 (40)	2.7±0.4
Taken Taiwan Associate of Medical education course before?
Yes	81 (74)	3.4±1.3
No	29 (26)	4.1±0.3

Degree of application was rated by Likert scale: (5: always; 4: frequently; 3: often; 2: occasionally; 1: rarely).

p Value <0.05 versus corresponding groups.

p Value for trend <0.01.

In addition, the degree of application of competency instructional skills was significantly higher in participants with previous experience of being PGY1 clinical instructors than those without such previous experience. However, there were no differences in the degree of application of skills learnt from training attributable to gender, having experience as a PGY1 mentor or having TAME course experience (table 3).

Average degree of application of skills learnt from activities of the faculty development programme in participants' teaching

In table 4, a comparison between teaching activities showed that clinical instructors reported that the skills always (100% of teaching times) used were mainly learnt from itinerant bedside teaching demonstrations, while the frequently (75–100% of teaching times) used skills were learnt from circuit bedside teaching demonstrations. Rarely (<25% of teaching times) used skills were learnt from PE teaching demonstrations. In the evaluation activities, clinical instructors reported that the always (100% of teaching times) and frequently (75–100% of teaching times) used skills were learnt from CbD evaluation demonstrations, while rarely (<25% of teaching times) used skills were learnt from OSCE evaluation demonstration. After addition of the percentages of always, frequently and often applied instructional skills, it was found that the most (>80% teaching times) commonly used skills were learnt from itinerant teaching, circuit bedside teaching and mini-CEX evaluation demonstration.

Table 4

Degree of avenge application of skills learnt from different teaching and training activities in their teaching (n=110)

Demonstration activities in training program	Percentage always (1)	Percentage frequently (2)	Percentage often (3)	Percentage occasionally (4)	Percentage rarely (5)	Percentage 1+2+3
Outpatient department teaching	11±2	37±4	25±5	19±9	8±0.7	73±1
Itinerant bedside teaching	27±3*	25±3	30±8	15±8	3±0.9	82±2
Physical examination teaching	13±4	28±5	19±7	19±5	21±0.8*	60±5
Circuit bedside teaching	23±1	48±3*	25±3	4±3	0	96±4**
Evidence-based medicine teaching	21±8	38±9	15±8	19±3	7±0.9	74±2
Case-based discussion evaluation	12±3*	39±7*	3±1	27±2*	19±0.1	54±3
Objective structural clinical examination evaluation	9±0.8	27±5	27±5	12±3	25±0.3*	63±6
Mini-clinical evaluation exercise evaluation	8±3	29±3	37±3*	23±4	3±0.4	74±8*
Training workshop of ‘how to teach ACGME competencies’	18±2	36±2	31±2	13±2	2±0.5	85±9
Training videotapes of ‘how to teach ACGME competencies’	12±3	37±4	35±4	15±4	1±0.7	84±3

Frequency of application (always: 100%; frequently: 75–100%; often: 50–75%; occasionally: 25–50%; rarely: 0–25% of teaching times of participants); The results were averaged data from questionnaires filled by participants at the end and follow-up 3-month after the training course.

*p<0.05 and **p<0.01 versus others (comparison the percentage of application of skills learnt from different teaching and training activities in their teaching).

ACGME, Accreditation Council for Graduate Medical Education.

Average percentage of acquisition of skills of ACGME competencies from activities of the faculty development programme

In teaching activities, participants reported that skills involving medical knowledge and system-based practice domains were mainly learnt from OPD teaching demonstrations, while skills involving interpersonal and communication skills domain were learnt from circuit bedside teaching demonstrations. Furthermore, skills involving practice-based learning and improvement domain were learnt from EBM teaching demonstrations, and skills involving professionalism domain were learnt from itinerant bedside teaching demonstrations. Finally, skills involving patient care domain were learnt from PE teaching demonstrations (table 5).

Table 5

Average percentage of acquisition of teaching and assessment skills of Accreditation Council for Graduate Medical Education (ACGME) competencies from training activities reported by participants (n=110)

Demonstration activities in training programme	Percentage medical knowledge (1)	Percentage interpersonal and communication skills (2)	Percentage system-based practice (3)	Percentage practice-based learning and improvement (4)	Percentage professionalism (5)	Percentage patient care (6)	Percentage 1+2+3+4+5+6
Outpatient department teaching	68±4*	12±2.7	15±0.4*	17±1	15±3	35±2	162±21**
Itinerant bedside teaching	31±2	3±0.3	1±0.03	14±2	42±11*	25±6	116±18
Physical examination teaching	27±3	5.1±0.2	8±0.05	3±0.9	10±3	31±4*	84±21
Circuit bedside teaching	12±7	53±0.9*	12±3	8±2	6±2	22±2	113±9
Evidence-based medicine teaching	9±0.6	8±0.7	9±0.8	22±4*	5±0.3	7±0.8	126±15*
Case-based discussion evaluation	11±1	5±1	29±2*	49±11**	9±0.7	23±3.7	60±8
Objective structural clinical examination evaluation	21±3	4±0.8	3±0.3	8±3	27±2*	13±3	76±8
Mini-clinical evaluation exercise evaluation	19±2	28±5*	2±0.6	3±0.6	8±0.9	51±9**	111±17
Training workshop of ‘how to teach ACGME competencies’	12±5	19±3	20±3	34±5	31±0.6	10±0.8	126±23*
Training videotapes of ‘how to teach ACGME competencies’	3±0.8	9±1.7	14±0.9	22±5	31±5	18±7	97±9

The results were averaged data of questionnaires completed by participants at the end of training course and 3 months after finishing the training course; participants were asked to choose one or two competencies for each activity.

*p<0.05 and **p<0.01 versus other teaching or evaluation activities.

In terms of evaluation activities, participants reported that skills involving interpersonal and communication skills and patient care domains were mainly learnt from mini-CEX demonstrations, while skills involving system-based practice and practice-based learning and improvement domains were learnt from CbD evaluation demonstrations. In addition, the skills involving professionalism domain were learnt from OSCE evaluation demonstrations.

Effects of postcourse training workshop on the participant's attitudes to ACGME competencies

Participants reported that postcourse training workshop effectively improved their agreement with the aspects of ‘should be learnt,’ ‘should be taught’ and ‘should be evaluated’ (figure 1). However, attitudes concerning ‘meeting social expectations’ of competencies were not changed by postcourse training workshop.

Figure 1

Participants, averaged attitude to the Accreditation Council for Graduate Medical Education six competencies (n=110), the degree of agreement with the asked questions was rated by Likert scale (1: strongly disagree 2: disagree; 3: neutral; 4: agree; 5: strongly agree). *p<0.05 versus finishing 40 h basal training course.

Effects of postcourse training workshop and videotape session on the participant's familiarity to ACGME competencies

Participants reported that both training workshop and videotapes significantly improved their familiarity with teaching and definitions of competencies among participants (figure 2A). In addition, the postcourse training videotapes also significantly improved participants' familiarity with assessment of competencies. Furthermore, participants felt that their familiarity with teaching and assessment of competencies improved progressively as the 40 h basic training course, postcourse training workshop and videotapes were completed.

Figure 2

Effects of postcourse training workshop and videotapes on the familiarity to (A) Overall and (B). Each domain of Accreditation Council for Graduate Medical Education six competencies. *p<0.05 versus finishing 40 h training course; †p value for trend <0.05 (progressively increase between different groups). ICS, interpersonal communication skills; MK, medical knowledge; P, professionalism; PBLI, problem-based learning and improvement; PC, patient care; SBP, system-based practice.

Participants reported that most of them were familiar with instructional skills in the domains of patient care, interpersonal and communication skills, and medical knowledge (Likert scale of >3.5) after the 40 h basic training course (figure 2B). However, participants felt that they were still not very familiar with the instruction in system-based practice, practice-based learning and improvement, and professionalism domains. Notably, participants felt that they were familiar with system-based practice, practice-based learning and improvement, and professionalism competencies progressively after finishing the postcourse training workshop and videotape session.14 15

Effects of postcourse training workshop and videotapes on the participant's training abilities with respect to ACGME competencies

After finishing the 40 h basic training course, postcourse training workshop and videotapes, participants felt that their ability to teach, assess and improve the PGY1 residents' ACGME competencies were improved progressively (figure 3A).

Figure 3

Effects of postcourse training workshop and videotapes on the establishment of the ability to train (A) Overall and (B). Each domain of Accreditation Council for Graduate Medical Education six competencies. *p<0.05 versus finishing 40 h training course; †p value for trend <0.05 (progressively increase between different groups). ICS, interpersonal communication skills; MK, medical knowledge; P, professionalism; PBLI, problem-based learning and improvement; PC, patient care; SBP, system-based practice.

Furthermore, participants felt that postcourse training workshop and videotapes significantly improved their ability to train PGY1 residents' system-based practice, problem-based learning and improvement, and professionalism domains (figure 3B). Nevertheless, participants reported that postcourse training workshop and videotapes did not further change their ability to teach PGY1 residents' medical knowledge and patient care competencies.

Percentage change from baseline in the degree of participant's application (>50% of teaching times) of skills learnt from activities in the follow-up questionnaire

In terms of teaching activities, participants reported in the follow-up questionnaire that the highest degree of improvement in the application frequency of skills was learnt from itinerant bedside teaching (figure 4). Among the evaluation activities, participants reported that the highest degree of improvement in the application frequency of skills was learnt from a CbD evaluation demonstration. Finally, participants reported that the degrees of improvement in the application frequency of skills learnt from postcourse training workshop and videotapes were similar.

Figure 4

Percentage changes from baseline of the degree of application of skills learnt from different (A). Teaching activities; (B). Training activities; (C). Postcourse training workshop/videotapes in follow-up questionnaires (3-month) reported by participants. p<0.05 and **p<0.01 versus other teaching/evaluation activities.

Discussion

After completion of the 40 h basic training course, participants did not agree strongly that they should learn ACGME competencies. This is probably due to the fact that clinical instructors considered that the ACGME competencies taught in our programme did not meet social expectations very well.4 15 In addition, participants felt that they did not acquire better instructional skills in the domains of system-based practice, problem-based learning and improvement, and professionalism domains to a significant degree after 40 h basal training course. Nonetheless, participants felt that the postcourse training workshop and videotapes (figures 2B, 3B) significantly improved their familiarity and teaching ability in the PGY1 residents' system-based practice, problem-based learning, and improvement, and professionalism domains.16 17 Accordingly, our programme directors should consider extending the hours of activity including itinerant bedside teaching and CbD evaluation demonstration that mainly trained the participants' instructional skills of the above three domians.16

Clinical instructors felt that both postcourse training workshop and videotapes markedly improved their familiarity and instructional skills of the above three domains. Previous studies had suggested that training videotapes consistently enhance trainees' observational powers, improves their ability to integrate different information and motivates them to learn.5 18–20 Our programme director should probably increase the hours of postcourse training workshop and videotapes session to improve training in the domains of system-based practice, problem-based learning and improvement, and professionalism in the future. Alternatively, the programme director can also eliminate the training sessions such as PE teaching and OSCE evaluation demonstrations that were not preferred by participants and spend more time on those training methods preferred by participants.

In addition, the participants also responded in the follow-up (3 months) questionnaires that the most skills that they applied in teaching had been mainly learnt from itinerant and circuit bedsides, and PE teaching demonstrations, postcourse training workshop and videotapes (table 4). Conversely, participants answered that they rarely used instructional skills learnt from CbD evaluation demonstrations. At the same time, our study suggested that the CbD evaluation demonstration mainly establishes teaching and assessment skills in the domains of system-based practice, problem-based learning and improvement. The lack of application of CbD instructional skills might have resulted from participants' poor familiarity with CbD instructional skills after completion of the programme. Thus, the programme directors should also consider extending the hours of CbD evaluation demonstration in future.

This study had several limitations. First, the questionnaire used to track and assess the participants' perception of the faculty development programme may suffer from information and recall bias. In other words, the evaluation of participants' perception to training should therefore be assessed immediately after each training session by persons other than the programme directors.7 In fact, a self-reported questionnaire might not reflect behaviours in authentic setting. Thus, instead of participants' feelings about the value of the various training sessions, we should assess the improvement in teaching ability of participants for PGY1 residents' competencies.

Second, we only assessed the participants' perception about levels 1–3 of the Kirkpatrick approach with regard to our programme. In other words, we did not evaluate the participants' basal attitude and familiarity with the ACGME six core competencies at the beginning of the faculty development programme. In order to validate our programme, a basal assessment of participants' perception is needed in the future.

Third, according to the ‘original’ definition, Kirkpatrick's third level evaluates ‘the amount of knowledge and skills learnt that participants actually use in everyday work.’ In our study, we modified the third level to be the ‘self-reported degree of participants’ application of skills learnt from different teaching and training activities.' Furthermore, in order to validate the results of our faculty development programme effectively, a clear definition of the desirable teacher's desirable behaviours needs to be given to participants in future works. The behaviours perhaps should include the following: creating an appropriate learning climate; being learner-centred; facilitating participants' learning; encouraging self-awareness through reflection; tailoring teaching to participants' needs and wants, etc. suggested by Hewson.8 Additionally, it is well established that when one goes through an evaluation process, the evaluation process becomes more difficult and time-consuming as one moves from Kirkpatrick level 1 to level 4, although higher level provides more information than lower level, that is, of increasingly significant value than lower level. Thus, the level 4 needs to be assessed in the future for our programme.9 12 It should be noted that, as yet, no controlled educational trials on this subject have been published, and this type of trial would be useful.18 Finally, this study involved only a short follow-up period, and as a result, the study's findings may represent only short-term changes in attitude, familiarity and teaching skills.9

Despite these limitations, the present study provides information about the participants' perception of the various training sessions of our ‘faculty development programme about competency for clinical instructors’ designed and organised by the Taipei VGH educational committee. To date, there are no well-established standards that specifically address the competency teaching and assessment skills of clinical instructors.21

Conclusions

The participants in our ‘faculty development programme of ACGME competencies for clinical instructors’ reported that our programme effectively increased their familiarity with various teaching and assessment skills of competencies for PGY1 residents. Additionally, clinical instructors also reported that they subsequently applied these skills in their work.