Utilization of evidence-based tools and medical education literature by Canadian postgraduate program directors in the teaching and assessment of the CanMEDS roles.

BACKGROUND: Researchers have shown that clinical educators feel insufficiently informed about how to teach and assess the CanMEDS roles. Thus, our objective was to examine the extent to which program directors utilize evidence-based tools and the medical education literature in teaching and assessing the CanMEDS roles.
METHODS: In 2016, the authors utilized an online questionnaire to survey 747 Canadian residency program directors (PD's) of Royal College of Physicians and Surgeons of Canada (RCPSC) accredited programs.
RESULTS: Overall, 186 PD's participated (24.9%). 36.6% did not know whether the teaching strategies they used were evidence-based and another third (31.9%) believed they were "not at all" or "to a small extent" evidence-based. Similarly, 31.8% did not know whether the assessment tools they used were evidence-based and another third (39.7%) believed they were "not at all" or "to a small extent" evidence-based. PD's were aware of research on teaching strategies (62.4%) and assessment tools (51.9%), but felt they did not have sufficient time to review relevant literature (72.1% for teaching and 64.1% for assessment).
CONCLUSIONS: Canadian PD's reported low awareness of evidence-based tools for teaching and assessment, implying a potential knowledge translation gap in medical education research.

Introduction

The introduction of competency based medical education (CBME)[1,2] is the facilitated development of competency frameworks, such as the CanMEDS Physician Competency Framework.[3,4] Various studies have shown that clinical teachers feel insufficiently informed about how to teach and assess the CanMEDS competencies.[5,6,7,8,9]

Program directors (PD’s) of postgraduate medical education programs in Canada are a prime example of ‘frontline’ clinician teachers. PD’s are responsible not only for the teaching and assessment of trainees at their local programs, but they are also responsible for implementing national standards—as exemplified by the CanMEDS competencies—at their individual institutions. As such, we felt PD’s represented a useful group to study, as they are actively involved with teaching and yet are also aware of new developments in their specialty on a national level.

Our study purpose was to examine the extent to which program directors in the midst of CBME utilize existing tools and the medical education literature in teaching and assessing the CanMEDS competencies. We also sought to examine the main barriers to the utilization of these tools and medical education literature in teaching and assessing the CanMEDS competencies.

Methods

Questionnaire development

We developed an online questionnaire that focused on PD’s awareness and utilization of established tools and the medical education literature in teaching the CanMEDS roles. The instrument consisted of 13 questions. Items on the questionnaire included questions regarding the types of teaching and assessment tools used and the extent to which the tools utilized were rooted in the medical education evidence. Participants were also asked questions regarding the number of years as program director, their area of specialty/subspecialty and whether they conducted medical education research. For questions regarding PD opinion, PD’s were asked to agree or disagree with various statements using a 5-point Likert scale (options included I don’t know, Not at all, To a small extent, To a moderate extent, or To a great extent). As part of content validation, three individuals with expertise and knowledge in the CanMEDS roles reviewed the questionnaire to ensure the contents of the questionnaire were appropriate and relevant to the topic area, and also checked for thoroughness in terms of pertinent items or content areas representation.[10] No changes were made to the questionnaire after expert review.

Sample

All PD’s of RCPSC accredited residency programs were eligible to participate in this study. Email addresses of all residency PD’s were obtained from the RCPSC website. Associate/Assistant PD’s were not included in the questionnaire.

Data collection

An information letter and the questionnaire link were sent to PD’s using Fluid Surveys. Two reminders were sent, at two-week intervals, following the initial questionnaire distribution in order to maximize response rate.[11] Consent was implied by participants’ completion and submission of the questionnaire.

Data analysis

Questionnaire data were analyzed in SPSS 24 using descriptive statistics. To assess the strength of association between years of practice as a PD or years of practice teaching in medical education and other variables, point-biserial correlation coefficients were computed. The number of years for each respective category (years as PD or years teaching in medical education) were treated as continuous variables and responses were dichotomized into binomial responses. The level for statistical significance was set a priori at ≤ 0.05. All statistical analyses were performed using R statistical software version 3.4.2 (R Core Team, Vienna, Austria.).

This study was approved by the Children’s Hospital of Eastern Ontario Research Ethics Board.

Results

The questionnaire was distributed to 747 residency PD’s. 186 PD’s participated (24.9%), representing 50 different specialties (See Appendix A). The mean years involved in clinical teaching for the participants was: 11.7 (±7.4) (range 1.5-35). The mean years in the role of PD was 4.3 (±4.0) (range 0.0 – 28.0).

Our questionnaire asked “To what extent are the strategies you use to teach the CanMEDS roles evidence based (i.e. guided by scientific knowledge and research evidence).” Approximately one third (36.6%) did not know whether the teaching strategies they used were evidence-based and another third (31.9%) believed they were “not at all” or “to a small extent” evidence-based.

Similarly, we asked “To what extent are the strategies you use to assess the CanMEDS roles evidence based (i.e. guided by scientific knowledge and research evidence)”? Approximately one third (31.8%) did not know whether the assessment tools they used were evidence-based and another third (39.6%) believed they were “not at all” or “to a small extent” evidence-based.

While PD’s were aware of research on teaching strategies (62.4%) and assessment tools (51.9%), most believed they did not have sufficient time to review the relevant literature (72.1% for teaching and 64.1% for assessment; Table 1).

Table 1

Responses of Canadian program directors: Awareness of and time available to review teaching strategies and assessment tools.

		I don’t know	Disagree	Agree
	N	n (%)
I am aware of research on teaching strategies	178	7 (3.9)	60 (33.7)	111 (62.4)
I have time to review research on teaching strategies	179	3 (1.7)	129 (72.1)	47 (26.2)
I am aware of research on assessment tools	156	14 (9.0)	61 (39.1)	81 (51.9)
I have time to review research on assessment tools	156	6 (3.8)	100 (64.1)	50 (32)

Out of 158 participants who answered our question as to whether they conducted medical education research, 108 (69%) said they did not. The use of pre-existing strategies or tools and/or the medical education literature was not correlated with years involved in clinical teaching or years spent as PD’s (See Appendix A).

Discussion

Our study demonstrates that the Canadian PD’s are not aware of evidence-based tools in the teaching and assessment of the CanMEDS competencies or did not know if the evidence behind the teaching and educational strategies utilized in their respective programs. This represents a major concern for medical education scholars, faculties of medicine, accreditation bodies and ‘front line’ clinician teachers.

For medical education scholars, our study demonstrates that medical education research has low utilization by ‘front line’ teachers and educators. It is also concerning for faculties of medicine, who, while promoting research in medical education, should note that their faculty are not utilizing the knowledge generated by this research. This situation is also important for accrediting bodies such as the RCPSC since we speculate that they would not want PD’s to implement teaching and assessment strategies simply for the purposes of fulfilling a ‘checklist’ of set accreditation criteria. Rather, we would all want PDs to have put thought into the teaching and assessments utilized and have evaluated the evidence behind the interventions they use. Finally, with respect to ‘front line’ clinician teachers, we note a contrast between the practice of evidence-based medicine and the practice of evidence-based teaching. We expect ‘front line’ clinicians to practice evidence-based medicine, but yet we do not see them practicing evidence-based teaching and assessment.

Concerns have emerged over time that there may be a knowledge translation problem in medical education[12-16] with this gap between research and practice having been identified in the medical education literature.[17-19] Indeed, our results are not dissimilar from a single centre study conducted by Nelson and et al almost 30 years ago,[20] where none of 14 medical school pre-clinical course directors were familiar with any current education research. The authors found that the course directors did not find medical education research particularly relevant to their teaching. The authors also found that the course directors were often looking for “simple answers” to complex educational questions.[20]

Studies concerning knowledge translation in medical education are limited. At the University of Toronto, Onyura et al,[13] interviewed faculty in education leadership roles actively involved in undergraduate teaching and curricular development. They found that these participants only occasionally engaged with medical education research knowledge. They found that many did not seek out information for their educational practices, and instead relied upon integrated experiential, historical, and institutional knowledge.[13] This is similar to what we found for PDs albeit, at a postgraduate as opposed to undergraduate level.

A recent study by Thomas et al.[19] surveyed 396 members of the Association for Medical Education in Europe (AMEE). In contrast to our study, 83% of their respondents stated that they personally drew from research findings when they made important decisions about educational practices. However, 59% felt that (other) educators in the health professions made little use of research findings. Sixty percent agreed or were neutral with the statement “The use of evidence hardly plays a role when decisions about health professions education practices are being made.”[19] Thus, in Thomas et al.’s study, their respondents personally tended to use medical education research, but felt that others often did not. This may have to do with the authors’ selection of AMEE members as their participants, as one would expect that AMEE members, by definition, would have an interest in medical education research. Our group consisted solely of program directors, a minority of which engaged in medical education research. As such, our sample may be more representative clinician teachers whereas Thomas’ et al.’s sample may be more reflective of clinician educators. Most PD’s in our study would not be classified as clinician educators, as only a minority of PD’s in our study engaged in medical education research.

Limited time to read and review the education literature was cited as a major barrier in our study. In effect, respondents may appear to be indicating that other duties -- such as clinical care, administration, paperwork, and bedside teaching as examples—are prioritized over reviewing the medical education literature. Similarly, Thomas et al,[19] found that 73% of respondents strongly disagreed or were neutral to the statement, “I have sufficient time to read all the literature on HPE” and 61% felt they could not keep up with the volume of health professions educational research. Onyura et al.[13] also found a lack of time to due to excessive workload was a major factor impeding access to literature.

Limitations of our study include the relatively small response rate, and the fact that our results do not provide reasons as to why, apart from limited time, program directors tend to not utilize the evidence base in medical education. It would also be useful to understand what helps program directors to utilize this evidence base and how we can get them to use this research more. Further research could help to provide insight into these aids and barriers. Moving forward, our group plans to conduct interviews with PD’s to better examine the aids and barriers in the utilization of evidence-based resources and the medical education literature. A strength of our study is the wide range of specialties that provided responses. An additional feature is that our sample may be more representative of clinician teachers as opposed to previous studies which may have focused more on clinician educators.

Conclusion

Our study is a large survey of postgraduate program directors examining how often they utilize existing tools and the medical education literature in teaching and assessing the CanMEDS competencies. We demonstrated that PD’s have a low utilization of existing tools and the medical education literature for the teaching and assessment of the CanMEDS roles, with lack of time being cited as a major barrier. This finding reinforces a potential knowledge translation gap in medical education research. Medical education researchers and administrators should examine methods to enable improved awareness and utilization of existing tools and the medical education literature.

Table 2

Association between years in practice as Program Director and use of evidence-based tools for teaching. N = 154

CanMEDS Role	Years as PD Mean (± SD)	To a moderate or great extent N (%)	r	P-value
Medical Expert, n = 153	4.2 (±3.5)	87 (56.9)	0.10	0.23
Communicator, n = 154	4.3 (±4.0)	47 (30.5)	0.03	0.73
Collaborator, n = 154	4.3 (±4.0)	39 (25.3)	0.04	0.66
Leader, n = 151	4.3 (±4.0)	31 (20.5)	-0.04	0.60
Health Advocate, n = 152	4.3 (±4.0)	27 (17.8)	0.07	0.42
Scholar, n = 152	4.3 (±4.0)	62 (40.8)	0.01	0.90
Professional, n = 152	4.3 (±4.0)	38 (25.0)	0.09	0.28

Table 3

Association between years teaching in medical education and use of evidence-based tools for assessment. N = 159

CanMEDS Role	Years Teaching Mean (± SD)	To a moderate or great extent N (%)	r	P-value
Medical Expert, n = 154	11.8 (±7.4)	74 (48.1)	0.04	0.65
Communicator, n = 153	11.7 (±7.3)	47 (30.7)	-0.07	0.38
Collaborator, n = 154	11.8 (±7.4)	39 (25.3)	0.03	0.69
Leader, n = 154	11.8 (±7.4)	29 (18.8)	-0.05	0.55
Health Advocate, n = 153	11.8 (±7.5)	34 (22.2)	0.02	0.81
Scholar, n = 154	11.8 (±7.4)	51 (33.1)	0.04	0.64
Professional, n = 154	11.8 (±7.4)	36 (23.4)	-0.03	0.70