Implementation and evaluation of an innovative leadership and teacher training program for non-physician emergency medicine practitioners in Uganda.

INTRODUCTION: Leadership and teaching skills are essential, but not often emphasized, components of medical training. As emergency care develops as a specialty in Uganda, two cadres of providers are being trained: physicians and non-physician clinicians (NPCs). Building formal leadership and educator training into these curricula is essential.
METHODS: A week long continuing education (CE) course on leadership and teaching is described and evaluated for effectiveness using Kirkpatrick's framework for learner-centred outcomes. The emergency care trained NPCs participated in a week-long course consisting of lectures, role-playing, and small group discussions, as well as a personality self-assessment. The evaluation process consisted of: 1) an immediate post-course survey to measure learner satisfaction, 2) a retrospective, pre/post self-assessment with a Likert-type scoring tool to measure knowledge gains, and 3) a three-month follow up survey and structured interviews to measure knowledge retention and behaviour change in practice.
RESULTS: All 15 NPCs participated in the evaluation process. Learner satisfaction was high with an average score of 9.3 (on a 1-10 scale) for course content, amount learned, and use of time. Participants reported gains in knowledge for each of the 24 competencies measured, with an average difference in pre- and post-course Likert scores of 1.11 (on a scale of 1-5). Lastly, all 15 participants shared detailed examples of using course content in practice three months after the course finished. The most frequently mentioned themes were "giving and receiving feedback," "delegating and assigning tasks," and "communication."
CONCLUSION: This course was a successful CE intervention in this setting as measured by Kirkpatrick's framework. The most frequently mentioned concepts used in practice point to the NPCs ability to take on leadership roles in this setting. Further research and evaluation methods should focus on the influence of culture and personalities on leadership education and translation into practice in an EM setting.

African relevance

Development and implementation of intervention took place in Uganda.

Intervention is culturally appropriate and accepted in Uganda.

Results can be generalizable to Uganda and possibly similar East African countries.

All participants are Ugandan.

Introduction

Emergency care (EC) services are an important foundation of any healthcare system. However, in most countries in sub-Saharan Africa, emergency care is not developed as a formal specialty. In 2017, Uganda successfully launched formalised advanced training for both physicians and non-physician clinicians (NPCs) in EC, through Mbarara University of Science and Technology (MUST).

The non-physician training program has been run since 2009 as a pilot by a non-governmental organization, Global Emergency Care (GEC), in partnership with a local Ugandan not-for-profit hospital, Karoli Lwanga Hospital. The graduates of this program are locally referred to as Emergency Care Practitioners (ECPs). The training model has graduates of the program actively participating in the training the new ECP students in order to more rapidly scale capacity for locally driven emergency care training and delivery. During the second year of the two-year program, the students develop a deeper knowledge of the course content and are taught basic teaching skills. The graduates of this program have been successful in reducing mortality of those seeking emergency care and the program is expanding [5], [9].

As the NPC program grows, these trainees are in a challenging position. They are among the only trained emergency care providers in Uganda and hence are placed into leadership and teaching roles without formal training in these areas. This situation is not unique to Uganda. A review of specialty training programs in SSA shows that traditionally, as in most emergency medicine training programs there is a focus on clinical knowledge and skills, while curriculum on leadership and teaching is mostly absent. Currently there are emergency medicine residency programs in ten African countries, with two non-physician clinician training programs. Many new providers from these programs have reported the lack in leadership and teacher training as a significant gap in their education, which has led to some providers leaving EC, as they felt unprepared for the leadership roles into which they were placed [3]. In a review of the available literature on faculty development within new emergency care programs in Africa, only three articles mention this training gap, and to our knowledge, there have been no program evaluations published addressing this gap [2], [3], [8].

The objective of this study is to describe a teaching and leadership course developed specifically to help close this knowledge and skill gap among the original cohorts of ECPs. Additionally, we intended to evaluate the effectiveness of the course, using a framework for learner-centred outcomes, in providing new knowledge and skills, which result in behaviour change in practice.

Methods

In an effort to close this training gap, GEC developed, implemented, and evaluated a continuing education (CE) course in leadership and teacher training specific to emergency care NPCs in low-resource environments. The sessions covered a breadth of topics ranging from basic teaching skills to complex interpersonal communication skills with a focus on leadership development.

This was a mixed-methods (quantitative and qualitative), retrospective pre/post, single-cohort design. Research ethics board approval was granted by the University of Washington in Seattle, Washington and Makerere University in Kampala, Uganda. While the course was mandatory CE for professional development for the staff, participating in the evaluation process for the purposes of this study was optional and anonymous. Each participant was eligible to provide feedback, and consent forms indicating their voluntary and willing participation were obtained from all participants. All the participants are Ugandan and fluent in English.

A short course in leadership and clinical teaching was developed by the local GEC support staff, foreign Emergency Medicine board-certified faculty, as well as input from experts in leadership and adult education from the University of Washington. The course content was developed with input and expert opinion from the GEC Board of Directors and the ECPs who were preparing to function in a teaching capacity for the NPC training program. The primary aim of the course was to improve the quality of the already-established GEC program by providing senior ECPs with additional knowledge and skills to support them as teachers and leaders.

A list of educational sessions was developed using various formats, including lectures, role-playing, and hands-on workshops (Appendix A). Additionally, as an introduction to teamwork and communication, a personality self-assessment was completed on the first day of the course. The results of this assessment were then discussed in a group setting as an exercise in understanding personalities and communication styles, and were considered during facilitation of group exercises, such as role-playing. At one training site, the course duration was approximately six hours of content per day each day for one week, while at the second site the course took place in 4-hour sessions twice a week over the course of three weeks.

The course evaluation is grounded in Kirkpatrick’s framework for measuring success of educational interventions via learner-centred outcomes (Fig. 1).

Fig. 1

Evaluation methods linked to Kirkpatrick’s framework.

The course was first evaluated using an immediate, retrospective, pre/post self-assessment and course survey (Appendix B). This was a paper survey which the participants completed anonymously, and returned to the facilitator. The facilitator was not an employee of GEC nor had influence on the participants’ employment. The survey was developed with the guidance of the Ugandan program director in order to ensure understanding of word choices and phrasing. To assess the first level of Kirkpatrick’s taxonomy – learner reactions and satisfaction – the ECPs were asked to choose which sessions were most and least helpful, and rate on a scale of 1–10 the overall content, amount learned, and use of time. Participants were then asked to retrospectively rate their comfort before and after the course with the core course competencies on a Likert-type scale from 1 to 5, with “1” being not at all comfortable with a concept and “5” meaning extremely comfortable. This reflects the second level of Kirkpatrick’s model: learner acquisition of knowledge, skills, and attitudes. Given a small N of 15, and this being a short-term output, the results of this survey are presented as differences in the mean scores without a statistical analysis.

To assess long-term gains in skills and competencies, as well as behaviour change in practice, qualitative data were collected three months after the course, via a follow-up survey and structured interviews (Appendix C). All participants were provided the list of core competencies covered in the course and asked to name the concepts they have used in the previous three months. In order to mitigate social desirability bias, participants were asked to provide descriptive details and examples to illustrate their responses. The survey was developed with input from experts in qualitative research, educational intervention research, and the local language and culture. Participants were allowed time to fill out the paper survey, and then the facilitator conducted structured interviews with the participants to review their answers and obtain any elaboration or clarification needed. Participant verbal responses were transcribed verbatim during the interview, and any alterations in word choice made by the interviewer were reviewed by the participant for accuracy in meaning and intent. The transcripts were then entered into Excel and analysed by the research coordinator for recurrent themes, key words, and phrases taught during the course. Most commonly mentioned concepts were counted and graphed for comparison.

Results

The short course took place at two clinical sites in Uganda, with all 15 qualified ECPs and second-year ECP students attending and completing the evaluations and surveys. Table 1 provides more detailed information about the course participants.

Table 1

ECP participant characteristics.

	MRRH	KLH
Gender
Male	2	6
Female	2	5
Age
<25	0	2
>/=25	4	9
Previous degree
Enrolled nurse	3	9
Registered nurse	1	2
Years working as an ECP
<2	0	4
>/=2	4	7

The ECPs were first asked to provide feedback about the course in general. They were asked to rate three statements regarding the course as a whole on a scale of 1–10, where 1 is “none or not good” and 10 is “a lot or great.” The statements were, “the amount I learned was,” “the content presented was,” and “use of class time was.” Overall the course was extremely well received; the average of the 15 participants’ responses ranged from 8.9 to 9.5 for each statement. Participants were also asked to pick the two sessions which they felt were the most valuable and least valuable and explain why, these results are summarised in Appendix D. When asked about overall value of the sessions, one ECP reported, “all [sessions] were most valuable” and another said, “everyone can become a leader and needs all skills as listed.”

All participants reported improved comfort with all competencies taught during the course. Fig. 2 shows the retrospective pre- and post- self-assessment scores and average difference in each score in all 24 competencies. The competencies with the most significant gains were “ask for feedback and listen to all members of the team,” with an average increase in score of 1.68 points, and “ability to make an effective and logical PowerPoint presentation,” with an average increase in score of 1.55. Other important gains were made in “ability to give feedback and constructive criticism” (average increase of 1.43), and “ability to assign roles and delegate tasks” (average increase of 1.31). Both asking for and giving feedback were also rated lowest overall in comfort level prior to the course.

Fig. 2

Retrospective pre/post self-assessment results.

The three-month post-course survey was completed by 100% of the participants, and asked about which content from the course they use most in practice and to provide examples. The three-month follow up results depicted in Fig. 3 focus on the most frequently mentioned major themes instead of specific skills used such as making a PowerPoint presentation.

Fig. 3

Frequency of most commonly mentioned concepts.

These major themes include giving and receiving feedback, delegating, direct communication, confidence, leadership, and teamwork and suggest personal growth and/or a better understanding of emergency care culture.

When asked to name examples of changes they have made to their lecture and/or bedside teaching styles, 8 of the 15 participants (53%) named techniques regarding feedback and/or communication as concepts they not only learned, but have utilised. The participants gave substantial examples for when and how they have put these important emergency care skills into practice including: “[While] giving feedback – talking to juniors – recognise that there are positives, but also include negative aspects.”; “I have developed confidence in my communication skills and good explanations.”; “[I] have improved in communication with other colleagues, ECPs and nursing students.”; “Asking for more feedback,” when asked to name ways they have improved their lectures and presentations.

Lastly, when asked to name concepts that have been most helpful and/or changes they have made in practice since the course, 11 of the 15 participants (73%) named concepts related to leading a team and delegating. Examples and explanations include: “[Improvement in] ability to delegate roles and tasks especially in simulations and in times when the ED is busy with sick patients.”; “[Improvement in] leadership skills – appreciating others abilities, being more understanding and patient.”; “Leadership and teaching skills have helped me be confident”; “[Improved in] ability to assign roles and delegate tasks”.

Discussion

To the best of our knowledge this is the first course targeting faculty development for emergency care providers in a low-income country. Here we synthesis the implementation and evaluation of the pilot course.

Kirkpatrick provides a widely accepted four-level training evaluation model, which includes additional key outcomes: learner reactions and satisfaction, the learning of knowledge, skills, and attitudes, and behaviour change in practice [4], [12], [11]. Using Kirkpatrick’s framework as a guide for this short course evaluation, the following learner-centred outcomes were proposed to measure effectiveness: 1) overall satisfaction with course content, 2) knowledge gains immediately following the course, and 3) knowledge retention and behaviour change in practice. This course proved successful at each of these levels of Kirkpatrick’s framework, suggesting that this course is an effective CE intervention for teaching leadership and teaching skills to NPCs working in emergency care in Uganda.

The high overall course assessment scores provide evidence for learner satisfaction, pointing to high cultural acceptability and relevance. The reported positive differences in comfort level for all categories in the pre/post assessment provide evidence for knowledge and skills gained. Furthermore, reported retention and use of concepts learned provides evidence for behaviour changes in practice. As the GEC program in Uganda grows, the current ECP trainers, ECPs, and future ECP students will benefit from the skills and knowledge learned during this course, as well as from the reported leadership growth the ECPs made during the three months following.

Over half of the participants described putting into practice what they had learned in the course about leadership, communication, delegation, and teamwork. In a hierarchical, physician-dominated hospital culture, it was surprising to hear NPCs were feeling more confident in taking on leadership roles and delegating [1], [7]. It could be argued that those who gravitate to emergency care in Uganda are similar to emergency care providers in Western countries where trends in personality types have been documented in the literature [10]. Traits that are most prevalent in the Western emergency care physician providers studied include enthusiastic, straight-forward, analytical, insensitive, and apt to loathe rules and guidelines [10]. In contrast, while there are no similar reports on clinicians in Uganda, the majority of participants in this evaluation self-identified as a “Dove.” Those who identify with this type are usually relaxed, slow-paced, and tend to avoid confrontation, change, and assertiveness [6]. The results of this self-assessment provides context for this cohort of clinicians and refutes the argument that they may have been predisposed to be direct and assertive prior to the course. Given this context, the three-month follow up results showing ECPTs adopting many of these behaviours is all that more profound. It demonstrates the potential for this course to influence individuals’ growth as leaders in the department and shift the traditional physician-led culture.

Further conclusions on the influence of culture and personalities on leadership education and development or department culture changes cannot be drawn as this course evaluation was not structured to measure these outcomes. Kirkpatrick’s framework, while well-utilised in the Western literature, does not account for these outcomes. Future similar course development and evaluation could benefit from a revised framework to better assess how these differences impact learning, subsequent translation of that learning into practice, and department culture. A revised framework is proposed for those conducting CE interventions in other countries that acknowledges the influence culture will likely have on these outcomes (Fig. 4).

Fig. 4

Revised framework for training evaluation.

Using this framework, future CE interventions should establish pre-course cultural norms and personality types. These results can guide evaluation methods, specifically long-term follow up to assess if and how this type of CE intervention can result in department culture shifts and individual personality changes.

It is difficult to generalise our results and the success of this course beyond this setting due to the limited number of participants and lack of a comparison group. Additionally, the influence of social desirability bias on participants’ responses cannot be excluded given the personal relationship between the participants and study investigators. In future evaluation processes, it would be preferred to obtain evaluation data via a researcher not formally associated with the program or in electronic form. The personality type self-assessments were utilised in an informal manner for teambuilding and course facilitation. However, this would have been a useful data point to correlate to outcomes. Future courses and evaluation processes could benefit from including and analysing this. Nonetheless, as it appears to be the first description and evaluation of a leadership and teacher training course for emergency care NPCs in sub-Saharan Africa, this paper lays important groundwork for future course development, implementation, and evaluation.

This course in leadership and teacher training was a successful CE intervention in this setting as measured by Kirkpatrick’s framework. Further research is needed to assess whether or not similar results can be achieved in other settings where emergency care is developing. If this work can be reproduced in other settings, incorporating this content into new emergency care program curricula can greatly benefit both the teachers and students, who will go on to provide emergency care to their communities. In continuing to build emergency care as a new specialty in this setting, future courses and research should build on our findings and evaluate how personality and culture influence teacher and leadership development from knowledge acquisition to behaviour changes in practice and department culture.