The Impact of a Communication Skills Workshop on Doctors' Behavior Over Time.

PURPOSE: Communication skills education is still relatively new in some non-Western countries. Further, most evaluation research on communication skills education examines only short-term results. In our communication skills program in Qatar, we aimed to: 1) assess the impact of the communication skills course on participant skills application; 2) assess the length of time since course completion associated with participant skills application; and 3) assess participant gender or clinical position associated with participant skills application.
METHODS: Seven hundred and thirty-eight physicians completed a seven-module communication skills course. Participants reflected on what they learned in the course and how the course had impacted their behavior through a nine-item online survey that included a four-item Communication Workshop Impact Scale (CWIS), three open questions, and two demographic questions. To assess the effect of time since workshop on outcomes, we stratified the respondents into five groups based on how long ago they had completed the course.
RESULTS: Three hundred and thirty-two physicians completed the survey. Participants reported agreement with the items on the CWIS: X=4.45 (range 1-5; SD=0.70). When asked which skill(s) they had been able to implement in their clinical practice, 235 gave a specific response, either a specific communication skill (eg, ask open questions), a higher-order category of skills (eg, questioning skills), or the name of one of the seven modules of the course. Only 28 participants listed the name of a skill or module name that they had not been able to implement. There was no evidence of difference in CWIS score based on time since course completion. There was no gender difference; however, residents had significantly lower CWIS scores than fellows (4.70 vs. 4.29, p<0.05).
CONCLUSION: Participants reported agreement with response items about the impact of the course on their skills application. Participant gender did not play a significant role, but residents had lower scores than did fellows. Furthermore, most physicians (92%) were able to name something specific that they had learned from the course and were currently implementing in their practice. Positive outcomes of the course did not seem to diminish over time. Future research should identify whether observable communication behavior matches the self-reported behavior.

Introduction

Patient-centered communication in healthcare is important to improving patient outcomes, ensuring patient safety, and increasing the quality of healthcare internationally.1–3 However, substantial research shows that communication in healthcare settings is often suboptimal (eg,4). Patients’ misunderstandings, non-adherence, and dissatisfaction, as well as medical errors and malpractice suits, are often linked to poor communication.5–7

Communication skills education is a well-accepted method for improving clinical communication and patient outcomes in medical education programs throughout the world,8 at undergraduate and graduate levels of education (eg,9,10), as well as with practicing physicians (eg,11). Experts in communication skills education agree that experiential methods (ie, role play with simulated patients) are critical to effective communication skills education.12,13

A widely used model for evaluating the success of training programs that proposes four steps to test for efficacy of training is the Kirkpatrick Assessment Model.14 This model has also been applied to medical education and communication skills education programs.15,16 The first step involves eliciting a reaction from the trainees about their feelings about the training; the second step proposes that participants’ learning be tested in an objective, quantitative manner; the third step suggests that a participants’ on-the-job performance be statistically compared after the training to their behavior before the training as it relates to the objectives of the training program; and the fourth step to test for efficacy of the program is to evaluate if the overarching desired result was accomplished. By quantifying evaluation in such a way, Kirkpatrick proposes that the efficacy of a training program is monitored adequately and the future of a program depends on the ability to enact the above steps and utilize the results.

The Kirkpatrick Assessment Model’s application to communication skills education programs has been found to be a good fit.17 The model does not explicitly consider the length of time that has passed in its evaluation levels, although it is somewhat implicit, particularly at the highest level. In other words, the highest level of “result” may take some time to become apparent. However, the evaluation of communication skills education generally examines only short-term impact,18 with very little attention paid to the long-term effect of such trainings. Research studies often do not allow for longitudinal designs and it can be difficult to obtain the resources necessary to follow participants over time. Conventional wisdom among healthcare communication educators is that learned skills drop off over time, but there are few data to support this, with only a few studies that show mixed results.17,19-21

We considered whether an innovative and feasible approach to gaining some insight into the long-term effect of the course would be to conduct a cross-sectional study of participants who had completed a communication skills course at varying times (6–36 months ago) to assess the long-term effect at the third level of the Kirkpatrick model – behavior. Specifically, we wanted to examine whether time since course completion was related to reported use of communication skills learned in the course, allowing for a snapshot of the effect of communication skills education on doctors’ behavior over time.

Communication Skills Training in the Middle East

In 2008, a large teaching hospital system in the Middle East began a partnership with a major US healthcare institution to adapt a communication skills course to fit the needs of its multinational community using accepted best practices.22,23 As we taught the course to hundreds of participants each year, we asked whether we were making an impact on the healthcare system as a whole. Anecdotally, we heard from colleagues and leaders across the system that the program was making a difference, but we had no data to show this.

Given the large number of physicians in varied specialties across eight hospitals, evaluation of behavior in clinics was not feasible. Our next best option was to ask the participants about how their behavior had changed as a consequence of the course. Specifically, we were interested in how the course impacted participants’ knowledge, attitudes and behavior, and if that differed depending on the length of time that had passed since participants completed the course. Thus, the aims of the present study were as follows:

Aim 1: To assess the impact of the communication skills course on participant skills application.

Aim 2: To assess whether the length of time since course completion is associated with participant skills application. Further, we hypothesized that there would be a negative association between length of time since course completion and ratings on the impact scale.

Aim 3: To assess whether participant gender or clinical position is associated with participant skills application.

Methods

Participants and Procedures

Between April 2014 and October 2016, 738 physicians completed a seven-module communication skills course over two days. These physicians were either residents or fellows (still in training), specialists (independent practicing physicians without teaching responsibility), or consultants (independent practicing physicians with teaching responsibility). For all groups except consultants, this was a required course before they could be promoted to the next level.

The program was made up of seven modules, focused on challenges in communicating with patients, including: 1) Breaking Bad News; 2) Shared Decision Making; 3) Responding to Patient Anger; 4) Working with Untrained Interpreters; 5) Discussing Prognosis; 6) Discussing DNAR orders; and 7) Conducting a Family Meeting. The content of the models was based on the Comskil Curriculum developed at our collaborating US institution, Memorial Sloan Kettering Cancer Center.24 Four modules (numbers 1, 2, 5, and 6) consisted of a short large-group lecture, video demonstration, and discussion (approximately 30 min), followed by a 60–75 min facilitator-led small-group role-play session with a standardized patient, in which participants practiced the strategies and skills relevant to the module. The three remaining modules (numbers 3, 4, and 7) consisted of a large-group lecture, followed by a large-group role-play discussion. Facilitators were physician volunteers who had completed the program previously and then underwent further training in facilitation of small-group work, based on the Comskil facilitation training model.25

This course was required for all residents and fellows (graduate trainees) as well as those in specialist positions applying for promotion to a consultant position. Although the program was initially developed in the West, we went through a process of culturally tailoring the program for it to be used in the Middle Eastern/Arabic setting.22,23 This included changing the names and contexts of role-play scenarios, making new videos, and, in some cases, changing the major focus of the module. For example, the US-based program has the module on Family Meetings focus on working with relatively functional families, usually at the end-of-life. In our culture, requests non-disclosure of cancer and other illnesses is very prevalent (eg, “Do not tell my mother she has cancer”), and so we focused the family meeting on working through requests for non-disclosure as well as working with families in conflicts.

In April 2017, we asked the 738 participants to reflect on what they learned in the course and how the course had impacted their behavior. Invitations to complete the online survey (Qualtrics) were sent out at the beginning of April 2017 and were available for completion for one month, with three emailed reminders. To assess the effect of time since workshop on skills application, we stratified the doctors into five groups based on how long ago they had completed the course (Table 1). The hospital’s Institutional Review Board approved the study.

Table 1

CWIS Score by Length of Time Since Course Taken

Cohort	n	Percentage	CWIS Score (1–5), mean (SD)
Cohort 1: 6–12 months	45	13.6	4.45 (0.60)
Cohort 2: 12–18 months	80	24.1	4.49 (0.63)
Cohort 3: 18–24 months	104	31.3	4.41 (0.78)
Cohort 4: 24–30 months	67	20.2	4.42 (0.82)
Cohort 5: 30–36 months	36	10.8	4.51 (0.54)
Total	332	100	4.45 (0.70)

Study Design and Setting

The study design was a cross-sectional survey, with a key independent variable being the length of time since the individuals completed the course (as a proxy for longitudinal data). Please see for the full survey. The study was conducted at a large teaching hospital which is in the not-for-profit public healthcare system in a Middle Eastern country, consisting of eight public hospitals and other healthcare services, with 19 residency training programs, 14 of which are accredited by the US Accreditation Council for Graduate Medical Education. The public healthcare system has also been accredited by the Joint Commission International. These accreditations have led to increased accountability for quality of care, education, and patient safety.

Measures

We were unable to find a published and validated scale to assess participants’ skills application six months post workshop. Consequently, we developed a four-item scale for this study, called the Communication Workshop Impact Scale (CWIS). The CWIS was made up of four items that we developed based on our learning objectives for the course. One of the items was reverse coded. Participants were asked to rate their agreement with the items on a scale between 1=Strongly disagree and 5=Strongly Agree. One item was removed as it was not reliable.

The remaining three questions (Table 2) were found to be a reliable measure of the impact of the communication skills course (Cronbach’s alpha of 0.876). In addition to the scale, the survey included three open questions and two demographic questions. The open questions asked the following:

Table 2

Communication Workshop Impact Scale

Item
1. I learned new skills from the communication skills course
2. I have been able to use what I learned in the communication skills course in my daily work with patients and families.
3. I never think about what I learned in the communication skills course.

Which skill or skills that you learned from the course have you been able to implement into your clinical practice most successfully?

Which skill or skills that you learned from the course have you NOT been able to implement into your clinical practice?

What have been the barriers to implementing these skills?

Analyses

Quantitative data were analyzed using SPSS v. 24. Tests used included Cronbach’s alpha reliability test and one-way ANOVA. The three open questions asked participants about their experiences implementing the skills. Open-ended data were content analyzed.

Results

In total, 332 physicians completed the evaluation survey. Table 1 shows the distribution of the respondents in terms of length of time since they took the course. The majority (70%) of respondents were male (Table 3). Nearly half of respondents were graduate medical trainees (either residents or fellows) when they took the course, about one-third were specialists, and about 20% were consultants (Table 3).

Table 3

Participant Demographics

	Count	Percentage
Gender
Male	201	70.3
Female	85	29.7
Total	286	100
Role When Course Taken	Count	Valid Percentage
Resident	103	36.0
Fellow	32	11.2
Specialist	91	31.8
Consultant	55	19.2
Other	5	1.7
Total	286	100

In response to RQ1, participants reported high scores on the CWIS: X=4.45 (SD=0.70). When asked which skill(s) they had been able to implement in their clinical practice, 232 respondents gave a total of 235 specific responses (Table 4). Some mentioned a specific communication skill (eg, ask open questions), a higher-order category of skills (eg, questioning skills), or the name of one of the seven modules of the course. When asked which skill or skills they had not been able to successfully implement, only 28 participants listed the name of a skill or module name (Table 4). When asked what barriers they encountered in trying to implement the skills, 109 barriers were listed by participants (Table 5). In response to RQ2, the mean CWIS scores per cohort are shown in Table 1. One-way ANOVA found no evidence of difference in CWIS score based on time since the course was taken. In response to RQ3, there was no gender difference in the CWIS score. However, the participant’s role at the time of the course was significant (F=3.411, p<0.05), such that residents had significantly lower CWIS scores than fellows (4.70 v. 4.29, p<0.05).

Table 4

Successful and Non-Successful Implementation of Skills

	Successfully Implemented	Not Implemented
Skills
Check patient understanding	18	0
Declare agenda	9	0
Ask open questions	7	0
Invite agenda	6	0
Check patient preference for information	6	0
Shift agenda	4	0
Invite questions	4	0
Summarize	4	0
Subtotal	49	0
Skill Categories
Shared decision making	35	0
Empathic communication	26	0
Information giving	20	0
Questioning	10	0
Checking	8	0
Agenda setting	6	7
Subtotal	105	7
Module names
Breaking Bad News	44	11
Responding to Patient Anger	15	4
Family Meeting/working with families	9	0
End of Life/DNR	0	6
Subtotal	68	21
Miscellaneous answers
Communicate effectively	4	–
None unsuccessful	–	117
Not applicable	–	5
No comment	–	5
Subtotal	4	127
Total	235	155

Table 5

Reported Barriers to Skills Implementation

Response to Open-Ended Question	Frequency
“No barriers”	74
Language	33
Time management	28
Not applicable	23
Privacy	9
Cultural and religious	7
Challenging patients	5
Need more practice	4
Total responses	183

Discussion

Significant efforts have been made to improve clinicians’ communication skills through education courses throughout the world. Although evaluations of reaction and learning, which are usually assessed soon following the training, are usually quite positive, many have questioned the enduring impact of such trainings,26–28 primarily asking if the skills that clinicians gain are retained over time. Answering that question is difficult from a methods perspective as longitudinal data collection is most often outside the scope of discrete research projects about communication skills training. The educational context described in this study is more amenable to addressing such a question as we developed a large communication education program that has been running for several years. Our unique cross-sectional design allowed us a longitudinal lens, as we studied participants who completed the course up to three years ago.

Participants in our study agreed with the items measuring the impact of the course on their practice. Further, our mixed-methods study design allowed us to gather more robust data about experience. We complemented the rating items measurement with open-ended questions that asked the participants to name specifically what they were doing in their practice that they had learned in their course, what they had not been able to implement, and what barriers they encountered. Participants’ ability to answer open-ended questions about what the effect of the course on their practice provided validation for the ratings near the top of the scale (means between 4 and 5).

Contrary to our hypothesis, there was no difference in participants’ CWIS scores based on when the course was taken. This finding suggests that the impact of the course was sustained over the long term. Future research could examine if there is an association between the use of the CWIS and observable doctors' behaviors. Positive findings would give added validation to the measure and more confidence in using the CWIS as an outcome measure for doctors' behaviors.

The only subgroup differences present in our study were that fellows had a significantly higher impact scores than did residents. One potential explanation for this finding is that in their role as trainees, residents may be more restricted in terms of clinical responsibilities and have had less opportunities to practice the skills and strategies taught and make them habits. Future qualitative research could explore how these two different groups of trainees experienced the training and its application afterward differently.

Limitations of this study include that it is based on self-assessments, which is subject to a social desirability bias. Certainly, observing the doctor’s behavior would be more beneficial, but it was not feasible in this case. A further limitation was that the response rate was low, particularly for those who had completed the course the least recently. The lower response rate for this group was likely an effect of turnover at the institution. A limitation of the training is that there were no consolidation or booster sessions.

Conclusion

Participants reported agreement with response items about the impact of the course on their skills application. Furthermore, most physicians (92%) were able to name something specific that they had learned from the course and were currently implementing in their practice. Participant gender did not affect ratings, but participant position did; residents had significantly lower scores of skills application than did fellows. Positive outcomes of the course did not seem to diminish over time. Future research should identify whether observable communication behavior matches the self-reported behavior.