In-process modification yields improved teaching outcomes for international emergency medicine.

AIMS: To determine if on-going modification of an international teaching course in emergency medicine will improve audience perception and receptivity.
METHODS: Over 4 years, we conducted surveys during emergency medicine symposiums given in Ghana, West Africa. In 2003, 2004 and 2006, symposium interventions of content modification (lecture topics/workshops), audience modification (smaller groups, designing workshops for different levels of providers) and in-process modification (modification of lecture technique in two sequential 2006 symposiums) were done. Survey responses were analyzed to determine what interventions resulted in greater participant satisfaction. When comparing the three groups, a one-way ANOVA test was conducted to determine differences among the 3 years. When comparing two groups, Levene's test for equality of variances was conducted for annual variance, and based on that result, an independent t-test was calculated.
RESULTS: The one-way ANOVA test indicated that there was a statistically significant difference among the three groups (2003, 2004, 2006) of symposium participants in four questions related to the participants' perception of the instructors. The independent t-test comparing the aggregate data of 2003 and 2004 with 2006 (audience modification and in-process modification between two sequential symposiums) showed a statistically significant improvement in the participants' receptivity. The data indicated that modifying the content of international symposiums and modifying the audience to which it is addressed are equally valuable tools to ensure a positive participant receptivity. Moreover, the data indicated that "in-process" modification conducted between the two sequential 2006 symposiums produced a statistically significant increase in positive perception, facilitating knowledge transfer.
CONCLUSION: Statistical analysis of surveys coupled with modification and intervention can improve participant satisfaction in international teaching programs.

Introduction

Background

The establishment of international medical teaching programs and academic affiliations has become increasingly popular [1-3]. While several processes, models and frameworks exist to investigate differences in belief and practice in a medical community, there has not been a process developed that examines how efficiently international programs are run to maximize the transfer of knowledge, technology and practices among international medical communities [4]. This process can be further complicated when two medical communities are not familiar with each other’s principle medical needs or cultural beliefs.

International symposiums are a well-established method to bring medical communities together to discuss medical issues [5]. These programs are often short, lasting a few days, but are seen as a way to develop longitudinal productive relationships among medical communities. Since 2000 the University of Southern California’s Division of International Medicine in the Department of Emergency Medicine has been conducting symposiums at Korle-Bu Teaching Hospital in conjunction with the College of Health Sciences, University of Ghana [6]. These symposiums offered health-care professionals a forum to discuss medical issues, establish valuable relationships and exchange medical information.

Emergency medicine in 2000 in Ghana was in its infancy and underdeveloped (Arnold scale). Ghana had a well-developed educational system, accredited universities, two medical schools and a peaceful developing economy. On an exploratory visit by one of the authors, contact with the medical professionals of Ghana was established, and the Department of Emergency Medicine of the University of Southern California was invited to begin a collegial relationship to develop this specialty. There was a special interest in development of trauma care and a pre-hospital care response system because of the rising morbidity and mortality statistics of motor vehicle accidents.

Importance

With an increasing number of international programs and 17 international emergency medicine fellowships, it is important to ensure that programs produce the best possible outcomes. The initial exchanges of information that take place among medical communities are imperative to establishing respect and cooperation in medical projects. Misconceptions or miscommunication can result in problems going unidentified, leading to poor outcomes. More importantly, when projects fail they can leave medical communities jaded and unwilling to participate in further programs or follow medical practices that might have otherwise been perceived as useful.

Interventions

In 2002 an attempt to export pre-assembled American educational programs with slight cultural modifications was the basis for didactic instruction in Ghana. The survey conducted at this time was very short, mostly soliciting comments and requests from the audience. Feedback indicated that the participants wanted the inclusion of more West African literature, more local lecturers from their college and an increase in small interactive hands-on workshops, more films and videos as well as more lectures in trauma and toxicology. The medical providers of Ghana were especially concerned with the delivery of trauma resuscitation and care. Statistics gathered in Accra showed an increasing number of motor vehicle accidents with the associated increased morbidity and mortality. Toxicology seemed to be a universal interest, and the Ghanaian medical providers were particularly interested in comparisons between the care in Ghana and in the US.

The 2003 symposium included local lecturers, topics that were suggested by 2002 symposium attendees and small interactive workshops. Since the symposium was a collaborative academic relationship, the audience wanted to hear from local lecturers. There may have been a feeling that local lecturers might also address local difficulties and concerns. After collecting data in 2003, a large number of non-physician health-care professionals were identified as participants in the symposium.

In 2004 the symposium provided non-physician health-care professionals with workshops that addressed their particular scope of practice. Specific nursing emergency workshops were included as well as the video/film presentations requested in 2002.

The leader of the 2004 symposium spent time after the symposium attending the emergency areas and saw that much of the training was not translating into the delivery of emergency care. This prompted a site survey to be conducted after the 2004 symposium. The site survey conducted in the Korle-Bu Teaching Hospital by the team leaders indicated that many health providers in the teaching hospital were not attending these symposiums because of the demands of their jobs and the fact that they could not afford to take time off work. Hence, many members of the staff of the emergency areas were not benefiting from the training. Meetings were then scheduled to bring the hospital administration into this initiative as a major stakeholder.

In 2005 there were no symposiums conducted because of the increased work load and lack of finances of the project leader.

In 2006 our process modification included training smaller groups composed of teams of on-line providers from the emergency areas of the Korle-Bu Teaching hospital. The College of Health Sciences of the University of Ghana talked with the Korle-Bu Hospital administration and convinced them that this would result in better delivery of emergency care by the hospital and increased satisfaction of the emergency care providers of the hospital. These participants were released for the first time from their hospital duties without loss of pay to attend the training. This represented a major stakeholder investment by the hospital and was extremely successful. In addition, as we had decided to give two similar back-to-back workshops, we were also able to introduce “in-process dynamic modification,” which allowed dynamic feedback so that presenters could adjust their presentation for maximum participant satisfaction. Feedback was analyzed and incorporated in evening debriefings on things such as articulation, speed of lecture delivery, and the use of abbreviations. Between the two workshops, participant evaluation was incorporated into the symposium (such as the lecturers speaking more slowly, abolishing the use of acronyms and abbreviations, and modifying lecture content). Ancillary staff workshops were continued, such as a repeat emergency nursing workshop and a workshop conducted by a medical social worker to discuss the social impacts of trauma and death notification on emergency staff.

Goals of this investigation

To improve the transfer of knowledge, we hypothesized that obtaining quantifiable audience feedback would allow us to alter future course content, methods and participation to increase the transfer of knowledge to Ghanaian health-care professionals. It is expected that the content of our presentations and the proper identification of participants were the most important factors that led to a successful symposium.

Methods

Study design

We conducted a post-symposium cohort study on all participants to evaluate how three kinds of modification affected participant perception and satisfaction for an international symposium. The three kinds of modification we evaluated were content modification (topics and workshops selected based on feedback from the audience), audience modification (inclusion of other health providers as a team) and “in-process” modification (modification mostly in delivery technique and response to audience interaction using the feedback from the first symposium to the next). Our study was approved by the University of Southern California Health Sciences Campus IRB after careful review.

Setting

Our symposium was conducted at Korle-Bu Teaching Hospital in Accra, Ghana. Korle-Bu Teaching Hospital is the main hospital in Accra and serves the approximately 1.6 million people of the capital city; it is affiliated with the University of Ghana College of Health Sciences. Attendees of the symposium were doctors, nurses, physician assistants, pharmacists and government administrators that worked at Korle-Bu Teaching Hospital, in the surrounding city and other West African countries. Each symposium lasted for 3 days. They began at 8:00 a.m. and ended at 5:00 p.m. each day with a 1-h lunch break. Attendees were required to have their own housing during the symposium. The 2003 and 2004 symposiums had had the same format – large group didactic in the morning followed by smaller workshops in the afternoon. A small fee was charged by the College in Ghana to cover the cost of the rooms and the food only.

Lecturers consisted of faculty and professionals from the University of Southern California’s Department of Emergency Medicine and some local faculty from the College of Health Sciences, University of Ghana. Lecturers participating flew over from the University of Southern California to Ghana 1 or 2 days before the symposium with their lecture material already completed and ready for distribution.

Selection of the participants

Any health-care professional that heard about the symposium was able to attend. Doctors and nurses were expected to be the primary attendees at the symposiums. All registered participants were asked to fill out a survey at the end of the symposium.

Methods of measurement

All participants were given a survey to fill out that was divided into two parts. The first part contained demographic information asking participants to state their profession, years in practice, employer and whether or not they had attended the symposium previously. The second part of the survey asked questions relating to the symposium itself. The first set of symposium questions asked participants to rate statements on agreement based on a scale from 0 to 5, with 5 being the highest level of agreement. These questions were: (1) Was the academic content valuable? (2) Was the class instruction excellent? (3) Were the instructors accessible? (4) Did the instructors treat students fairly? (5) Did instructors care about you as an individual? (6) Would you attend this program again? (7) Did your experience meet your expectations? (8) Could the instructor adequately answer your questions? The second set of symposium questions was open-ended questions asking for suggestions to improve the symposium and asking whether participants would be willing to collaborate on future projects.

Data collection and processing

Responses to the surveys were gathered and reviewed by two individuals. All information on the surveys was transferred to a Microsoft Excel (Microsoft Inc., Redmond, WA) spreadsheet. Each individual took turns entering the data into spreadsheets with a second person reviewing the data for accuracy. After all data were entered into basic spreadsheets, they were transferred to SPSS 11.0 for Windows (SPSS Inc., Chicago, IL) for all statistical analyses. The statistical analyses were conducted by a health-care professional with a Master’s Degree in Statistics and were overseen by two study physicians.

Following each individual symposium, a report was made available to key stakeholders. In 2003, only descriptive data and mean responses were reported to stakeholders. In 2004, data from 2003 and 2004 were collected and analyzed. In 2007, all data for 2003, 2004 and 2006 were gathered and entered into the computers for comparative analysis. All participants who had attended the symposium previously were excluded from the analysis. When comparing the three groups, a one-way ANOVA test was conducted to determine differences between the groups. When comparing two groups, Levene’s test for equality of variances was conducted to determine if the responses’ variances differed throughout the years. Based on this result, independent t-test for continuous variables was calculated assuming either equal variances or unequal variance.

Results

Characteristics of study participants

Data were collected from 275 responses over the three symposiums. In 2003 there were 97 attendees, of which 87.4% said that this was their first symposium. All registered attendees submitted responses, although there were other unregistered attendees, for instance, medical students, who could come and audit for free. Participants included 56 physicians, 27 nurses, 4 pharmacists, 5 physician assistants and 18 participants that identified themselves as other. The average number of years working in health care was 12.1. In 2004 there were 96 attendees, of which 90.6% said that this was their first symposium. Participants included 30 physicians, 58 nurses, 2 pharmacists and 5 people who identified themselves as other. The average number of years working in health care was 12.7. In 2006 we split the symposium into two distinct groups. Group 1 had 10 doctors, 26 nurses and an average of 15.7 years of experience in health care, with 83.3% of participants reporting that this was their first symposium. Group 2 had 12 doctors, 34 nurses, a medical student and an average of 12.68 years of experience working in health care, with 80.9% of participants reporting that this was their first symposium. After stratifying the data for first-time attendees, we were able to include 83 participants from 2003, 87 participants from 2004 and 68 participants from 2006 for a total of 275 in our analyses.

Average responses

The mean scores that were recorded on the surveys indicated that participants believed the program was overall excellent (Table 1). In 2003 and 2004 we continued to score between 3.9 and 4.6. In both 2003 and 2004 the highest reported mean value was participants’ agreement that they would attend the symposium again and the lowest reported value was due to participants not believing that we cared about them as individuals. Our statistical analysis indicated that there was no statistically significant difference in reception by attendees when comparing content modification in 2003 with audience modification in 2004.

Table 1

Computation of mean responses

Group number		Was the acadmic content valuable?	Was the class intruction excellent?	Were instructors accessible?	Did the instructors treat you fairly?	Did instructors care about you as an individual?	Would you attend the program again?	Did your experiences meet your expectations?	Could the instructor adequately answer questions?
2003	Mean	4.51948	4.25316	4.37037	4.32911	3.92500	4.59494	4.20253	4.41892
	N SD	77.576066	79.669258	81.557773	79.674090	80.896900	79.610149	79.627975	74.619473
2004	Mean	4.47674	4.32099	4.45783	4.37209	3.94186	4.51190	4.16279	4.38824
	N SD	86.608310	81.629030	83.668110	86.614463	86.725253	84.768256	86.683597	85.619026
2006	Mean	4.64466	4.54666	4.59563	4.76721	4.42403	4.93874	4.14222	4.64466
	N SD	68.914170	68.915492	68.981711	68.688135	681.172009	68.262176	681.295581	68.936567
Total	Mean	4.54042	4.36479	4.46769	4.47283	4.07621	4.66595	4.17026	4.47505
	N SD	231.703371	228.745425	232.744246	233.681212	234.953644	231.625832	233.888341	227.734128

ANOVA

The one-way ANOVA test indicated that there was a statistically significant difference among the three groups in 2003, 2004 and 2006 on four of the questions (Table 2). The analyses of the three groups' data indicated that there was a difference in perceiving whether the class instruction was excellent (p < 0.05), whether instructors treated people fairly (p < 0.001), whether instructors cared about attendees as individuals (p = 0.001) and whether attendees would attend the program again (p < 0.001). Despite these perceived differences, there were no statistically significant changes in perceptions as to whether or not participant’s expectations were met, the value of the academic content, whether instructors were accessible or whether instructors adequately answered questions.

Table 2

One-way ANOVA for 2003, 2004 and 2006

		Sum of squares	df	Mean square	F	Sig.
Was the acadmic content valuable?	Between groups	1.21	2	0.561	1.135	0.323
	Within groups	112.667	228	0.494
	Total	113.788	230
Was the class intruction excellent?	Between groups	3.389	2	1.695	3.106	0.047
	Within groups	122.745	225	0.546
	Total	126.134	227
Were instructors accessible?	Between groups	1.888	2	0.944	1.715	0.182
	Within groups	126.063	229	0.550
	Total	127.951	231
Did the instructors treat you fairly?	Between groups	8.397	2	4.199	9.728	0.000
	Within groups	99.263	230	0.432
	Total	107.660	232
Did the instructors care about you as an individual?	Between groups	11.608	2	5.804	6.694	0.001
	Within groups	200.291	231	0.867
	Total	211.899	233
Would you attend the program again?	Between groups	7.452	2	3.726	10.281	0.000
	Within groups	82.631	228	0.362
	Total	90.083	230
Did your experiences meet your expectations	Between groups	0.141	2	0.070	0.088	0.915
	Within groups	182.942	230	0.795
	Total	183.083	232
Could the instructor adequately answer questions?	Between groups	2.830	2	1.415	2.664	0.072
	Within groups	118.971	224	0.531
	Total	121.801	226

Independent t-test

When comparing the aggregate data of 2003 and 2004 with the data obtained in 2006, we found that there were five statistically significant improvements to participants’ receptivity (Table 3). The mean perception as to whether the class instruction was excellent increased from 4.29 to 4.55 (p < 0.05) in agreement (95% CI 0.05 to 0.47). The mean perception as to whether the instructors treated participants fairly increased from 4.35 to 4.77 (p < 0.001) in agreement (95% CI 0.22 to 0.61). The mean perception as to whether the instructors cared about you as an individual increased from 3.93 to 4.42 (p < 0.01) in agreement (95% CI 0.18 to 0.80). The mean perception as to whether the participants would attend this program again increased from 4.55 to 4.94 (p < 0.001) in agreement (95% CI 0.26 to 0.51). The mean perception as to whether lecturers could adequately answer participant’s questions increased from 4.40 to 4.65 (p < 0.05) in agreement (95% CI 0.34 to 0.45). However, there was still no statistically significant change in their perception as to whether the symposium had met their expectations, whether the academic content was valuable and whether instructors were accessible.

Table 3

Independent t-test comparing the combined mean of 2003 and 2004 vs. 2006

		Levene’s test for equality of variances		t-test for equality of means
		F	Sig.	t	df	Sig. (two-tailed)	Mean difference	Std. error difference	95% Confidence interval of the difference
									Lower	Upper
Was the acadmic content valuable?	Equal variances assumed	0.025	0.874	1.458	229	0.146	0.14773	0.101293	−0.051857	0.347315
	Equal variances not assumed			1.229	91.326	0.222	0.14773	0.120162	−0.90946	0.386405
Was the class intruction excellent?	Equal variances assumed	0.093	0.760	2.427	226	0.016	0.25916	0.106764	0.048781	0.469543
	Equal variances not assumed			2.120	96.727	0.037	0.25916	0.122271	0.016479	0.501844
Were the instructors accessible?	Equal variances assumed	0.690	0.407	1.693	230	0.092	0.18100	0.106915	−0.029659	0.391655
	Equal variances not assumed			1.410	89.662	0.162	0.18100	0.128394	0.074091	0.436088
Did the instructors treat students fairly?	Equal variances assumed	12.235	0.001	4.399	231	0.000	0.41569	0.094501	0.229497	0.601884
	Equal variances not assumed			4.273	117.524	0.000	0.41569	0.097274	0.223053	0.608328
Did the instructors care about you as an individual?	Equal variances assumed	5.156	0.024	3.665	232	0.000	0.49029	0.133782	0.226711	0.753878
	Equal variances not assumed			3.155	94.321	0.002	0.49029	0.155412	0.181733	0.798856
Would you attend the program again?	Equal variances assumed	79.473	0.000	4.450	229	0.000	0.38659	0.086866	0.215430	0.557746
	Equal variances not assumed			6.131	227.383	0.000	0.38659	0.063058	0.262335	0.510841
Did your experiences meet your expectations?	Equal variances assumed	17.911	0.000	−.309	231	0.758	−0.03960	0.128265	−0.292317	0.213122
	Equal variances not assumed			−.240	81.528	0.811	−0.03960	0.165201	−0.368263	0.289068
Could the instructor adequately answer questions?	Equal variances assumed	0.090	0.764	2.298	225	0.022	0.24215	0.105380	0.034488	0.449804
	Equal variances not assumed			1.958	92.864	0.053	0.24215	0.123682	−0.003467	0.487759

Discussion

Audience modification and content modification had little effect on mean scores in 2003 and 2004. The target audiences uniformly graded the programs offered as excellent, but there were persistently lower scores for our connection with the audience and their perception of whether we cared about them as individuals. Moreover, on-site surveys after the symposiums indicated that not all target audiences were being impacted.

In 2006, we totally re-designed the symposiums. The 2006 participants were health provider teams from the emergency areas of Korle-Bu Hospital who had been freed up from their duties by the hospital to allow them to attend the symposium without losing salary. The 2006 symposium was not open to a larger audience from all of Ghana, so the 2006 audience was a smaller group limited to the emergency care providers of the emergency areas of the Korle-Bu Teaching Hospital. The groups were much smaller, and in-process modification between the two back-to-back symposiums occurred in de-briefing sessions to give feedback to the instructors.

The 2006 survey data showed that mean perceptions increased in five major categories when the audience was smaller to allow more one-on-one focused interaction with the instructors. In addition, in-process modification of delivery and content between the back-to back symposiums resulted in improved mean scores in audience satisfaction, although because of small participant numbers in each of the 2006 groups, this did not reach statistical significance. Combining all participants of the 2006 audience in an one-way ANOVA test with the 2003 and 2004 audiences demonstrated that smaller focused groups and in-process modification demonstrated improved scores in perceiving whether the class instruction was excellent (p < 0.05), whether instructors treated people fairly (p < 0.001), whether instructors cared about attendees as individuals (p = 0.001) and whether attendees would attend the program again (p < 0.001). An independent t-test comparing the aggregate data of 2003 and 2004 with the data obtained in 2006 demonstrated improved scores in the same four areas and also an improved score as to whether lecturers could adequately answer participants’ questions.

How a message is crafted and disseminated and how it is perceived and comprehended by the target audience are extremely important in knowledge transfer and subjective satisfaction with the message. This concept has been utilized very effectively in public health and political and other media communications. Audience perception and comprehension improves with addressing audience composition, content of lectures and workshops, delivery by the lecturers and on-going dynamic modification based on feedback from all participants. Also barriers to participation in this process must be assessed and dealt with by the stakeholders to allow the information to reach the intended target audiences.

While certain statistical analysis has been done on courses conducted in other countries, it has usually examined the acquisition of practical skills and knowledge by the use of pre and post tests on short-term and long-term skills [7-10]. Statistics looked at educational parameters, pre- and post-test scores and analyzed the program effectiveness by p values. The emergency medicine literature does not describe an on-going process that would examine how international programs could be dynamically modified to maximize the transfer of knowledge and practices among international medical communities, although several of the adaptive principles have been recommended by researchers in the past [11, 12].

Limitations

The following limitations and assumptions should be considered in interpreting our findings. Our research on perceptions was based on interactions between lecturers and attendees at the University of Southern California Department of Emergency Medicine and the College of Health Sciences University of Ghana annual symposia over a 6-year period. International symposiums that are held in different countries will have cultural issues and language and other barriers particular to their setting.

Conclusion

Statistical analysis of surveys coupled with modification and intervention can improve participant satisfaction in international teaching programs. On-going in-process modification can produce a statistically significant increase in positive perception, thereby facilitating knowledge transfer among medical communities and the development of long-lasting affiliations. Further, there is a universal need to adapt and modify symposia to meet the needs of the country as identified by the health-care providers of that country. Pre-packaged courses imported from the developed world may not contain material or be relevant to the concerns or resources of the host country.

Although this paper only describes increased participant satisfaction of short symposia from 2002 to 2006, the authors believe the improvement of the symposia also translates into improvement in daily running of emergency medical services in Ghana. Ghana has made enormous advances in the development of a pre-hospital care system in the last 3 years. The increased buy-in and motivation of emergency care providers will continue to drive and maintain development in Ghana.