Targeted Needs Assessment of Off-service Residents in Emergency Medicine.

PURPOSE: The purpose of this study is to evaluate the needs of internal medicine residents rotating through the emergency department (ED).
METHODS: A survey was distributed to 100 internal medicine residents (post-graduate years 2 and 3) from two different residency programs before the start of their emergency medicine (EM) rotation. Residents ranked the level of importance and the level of preparedness for 23 different EM topics, using a Likert-type scale ranging from 1 (least important/least prepared) to 4 (most important/most prepared). We calculated delta values (Δ) from the difference between importance and preparedness and undertook significance testing of this difference.
RESULTS: A total of 71 out of 100 surveys were completed properly and returned. Internal medicine residents felt most ill-prepared in the areas of orthopedics, environmental emergencies, otolaryngology, airway management, and ophthalmology. The largest perceived gaps between importance and preparedness lay within the areas of airway management (Δ=1.30), ophthalmology (Δ=1.10), environmental emergencies (Δ=0.96), and orthopedics (Δ=0.96).
CONCLUSION: Our data suggest that internal medicine residents are inadequately prepared for EM topics that they feel are important to their education, specifically airway management, ophthalmology, environmental emergencies and orthopedics. It is quite possible that other specialty residents are also poorly prepared for similar core EM topics. These data will hopefully guide future curricular change for off-service residents in the ED.

INTRODUCTION

Emergency departments (ED) throughout the country are experiencing ever-increasing patient volumes, longer wait times, strained personnel and resources, and increased pressure to reduce medical error. Teaching programs face the additional challenge of maintaining a productive educational environment for emergency medicine (EM) residents and medical students. Despite a strong desire to learn critical thinking in the ED, the educational objectives and needs of off-service residents are often overlooked.1

Standards in curriculum do exist for EM residents but not for off-service residents, who typically spend only one month of their residency in the ED.1–2,3–4 This is in direct contrast to developed goals and objectives for EM residents while on off-service rotations.5–7 Some off-service rotators attend weekly educational conferences, while others only experience bedside teaching.8 The goal of the EM rotation is to provide off-service residents an appreciation for the limited time frame in which to complete a work-up, evaluate for emergent conditions, develop a treatment plan, communicate effectively, and make an appropriate disposition.9–10 Coming from various types of residency programs (obstetrics and gynecology [OBGyn], internal medicine [IM], pediatrics, etc.), the learning needs for each off-service resident are unique. Identification of those needs will not only provide for a more complete educational experience, but also change physician behavior and potentially improve patient outcomes.11 To tailor the curriculum to reflect this new educational directive, we undertook a needs assessment.

We used David Kern’s12 six-step model for curriculum development as a guide and framework for this project. Step one of the six steps is to identify the problem and perform a general needs assessment. In this sense, we are beginning to identify the gap between the current approach and the ideal approach, specifically with IM residents making up the largest group of off-service rotators in the ED. These individuals do not have a standardized curriculum to focus education about EM concepts and principles. Step two in Kern’s six-step model is to develop a targeted needs assessment to ensure the most relevant information is obtained from the true stakeholders, in this case IM rotators.12 We performed a survey analysis to gather this information. Our goal was to identify areas of knowledge gaps between the learning needs of the rotating IM residents and what is provided by the current curriculum.

METHODS

From January to March 2007, surveys were distributed to 100 IM post graduate years (PGY)-2 and PGY-3 IM residents (a convenience sample from two IM residency programs) who had not previously rotated in the ED. The survey evaluated 23 EM core topics. These topics were identified in the following fashion: a group of associate or full professor ED physicians, experts in their respective EM subspecialty, created an expansive qualitative list of topics. This list was then validated in a review by a random sample of associate/full professor EM physicians in addition to IM program directors. After the review, appropriate corrections were made; the resultant product became the final topic list to be used in the study.

Surveys were distributed to PGY-2 and PGY-3 residents from two academic urban IM residency programs before the start of their EM rotations. These surveys were completely voluntary, anonymous, and had no bearing on evaluations of the residents during their rotations. To obtain a higher response rate, multiple requests were attempted via email and at noon conferences.

IM residents were asked to rank the importance they placed in and self-preparedness for 23 EM core topics. The responses were based on a modified Likert-type scale: 1 (unimportant/poorly prepared) to 4 (extremely important/well prepared).

Only fully completed returned surveys were used as eligible data. We used a two-tailed, paired student t-test to compare means of importance versus preparedness to identify the largest gap, using an online stats calculator and Microsoft Excel 2008 for Macintosh.8

RESULTS

A total of 100 IM residents were given the survey, and 83 returned it for a response rate of 83%. Twelve surveys were discarded as the residents failed to complete the entire survey. Therefore, 71 surveys were deemed viable for use in this study.

The results of the study are summarized in Table 1. Demographic information of the survey respondents was not collected, as all were either PGY-2 or PGY-3 IM residents from academic programs. IM residents felt least prepared in the areas of airway management, environmental emergencies, orthopedics, ENT/dental emergencies and ophthalmology. Of the five topics, the largest gap between importance and preparedness (i.e., those most important topics that the residents felt least prepared for before their rotation) lay within airway management, orthopedics, environmental emergencies and ophthalmology.

Table 1

Questionnaire snalysis: importance vs. preparedness (n=71)

Topics	Importance		Preparedness		Δ Mean	p-values (95% CI)
	Mean	SD	Mean	SD	MI-MP
Abscesses	2.73	0.72	2.54	0.77	0.20	0.08 (0.02–0.42)
ACLS/shock/codes	3.97	0.17	3.23	0.74	0.74	<0.0001 (0.55–0.93)
Airway management †Ω	3.41	0.73	2.11	0.83	1.30	<0.0001 (1.09–1.50)
Allergy	3.53	0.56	2.90	0.70	0.63	<0.0001 (0.42–0.84)
Cardiology	3.99	0.12	3.40	0.71	0.59	<0.0001 (0.42–0.76)
Dermatology	3.10	0.71	2.26	0.70	0.84	<0.0001 (0.63–1.06)
Endocrine	3.86	0.39	3.41	0.65	0.44	<0.0001 (0.28–0.61)
Eare, nose, throat/dental Ω	2.61	0.79	1.82	0.66	0.80	<0.0001 (0.58–1.02)
ER procedures	3.63	0.62	2.94	0.71	0.69	<0.0001 (0.45–0.92)
Environmental †Ω	2.79	0.63	1.82	0.70	0.96	<0.0001 (0.76–1.15)
Gastrointestinal	3.96	0.20	3.66	0.48	0.29	<0.0001 (0.16–0.41)
Genitourinary emergencies	3.09	0.58	2.55	0.75	0.54	<0.0001 (0.34–0.75)
Intoxication	3.31	0.65	3.00	0.64	0.31	.0014 (0.13–0.50)
Neurology	3.66	0.59	2.74	0.65	0.91	<0.0001 (0.72–1.11)
Obstetrics/gynecology	2.64	0.90	2.14	0.80	0.50	0.0003 (0.24–0.76)
Ophthalmology †Ω	3.06	0.74	1.94	0.59	1.11	<0.0001 (0.92–1.31)
Orthopedics †Ω	2.71	0.82	1.76	0.71	0.96	<0.0001 (0.73–1.18)
Pain management	3.23	0.76	3.06	0.81	0.17	0.1529 (0.07–0.41)
Pulmonary	3.96	0.20	3.56	0.53	0.39	<0.0001 (0.26–0.52)
Psychiatry	2.83	0.78	2.69	0.81	0.14	0.2413 (0.10–0.38)
Surgical abdomen	3.53	0.65	3.13	0.67	0.40	0.0002 (0.20–0.60)
Toxicology	3.20	0.60	2.32	0.60	0.87	<0.0001 (0.67–1.07)
Vascular	3.30	0.69	2.57	0.71	0.73	<0.0001 (0.50–0.95)

top four largest gap between importance and preparedness

top five topics residents felt least prepared to manage

ACLS, advanced cardiac life support; M, mean of importance; M, mean of preparedness; SD, standard deviation

All topics except for abscesses, pain management and psychiatry had a statistically significant difference between importance and preparedness. However, upon further statistical analysis, only airway management and opthalmology maintained a >1 Likert scale difference between importance and preparedness based on the 95% confidence interval (Table 1).

DISCUSSION

In April of 2007 the University of California at San Francisco identified five phases of curricular change. The first two phases of change included recognizing a need for change and creating a vision for a new curriculum.14 Moving forward, a focused needs assessment is a very effective method of obtaining information for such change. Although EM programs have a specific curriculum for their residents, the challenge is to tailor education for off-service residents. While certain learning objectives are bound to overlap with current protocols, off-service residents can be unfamiliar with this material. A needs assessment successfully accomplishes the following: 1) it ascertains whether a revised curriculum is necessary; and 2) it supports collaboration between departments and residency programs in a multi-disciplinary approach, benefiting residents and strengthening interdepartmental communication.15

Our needs assessment demonstrates multiple areas of deficiency in the EM curriculum for IM residents. In this study, nearly every core topic received a mean score of “somewhat important” to “moderately important.” Airway management, orthopedics, environmental emergencies and ophthalmology had the highest significant difference between their means, indicating the largest gaps in IM knowledge base. Furthermore, airway management and opthalmology had a >1 Likert scale difference between importance and preparedness, suggesting a correlation with clinical relevance. This cut off was determined by a team of experts made up of EM and IM attending physicians through informal consensus at one of many sessions during the course of the project. The identified gaps should drive the development of goals and objectives and further education of an off-service curriculum.

LIMITATIONS

There were some limitations to this study, most notably being the population of residents used in this study. We acquired data from IM residents and not from residents of other specialties. However, it should be noted that IM residents were chosen as the initial study group as they comprised the largest population of off-service residents rotating through the ED. Also, we did not conduct a power analysis; instead, the number of residents surveyed was based on a convenience sample of the two IM residency programs. In addition, the background of the individual residents involved may differ. Although responses tended to be similar, some residents may have more experience in the ED (as a medical student or during their intern year) than others. Similarly, during their rotation, some residents may not have studied as rigorously as others. While there is access to all major EM texts in the ED, resident use may have varied.

Moreover, there are minor differences between the two IM training programs. Furthermore, residents may not be able to appreciate the value of the learning experiences they have had thus far until they actually practice in the future and have the opportunity to test their knowledge. Finally, the educational goals of each IM resident can vary and affect their rankings of topics. The priorities of a PGY-3 entering endocrinology may vastly differ than those entering Pulmonology/Critical Care, leading to respondent bias.

This need assessment indicates two main areas of future research. It is clear that information obtained can be useful in curriculum development. In addition, such studies can easily be replicated on residents of other residency programs (i.e, surgery, psychiatry, OB/Gyn) to identify the needs of a variety of off-service rotators. We must recognize that this needs assessment is only a first step towards future curriculum change. More data from residents in other types of residency programs are necessary to gain a complete understanding of off-service rotators’ needs.

CONCLUSION

A needs assessment is a vital organizational tool in developing and targeting curriculum change for individual learners. This study ascertains statistically significant needs in almost all of the areas included in the survey, with particular significance in the fields of airway management, ophthalmology, environmental emergencies, and orthopedics, which illustrate a gap between importance of the topic and level of preparedness. Furthermore, this needs assessment can act as a guide and a first step towards progressive curricular change for off-service residents in the ED.