Written and Online Residency Guidebook to Improve Resident Efficiency and Knowledge of Best Patient Care Practices.

INTRODUCTION: Residents at most institutions change rotations every 2 to 4 weeks. It often takes significant time for residents to become acclimated to the different protocols, expectations, and environments of each unique rotation. As a result, residents often spend time searching for answers, time that could be spent in outside learning and direct patient care. The goal of this resource is to provide a novel guidebook that improves residents' efficiency and knowledge of best patient care practices.
METHODS: The guidebook begins with an introductory chapter with key contact information that can be filled in for the user's institution, which is followed by 16 rotation-specific chapters. A rotation-based approach was chosen as it focuses the content on the most pertinent information. Thus, trainees can quickly read a chapter to cover the most pertinent content for their current rotation. As a surrogate marker for efficiency, noon-conference attendance logs were queried to assess improvement in on-time attendance after introduction of the guidebook.
RESULTS: After introduction of the learning resources, on-time arrival to noon conference improved for all residents and interns. Guidebook survey results were universally favorable; however, around half of respondents stated that they used the guidebook once or less per rotation. DISCUSSION: Underutilization of these resources potentially contributed to the lack of a statistically significant improvement overall. Future directions should focus on augmenting the quality and utilization of the guidebook and then reevaluating if, once well adopted, there is a sustained benefit.

Educational Objectives

By the end of this module, the learner will be able to:

Enumerate the rationale for developing novel learning resources for physicians in training.

Recognize the defined benefits and limitations of our learning resources (i.e., guidebook, online wiki) and evaluation metrics (i.e., noon-conference attendance logs, knowledge-based assessments, survey).

Appraise whether implementing similar learning resources would be valuable for his/her training program.

Introduction

Residents at most institutions change rotations every 2 to 4 weeks. It often takes significant time for residents to become acclimated to the different protocols, expectations, and environments of each unique rotation. At the start of a new rotation, trainees are inundated with new schedules, educational objectives, and hospital pathways, as well as a myriad of external resources. The high-yield information in these various documents may not be easily accessible in a single repository. As a result, residents often spend time searching for answers, time that could be spent in outside learning and direct patient care.

A core challenge in graduate medical education is finding the balance between trainee education and providing quality patient care. Addressing this challenge has become even more important following the 2003 and 2011 duty hour regulations. Following these regulations, residents have reported decreased availability for educational activities and the perception of a worsening in quality of care.[1,2] Recent studies of internal medicine and pediatric residents highlight that 15% to 31% of a resident's day is spent in educational activities and only another 12% to 17% in direct patient care.[3-5]

Programs at several other institutions have attempted to leverage new technologies to address this challenge. For example, providing residents with tablet computers has been shown to improve trainee efficiency.[6,7] Unfortunately, these new technologies may be cost prohibitive. Web-based, collaborative programs such as podcasts, blogs, and wikis have been shown to be accessible and inexpensive options to enhance the education of medical trainees.[8] We felt that online wikis—websites with editable shared content—would be the best option as these customizable websites have been shown to be accepted and well utilized by residents.[9] Moreover, involvement of trainees in the creation of an online wiki has been shown to improve their scores on standardized knowledge-based assessments.[10] A recent systematic review of online wikis revealed that there were 18 studies of wiki use targeting medical learners.[11] One study showed that an online wiki could be scaled up to a national level, acting as a core reference nexus for all trainees in obstetrics and gynecology in Ireland.[12]

When physicians have clinical questions, they rely on resources that are familiar, available, and high yield.[13] We hypothesized that creating a handheld residency guidebook might evolve into this familiar, available, and high-yield repository of core information. While the guidebook would have nominal financial impact, its creation and maintenance would be a major investment in time.

Methods

The content and structure of the guidebook used at Children's National Health Center (Appendix A) were developed by surveys of both residents and core rotation faculty. In September 2010, all pediatric residents at Children's National Medical Center were queried regarding what content would have been most useful to them to improve efficiency and address knowledge deficits on their last three clinical rotations. Examples of recurrent efficiency items included how to place common orders, where to go for certain daily activities, and helpful phone numbers or contact persons. We provided information from our own institution in the guidebook, but it can be replaced with information from the user's institution using the template (Appendix B).

Knowledge items that the pediatric residents recommended for the guidebook were important concepts for those rotations (e.g., gastrostomy tube replacement algorithm on the gastroenterology rotation, acute medical management of status epilepticus on the neurology rotation, heart murmur descriptions on the cardiology rotation). A subsequent survey was sent to key educational faculty in each division requesting a list of common mistakes made by residents on their rotation. A focus-group discussion consisting of eight residents, representing all 3 years of residency classes, was facilitated by a nonphysician to determine how to best incorporate topics generated from the surveys into a useful format. The focus group created a content outline in which each rotation was covered by a separate chapter. Using this outline, the guidebook was created and subsequently edited by a group of 26 residents and core faculty. Each chapter was generally written by two authors who further refined the content outline while editing each other's work. Once each chapter was completed, a separate resident and faculty editor reviewed each section in detail for accuracy and completeness. The guidebook was ultimately completed in November 2011, 14 months after the original surveys were dispersed. The guidebook was distributed to each resident in the form of a 90-page, pocket-size print version.

The guidebook begins with an introductory chapter with key contact information that can be filled in for the user's institution. This is followed by 16 rotation-specific chapters. A rotation-based approach was chosen as it focuses the content on the most pertinent information. Thus, trainees can quickly read a chapter to cover the most pertinent content for their current rotation. The guidebook was subsequently converted into an online wiki housed on our program's interactive online learning community, initially hosted by Igloo and subsequently moved onto our institutionally developed, custom-designed learning portal.

The efficacy of the guidebook can be judged based on the assessment forms (Appendices C & D). Core faculty helped develop the 20-item multiple-choice knowledge-based assessments focusing on best patient care practices for the rotations of gastroenterology and neonatology. Questions were pilot tested with senior residents, chief residents, and content experts in an iterative fashion to maximize content validity. These paper-based assessments were administered at the beginning and culmination of each rotation. For those residents who completed both the pretests and posttests, average change in pretest to posttest scores was compared between residents who had access to the guidebook and controls from before the guidebook was introduced.

After the knowledge-based assessments had been completed, all current residents (given the length of the study, all of these had at some point been a part of the pre- or postintervention) were surveyed to determine if they perceived that these interventions had improved their efficiency and knowledge of best patient care practices.

Results

Residents at our hospital have a daily noon conference that they must sign into electronically upon arrival. As an indirect marker for efficiency, these noon-conference attendance logs were queried to assess improvement in on-time attendance after the guidebook intervention. We defined on-time attendance as the number of days per rotation that a resident arrived before 12:15 p.m. We theorized that if the guidebook acted as a familiar, available, and high-yield source of information, then it should also help residents to complete their numerous morning tasks and arrive at noon conference on time. For our inpatient rotations, morning tasks that must be accomplished before attending noon conference include receiving sign-out, prerounding on all patients, writing progress notes, rounding with the attending and team, and completing key patient care activities (e.g., calling consultants, evaluating morning admissions) after rounds. There were four noon conferences per week; thus, in a standard 4-week rotation, a resident could attend a maximum of 16 conferences.

Preintervention evaluation of noon-conference attendance and knowledge-based assessments occurred from March 2010 through October 2011, thereby including residents at the start and end of two different academic years. The guidebook was released in print form to the residents in November 2011, and by December 2011, it had been fully converted into an online wiki. Postintervention assessments evaluated interns from February 2012 to October 2012, again covering two different academic years. This was purposely done to ensure that that measured improvements were not simply due to gains in efficiency from increased resident experience as the academic year progressed. The survey of resident perception of the guidebook and online wiki utility was conducted in January 2013.

Table 1 depicts the average number of days that residents arrived on time to noon conference over the course of a 4-week-long rotation before and after the release of the guidebook and online wiki. Out of a maximal 16 days per rotation, average on-time arrival to noon conference improved from 3.9 days preintervention to 4.7 days postintervention (p = .06). Interns (n = 40) specifically improved from 3.9 days to 5.5 days, and evaluating this subgroup separately, there was a statistically significant improvement (p = .04) in on-time attendance at noon conference.

Table 1.

Average Number of Days of On-Time Noon-Conference Arrival per 4-Week Rotation Before and After Guidebook Introduction

Year	Before	After	p
PGY-1	3.9	5.5	0.04
PGY-2	4.1	4.3	0.11
PGY-3	3.6	4.4	0.06
Overall (n = 100)	3.9	4.7	0.06

As shown in Table 2, resident performance on knowledge-based assessments improved on the pre-post rotation-specific multiple-choice tests by 3.7% (n = 16) prior to the guidebook and online wiki introduction, compared to 11.2% (n = 18) after the educational intervention; however, this was not statistically significant (p = .17).

Table 2.

Average Percent Improvement From Prerotation to Postrotation Testing on Knowledge-Based Assessments Before and After Guidebook Introduction

Assessment	Before	After	p
Gastroenterology	2.3% (n = 7)	14.0% (n = 8)	0.19
Neonatology	4.7% (n = 9)	9.0% (n = 10)	0.49
Overall	3.7% (n = 16)	11.2% (n = 18)	0.17

Fifty-four out of 100 residents responded to the summative survey. The first question asked, “How frequently do you use the guidebook?” Responses were never (9%), once per rotation (43%), two to four times per rotation (26%), at least weekly (20%), and at least daily (2%). The remaining results are shown in Table 3.

Table 3.

Evaluation of Guidebook by Residents on the Summative Survey Instrument

Survey Question	Average
The residency guidebook was useful to me.	3.9
The residency guidebook improved my knowledge of best patient care practices.	3.6
The residency guidebook improved my efficiency.	3.7
The residency should continue to improve and maintain the guidebook.	4.4
The residency should continue to improve and maintain the online wiki.	3.8

In summary, although we did not show a statistically significant improvement in knowledge-based assessments or overall efficiency, the trend was consistently favorable. On subgroup analysis of interns, guidebook and online wiki introduction was associated with an improvement on a surrogate marker of efficiency, attendance at noon conference. Residents self-reported that the guidebook and online wiki improved their efficiency and knowledge of best patient care practices.

Discussion

Introduction of our comprehensive, residency program–wide guidebook in both written and online wiki formats was associated with an increase in on-time arrival to noon conference by pediatric interns. Although we were unable to demonstrate a statistically significant improvement in knowledge by formalized assessments, residents self-reported that the guidebook helped to improve their knowledge of best patient care practices on their rotations.

We surmise that the association we observed between guidebook introduction and improved on-time noon-conference attendance by those residents with the least experience, namely, interns, might be secondary to efficiency gains. Anecdotally, interns at our institution feel that the busiest part of their day on inpatient rotations is during the morning hours. Consequently, our guidebook focused on topics essential for interns to get their morning tasks accomplished.

To our knowledge, no prior studies have shown that guidebooks or online wikis objectively increase resident efficiency gains across an entire program or improve performance on knowledge-based assessments for those not involved in the creation of the learning resource. In our study, the fact that those who did not help create the guidebook or wiki did not show statistically significant gains on objective assessments of knowledge acquisition aligns with constructionism learning theory. Ackerman stated that in constructionism learning theory, “the cycle of self-directed learning is an iterative process by which learners invent for themselves the tools and mediations that best support the exploration of what they care most about.”[14] In the future, we will encourage residents not only to add to the guidebook and online wiki based on their experiences in order to keep it up to date but also to customize it and make it their own.

In trying to reconcile the lack of statistically significant improvement in knowledge-based assessments and overall efficiency with the positive survey results, we noted that our pediatric residents used the online wiki comparatively less than other studies of resident usage of online wikis.[9] A study of the utility of UpToDate and personal digital assistant–based applications highlighted the fact that underutilization of high-yield and, particularly, Internet-based resources is common for residents even when they perceive these sources to be of value.[15] Phua and Lim highlighted in their 2007 study that despite the electronic information boom, residents still preferred traditional sources of medical information over electronic resources.[16]

The natural history of learning to use educational resources mirrors the progression that occurs in solving clinical problems.[17] That is, only after repeat episodes of utilizing a resource to address a clinical problem does a resident become adept at utilizing this resource. We began evaluating residents promptly after the guidebook and online wiki were introduced; thus, we suspect that some residents did not have an adequate opportunity to acclimate themselves to this novel learning resource. We hypothesize that according to constructionism theory, the more senior residents had used other learning resources situated on specific rotations, developing their own personal favorites before the guidebook and online wiki were introduced. Although the residents reported that the guidebook and online wiki were valued, they may not have been more useful than the residents' current regimen of learning resources. Underutilization of these resources, whether because of novelty or unsuitability, potentially contributed to the lack of an improvement in overall efficiency and knowledge of best patient care practices.

In addition to the noted underutilization of the guidebook, there are several other limitations to this study. First of all, this was a single institutional study with a limited sample size of 100 pediatric residents. The survey itself had a response rate of only 54%, and self-reported perceptions may be fraught with biases. Additionally, the measure of efficiency (i.e., on-time arrival at noon conference) is only a surrogate marker for resident efficiency. However, we are unaware of any other changes in the structure of the residency program over the study periods, and the study was designed to overlap academic years to account for natural improvements in knowledge and efficiency. The formalized assessments of knowledge might have been underpowered to detect a true difference, and they assessed many topics not explicitly covered in the guidebook. As the trend was favorable, perhaps assessment of more residents on additional rotations might have demonstrated a positive relationship. Finally, as results were deidentified, we were unable to correlate whether those who used the guidebook more frequently had higher on-time attendance and improvement on knowledge-based assessments.

Future directions should focus on augmenting the quality and utilization of the guidebook and our online wiki, then reevaluating if, once well adopted, there is a sustained benefit. It would be interesting to consider alternative metrics of evaluating trainee efficiency, for example, a comparative time-motion study. For programs that already have a well-utilized guidebook, it would be valuable to ascertain if converting this into an online wiki confers efficiency or knowledge benefit upon the residents.

A. Guidebook Childrens National Version.pdf

B. Guidebook Template.docx

C. Assessment I (Gastroenterology).pdf

D. Assessment II (Neonatology).pdf

All appendices are peer reviewed as integral parts of the Original Publication.