An innovation tournament to improve medical residency.

Two large national studies of resident duty hours incidentally revealed surgical and medical resident dissatisfaction with residency training. Aiming for an inclusive and democratic approach to improve graduate medical education, we conducted a national innovation tournament--reaching out to the program directors of all 474 US internal medicine residency programs to invite them and their residents and associate program directors to participate. Participants could submit multiple ideas as individuals or teams in four domains: [1] resident well-being and personal and professional development; [2] resident education and clinical preparedness; [3] resident sleep and alertness; and [4] patient safety. Residents and program directors were reinvited to rate ideas, whether they had submitted ideas themselves or not. We used a schedule of lottery-based prizes to stimulate the submission and rating of ideas and encourage engagement. 164 residents and program directors from 51 different programs submitted 328 ideas. 153 residents and program directors from 48 different programs submitted 15,345 ratings of ideas. Winning ideas aimed to reduce residents' work burden or improve their mental health, sleep, eating, or relaxation or reflected technical fixes to the operations of residency, such as changing vacation schedules and the timing of pay. The results of this tournament provided actionable suggestions to improve residency training now being tested in our own residency programs. Innovation tournaments drive engagement and generate value by their opportunities for inclusion and by shifting problem solving to the end user.

The context and problem

In 1984, an 18-year old college student named Libby Zion died while an inpatient at The New York Hospital. Her father, Sidney Zion—an influential commentator and lawyer in New York—blamed under-supervised and overworked resident physicians. The result was a cascade of commissions and regulations most notable for limiting resident duty hours and shift lengths, largely in the interest of patient safety.[1,2] In 2011, the Accreditation Council for Graduate Medical Education introduced additional 16-h shift length restrictions on first year residents,[3] creating further tension between the regulations and a graduate medical education community worried that these duty hour restrictions would reduce the quality of medical education. The arguments on both sides were well-meaning and plausible; yet, four decades after Libby Zion’s death, no large prospective trials had provided the kind of evidence that could inform such policy at the level we expect for the introduction of a new drug or vaccine.[4]

iCOMPARE and FIRST were large pragmatic randomized trials that compared limited shift lengths in medical and surgical residency training programs with unrestricted (flexible) shift lengths.[5,6] The trials found that flexible shift lengths were not inferior to the 2011 limited duty hour rules in patient safety, sleep and alertness, and many educational experiences.

However, the two studies also revealed substantial disenchantment with residency training in general—more so in internal medicine than surgery. Indeed, an editorial accompanying one of the iCOMPARE reports concluded with a call to action: “The contribution of the iCOMPARE trial may not be the determination of whether flexible or standard duty hours are preferred, but rather whether health system and education leaders hear the sentinel plea of residents to reform our clinical learning environments to prioritize people.”[7]

The approach

Hearing that plea, we sought to engage internal medicine residents and residency program faculty in an innovation tournament to crowd-source and rate ideas to improve internal medicine residency. Innovation tournaments provide a structured, competitive environment for ideas to be shared and evaluated.[8] In a typical tournament, participants are invited to submit ideas, usually through a web-based platform, and then those ideas are rated—often by the same stakeholders submitting ideas in the first place. These tournaments drive engagement and generate value by shifting problem solving to the end user. Because they typically allow anonymous submissions and ratings, they are highly inclusive and highly democratizing.

We invited residents and directors of every internal medicine residency program in the United States to participate in an online innovation tournament to improve internal medicine residency in four domains[1]: resident well-being and personal and professional development[2]; resident education and clinical preparedness[3]; resident sleep and alertness; and[4] patient safety. Here we present our experience using this format, and the ideas received, to improve resident training.

Participants

In October 2019, we asked all 474 US internal medicine residency program directors (identified through the Accreditation Council for Graduate Medical Education) to distribute to their current residents and associate or assistant program directors an invitation to participate in an innovation tournament to improve medical residency. A video link (https://vimeo.com/368165739) gave further information on the tournament. Program directors were reminded several times by email to encourage participation and were provided with sample email text to invite their residents and program faculty.

Idea submissions

Residents and program faculty received a code allowing access to the tournament, programmed on the “Your Big Idea” platform operated by the University of Pennsylvania (Fig. 1). Participants could submit as individuals or in groups as many ideas as they wanted in any or all of the four categories. Each submission required a title, a description of the problem, a summary of the idea, and an explanation of why the idea could work. Submissions were limited to 300 words. The platform included pull-down menus to self-identify post-graduate year (PGY) or faculty and the specific internal medicine program. The submission phase remained open from November 4, 2019 through January 15, 2020.

Fig. 1.

Idea submission.

Idea ratings

After the submission period closed, all residents and program faculty were invited to return to the platform to rate submissions—whether or not they themselves had submitted ideas. Ideas were presented to raters anonymously and in random order and rated with 1–5 stars (Fig. 2). Participants could rate ideas only once, but could rate as many of the ideas as they wanted and were invited to submit written comments in addition to the star ratings. Participants could see only their ratings, and not the ratings others may have provided. The top 10 winners were selected in order by the mean rating.

Fig. 2.

Idea rating.

Participant incentives

We aimed to encourage participation in both submission and rating and designed incentive structures. We used a lottery-based approach that rewarded volume of participation and quality of submission while still making it possible to win randomly[1]: Each submission received a chance of receiving one of five $5000 prizes selected at random.[2] The top three most popular ideas based on ratings were eligible for prizes of $10,000, $5,000, $2,500, and the next seven for prizes of $1000.[3] Each submission was eligible for one of ten special prizes of $1000 in categories such as “So crazy it might work,” “Most likely to go viral on Twitter,” “Most socially inclusive,”—all judged by a panel of medical education experts.[4] Each participant rating ideas received one chance to win one of ten $2500 randomly selected prizes for every 50 ideas rated. The four different incentives summed to $84,500.

Participants were informed that their ideas could be publicized. The approach was reviewed by the Institutional Review Board at the University of Pennsylvania. The sponsors had no role in the design or conduct of the tournament.

Outcomes

128 residents (roughly equally divided across PGY-1 to PGY-3) and 36 program directors from 51 different programs submitted 328 ideas: 155 about resident well-being and personal and professional development; 95 about resident education and clinical preparedness; 45 about resident sleep and alertness; and 33 about patient safety. 153 residents and program directors from 48 different programs submitted 15,345 ratings of ideas. Each idea received a mean of 47 ratings (SD 4.4). The range of mean ratings was 1.51–3.80 (mean 2.83, SD 0.4).

Popular and winning ideas

The ten most popular ideas and the ten ideas winning in special categories are listed in Tables 1 and 2. The entire list of 328 ideas is available in an online supplement. Several winning ideas aimed to improve clinical skills, such as the performance of procedures or experience with telemedicine or the roles of ancillary health care professionals. One idea focused on learning more about personal finance. Several ideas addressed improvements in patient care directly, such as ways to help patients with medication adherence or housing needs, or ways to improve discharge summaries to advance continuity.

Table 1

Top 10 ideas by ratings.

Tell us about the problem you’re trying to solve	Idea Summary	Tell us why you think your idea would work
When nurses cannot obtain IV access, they turn to residents to try to obtain access. Given that residents are generally busy with running a team during the day, finding time and finding someone to observe the procedure if one is not certified can be difficult.	There can be an IV Access Elective in which interns and residents can have dedicated time to practice US guided IV placement, Midline placement, PICC placement, and Central Line placement. This would be under the guidance of a senior who is certified, and therefore, interns may be certified much earlier during their residency and feel comfortable with these procedures when called upon by nurses to get access.	Having dedicated time to practice procedures will allow interns and residents to become comfortable and even proficient at doing essential procedures, which will lead to better patient care.
Lack of flexibility in scheduling doctors appts and running errands during regular business hours.	Most programs have implemented block scheduling. During the outpatient block there should be 1 half-day session that a resident would be excused from clinic duties. This would allow for “free” time to use at residents’ discretion to schedule appts, run errands, sleep or workout.	By allowing half days without clinic duties and without using precious vacation time this will help in achieving a balanced life outside of the hospital.
Personal finances have a potential impact on all aspects of a resident’s personal and professional life; and inadequate practices, goals, or expectations could impact future professional decisions. With increasing amounts of debt, a better understanding of finance may also open up residents to pursue traditionally “lower-income” but important career choices.	Residency programs often include a short curriculum in finance and practice management, often by the end of the residency. A more ideal scenario would be a longitudinal course that could span the 3 years and progress from basic topics as credit, budgeting, and loan management to investment options, retirement funds, contract negotiation, etc. Most residents have lived for years on student loan income and in several years have a significant increase in salary without significant experience in asset management. Wrong choices in early career stages could have long-term impacts yet are easily avoidable with some education, guidance and mentoring.	It is known that even basic education on personal finance can have long-term positive impacts, and it is particularly true with residents facing management of high loan burdens, a variety of contract options, and high-incomes. There are multiple options to incorporate this into our existing academic curriculum.
Majority of the patients we see during residency often have multiple comorbidities and have multiple medications that are often changed after hospitalizations or clinic visits. They often forget the names and are unable to carry the pill bottles to office visits due to various reasons, increasing concern for non compliance.	After discharge from the hospital, a small card with a list of medications that can fit into a patient’s wallet or purse would be an easy solution for medical reconciliation on repeat office visits. Patient’s would just have to show the card with a list of their medications, and it can be updated as required. It can be reconfirmed with the pharmacy refill history and medications which are not required can be striked off to ensure patient discontinues taking them. Patients can show the card at the pharmacy or to a different specialist who may be outside the hospital EMR.	Medication names are hard for people who are not in the medical field to comprehend. A card that fits in the pocket is easy to carry and assures patients and physicians of their medications and reduces cumbersome storage of paperwork from hospital or clinic visits.
Discharge summaries assist in the transition from inpatient to outpatient care by communicating key diagnoses, medication changes, and follow-up instructions. Despite use of standard formats for discharge summaries through electronic medical records, PCPs report dissatisfaction with locating important information in discharge summaries which often requires obtaining information from other sources.	Create a template at the beginning of discharge summaries with key details primary care physicians desire to know upon discharge in an efficient, easy to find format. Example:o Main diagnosis/reason(s) for hospitalization: ○ New Medications and Medication Changes and why (why med was added or dose was changed, include end dates for short-term meds): ○ Medications Discontinued and why: ○ Labs/tests to follow up after discharge: ○ Which consultants saw the patient inpatient: ○ Follow-up required with the following specialties: ○ Instructions for outpatient providers: ○ Non-acute issues that need to be addressed as outpatient (e.g. Incidental imaging findings):	IM residents surveyed report it is not more difficult to use the new template (90%) and not more time consuming (55%). Residents thought that discharge summary quality improved with incorporation of the template (93%) and would like to see the template be permanently made a part of discharge summaries (93%).
Patients on inpatient medical service often have “non-medical” issues such as financial/food/housing insecurity, inadequate support to care for self, lack of mobility that require intensive out-of-hospital resources to be set up for safe discharge planning. This process can increase LoS and is outside the scope of most resident’s training.	When patients are medically ready for discharge, they can be transferred to a non-resident service or given a “status” that has a more intensive social work/care coordination following. There would still be an MD following but the common structure of a detailed medical rounds supplemented by brief care coordination rounds would be inverted to better represent the patient’s changing needs.	This process would allow care coordination and social work services to be more judiciously/appropriately allocated to patients whose progress in the health system largely depends on them. It would also offload this responsibility from residents - as it is a large source of burnout and not within their expertise.
Residents commonly receive untimely pages that limit the ability to catch some sleep on call or interrupt valuable planned didactics such as noon conference.	We propose a change to the pager system where one can switch paging status to “sleeping” or “at didactic”. In this mode, we would program the paging system to only allow pages sent that are labelled as “urgent” or “STAT”. We believe this would limit pages sent by nursing/other providers about non-urgent issues that can addressed during the day or after didactics. In order to limit “click fatigue” changing paging status back and forth, this could be easily automated (i.e., status is switched to “at didactic” at noon and switched back to “on page” at 1 p.m.).	By disallowing pages not labelled as “urgent” or “STAT”, it prompts nurses/other providers to reconsider whether the issue warrants the “urgent” or “STAT” designation at the moment, thus reducing the amount of non-urgent pages during critical times of rest and learning.
Residents are often looking for procedure opportunities (i.e. central lines, paracentesis, arterial lines, lumbar punctures, thoracentesis) or need help getting a procedure performed. Communicating these needs and interests is difficult especially in large hospitals.	Create a procedure request board for residents looking for a procedure to do, update it on said board so the rest of the class knows to give it to you when the opportunity arises. Also include a request supervisor function - sort of like an uber app for calling supervisors for procedures.	This will help those who need procedures done get them done quickly, and helps those who want to complete more procedures find what they’re looking for.
Residents often find it near impossible to effectively connect with healthcare providers for their own needs (primary care, dental, mental health etc). A great deal of time and effort is spent in coordination of these basic needs.	Great success has been shown in the utilization of patient navigators for multidisciplinary clinics such as cancer centers. One of these pre-existing patient navigators could be assigned part time to the residency program. They would serve as a point person for residents to reach out to in order to help coordinate visits and be connected with supplemental resources. They could even proactively reach out with access to resident schedules to facilitate more PCP visits among residents to help ensure well being.	Many hospital systems already have this resource and given the relatively small number of residents this rapid intervention would make residents feel cared for by unburdening them in the stress of managing their basic healthcare.
Burnout, anxiety, and depression are very common among residents. Through an opt-out MH program, we can improve access/reduce barriers to mental health services and reduce stigma related to mental health.	All residents would automatically be assigned to a mental health therapist and would have an appointment scheduled for them during one of their lighter blocks (outpatient or elective rotations). Residents have the option to opt-out of this program.	We have piloted this idea among interns at our program, and it has been received with great support. We hope it will reduce access to mental health services and reduce stigma related to mental health.

Table 2

Winners in 10 special categories.

Tell us about the problem you’re trying to solve	Idea Summary	Tell us why you think your idea would work
Winner, “So crazy it might work.” The boost in happiness experienced by giving is under recognized. Current approaches to clinician well-being address system and individual factors. Expressing gratitude is an endorsed practice but giving to others hasn’t been.	Weekly, 2–3 residents from the IM program will be selected to spend $20 on someone else within 2 days. By the end of the year, all residents in the program will have been selected and participate. A brief survey will be completed by each participant to capture what was done and a reflection of what it meant to them. They will also be invited to post their reflection on social media with #givingmakesuswell.	Michael Norton’s TED talk https://youtu.be/PsihkFWDt3Y and research evaluated giving students money and asking them to spend it on either themselves or someone else by the end of the day. Giving to others boosted happiness.
Winner, “Most likely to go viral.” Residency is a stressful period of time with demanding schedules. Individuals who enter residency without established partners often find difficulty dating. There exists data that people in medicine are more likely to end up with partners in medicine.	Swiping-based matchmaking apps have taken the layperson’s dating world by storm. We would like to extend this to the residency world. We want to create an dating app exclusively for residents to help facilitate relationships among medical professionals. The app will be geared less towards casual encounters and more towards building meaningful relationships. It will have the ability to set preferences based on location, age, specialty, and PGY year.	This is an idea that residents are craving. Residents often express frustration at the current dating app climate and are looking to meet like minded people with similar goals.
Winner, “Most geeky.” Decrease fatigue in the workplace through relaxation	Virtual reality for the Doctor. This will be a corner/space with virtual reality headset(s) which residents may use when needed to take a timed break eg (15 mins). The software will allow residents to either be transported to a peaceful location (beach, mountain, garden) or choose a short game to engage in.	Working for extended, consecutive hours may get monotonous and decrease alertness. Allowing for timed breaks with virtual reality, will not only allow residents to take breaks when needed but provide a visual stimulus to enhance the break experience.
Winner, “Best use of high tech.” Residency and sleeplessness are more often than not, synonymous. This hampers resident health and potentially jeopardizes patient care.	Residents are to have the option to enroll in a sponsored program where a fitness/sleep tracker is offered to residents in exchange for enrollment. Participants’ sleep data may then be analyzed quarterly revealing considerable insight into frequency of sleep inadequacy and possibly even clinical conditions that may require intervention. Based on this information, flagged residents maybe offered counseling on sleep hygiene, referral to a sleep specialist or help as appropriate to mitigate the issue. This information may also be used to assess problem rotations/services which may require adjustment.	Most people including physicians do NOT give sleep the importance it is due. They also lack insight into their own sleep habits. Sponsored monitoring encourages people to shift focus into their own sleep habits and make changes. This also gives programs the opportunity to address resident health.
Winner, “Promotes the most engagement.” Most people gain weight in residency. We spend lots of money and calories on eating out and delivery food.	It would be fun to have a residency cookbook where people are encouraged to submit their own tried-and-true recipes. More points if the recipes have fun inside jokes related to the program. Depending on how many people submit a recipe, could give out prizes for best dessert, best main dish, etc.	This is a fun initiative that I think many people could benefit from, and it is something that we could hold onto after graduating.
Winner, “Just might get the Nobel Peace Prize.” New resident physicians face extreme financial hardship in the period between graduation from medical school (during which it is nearly impossible to be gainfully employed) and beginning salaried residency training. During this time of financial hardship, many are expected to shoulder the expense of relocation.	New residents should be paid their first paycheck at the beginning of the initial pay period. This would prevent homelessness and food insecurity of new residents who in many cases need to pay large sums for moving expenses leaving no finances available for food or supplies for their new position.	Trainees in their first month have a unique difficulty with affording food and housing. Providing their first paycheck upfront rather than waiting until the end of the pay period would help people and would carry low risk because of the very low dropout rate of residents in the first month.
Winner, “Back to the future.” Telehealth is an expanding market bringing face-to-face to patients who otherwise might not be able to come to an appointment as well as enhance quick patient-physician communications beyond that of a phone call. However, residents are woefully ill-prepared with how to best handle these patient encounters.	I propose that IM residencies include Telehealth experiences during the residency training period to increase nascent physician familiarity with the tool. Many residencies are already attached to health systems that are already employing TeleHealth for patients. This could be particularly advantageous in resident clinics with chronic “no show” patients. This also could be used as a quick geriatrics consult for a patient at a nursing home to avoid unnecessary and potentially harmful ER admissions for otherwise stable patients. Attendings would be present as in clinic for case presentation and review alongside the resident.	I think that as medicine transforms more into patient satisfaction metrics, the convenience of not having to be physically present and waiting in the office for the doctor will become more popular. I think it prepares IM residents for the coming wave of care delivery.
Winner, “Most socially inclusive.” High quality medical care is delivered by multiple indispensable health care providers; nurses, respiratory therapists, unit clerks, physician assistants, to name a few. We all have unique areas of expertise. However, when working together in daily operations, misunderstandings and miscommunication occur when we do not understand anothers perspective.	A devoted one week rotation through Ancillary services during residency where you shadow different providers, to see what their day-to-day responsibilities and work flow are. Questions like, “Where is the DKA patients BMP?”, “How come this patient has not gotten his SAR placement?”, “Will this patient ever get their medication”. These are the	This Ancillary Rotation would provide depth and perspective about what goes into health care delivery, provide institution wide team building, and ultimately find solutions to make hospital care more efficient. frustrations we hear when working with our fellow co-residents. We propose: days spent with social worker, nursing, phlebotomy, laboratory technician, and respiratory therapist. In doing so we would gain an appreciation for the difficulties of their role, and what can stunt patient care.
Winner, “Best use of low tech.” Although lunches are covered in residency, there is usually no dinner provided for late night calls. This means residents bear the time-consuming burden of preparing food or spending their limited income purchasing their dinner.	Hospital food services generate up to three pounds of food waste per hospital bed. A significant portion of this waste is due to overproduction of food. Instead of throwing away the extra food, hospitals can use this food for meals for housestaff. At the end of each day after dinner service (i.e., 9 p.m.), overproduced food in both the food service and visitor cafeteria can be delivered to a specified location. Residents can then help themselves to a hot plate of food for free.	By leverage existing overproduced food that would simply lead to food waste, this would place no extra cost on the hospital system. The amount of food overproduced in the hospital would instead be put to positive use in satisfying hungry housestaff.
Winner, “Most promising low hanging fruit.” Sleep and fatigue	Having rotated in several internal medicine rotations at different hospitals-all of them had one thing in common-uncomfortable beds. If a resident is on call, it is already difficult to sleep, but in that 1 hour they have of free time it is difficult to take a nap because the beds are equivalent to sleeping on the floor. We should have comfortable beds.	Comfortable beds = easier to sleep = residents have more rest

The majority of the winning ideas focused on resident well-being. Some of these ideas aimed to reduce residents’ work burden or improve their mental health, sleep, eating, or relaxation. Among these was the idea that residents should automatically be provided with mental health services, allowing them to opt out rather than requiring them to opt in and seek care. Other ideas reflected technical fixes to the operations of residency: one recognized that in other industries a weeklong vacation typically spans 9 days because of the weekends on either side, but only 7 days for residents who can’t rely on weekends off. Another suggested that residents be paid their first month’s salary in advance, because otherwise many have no income or resources at precisely the time they must relocate.

What links these entries is that nearly all of them seem actionable, reflecting relatively easy opportunities to improve residency. The suggestion of a swipe-based resident matchmaking app named, “The Call Room,” might be harder for residency programs to get behind, but market-based alternatives exist.

Lessons for the field

This study offers four key lessons. First, online innovation tournaments are effective ways to engage large audiences to address focused problems. We used a program of our own design (https://bigidea.pennmedicine.org/) but other innovation tournament platforms are freely available to facilitate this work (e.g., http://darwinator.com/). Innovation tournaments are like high powered suggestion boxes. They achieve their goals by facilitating the collection and evaluation of many ideas, allowing people to participate in either or both of those processes. They are inclusive and democratic, and often fun. The competition of these tournaments can add excitement beyond conventional approaches of submitting ideas and can bring stakeholders into the rating process as well as the submission process.[4]

Second, our process was engaging, although not completely. We easily engaged participants from 51 different internal medicine training programs, receiving 328 ideas and 15,345 ratings of those ideas with little effort for scale. No doubt we were advantaged by our ability to offer monetary prizes for participation. However, those 51 programs represented only about 10% of the 474 we targeted. We recruited residents and faculty through email messages to their program directors suggesting that low overall engagement represents the challenge of reaching residency program directors and getting them to encourage the process in the first place. Part of the fundamental problem motivating this study is that residents have limited extra time. Program directors also have limited time. We considered the possibility of creating prizes for programs themselves, based on the proportion of their residents they engaged. We ultimately set that idea aside but perhaps future efforts could test whether that approach could broaden participation.

Third, we received ideas that were on-topic and in many cases promising and actionable. The most meaningful measure of a tournament’s success is programmatic change leading to an improvement in goals. Although our tournament targeted the submission and rating of ideas and not their implementation, those ideas are now available so that others can consider them for implementation. The programs at Penn and Hopkins are already making changes. Penn’s program implemented: required meetings with a mental health specialist starting in internship but sunset the program after it was less successful than anticipated; a weekly night float brunch; themed social groups; and greatly enhanced longitudinal mentorship. The Hopkins program has implemented opt-out mental health, clinical skills in telemedicine, a busy status in paging during conferences, and personal time half days during ambulatory rotations.

Finally, innovation tournaments are useful not just for identifying solutions but also for identifying problems. We learn what residents want both when the same general idea is submitted often (e.g., support for ridesharing services after late night shifts, time off for personal appointments) and when ideas receive many votes (e.g., improving procedural skills, getting half days off for personal health care, learning about personal finance).

The summary lessons from this experience is that it isn’t that hard to elevate our ascertainment of the problems and potential solutions to improve residency education, that one can do so in a democratic and inclusive way that engages the community of residents and educators, and that many of those solutions seem plausible and actionable.