ALiEM Connect: Large-Scale, Interactive, Virtual Residency Programming in Response to COVID-19.

PROBLEM: The COVID-19 pandemic restricted in-person gatherings, including residency conferences. The pressure to quickly reorganize educational conferences and convert content to a remote format overwhelmed many programs. This article describes the pilot event of a large-scale, interactive, virtual educational conference modeled, designed, and implemented by Academic Life in Emergency Medicine (ALiEM), called ALiEM Connect. APPROACH: The pilot ALiEM Connect event was conceptualized and implemented within a 2-week period in March 2020. The pilot was livestreamed via a combination of Zoom and YouTube and was archived by YouTube. Slack was used as a backchannel to allow interaction with other participants and engagement with the speakers (via moderators who posed questions from the backchannel to the speakers live during the videoconference). OUTCOMES: The RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework was used for program evaluation, showing that 64 U.S. Accreditation Council for Graduate Medical Education-accredited emergency medicine residency programs participated in the pilot event, with 1,178 unique users during the event (reach). For effectiveness, 93% (139/149) of trainees reported the pilot as enjoyable and 85% (126/149) reported it was equivalent to or better than their usual academic proceedings. Adoption for ALiEM Connect was fairly good with 64/237 (27%) of invited residency programs registering and participating in the pilot event. Implementation was demonstrated by nearly half of the livestream viewers (47%, 553/1,178) interacting in the backchannel discussion, sending a total of 4,128 messages in the first 4 hours. NEXT STEPS: The final component of the RE-AIM framework, maintenance, will take more time to evaluate. Further study is required to measure the educational impact of events like the ALiEM Connect pilot. The ALiEM Connect model could potentially be used to replace educational conferences that have been canceled or to implement and/or augment a large-scale, shared curriculum among residency programs in the future.

Problem

The COVID-19 pandemic broadly impacted residency education due to limitations to in-person gatherings. [1] The Accreditation Council for Graduate Medical Education (ACGME) Program Requirements for Emergency Medicine mandates that residency programs provide a minimum of 5 hours of planned educational conferences weekly. [2] These conferences are generally presented live to colocated audiences of resident and faculty participants. However, physical distancing measures prompted the rapid conversion of didactic sessions from in-person to remote learning platforms, which presented an opportunity to create a national platform for harmonizing emergency medicine education. [1]

Despite the unexpected challenges COVID-19 presented, resident education remains a priority and mandate. The pressure to quickly reorganize conferences and convert content to a remote format overwhelmed many programs. While successful virtual conferences have been previously described as an effective solution for delivering residency didactics, [3,4] prior implementations failed to scale up or facilitate real-time interaction between speakers and attendees.

This article describes the pilot event of a large-scale, interactive, virtual educational conference modeled, designed, and implemented by Academic Life in Emergency Medicine (ALiEM), called ALiEM Connect.

Approach

ALiEM Connect aimed to deliver large-scale, remote residency education in a synchronous, interactive fashion. [5] We also wanted to alleviate the burden of last-minute conference planning for clinician educators during the COVID-19 pandemic and provide access to nationally renowned speakers in a cost- and time-efficient way. The pilot ALiEM Connect event was conceptualized and implemented within a 2-week time frame that began with format and platform investigation in mid-March 2020. Speaker invitations closely followed, event promotion began March 16, 2020, and the 1-day pilot event was held on March 25, 2020.

Based on ALiEM’s prior experiences with online education via social media platforms, [6,7] we believed there were 3 elements necessary to make this experience successful: (1) the speakers had to be able to interact with each other (and the moderators in a “backstage” virtual holding area) and reach their audience, (2) the audience had to be able to interact with each other, and (3) the audience had to be able to engage with the speakers. Details of our approach are depicted in Figure 1.

Figure 1

Schematic of the synchronous and asynchronous components of the ALiEM Connect pilot. The ALiEM Connect pilot was a 1-day event on March 25, 2020, aimed at providing a large-scale, interactive, virtual residency educational conference. Abbreviation: ALiEM, Academic Life in Emergency Medicine.

First, our model required that speakers interact with each other and reach the audience using a videoconference platform (e.g., Zoom, WebEx, Skype, GoToMeeting). After examining multiple platforms, we selected Zoom for the event and scheduled a conference. Speakers joined the Zoom videoconference feed that was also livestreamed to participants via a public YouTube video. Using YouTube as well as Zoom allowed us to overcome the scaling limitations inherent to videoconference platforms alone, as only the 6 speakers and 2 moderators needed to be on Zoom, not the entire audience. Furthermore, the benefits of streaming content via YouTube included free accessibility to a large viewership and easy integration with Zoom, overcoming concerns about malicious or disruptive participants, and automatic archival (see below).

Second, the ALiEM Connect model required a mechanism for interactivity between participants. “Digital backchanneling” was previously described as a virtual mechanism for informal discussions that occur alongside formal proceedings and has become increasingly popular among medical conference attendees. [8] While many platforms are used for live discussions and question-and-answer sessions (Twitter, WhatsApp, Microsoft Teams, etc.), we chose Slack due to our speakers’ and moderators’ existing familiarity with it, its compliance with Health Insurance Portability and Accountability Act privacy standards, its compartmentalization of discussions by residency program and by speaker or topic using different channels, and its ability to synchronize with other applications, including calendars, supporting documents (e.g., .jpg, .pdf, and .docx files), and surveys.

Third, our model also focused on the audience engaging with the speakers. To optimize this interactivity, moderators facilitated the Slack-based discussions. They monitored these discussions for audience questions, which were then posed to the speakers live during the videoconference. Some moderators also engaged in real-time microblogging of the event via Twitter.

Curriculum design and implementation for the event were led by a team of 8 ALiEM faculty collaborators (our organizing team) with experience and expertise in medical education and previous experience delivering large-scale, remote learning programs for faculty development and chief resident training purposes. Residency programs were invited to participate via the Council of Residency Directors in Emergency Medicine communities portal and through ALiEM social media posts. We designed the didactic program content to be 2 hours in length, divided into six 20-minute lectures, each by a different speaker. Topics for the ALiEM Connect pilot were selected collaboratively by the speakers and our organizing team, guided by current events and subject matter content from the American Board of Emergency Medicine (ABEM) Model of the Clinical Practice of Emergency Medicine (MCPEM). Invited speakers were nationally recognized emergency medicine educators who were free to select their own lecture content based on the ABEM MCPEM; one COVID-19 lecture was purposefully included.

As an added benefit, the livestreaming of our event on YouTube allowed for automatic archival of content for later viewing and asynchronous interaction (Figure 1). Viewers of the archived YouTube video could interact in an asynchronous format with the moderators and/or speakers of the event via Twitter.

The event was made available for free to all U.S. ACGME-accredited emergency medicine residency programs through the generous donation of upgraded licenses from Zoom and Slack and funding from the ABEM.

An ethics review exemption for this work was provided by the Hamilton Integrated Research Ethics Board.

Outcomes

We used the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework (https://www.re-aim.org/), which emphasizes essential program elements that can improve the sustainable adoption and implementation of effective, generalizable interventions, to evaluate our program outcomes. The RE-AIM framework includes 5 components: reach, effectiveness, adoption, implementation, and maintenance. [9,10] For the purposes of this article, we report results for the first 4 components of the RE-AIM framework. For this article, the maintenance phase was deemed out of scope. Supplemental Digital Appendix 1 (at http://links.lww.com/ACADMED/B98) contains a matrix of our data sources and analyses. Conference attendance was tracked by individual residency programs. Additionally, we solicited program evaluations from participants (both residency leaders and residents) via anonymous online surveys (Google Forms, Google, Mountain View, California; see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/B99).

Reach

All 237 U.S. ACGME-accredited emergency medicine residency programs were invited to participate; 64 (27%) programs participated. On average, 36 (standard deviation [SD] = 15) residents per participating program watched the YouTube livestream, with 1,178 unique users and an average of 1,070 viewers/hour (Figure 2A). At our start time, 277 participants were in attendance and the number increased logarithmically until the 30-minute mark. YouTube analytics for the week following the event (March 25–April 1, 2020) identified 2,271 unique viewers who watched the event, yielding a total of 3,122 views between the livestreamed and recorded versions.

Figure 2

Infographic detailing the (panel A) reach, (panel B) effectiveness, and (panel C) adoption results from the ALiEM Connect pilot program evaluation. The ALiEM Connect pilot was a 1-day event on March 25, 2020, aimed at providing a large-scale, interactive, virtual residency educational conference. Reach, effectiveness, and adoption are 3 of the 5 components of the RE-AIM (Reach, Effectiveness, Adoption, Implementation, Maintenance) framework. The emoji cloud in panel B was generated on McMaster University–licensed Microsoft Windows 10 (Microsoft Corp., Redmond, Washington) software. The emoji character set used is part of the Windows 10 operating system and was used according to Microsoft’s general use agreement. Abbreviation: ALiEM, Academic Life in Emergency Medicine.

Effectiveness

We surveyed residency leaders (e.g., chief residents, program directors, assistant program directors) about their experience with the ALiEM Connect pilot in comparison with their usual weekly educational conferences, with a 55% response rate (35/64 programs). All residency leaders rated their overall experience with the ALiEM Connect pilot positively (100%, 35/35). We also surveyed trainees (response rate: 13%, 149/1,156, which is a low survey response rate that would be prudent to improve in the future). A majority of trainees who responded to the survey (93%, 139/149) rated the ALiEM Connect pilot as enjoyable. Compared with typical in-person offerings, the majority of residency leaders (86%, 30/35) and residents (85%, 126/149) felt that the pilot was equivalent to or better than their usual academic proceedings.

Additionally, to measure participant engagement with our digital backchanneling, we employed a cloud analysis of reactions (Figure 2B). That is, we analyzed participants’ use of emojis when reacting and responding to comments, questions, and insights from their peers during the pilot. Participants used 120 unique emojis a total of 1,531 times.

Adoption

As noted above, 64 (27%) of the 237 invited U.S. ACGME-accredited emergency medicine residency programs registered and participated in the ALiEM Connect pilot. We anticipated an upper limit of approximately 109 (46%) programs being able to participate, as this was the number of programs surveyed during the development process (before March 25, 2020, using Google Forms, Google, Mountain View, California) that reported Wednesdays (our event day) were their typical conference day (see Supplemental Digital Appendix 2 at http://links.lww.com/ACADMED/B99 for the survey). The other 128 (54%) programs did not have protected conference time for residents to attend our event.

To examine the adoption of the technologies, we analyzed Slack usage data to see how many participants interacted by sending messages (see Figure 2C for data) and program evaluation survey results. Our analysis indicated that trainees responded favorably to the combination of a livestream video with backchannel interaction. For example, the majority of trainees (66%, 98/149) reported a positive reaction to Slack. Residency leaders reported that the ALiEM Connect pilot achieved a number of perceived benefits, suggesting that they would be willing to continue participating in ALiEM Connect events. Some residency leaders described their experience as follows:

You did a great job, and I appreciated the content. We tend to have very hands-on small group, engaging conferences so the change to virtual conference[s] has been a huge adjustment for us, and we struggle when it is just lectures. Your content was great but with Slack I think there is a ton of space to get even more interactive and engaging.

I just want to say how much I enjoyed ALiEM Connect yesterday. I think that this could be a game changer in how we do conference[s] and resident education.

Implementation

Within the RE-AIM framework, evaluating implementation includes determining if your intervention was used as intended. About half of livestream users (47%, 553/1,178) interacted in the digital backchannel discussion, sending a total of 4,128 messages in the first 4 hours of the event. Most participants who interacted (47%, 261/553) sent a range of 1–3 messages.

There were 2 types of channels available for communication in the ALiEM Connect Slack workspace: one solely for members of a specific residency program and another speaker- or topic-specific channel (one for each of the 6 speakers) for all conference participants to communicate with the speakers and about their topics. A total of 1,156 participants enrolled in the ALiEM Connect Slack workspace channels and, taken all together, sent 4,128 messages through backchannel interactions (i.e., no additional messages were sent after the first 4 hours of the event). Of the 64 participating residency programs, 17 (27%) programs did not have any participants in their program-specific channels. The other 47 (73%) programs had widely variable numbers of participants in their program-specific channels, ranging from 2 to 64.

To foster engagement within residency programs, we included 1 breakout session midway through the event. Participants were instructed to discuss a specific topic related to the lectures being given. Thirty-nine (61%) of the 64 programs had active backchannel discussions during this breakout session. The participating programs sent a mean of 103 (SD, 73; range, 1–229) messages during the breakout session. A large proportion of all messages (95%, 3,914/4,128) took place as part of participants’ interactions within their program-specific channel as opposed to the speaker- or topic-specific channels open to all attendees (3%, 138/4,128). Discussions during the breakout session included the topics of the lectures as well as storytelling from clinical experiences, literature sharing, and humor in conversation, mimicking real-life, in-person scenarios.

Next Steps

The final component of the RE-AIM framework is maintenance, which will take more time to evaluate. Will this be a sustainable model for large-scale, online, interactive conferencing? It is not yet clear. At the time of drafting this article, we had held successful second and third ALiEM Connect experiences on different days of the week, with the same number of organizers (8, although the individual organizers were different), 1 more speaker (7), and slightly fewer (51) participating residency programs.

Our collaboration with our specialty’s medical board (ABEM) on ALiEM Connect enhanced the infrastructure and visibility needed to maintain future events and provided a unique opportunity for conveying critical information to a large number of trainees. Further research is needed to thoroughly measure the educational impact of such events, especially in comparison to other analogous remote or in-person academic events. Continued study is also required to elucidate remote teaching strategies that best optimize engagement and content retention.

Technical difficulties remain a challenge at the forefront of remote learning and require trial and error to learn how to mitigate. In our experience, the live format required in-the-moment adaptability to unanticipated obstacles, such as the unpredictable initial linkages between Zoom and YouTube, the ALiEM website crashing due to the initial surge of traffic, and speakers being unable to share presentation slides on their screens. These brief technical challenges were managed quickly and did not seem to detract from participants’ overall experience. Involving support staff and developing a record of previous experiences can increase preparedness for future events by informing strategies to mitigate such challenges. See Supplemental Digital Appendix 3 (at http://links.lww.com/ACADMED/B100) for technical tips and best practices that we compiled based on our experience with this and 2 other large-scale, interactive, virtual ALiEM Connect events.

The COVID-19 pandemic fueled an unprecedented urgency to innovate with new and potentially improved strategies for residency education. The ALiEM Connect model could potentially be used to replace institutional or professional society educational conferences that have been canceled or to implement and/or augment a large-scale, shared curriculum among residency programs in the future.

In summary, our virtual conference program offered a solution to addressing an immediate gap in residency education due to COVID-19 restrictions; however, some demand for a national residency didactic program likely existed already. Residency education often occurs in silos worldwide, leading to redundant and substantial financial and opportunity costs. What was once a dream of medical educators—a high-quality, engaging didactic conference viewed simultaneously by residents across the nation—is now achievable.

Acknowledgments:

The authors would like to acknowledge the hard work of all the educators who have risen to the occasion to provide virtual or remote education during the COVID-19 pandemic, especially those who contributed their time, energy, and money to ensuring that Free Open Access Medical education (FOAM) continued to be possible. Thank you also to Dr. Michelle Lin and ALiEM (Academic Life in Emergency Medicine), LLC, for their permission to use their various logos in the exhibits of this report.