Implementing Value-Added Medical Education: Lessons Learned From the Student-Initiated Stanford Frontline COVID-19 Consult Service.

PROBLEM: Value-added medical education (VAME) has been difficult to implement due to student and educator constraints. The COVID-19 pandemic caused mass transitions to online learning, removed students from clinical settings, and underscored students' desires for meaningful VAME opportunities. The authors introduced the Stanford Frontline COVID-19 Consult Service (SFCS), through which off-service medical and physician assistant (PA) students provided assistance to clinicians in the form of rapid research regarding COVID-19 clinical questions. APPROACH: The SFCS, a student-derived VAME initiative, was implemented from March to May 2020 by Stanford University medical students, PA students, and faculty. SFCS aligned with not only the interests of clinicians and students but also national accreditation standards. Students attended weekly editorial meetings, didactic sessions on literature reviews and information management, and they underwent rigorous training on the peer review process. After 2 months, the authors expanded the service to local community clinicians. OUTCOMES: The SFCS enrolled 16 students, was supported by 13 faculty members, and produced 87 peer-reviewed evidence syntheses. Of the 16 SFCS students, 13 (81%) completed evaluations; of 128 Stanford Primary Care and Population Health clinicians, 48 (38%) completed evaluations. Overall student satisfaction with the SFCS was 4.9/5 (standard deviation [SD] 0.3). Self-assessed achievement of SFCS learning objectives exceeded 90% for all objectives. Overall faculty satisfaction with the SFCS was 4.4/5 (SD 0.8). Most faculty (40/46 [87%]) planned to use the database to answer future COVID-19 questions. NEXT STEPS: The SFCS is a novel, student-initiated VAME curriculum focused on increasing students' meaningful contributions to patient care. The authors will track SFCS students throughout their clerkships to gauge clerkship performance/preparedness, and they will develop training for integrating VAME into preclerkship curricula at other institutions. Given its adaptive, student-driven design, the VAME framework used to develop the SFCS empowers students to create their own personalized, experiential learning.

Problem

Implementing “value-added” medical education (VAME)—that is, experiential learning through which students meaningfully add capacity to patient care—has been challenging from both a student and educator perspective. [1] VAME asks full-time medical students to adopt innovative service roles on top of completing their existing coursework. [2] Similarly, VAME asks busy medical educators to develop new mentorship roles to help students more directly contribute to patient care. [2] Moreover, traditional VAME relies on educators preidentifying student learning objectives, which may not align with student interests. [3] Given these challenges, published descriptions of successful VAME programs in U.S. and Canadian medical curricula are scarce. [4] However, when, early in the COVID-19 pandemic, clerkship students were removed from frontline clinical care and all didactic curricula moved online, we identified a unique opportunity to develop adaptive, student-driven VAME programs.

We realized, based on our own experiences and conversations with fellow students, that students desired service experiences that meaningfully influenced patient care, especially during the pandemic. Meanwhile, frontline clinicians (physicians with MDs and DOs, advanced practice providers) had unanswered patient-driven questions related to COVID-19 and were hard-pressed to keep up with the rapidly evolving body of peer-reviewed and preprint literature, the barrage of information from the press, and guidance from governmental agencies. We, therefore, created the Stanford Frontline COVID-19 Consult Service (SFCS) to meet the needs of not only students seeking value-added service learning but also frontline clinicians requiring vetted, accurate COVID-19 information.

In this report, we share the development process and lessons learned from this student-driven initiative, which we hope will inform future VAME activities both at Stanford and elsewhere.

Approach

Early in the pandemic, faculty within the Stanford Division of Primary Care and Population Health (PCPH) organized COVID-19 Daily Digest emails to help the community process the enormous body of COVID-19 information; however, faculty providing care to patients with COVID-19 still had critical unanswered questions about the disease. These frontline providers were challenged to keep up with the primary literature and information from additional sources. Six Stanford clerkship medical students (including B.Z., C.C., T.J., S.C., and T.R.), responding to a call for volunteers issued by PCPH faculty, formed the Stanford Frontline COVID-19 Consult Service (SFCS). Using a student-driven VAME framework and leveraging their own interests and skill sets, students independently designed the SFCS process, including its integration with clinical work; developed the SFCS curriculum; and created an evaluation plan—all of which were then enhanced through faculty mentorship.

SFCS is a means to provide succinct syntheses of evidence to answer COVID-19-related clinical questions from primary care clinicians. Using snowball recruitment, we recruited 16 first-, second-, and third-year Stanford medical students and physician assistant (PA) students and 13 volunteer faculty mentors to staff the consult service. We assigned students to 1 of 7 teams, each responsible for questions within the following topic areas:

Clinical Trials and Management;

Epidemiology, Risk, and Spread;

Healthcare Worker Safety and Prevention;

In the News;

Testing;

Symptoms; and

Vaccines.

Evidence syntheses and recommendations were peer reviewed before release.

Competency alignment

Because students were pulled from direct patient care activities, we were concerned that they might not be able to achieve the competencies required for graduation. We therefore aligned the SFCS service-learning objectives with the 2020 Liaison Committee on Medical Education (LCME) accreditation standards [5] (Chart 1).

Chart 1

SFCS Learning Objectives Mapped to LCME Standards

Curriculum for evidence synthesis

Prior studies of teaching evidence-based medicine (EBM) suggest that librarian-supported curricula that blend experiential, problem-based learning with online didactics are effective for increasing learners’ competency in EBM. [6] Using these principles and information on online curricula [7] to guide us, we designed online SFCS curricular elements consisting of virtual didactic sessions, weekly editorial meetings, faculty feedback, and a peer review process (see below). Students participated in two, 30-minute virtual didactic sessions to learn how to source and evaluate evidence. These sessions, codeveloped with Stanford Lane Medical Library Information Management Services, covered commonly used and COVID-19-specific resources, search strategies, frameworks for evaluating evidence, and citation management tools. Students practiced navigating databases and managing sources during the sessions and were encouraged to engage the librarians with further questions. Students also met weekly as a group with core faculty to share best practices, identify ways to improve SFCS, and share feedback about the service. Lastly, students coproduced a template for generating succinct evidence syntheses with 3 features: top-level recommendations for care, a summary paragraph of key findings, and a detailed, referenced evidence summary (see Supplemental Digital Appendix 1 at http://links.lww.com/ACADMED/B113).

Peer review

Frontline primary care clinicians used SFCS evidence syntheses to directly inform patient care. To ensure the SFCS’s quality, accuracy, and relevance for outpatients, we implemented a peer review process involving students and faculty alike. Two student editors-in-chief (EICs) daily reviewed questions submitted by clinicians and distributed each question to the appropriate topic lead (TL). TLs assigned each question to a student team member while also reviewing the literature daily in an effort to anticipate updates that might be required and/or questions from faculty that might arise. One or 2 of the recruited faculty members supported each topic team, providing clinical feedback on each evidence synthesis the team produced. We also assembled a faculty consensus panel to advise teams on challenging questions. Once completed, evidence syntheses were reviewed by the TL and EIC before they were uploaded to an external database for clinician access.

Clinical integration

We designed the SFCS closely around existing primary care workflows to ease use (Figure 1). First, clinicians submitted questions through a Qualtrics (vMay2020; Qualtrics, Provo, Utah) survey link embedded in the COVID-19 Daily Digest emails and available through the PCPH COVID-19 portal. Students uploaded their syntheses as PDFs into externally viewable file-sharing folders, so clinicians could easily access them on their cell phones. The next Daily Digest featured recent, high-impact evidence syntheses. If clinicians submitted questions that the SFCS had already addressed, they would receive personalized answers with links to the appropriate syntheses.

Figure 1

The Stanford Frontline COVID-19 Consult Service Process: Clinical question generation and peer review. The authors developed this process over 8 weeks (March through May 2020) by incorporating feedback and adapting to the changing needs of students and clinicians. If needed, faculty referred challenging questions to a consensus panel of senior faculty members via email. Abbreviation: EIC, editor-in-chief. aEvidence syntheses developed in response to clinician questions were stored on a publicly accessible Smartsheets (Bellevue, Washington) database. bFaculty feedback occurred via email and during weekly Zoom (San Jose, California) calls. EIC communication with topic leads and authors occurred via email, weekly Zoom calls, and Slack (San Francisco, California) instant messaging platform.

Expansions to community clinics

After 8 weeks, we had standardized our consult service structure and peer review process, and we had integrated early clinician feedback into our work. In response to emerging data about communities of color being disproportionately affected by COVID-19, existing SFCS team members crafted responses specific to vulnerable populations and expanded the SFCS to 9 local Federally Qualified Health Centers. These students followed updated epidemiology reports and clarified the availability of local community resources for members of marginalized populations. We concurrently expanded the consult service to Stanford-affiliated community clinicians.

Outcomes

From March 20 to May 29, 2020, 16 off-service Stanford medical and PA students and 13 faculty mentors wrote 87 evidence-based syntheses in response to questions from frontline clinical care providers. Each student spent 3 to 4 hours per evidence synthesis, each of which averaged 2 pages in length. EICs each spent 5 hours per week distributing and reviewing the syntheses.

Students were surveyed at 8 weeks regarding their satisfaction with SFCS and their self-assessed attainment of the learning objectives. Responses were based on a traditional 5-point Likert scale (where 1 = strongly disagree and 5 = strongly agree). We surveyed PCPH faculty during a retreat to evaluate SFCS’s efficacy and integration into their daily workflow (see Supplemental Digital Appendixes 2 and 3 at http://links.lww.com/ACADMED/B113). All data were analyzed using Qualtrics (vMay2020; Qualtrics, Provo, Utah). Because students constitute a protected class, we sought approval to survey them from the Stanford Institutional Review Board (IRB). The evaluation was approved by the Stanford IRB (protocol 56112), the Student Data Oversight Committee, and the COVID-19 Clinical Research Committee. The faculty evaluation was deemed not human subjects research (protocol 56747).

Student outcomes

Of the 16 SFCS students, 13 (81%) completed evaluations. Of these 13, 10 (77%) identified as female, 3 (23%) as male, 9 (69%) were medical students, and 4 (31%) were PA students. Overall student satisfaction with the SFCS process was 4.9/5 (standard deviation [SD] 0.3). All students agreed that SFCS participation enhanced their literature review skills: 10/13 (77%) strongly agreed and 3/13 (23%) somewhat agreed. Specifically, all students (13/13) found the peer review process helpful, and 11 of 12 (92%) students found the didactic sessions helpful. Almost all the students (12 of 13; 92%) reported achievement of each SFCS learning objective. Lastly, most students (10 of 11; 91%) agreed that participation in SFCS helped prepare them for clerkships.

Primary care clinician outcomes

Of 128 PCPH clinicians, 48 (38%) responded. Of these 48, 43 (90%) were an MD or DO, 2 (4%) were PAs, 2 (4%) were nurse practitioners, and 1 (2%) identified as “other.” Of the 48, 35 (73%) identified as female, 11 (23%) as male, and 2 (4%) either identified as gender nonconforming or preferred not to answer. Respondents’ overall satisfaction with the SFCS’s utility for patient care was 4.4/5 (SD 0.8). Of 43 responding to the relevant question, over half (n = 24; 56%) were very satisfied and 12 (28%) were satisfied. In addition, the average satisfaction with information quality was 4.3/5 (SD 0.7). Of the 45 who responded to the question, 21 (47%) were very satisfied, 17 (38%) were satisfied, 7 (16%) were neither satisfied nor dissatisfied, and 0 were unsatisfied with the quality of the information provided. Of the 48 answering the relevant question, the majority (n = 34 [71%]) reported using the SFCS syntheses frequently (n = 10), often (n = 12), or sometimes (n = 12) over the past 2 months. Additionally, of the 46 faculty members responding to the relevant question, most (n = 40 [87%]) planned to use the database frequently (11), often (9), or sometimes (20) for answering future COVID-19 questions.

Next Steps

SFCS, which allows medical and PA students to meaningfully contribute to patient care through experiential learning, is a promising example of VAME. We rapidly created and refined a service to provide peer-reviewed, evidence-based answers to frontline clinicians’ questions. In 2 months, we answered 87 clinical questions in 7 topic areas. This service aligned with national accreditation standards, evolved based on student interests and population health needs, and was carefully integrated into the clinician workflow. We found that program participation allowed students to engage with frontline patient care and improved students’ confidence in their ability to perform clinical literature reviews for future clerkships. Faculty were satisfied with the quality of the consult service and planned to continue using the evidence syntheses in their patient care.

The SFCS has illustrated how students can codirect their own curricular development (Figure 2). Using a VAME framework, students directly saw a clinician need and then, along with faculty mentors, developed an innovative, adaptive curriculum to meet that need. The curriculum aligns with national accreditation board [5] and service-learning standards.

Figure 2

Updated conceptual framework for VAME: Iterative student-driven curriculum development and implementation. The authors developed this VAME framework based on their experience with the Stanford Frontline COVID-19 Consult Service, or SFCS (March–May 2020). The initial, student-driven SFCS patient objectives and curricular design were enhanced by faculty input for clinical integration and evaluation, representing the first cycle of coproduction. After 8 weeks of operation, the SFCS’s expansion to local community clinics represents a full iteration of this cycle, at the end of which students reassessed the SFCS’s patient-centered objectives with health disparities in mind to create an externally facing database and community provider consult service. Abbreviations: VAME, value-added medical education; SFCS, Stanford Frontline COVID-19 Consult Service.

Even in the absence of a global pandemic, medical students and PA students completing clerkships are called upon to update their residents and attendings on the latest clinical research during team rounds. By answering actual clinical questions, the SFCS prepared students to create focused evidence syntheses regarding a variety of novel topics immediately relevant to patient care. We plan to follow student participants throughout their clerkships to understand how participating in SFCS has influenced their approach to patient care, their ability to conduct literature reviews, their perceptions of VAME, and their interests in PCPH. We are now working with Stanford preclerkship curriculum design leaders to incorporate the SFCS research model into existing EBM curricula. The virtual and interactive format of the SFCS allows for flexible integration into existing clinical reasoning courses to prepare students for clerkships and their future careers.

We have reported on a single-institution innovation implemented during an unprecedented time of crisis, which may limit program generalizability. As other courses integrate SFCS, more outcomes data will become available. All participants were self-selecting, but each entered the SFCS with different levels of experience conducting literature reviews; further data on SFCS students’ experiences during their clerkships and assessments of future SFCS students will provide information on how the service influences skills. Although we have developed a standardized curriculum to train participants to independently evaluate information and assemble high-quality evidence syntheses, each synthesis still requires faculty review because of its direct influence on patient care. The structure of the SFCS was developed iteratively to meet an urgent, well-recognized clinical need. As such, we did not undertake a formal stakeholder needs assessment or preprogram survey before launch. Given this time frame, we also did not formally align SFCS with the Association of American Medical College (AAMC) Entrustable Professional Activities (EPAs) but suggest that future student-driven VAME programs be mapped to EPAs. [8] As we look toward a post-pandemic future, we will face sustainability challenges; that is, students will have to balance their participation in SFCS at its current scale with their required board exams and clerkship responsibilities. Finally, to support adoption of sustainable VAME programs such as SFCS at other institutions, we plan to develop training for faculty and students around best practices for VAME development and implementation.

The SFCS curriculum was optimized to support students’ existing expertise, interests, and level of independence. Student-driven VAME empowers students to take ownership of their education and begin honoring their professional responsibilities using their current skills rather than waiting for their medical license. The SFCS, developed through a student-driven VAME framework, shifts the relationship between educator and student from one of unidirectional oversight and, instead, suggests how motivated students and educators might collaboratively develop innovative learning objectives, instructional materials, and assessments that align with student interests and talents while also satisfying LCME accreditation standards and AAMC core EPAs.

We hope that the SFCS, borne out of necessity, will continue to inspire students to meaningfully engage with patient care and effect radical changes in the design and implementation of VAME programs long after the COVID-19 pandemic ends.

Acknowledgments:

The authors gratefully acknowledge their colleagues for their assistance in implementing and disseminating the Stanford Frontline COVID-19 Clinical Consult Service: Christopher Stave, MLS, Sang-ick Chang, MD, and Kirsti Weng, MD. They also want to thank the following (alphabetically): Tasnim Ahmed; Gabriela Ruiz Colon; Nancy Cuan, MD; Alexis Doyle; Elizabeth Greyber, MD; Vivian Ho; Kathleen Kenny, MD; Noemi Levy; Grace Li; Bryant Lin, MD; Steven Lin, MD; Karl Lorenz, MD; Justin Lotfi, MD; Kara Lubieniecki; Kim Osborn, MPA; Danielle Pedersen; Alexandra Ross; Amelia Sattler, MD; Ashley Stevenson; Abd Al-Rahman Traboulsi; and Songan Wang.