Robust Institutional Support and Collaboration Between Summer Training Programs in Cancer and Biomedicine Drive the Pivot to a Virtual Format in Response to the COVID Pandemic.

Summer internships serve important roles in training the next generation of biomedical researchers and healthcare providers through laboratory and clinical experiences that excite trainees about these fields and help them make informed decisions about career paths. The SARS-CoV-2 (COVID) pandemic and associated physical distancing restrictions precluded implementation of traditional in-person summer curricula and led to the cancellation of many internships across the USA. COVID-related disruptions also created opportunities for trainees to engage in remote research, become proficient in online learning platforms, and explore multidisciplinary topics. These skills are highly relevant to trainees as virtual interfaces occupy an increasingly mainstream role in their professional paths. The response to the COVID pandemic required real-time adaptations at all levels for major biomedical institutions including the University of Maryland Baltimore (UMB). Pivoting summer programs to a virtual format as part of this response provided a "teachable moment" to expose trainees to the innovation and resilience that are essential components of the biomedical profession. UMB summer programs, which span diverse biomedical disciplines from cancer research to diabetes, consolidated resources and identified mentors with online research projects to develop a robust virtual curriculum. Herein, data from a cancer-focused internship illustrate the collaborative adaptations to established components and creation of new learning modules in the transition to, and implementation of, online training. Outcomes are presented in the context of the COVID pandemic and significant societal issues that arose in the summer of 2020. The utility of virtual components and their impact on future programs is discussed.

Background

UMB Summer Research and Education Programs: Adapting to the COVID Pandemic

A robust pipeline of training supported by University of Maryland Baltimore (UMB) professional schools, research institutes, and medical centers spans middle school through post-graduate education and includes integral summer components (Fig. 1). The outbreak of SARS-CoV-2 virus (COVID) in the spring of 2020 and its subsequent spread as a developing pandemic led to the partial closing of research facilities at UMB and across the country and created unprecedented challenges for the biomedical research enterprise [1-3]. The resulting restricted access to laboratories that are essential for most summer research training precluded implementation of traditional in-person programs and led to the cancellation of most summer training programs [4-6]. Despite these challenges, some institutions, including UMB, utilized opportunities presented by COVID-associated restrictions to adapt existing components to an effective online platform and develop completely novel virtual projects and program elements [5-9]. All aspects of UMB summer programming including financial support, research and clinical training options, mentor-trainee interactions, education modules, and networking forums were affected by the government and institutional regulations imposed in response to the rapidly emerging pandemic. Therefore, UMB program leaders evaluated the collective impact of COVID-precipitated changes on the capacity to adapt and implement each program to an online format as a critical determinant in the decision to offer summer training. Specific steps and factors considered by all programs in this process are shown through the example of the Nathan Schnaper Intern Program in Translational Cancer Research (NSIP) in Fig. 2A [10]. In response to these challenges, strong institutional support, dedicated faculty mentors, innovative inter-programmatic collaboration, and engaged trainees combined to create a rich virtual curriculum for all UMB summer programs. A timeline of summer program development and milestones in the context of new COVID cases in Maryland illustrates the rapid adaptation of UMB resources in response to COVID-associated regulations (Fig. 2B). This report focuses on programs that are implemented exclusively during the summer in conjunction with full-time training programs (Fig. 1). The modification of existing training components and creation of novel research and education modules to accommodate an online platform are described using outcomes from the cancer-focused NSIP to highlight the impact of these changes on the full profile of UMB summer programs. The expanded training approaches and content scope that resulted from a COVID-adapted curriculum, and how these new components may enhance future in-person programs, are discussed.

Fig. 1

UMB training programs. All programs include summer components and contributed to the COVID-adapted virtual curriculum; a red asterisk (top) and bold font (bottom) denote programs conducted exclusively during the summer. Program acronyms are CURE, Continuing Umbrella of Research Experiences; NSIP, Nathan Schnaper Intern Program; ASCEND, A Student-Centered Entrepreneurship Development training model; STAR-PREP, Science Training for Advancing biomedical Research-Postbaccalaureate Research Education Program; PRISM, Program for Research Initiated by Students and Mentors; and SPORT, Summer Program in Obesity, nutrition and diabetes Research Training. Photos (from left) show trainees in UMB CURE, NSIP, STAR-PREP, Bridges to the Doctorate, and Cancer Biology programs

Fig. 2

Steps in the development of a virtual summer curriculum. A Decision tree for program transition to online training illustrated with data from the NSIP; a similar process was followed by all programs. B Timeline of new SARS-CoV-2 cases in Maryland (y-axis) (https://usafacts.org) and regulatory actions (yellow shaded text), virtual summer program development (red shaded text), and key UMB COVID responses (blue shaded text)

Methods

Study Participants

Trainee Recruitment and Selection

The composition of different summer training programs that are administered through the University of Maryland School of Medicine (UMSOM) Office of Student Research (OSR) is shown in Table 1. Trainees for the different programs are recruited from institutional (PRISM), local (UM Scholars), and national (NSIP, SPORT) applicant pools through program websites, presentations by program leadership and representation at national meetings (e.g., International Cancer Education Conference, Annual Biomedical Research Conference for Minority Students). Internship applications are reviewed for program-specific criteria by a selection committee composed of program leadership and mentors. Each application is reviewed by multiple committee members and scored on a 1–9 scale (1 = best) based on short answer questions, a personal statement, and letters of recommendation. A genuine interest in biomedical research, rather than extensive prior experience, is a focus of applicant evaluation in all programs. This approach ensures that talented students from diverse backgrounds who have not had the opportunity or resources to conduct research in their training to date have access to UMB  programs. The committee considers the applicant’s interest in biomedical research and how the training experience will advance their educational and career goals as driving criteria for admission. All training with human participants (i.e., trainees) is performed in accordance with the ethical standards of our institution and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Trainee participation in OSR programs does not require formal consent and is not subject to approval by our Institutional Review Board. No personal identifying information is presented in this report; only summary data are analyzed.

Table 1

The 2020 OSR summer program participants

Program1	Period	Trainees			Mentors
		Total	Female	URM	Total	Female	URM
NSIP	2020	18	89%	17%	13	69%	8%
	avg/yr2	21	79%	27%	21	38%	9%
UM scholars	2020	16	38%	38%	11	44%	6%
	avg/yr	14	52%	17%	12	41%	19%
PRISM	2020	56	57%	2%	42	46%	0%
	avg/yr	38	58%	4%	30	41%	5%
SPORT	2020	12	92%	25%	13	31%	8%
	avg/yr	12	67%	25%	9	56%	11%

1Data are for participants recruited for OSR programs that are fully implemented during the summer. Data on participants in year-round OSR programs are not shown

2Data from 4-year averages are shown (2015–2019). NSIP Nathan Schnaper Intern Program; UM Scholars University of Maryland Scholars; PRISM Program for Research Initiated by Students and Mentors; SPORT Summer Program in Obesity, nutrition and diabetes Research Training

Mentors

Mentors are faculty from UMB professional schools, primarily in the UMSOM, and partner institutions. Mentors sign a formal agreement articulating their responsibilities that include guiding the research project, meeting regularly to review progress, assisting in the preparation of intern presentations, attending program events, and evaluation of program components.

Outcome Evaluation

Evaluation of trainee learning and content acquisition was performed by pre- and post-activity surveys using a 5-point Likert scale in which 1 = Excellent, 2 = Very Good, 3 = Good, 4 = Fair, and 5 = Poor (unless otherwise defined in the figure legends). Surveys were administered online and via a smart phone application. The percent increase in content acquisition over baseline (pre-survey) Likert values was calculated using the formula (post-mean − pre-mean/pre mean) × 100. The statistical significance of the differences in pre- and post-survey mean values was determined using a two-tailed unpaired t-test.

Results

Institutional, Government, and Private Sources Combine to Provide Strong Support for a Pivot to Virtual Summer Programming

UMB summer programs are supported by institutional, government, and private sources and serve trainees that span a continuum of educational levels (Fig. 1). These programs provide a corresponding array of experiences including academic enrichment, career path exploration, research, and clinical exposure that are important for trainee educational and career advancement. COVID-associated cancellation of most summer programs across the USA made the need for training that could be implemented in a virtual setting particularly acute [4-9]. Indeed, by the time COVID-related closings began, most OSR summer programs had reviewed and accepted full cohorts of trainees who were anxious to secure summer training amidst a backdrop of dwindling opportunities. Initial impetus towards an online platform was provided by the UMSOM, as medical students conduct independent research as part of their degree requirements and rely on summer research to complete this experience within a 4-year window. To provide this opportunity, the UMSOM committed to adapt medical student research programs to a virtual format through the OSR that serves as an administrative umbrella for all summer programs. This institutional support opened the door for other programs to follow suit and contribute resources to a robust developing virtual curriculum. The OSR-led PRISM (medical students) and state of Maryland-sponsored UM Scholars undergraduate programs were among the first to “opt in” to the OSR virtual summer platform. Eight additional programs comprise the OSR portfolio and are supported by NIH grants; many of which (e.g., SPORT, STAR-PREP, and CURE Connections) approved the development of modified curricula that would accommodate COVID restrictions. Support for other NIH-funded programs, in which hands-on research is integral to training experiences and required for program implementation (NSIP and B2D), was deferred. Private and philanthropic resources provided key mechanisms to fill the gap in support for programs with admitted students, willing mentors, and virtual projects in which NIH funds were deferred. In particular, the cancer-focused NSIP receives partial support from donations by colleagues and former patients of the late Dr. Nathan Schnaper, the program’s namesake. The Schnaper Fund supported a cohort of 18 interns in 2020 which, in turn, fueled community enthusiasm for the program that resulted in an increase in donor contributions. Independent donations from UMSOM faculty provided further private support for other OSR summer programs. Together, this financial support and institutional commitment provided an essential foundationto leverage additional resources and build a strong virtual program.

Virtual Program Participants: Recruitment of Students and Mentors

Admission to UMB summer programs is competitive with trainee recruitment occurring in the fall preceding summer program implementation. Most application deadlines are set by the end of the year, and cohorts are selected by February in each program year. Accordingly, COVID did not impact trainee recruitment as a full contingent of accepted trainees was already in place at the time of the outbreak and did not differ markedly from participant demographics seen in typical summers (Table 1). However, the number of UMSOM medical students in the PRISM and Radiation Oncology programs was markedly increased in 2020 reflecting the combined impact of an increased demand for summer experiences in light of COVID-associated cancellations of research programs at most other institutions and an increased capacity of SOM faculty to mentor multiple students in a virtual format (Table 1). Upon learning that programs would be conducted in a virtual format, over 90% of accepted trainees remained committed to participation across all programs. The pool of 2020 summer mentors was also largely established pre-pandemic; however, most of the proposed projects involved in-person laboratory research and were not readily adaptable to an online platform. The transition to a virtual format thus potentially limited the projects available and corresponding number of trainees able to participate in summer programs. A tripartite mentor response effectively addressed this need for an increased selection of virtual projects and allowed the development of an online curriculum to move forward. Specifically, some mentors had projects, such as those incorporating in silico database analysis, that were easily transitioned to a remote research platform and provided the first source of projects. A second category of projects had the potential for a robust online component but required some level of prior experience by trainees (e.g., programming, statistical analysis, bioinformatics). To fully utilize this subset of mentors, trainees with these skillsets were identified and matched with corresponding mentors. In addition, new education modules on bioinformatics, “R” platform and Python were developed for trainees to gain or enhance relevant skills in real time, and directly apply their learning to projects requiring this background. The third source of projects to fill the need for virtual research experiences came from new mentors who stepped up as part of a campus-wide effort to sustain summer programming. Mentors in all summer programs were distributed among these categories; for example, in the NSIP just 14% of mentors were not able to offer projects amenable to a virtual setting, 72% had projects that could be directly converted to a virtual format, and 14% were able to offer online projects to students who received new skills training (Fig. 2A). Notably, virtual NSIP projects aligned with a programmatic goal of increasing the scope of trainee cancer research beyond laboratory investigation and led to an expansion of the mentor pool in which 62% of mentors were new to the program (Table 1). Through the concerted efforts and enthusiastic support of the UMB academic community, over 90% of accepted summer program trainees were able to gain this research experience in the face of challenges posed by the COVID pandemic.

Building a Virtual Module Menu Through Inter-programmatic OSR Collaborations

In addition to trainee research, each of the OSR programs includes a rich curriculum comprised of program-specific seminars, workshops, and journal clubs as well as joint, OSR-wide educational components. To provide these modules in a virtual setting, resources from all OSR programs were combined to develop a robust menu of online educational experiences that were open to trainees from all programs (Table 2). A comprehensive central schedule was generated using the Teamup application to provide trainees with information on each session that included the topic, title, reading material, PowerPoint presentation, links to the synchronous Zoom presentation, and post-session evaluation survey (Fig. 3). Each program identified the modules that were required and optional for their trainees based on specific thematic goals (e.g., cancer-focused components for NSIP), and these were color coded to facilitate schedule navigation and use. This open access to an array of virtual learning resources combined with the inherent flexibility of online research provided the opportunity for trainees to explore interdisciplinary subjects and training as a collateral benefit of the virtual setting. This level of extra-programmatic participation is difficult, if not impossible, in the context of a traditional in-person summer format in which schedules are restricted by real-time laboratory research. In fact, the OSR open access menu increased the number of optional training by 8–12 modules depending on the program. These joint sessions were well attended by trainees; therefore, synchronous and recorded Zoom sessions will be retained in future summers as mechanisms to augment in-person learning. While some aspects of the traditional summer curricula were amenable to virtual implementation, others involved essential in-person aspects that could not be replicated online and were either modified or deferred until traditional summer programming is resumed. For example, clinical exposure, which is an integral component of several programs and was prohibited by COVID restrictions, was replaced by trainee-organized physician panel discussions to provide insights into diverse medical specialties. Beyond the online versions of existing curriculum components contributed by each OSR program, new workshops and nanocourses were developed to provide interns with the skills required for specific online projects. For example, faculty from the UM Institute for Genome Science (IGS) developed a nanocourse with sections on bioinformatics, “R” platform, Python, and Biostatistics. Furthermore, a NSIP mentor from the Microbiology and Immunology faculty organized a new Immunology Overview nanocourse given by advanced immunology graduate students (see “7” section). The real-time application of skills gained in these modules to research projects fostered intern understanding and retention of fundamental concepts as an added benefit of this format. The virtual curriculum retained traditional OSR components that punctuate the summer and bring together trainees from all programs in orientation,mid-summer retreat, and final research presentation forums (Table 2). These events were readily converted to a virtual format and utilized online breakout rooms to provide the small group interactions that facilitate community building and networking as a key outcome. The trainee presentations that are the culmination of summer research were reduced to 10 min (6-min presentations with 4 min for questions) and spread over 3 days to minimize online fatigue. Trainees benefitted from synchronous science communication workshops and numerous online resources that provided high-impact/short-duration content. A full comparison of components offered during in-person and virtual program settings is shown using the NSIP as an example in Table 2 and illustrates the contributions of new and modified modules to the online curriculum. Other programs utilized a similar schedule that differed primarily in the designation of required and optional modules.

Table 2

Robust curricula augment summer trainee research in traditional and virtual settings*

*Curriculum shown is for the NSIP with required components in bold; curriculum for other OSR programs is similar but differs in modules required. Module colors designate formats in which they are offered and identify new components developed for 2020: green — traditional or both formats, blue — virtual format, yellow — new component developed for virtual format, red — not offered in format listed

Fig. 3

An online module menu coordinates schedules for all summer programs

Effective Mentoring, Developing a Sense of Community, and Fostering Trainee Wellness in a Virtual Environment

The regular interactions between mentors and trainees, as well as community building among interns within and between OSR programs, are integral components of the summer experience. Indeed, for most trainees the relationships formed through these connections extend well beyond the summer and provide the foundation of a developing professional network that is a highly impactful program outcome. In traditional in-person curricula, trainee interactions occur in a series of seminars, workshops, forums, and social functions over the course of the summer and serve to develop a strong sense of community within the group. Beyond promoting cohort cohesiveness, these structured events provide a daily and weekly framework that facilitates the development of time management skills and function as milestones for trainees to monitor their progress. While the content of many of these events was readily transitioned to a virtual format, replicating the interpersonal interactions that are central to effective community building in an online setting required novel approaches and mechanisms [5, 6, 9]. To address this challenge, we took a multifaceted approach in which new online components focused on mentoring, trainee interactions, and holistic wellness. These virtual mechanisms are described below and complemented the traditional approaches resulting in more in-depth interactions that will be continued as part of future in-person summer curricula.

Mentor-Trainee Interactions

The development and support of mentor-trainee relationships is integral to program success and particularly important in a virtual setting as documented in studies of mentoring outcomes during the pandemic [11-13]. To enhance this critical channel of communication, mentors connected with interns on a daily basis to review progress, answer questions, and discuss problems. The length of these meetings varied depending on the stage of the research project; more important than duration, however, was their consistency that added critical structure to an online research environment. A post-doctoral fellow or senior-levelstudent working closely with the trainee often served as an alternate daily point of contact. These interactions, as well as virtual lab meetings, served to enhance the trainee’s sense of inclusion through direct connections with members of the research team. This team-building component of this relationship has emerged as an important aspect of effective mentoring in a virtual setting [11-13]. An increased focus on mentor interactions was successful in enhancing mentor-trainee relationships as exemplified in trainee survey comments such as “I have continued working with my mentor after the program ended and he just submitted our paper. Working with him has been wonderful — [he is] a fantastic mentor” (also see “13” section). A unique aspect of the virtual setting is that trainees do not utilize laboratory space; the absence of this physical requirement allows faculty with sufficient time and appropriate projects to mentor more than one trainee. In fact, 36% mentors across all 2020 OSR programs mentored more than one trainee. This training strategy can involve trainees from one or multiple programs and offers benefits for both the mentor and trainees that are particularly relevant in a virtual environment [13]. Specifically, trainees may work as a team on complementary aspects of one project or may have individual projects that are related to a larger research question. In either case, project coordination and potential interdependence increases trainee interactions, hones communication skills, and broadens their scope of engagement. Beyond interactions with their individual mentors, trainees connected with mentors in multiple modules throughout the summer (Table 2), beginning with a “Meet the Mentors” session, continuing with panel discussions and seminars and concluding with trainee research presentations. These interactions provide trainees with an informative perspective of diverse faculty research and career paths while gaining an understanding of their roles in the context of UMB structure and organization.

Cohort Cohesion

OSR-wide and program specific components of the traditional curriculum are designed to promote development of the trainee community and were expanded to achieve this goal in a virtual setting. The summer is punctuated by three major OSR events — Orientation, a Mid-summer Retreat, and the Summer Research Forum — that include participants from all summer programs and serve an important community-building function. These events were successfully converted to an online format in 2020. The effective use of Zoom breakout rooms supplemented with “icebreaker” activities provided the small group dynamic that is important to foster informal interactions as part of these events. In addition to OSR-wide events, each program has independent functions tailored to its trainees, research focus, and history. For example, the Bench to Bedside and Backseries on translational cancer research is a signature component of the NSIP that is held each Monday morning and is followed by time for an informal meeting between interns and the program director and coordinator. For the virtual setting, content of this post-meeting time was expanded beyond the standard preview of the week’s schedule to include discussion of wide-ranging topics from weekend activities to current events and popular culture. These regular interactions helped trainees get to know one another while providing a defined start to the week. Some programs (e.g., SPORT) convened trainees 2–3 times each week with the goal of providing a regular informal connection with fellow trainees in the absence of any programmatic agenda. Other programs incorporated trainee interactions into multiple scheduled events each week. Segments in which trainees described their own background and interests, educational goals, and career aspirations were integrated into online modules to both help the speaker tailor the presentation to the audience, and to provide trainees with information that fueled interactions in subsequent sessions. Inclusion of this component particularly benefitted panel discussions with researchers, physicians, and biotech industry leaders; indeed, after just a few sessions, the effectiveness of this format was evidenced by the ability ofmost traineesto introduce and provide a brief bio for anyone else in their cohort. Trainee feedback reflected the collective success of these approaches to facilitate the pivot to a virtual format:

“Thank you for creating a very smooth transition [to a virtual format].”

“I liked that there was an interactive component to the talk—it really helped with engagement.”

Intern Wellness Components and Resources

New and existing approaches promoted online trainee engagement in group settings that included inter-programmatic, program-specific, or research group contexts. However, COVID restrictions necessitated an increase in research and education components that were performed independently and, in turn, presented a trainee-specific set of challenges that reflected individual learning styles and prior experience in a virtual research environment [6, 12]. To address such issues that may not be evident in group settings, program directors and coordinators scheduled a series of one-on-one meetings in which any aspect of the program, the trainee’s educational path, or other concerns could be discussed in a private setting. The trainees were highly engaged in these sessions and appreciated the dedicated time, opportunity to get to know program leadership, and advice on career trajectory. Beyond program-related topics, UMB offers a robust array of professional online wellness resources to help with broad COVID-related challenges. A spectrum of institutional services, which span online fitness, yoga, and meditation classes at the Fitness Center Wellness Hub, to a COVID Coping Kit from Human Resources and a Mental Health and COVID-19 resource page by the Student Counseling Center, were available to summer trainees. Trainees were connected to these resources as part of orientation with weekly reminders and updates on new offerings.

Outcomes and Evaluation

The transition of OSR summer programs to a virtual format in 2020 impacted all aspects of the curriculum. Evaluation of new components and feedback from participants provided insights into the utility of online platforms to add flexibility and content as a potential mechanism to complement in-person curricula. In addition, comparing outcomes in which identical modules were delivered in-person (2019 programs) and online (2020 programs) provided valuable information on the impact of COVID restrictions on program implementation and trainee engagement. Pre- and post-session surveys used a Likert scale to assess content learning by participants in different modules. Questionnaires and informal discussion with program leaders provided mechanisms to collect feedback on program format, organization, delivery, and trainee-specific components (e.g., research projects and mentoring). Outcomes in three major categories, and how best practices from virtual and in-person formats may inform future programs, are discussed.

Mentored Trainee Research

A virtual research setting necessitated the identification of projects that analyzed existing datasets and included meta-analyses of patient data, epidemiologic studies, bioinformatic analysis, Genome Wide Association Studies, and literature reviews among others. Examples of projects from the different OSR programs are shown in Table 3. The increase in studies using “big datasets” and population-based analysis as compared to “wet lab” research that is typically a mainstay of trainee projects provided a collateral benefit for several programs, including the NSIP, which sought to expand projects in these areas and provide trainees with a broader perspective of cancer research. In support of this goal and to foster the development of online-compatible projects, the new summer IGS nanocourse introduced trainees to statistical analysis and related tools that are central to the success of data-intensive research (see “Educational Modules, Professional Development Workshops, and Research Forums” section).

Table 3

Examples of virtual trainee research projects in OSR summer programs

Program	Research project
NSIP	The impact of ethnicity on single-nucleotide polymorphisms (SNPs) that regulate immune signatures in prostate cancer
	Measuring financial toxicity of cancer in young active-duty military patients
	Protein expression patterns in monolayer and spheroid melanoma cell cultures with acquired resistance to BRaf and MEK1/2 inhibitors
UM Scholars	Producing three-dimensional models of the interactions among semaphorin 4D, plexin B1, and neuropilin 2
	Using induced pluripotent stem cells (iPSC) to model GBA1-associated neurodegeneration in Gaucher and Parkinson’s diseases
	Identifying targets of natural immunity within a protein associated with severe malaria pathogenesis
PRISM	Psychological predictors of virtual reality responsiveness in chronic pain patients
	The impact of intermediate anti-drug antibodies to infliximab and adalimumab on clinical outcomes in patients with Crohn’s disease or ulcerative colitis
	Use of intra-arterial contrast injection in computerized tomography in a porcine model
Radiation Oncology	Proton radiation package time and outcomes in patients undergoing curative treatment for head and neck cancers: a single-institution experience
	PD-L1 expression as a prognostic marker for local control following immune checkpoint inhibition and stereotactic radiation for brain metastases
	Correlating dosimetric parameters with clinical toxicity outcomes between proton and photon-based treatment for gynecological cancers
SPORT	Evaluating how high glucose affects epigenetic mechanisms in diabetic embryopathy
	Examining geographic disparities in obesity and physical activity among children and adolescents living in rural, urban, and suburban areas
	Assessing the validity of estimating resting metabolic rate (RMR) in chronic stroke patients

A primary outcome of trainee research is authorship on research publications and conference abstracts that provides clear evidence of their contributions, strong credentials to support their professional advancement, and is an important metric of program success. In a traditional in-person setting, between 20 and 50% of the trainees co-author publications on their summer research with the exact number depending on many factors including the participants’ level of training, the type and stage of research projects, and prior research experience. Remarkably, trainee research productivity in a remote format was comparable to prior in-person programs with publication rates to date in 2020 between 10 and 33% for the different OSR programs; manuscripts in preparation are expected to further increase those numbers. The minimal impact of a virtual setting on this outcome may reflect the fact that, for most virtual projects, the generation of publishable results was expedited by utilizing existing data rather than having to generate it de novo through laboratory experimentation. Independent of the type of the project, however, trainee research requires dedicated mentoring support; thus, the productivity observed suggests that mentor-trainee engagement was effective and not compromised by interactions that were limited to online platforms. Consistent with this robust trainee output, feedback from both mentors and trainees was positive; for example, over 2/3 of PRISM trainees cited engagement by their mentors as a primary factor in their decision to do summer research. Furthermore, trainees at all levels recognized the value of an immersive virtual experience to their diverse professional paths and did not hesitate to accept an offer to participate in this format [14]. Trainees on a MD track honed remote communication skills that are essential for effective telemedicine — something many had experienced from a patient’s perspective during the pandemic. Research-aspiring trainees directly experienced the essential nature of access to remote datasets in conducting their research as well as the integral role of online collaborators in making these connections. Mentor enthusiasm to provide these virtual research experiences was evidenced by the marked increase in new mentors; 62% of 2020 NSIP mentors were first time participants who “stepped up” as part of the robust campus-wide support for virtual summer programs. Importantly, 100% of new mentors across all programs plan to continue their participation in future programs providing evidence of their satisfaction with program implementation and trainee productivity.

Educational Modules, Professional Development Workshops, and Research Forums

The development of online research projects increased schedule flexibility which, in turn, enhanced opportunities for trainees to access virtual educational modules via an internet-based scheduling platform (Fig. 3). Compared to traditional summer programs in which trainees primarily attend modules hosted by their specific program, the online format led to increased inter-programmatic participation, with trainees from multiple programs attending the different modules. For example, NSIP Bench to Bedside and Back (B2B) seminars, the IGS nanocourse, and the Immunology Overview nanocourse were all attended by trainees from 80% or more of the summer programs. The interdisciplinary learning fostered by this open format will be continued in future in-person curricula through the inclusion of virtual synchronous and asynchronous sessions.

Established program modules were adapted to a virtual format and trainee understanding of the material was measured through pre- and post-module surveys. Trainee understanding was increased over baseline for all modules and reflected effective content delivery as illustrated in data from the B2B translational cancer research educational forum (Fig. 4) and the Scientific Communication professional skills workshop (Fig. 5). In line with this outcome, over 90% of trainees rated faculty presentation delivery and participant engagement in the top two categories of a five-point Likert scale. Trainee acquisition of content was measured by responses to survey questions such as rate your agreement with the statement: I am confident that I can name two hallmarks of cancer and describe how they contribute to oncogenesis. The post-lecture means were consistently positive for virtual modules; however, the exact increase over baseline understanding for a given topic varied and is influenced by a number of factors (e.g., compare patient outreach and hallmarks of cancer topics in Fig. 4). Trainee research experience and coursework are key determinants of baseline subject knowledge and can differ significantly between programs (e.g., undergraduate, post-bac, and grad/medical) and within program cohorts (e.g., junior/senior vs freshman/sophomore undergraduate). This diversity in trainee background can thus impact net changes in comprehension when averaged over all participants as was done in this study. The increased participation of trainees from multiple programs in virtual modules enhanced the heterogeneity in participant experience levels and may have further impacted the mean group change in trainee understanding measured. However, this same diversity in trainee makeup that can act as a confounder in assessing content acquisition for multiprogrammatic settings was offset by its enrichment of these forums through increased near-peer interactions.

Fig. 4

Trainee understanding of B2B module topics was measured in surveys given before and after each session. A Likert scale was used to assess trainee understanding through their responses to two or three topic-specific questions on key aspects of the lectures. The percentage change in the mean class score for each question is depicted by grey, yellow, and orange bars in the graph. The post-module increase in mean scores over baseline was significant (p < 0.01) for all questions

Fig. 5

Trainee increase in scientific communication skills was measured in surveys given before and after virtual and in-person workshops. A Likert scale was used to measure trainee assessment of their skill levels in four areas. The percent change in the mean class score for each question is depicted by orange and blue bars. The post-workshop mean scores significantly increased in all skill categories (p < 0.01); however, the percent increase did not significantly differ between the two implementation formats (p = 0.26)

A comparison of module components in which identical content was delivered in-person (2019) and online (2020) provided the opportunity to compare outcomes in these distinct formats. Interestingly, learning outcomes for both Scientific Communication (Fig. 5) and B2B (Table S1) modules failed to reveal a clear format-dependent difference in trainee understanding. Some topics showed a greater increase in content acquisition in-person whereas others fared better in a virtual setting and some did not change. The reduced distractions in an online setting may enhance trainee focus to, in part, offset the higher level of information uptake expected for a stimulating in-person environment. Furthermore, our data are from short-term learning experiences in which the impact of format on content delivery may be reduced as compared to full-semester courses. Indeed, online fatigue is well documented [15], suggesting that a virtual format is best implemented not as the sole instructional mechanism, but to complement an in-person curriculum.

To increase intern skills that enhance virtual research opportunities and to expand program content, new online modules were developed. As many virtual research projects involved intensive data analysis requiring skillsets that not all trainees possessed, IGS faculty designed a 5-week nanocourse to provide essential training that could be directly applied to their projects. The most widely utilized resources were covered through sections on Bioinformatic programs, “R” platform, Python, and Biostatistics, with more specialized topics addressed via individual consultations. Pre- and post-session surveys demonstrated an increase in trainee competence in these subjects (Fig. 6). Most importantly, over 60% of trainees indicated that the nanocourse provided skills that they used in their summer research projects across all topics covered. Consistent with this outcome, subject material delivery and audience engagement of IGS session leaders received a mean score of 4/5 in trainee surveys (5= best score). Beyond summer research, nanocourse content was valuable in subsequent applications as exemplified in trainee feedback such as “at the beginning of the summer I was intimidated by the ‘R’ program, but I now find it very versatile; it is [now] my go-to application for data representation.”

Fig. 6

Trainee understanding of IGS module topics was measured in surveys given before and after each session. A Likert scale was used to assess trainee understanding through their responses to three topic-specific questions on key aspects of the lectures. The percentage change in the mean class score for each question (depicted by grey, yellow, and orange bars in the graph). The post-module increase in mean scores was significant (p < 0.01) for all questions except Biostats question 1 (p = 0.096)

A second module that was created de novo for the virtual environment broadened the scope of educational offerings with a focus on immunology, a topic that was relevant to academic coursework and integral to many trainee projects, particularly those in the area of tumor immunology. The Overview of Immunology 5-week nanocourse was led by senior graduate students in the Molecular Microbiology and Immunology program and included didactic and small group breakout components covering key basic and clinical aspects of this subject. The nanocourse had strong inter-programmatic participation, with eight programs represented across a training spectrum that spanned undergraduate through postdoctoral education, and an average attendance that exceeded 70 trainees per session. The near peer “mixing” that occurred between trainees at different stages of their education enriched the sessions by providing an informal setting that facilitated discussion of basic concepts and their translation to clinical applications [6, 11]. The effectiveness of the nanocourse is evidenced by strong learning outcomes that are even more striking given that nearly one quarter of the audience had previously taken immunology as part of the medical school curriculum (Fig. 7). The student-led format fostered a collective “all for one/one for all” learning approach and a trainee-centric virtual environment in which all student leaders received high evaluations of 4.6/5 on their engagement and delivery.

Fig. 7

Trainee understanding of immunology topics was measured using the Likert scale defined in the key for post-module surveys. Surveys included the option of “1 — strongly disagree”; however, this score was not selected in any of the surveys. The percent of responses in each category is shown for two questions on each topic

Contextual Components Provide Collateral Benefits as Key Outcomes

Unanticipated events in any given summer provide opportunities to engage trainees on impactful subjects that are relevant to the societal and global issues they will face in their future roles as professional researchers and caregivers. The summer of 2020 witnessed a confluence of high-profile issues including polarizing politics, a viral pandemic, and racial injustice that sparked movements and touched all facets of the population including UMB faculty, staff, and trainees. The decision to offer OSR summer programs in a virtual format allowed trainees to view these pivotal events through the lens of a major biomedical research institution that played an integral role in the national response to these challenges. This experience provided “teachable moments” for the next generation of researchers and caregivers that illustrated the importance of resilience and flexibility as essential traits of the biomedical profession and would not have been possible had programs been cancelled. For example, trainees gained invaluable perspectives of the multifaceted response to the pandemic through synchronous learning modules by front line providers conducting basic SARS-CoV-2 research, infection testing, vaccine development, and patient care at UMB professional schools and medical centers. These forums included panel discussions and journal clubs (Table 3) that were among the most widely attended events of the summer. All UMB pipeline programs integrate components that foster diversity equity and inclusion values in faculty and trainees and promote recruitment of applicants from diverse groups. Indeed, five programs are “diversity-focused” in which all trainees are from UR groups. This culture provided valuable opportunities at UMB and other institutions to expand program scope and address COVID-related issues that particularly impacted UR groups [3, 5, 16] through remote learning [17, 18]. Together, innovative initiatives at OSR and programs across the US embraced a “never let a crisis go to waste” [18] attitude to successfully implement training during the COVID pandemic.

Beyond pandemic-associated public health concerns, societal issues also provided a major contextual backdrop to summer training [3, 15]. Black Lives Matter-inspired actions coincided with summer programming and provided a valuable opportunity for UMB program leaders to create an open space for the campus community to come together, discuss events, and share perspectives. Summer trainees were invited to campus-wide dialogues with the UMB President and to seminars by experts in equity and racial justice. Furthermore, OSR leadership held an “open mic townhall” discussion to mark the Juneteenth Real Independence Day and #Shut Down STEM/#Shut Down Academia events in the context of ongoing racial issues. These forums connected UMB trainees to diverse academic institutions across the country who were also addressing these issues and gave trainees powerful involvement as part of national and international responses to these events [19, 20]. Furthermore, the forums modeled the inclusivity and civility that are central to UMB institutional culture and represent important values to instill in the next generation of biomedical professionals. Trainee appreciation for the value of human interaction and its importance for wellness, especially in a virtual setting, emerged as a key outcome of these gatherings. The opportunity for trainees to voice their anger, sadness, frustrations, and internal conflicts in these forums led program leaders to expand subsequent pre- and post-module open zoom time — as opposed to using a “wait room” mechanism — which proved an effective outlet for continuing trainee discussions. Indeed, feedback indicated that these informal interactions helped to mitigate trainee isolation and anxieties while displaying a powerfully human side of a typically intense R1 training environment. The collective positive impact of curriculum components developed as real-time responses to significant national events reinforced the intangible benefits of summer training programs that extend beyond traditional academic components.

Discussion

Lessons Learned From Adaptations to a Virtual Format in 2020

The transition to remote work and education settings has impacted all facets of our society and identified effective approaches that will be retained in a post-COVID world. This study described the approaches used to transition summer training programs to a virtual format in response to COVID restrictions. The impact of these adaptations on program outcomes utilized data from the cancer-focused NSIP while highlighting common themes that were successful for a profile of UMB summer programs spanning an array of biomedical disciplines. Outcomes from the NSIP illustrate how collaborative learning modules and research forums can enrich the curriculum for all programs. This broader interdisciplinary exposure is particularly relevant to cancer education as malignant disease affects essentially all organ systems. An advantage of the virtual setting that was evident from the outset is the ease with which seminars and research forums can be scheduled using online platforms that, in turn, greatly expanded the opportunities for trainees to participate in extra-programmatic learning. This flexibility also allowed the inclusion of presentations by renowned experts from diverse fields without the costs and travel complications of in-person seminars. Evaluation surveys from specific training modules noted the importance of providing advance reading material to accommodate the different educational backgrounds in a more diverse remote audience. Trainees appreciated the increased mentor interactions that were designed to enhance connections in a virtual environment suggesting that more dedicated mentor time would also benefit an in-person format. Taken together, the feedback gained from a virtual curriculum indicated that online components are best utilized as part of a hybrid format to augment an in-person curriculum and minimize computer fatigue. These findings helped to inform the design of future summer curricula in cancer education and broader biomedical fields.

Impact of Virtual Components on Subsequent Programs: an Update From 2021

The development of effective vaccines against SARS-CoV-2 in record time significantly curbed infection rates and mortality. This positive trajectory led to the approval of in-person laboratory research for 2021 summer trainees. The resumption of hands-on mentored research restored trainee funding from sources that required this in-person component (e.g., NCI) and largely eliminated funding gaps that existed in 2020. Learning modules and research forums that involved large groups of trainees were initially implemented in a virtual format due to continued physical distance and density restrictions necessitated by emerging virus variants. However, an improving public health profile over the course of the summer allowed a progressive easing of regulations and a transition to an in-person format for approximately 80% of NSIP components. The option for trainees to participate in an array of virtual extra-programmatic modules was retained in 2021 as surveys from 2020 trainees indicated that this was a value-added aspect of the online curriculum. In practice, while some virtual modules (e.g., the Health Disparities Journal Club) continued to enjoy significant inter-programmatic participation in 2021, most trainees were excited to return to in person research and utilized any free time to advance their projects rather than on computer-based exploration of interdisciplinary topics. Indeed, after over a year of academic curriculum in a virtual format, this robust embrace of any opportunity for in-person engagement by trainees was to be expected. The enthusiasm for in-person projects included both “wet bench” laboratory work and in-silico studies suggesting that the interpersonal connections gained through on-site projects, rather than the nature of the research per se, is paramount to the trainees’ experiences. This trainee “deficit” in in-person interactions was further addressed through increased time with mentors and opportunities for one-on-one meetings with NSIP leadership — two aspects of 2020 that were retained in 2021. While intern feedback indicated that the expanded in-person contact was appreciated, the value of continued virtual curriculum components, such as alumni and faculty panel discussions, to extend program capacity via online channels was recognized, and favored for inclusion in future curricula. Outcomes from 2021 thus support our predictions at the conclusion of 2020 that, as undergraduate and graduate institutions return to in-person learning, the virtual–in-person education pendulum will equilibrate to a hybrid that combines optimal features of both formats.

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 79 KB)