Spiralling Pre-clerkship Concepts into the Clinical Phase: Augmenting Knowledge Transfer Using Innovative Technology-Enhanced Curriculum Activities.

Background: Clerkship is a challenging transition for medical students where they learn to apply functional knowledge and diagnostic reasoning skills learned in the pre-clinical phase into the clinical environment. Rather than a smooth continuum to facilitate application of knowledge, clerkship blocks are discrete, fragmented structures with little integration. Developments in cognitive psychology and increasing attention to the student learning environment are driving more purposeful integration in medical education. We sought to enhance knowledge transfer in the Family Medicine clerkship by developing an e-learning pathway with both asynchronous and synchronous components to integrate pre-clerkship problem-based learning (PBL) cases into more complex clinical scenarios.
Methods: A parallel-convergent mixed methods evaluation was conducted, which included comparison of learning outcomes (exit exams) relative to the prior class, knowledge check quiz performance, and qualitative analysis of student and faculty perceptions.
Results: Analyses revealed no significant difference between exit exam scores of the intervention and pre-intervention clerks (p = 0.30). There were statistically significant differences in mean quiz scores over the rotation (p = 0.0001). Moreover, learners and faculty each perceived the integration components as facilitating the transfer of pre-clinical learning into clerkship activities.
Conclusion: The novel e-learning pathway firmly anchored FM clerkship learning and will continue to ensure learners are ideally primed to optimize their direct clinical learning opportunities. © International Association of Medical Science Educators 2021.

Introduction/Reasons Why

Clerkship, a fixture of medical education, requires medical students to transfer their functional pre-clinical knowledge and diagnostic skills, which are acquired primarily in virtual and physical classrooms, into real-life clinical care situations [1]. Clerkship is most frequently organized as a series of clinical rotations that highlight each of the core medical disciplines. The transition from preclinical to clinical training can be challenging for medical students for several reasons: they must transition to a number of different clinical learning environments [2], navigate new hierarchical learning structures [3], contribute to patient care, all while simultaneously understanding and meeting the expectations of frequently changing preceptors [4-7]. To support student learning through this transition, clerkship activities in most core medical disciplines involve discipline-specific tutorials designed to stimulate students’ acquired knowledge in a way that helps them apply the core principles learned in the preclinical phase in the context of the relevant clinical environment [8]. Notably, these tutorials are typically organized and developed within the bounds of discipline-specific knowledge without consideration for the totality of students` clerkship experiences. Accordingly, they are regularly experienced by students as discrete and fragmented learning opportunities, which are distinct from their clinical activities and so do not facilitate the development of coherent and integrated medical knowledge [3].

In 2005, the Undergraduate MD (UGME) Program at the Michael G. DeGroote School of Medicine at McMaster University (Hamilton, Canada) aimed to better integrate learning across the pre-clinical and clinical phases (clerkship) with its innovative COMPASS curriculum [9]. The acronym dissected with liberal poetic license, means-Concept Oriented, Multidisciplinary, Problem based, Practice for transfer, Simulation and Streaming [9]. The COMPASS curriculum, building on McMaster’s trademark Problem Based Learning (PBL), was structured to provide horizontal and vertical integration of fundamental basic science into problem-based clinical examples [3]. Horizontal integration considers the way in which several to-be-learned topic areas are presented with respect to one another. Horizontal integration means that material related to a single topic or subject is delivered in parallel with other topics, taught in multiple ways concurrently and with the same level of complexity. Vertical integration, on the other hand, leverages ideas from adult learning theory and cognitive psychology to promote the integration of knowledge [10, 11]. Specifically, the vertical approach builds on the initial foundation of knowledge in any one topic area and presents the materials with increased complexity so that learners gradually develop more capacity for applying acquired knowledge into novel and more difficult contexts [1, 10, 12]. When combined effectively, horizontal and vertical integration are understood together as “spiral integration,” with defined topics and subject areas interwoven to help learners develop a more comprehensive understanding of the collective knowledge [10, 11].

While many components of COMPASS were integrated horizontally across the pre-clerkship phase with success, others proved more challenging; notably, the practice-to-transfer and simulation components. The promise of vertical integration was also not fully realized, as there were few opportunities to apply increasingly complex knowledge in clerkship, and there was no opportunity to revisit foundational concepts with diverse clinical scenarios [13]. The addition of two distributed campuses, Kitchener-Waterloo (KW) in 2007 and Niagara in 2008, added additional difficulty around ensuring curricular equivalency at all teaching sites. Clerkship tutorials, similar to pre-clinical phase tutorials, are meant to facilitate the application of foundational knowledge to novel clinical situations. However, the lack of integrated curriculum, variability of teaching effectiveness amongst tutors, and the challenge of delivering equivalent face-to-face tutorial experiences across three distributed campuses made for a discrepant and potentially suboptimal experience for students. Given some of these issues, such as the newly distributed landscape, in addition to updates in medical and healthcare education systems, a multi-year process of curriculum renewal was undertaken with gradual rollout late in 2019. Here, COMPASS was modified with the aim to “more efficiently build, consolidate and integrate … knowledge and skills, thus offering an opportunity to measurably enhance students’ clinical transfer of concepts introduced in the pre-clinical curriculum” [9].

Accordingly, we dissected the various curricular components of COMPASS, with the aim of developing a series of new rotation-specific curricula. We began this exercise with a focus on improving the Family Medicine clerkship.

Specifically, we identified and emphasized core family medicine concepts reflected in multi-disciplinary problem-based learning cases from pre-clerkship, spiraling them from simple to more complex and revisiting them in clerkship. To enhance knowledge transfer, we referenced evidence that supports popular cognitive psychology learning theory and introduced multiple opportunities for retrieval-based practice, which employed well-established blocked, spaced, and interleaved presentation techniques [14, 15]. Further, informed by adult learning theory, we added interactive multimedia e-learning experiences to broaden exposure to clinical problems that may not be encountered in a clinical rotation and to optimize knowledge retention [16, 17]. This latter innovation involved a blended teaching approach that included synchronous and asynchronous online instruction. Students engaged with asynchronous e-learning modules housed in the Undergraduate MD Program’s learning management system (LMS), which were followed by synchronous faculty-led sessions delivered to all campuses via a web-based video-conferencing platform.

This curricular intervention took the form of the “Family Medicine Clerkship e-Learning Pathway” (see Appendix 1) with the following modular components:

Asynchronous Content

Five PBL cases, which were previously encountered during pre-clerkship, were purposefully selected for inclusion in the new curriculum These cases represent key health themes throughout the lifespan—Pediatrics, Care of the Adolescent, Care of the Adult, Care of Elderly, and Palliative Care—and served as individual modules for this asynchronous content. The education pathway required the clerks to work through these adapted PBL cases, which were modified to present additional complexity relative to their original presentation. This encouraged application of foundational concepts into novel clinical situations. Each module was designed to emphasize core content, in preparation for the associated synchronous tutorial.

Synchronous Content and Format

At designated times each week during the clerkship rotation, students participated in a 3-h synchronous tutorial that was facilitated by faculty leaders – also known as “maestros”—who were both subject and content experts in that week’s topics. Tutorials were conducted using a web conferencing platform (WebEx, Cisco Systems, California, USA) with the maestro remote from all the student groups. As this was pre-pandemic, students would meet in-person, in small groups (maximum eight students), to participate in the web conferences: for example, one small group in Niagara and KW, and two in Hamilton. The maestro joined from their own remote location. During tutorial, the asynchronous module content was reviewed, provocative cases were discussed, and important key concepts were articulated. Both the asynchronous modules and synchronous tutorials addressed the inherent challenge of providing an equivalent learning experience across all campuses.

Assessment

Embedded within the modules were knowledge check exercises. These took the form of multiple-choice quizzes comprised of 8 questions, which were selected randomly from an existing formative assessment bank and mapped to core family medicine clerkship topics. This bank was developed by the Canadian Family Medicine Education Directors (CUFMED) division of the College of Family Physicians of Canada (CFPC). These MCQs were administered pre-, mid-, and post-rotation and functioned as a type of formative progressive performance index. Further, each PBL module concluded with a Concept Application Exercise (CAE), which required a narrative essay-type short answer response to questions designed to encourage the integration and application of knowledge [13, 18, 19]. Individual feedback was provided to the students by the maestros.

Lastly, an established exit exam comprised of fifteen MCQ questions based on clinical decision making was administered electronically using the ExamSoft (ExamSoft Worldwide LLC, Texas, USA) platform within the final days of the clerkship rotation. This assessment is consistent with that administered prior to this curriculum redesign.

A complete overview of the innovative clerkship curriculum, linked with learning objectives, is shown in Appendix 1.

In this report, we present an evaluation of this new Family Medicine Clerkship e-learning pathway. Our goals were to evaluate the impact of the e-learning pathway on student learning and to understand student and faculty perceptions of the learning pathway and its implementation. We anticipated that the new curriculum intervention would increase students’ ability to apply foundational knowledge in different clinical situations (i.e., knowledge transfer), and that such learning would facilitate better exit exam performance. We also reasoned that the integrated online blended teaching approach would be perceived favorably by students and faculty and refine clerkship curriculum for the future.

Methods

Study Design

This study was carried out using a three-armed, parallel, convergent, mixed-methods study design [20], which included quantitative review of student performance on tests of knowledge and qualitative analysis of interviews with students and faculty to ascertain their perceptions of the new e-learning pathway. This integrated approach was chosen for its ability to offer multiple forms of evidence as to the impact of the curriculum on student learning and student experience, as well as an opportunity to appraise the features of experiences that impacts the learning. Results are reported informed by the Good Reporting of a Mixed Methods Study (GRAMMS) framework. These analyses were conducted separately and integrated at the end of the study [21].

The first and second arms of the study quantitatively compared (1) in-progress and (2) final exam scores using results from (1) a multiple-choice question quiz administered before, during and after clerkship, and (2) final exam scores between students who participated in this innovative clerkship curriculum versus students from a previous cohort, before the curriculum was introduced. The third arm employed a qualitative descriptive approach [22, 23] and individual interviews to understand the experiences of students and maestros.

Study Population/Sampling

This study was conducted at the Hamilton, Kitchener-Waterloo and Niagara UGME campuses of McMaster University (Hamilton, Canada). Research Ethics Board approval was obtained from the Hamilton Integrated Research Ethics Board. All medical students entering their Family Medicine Clerkship rotation at all three UGME campuses between May 2019 and May 2020 were informed about the study via a recruitment email, an informational flyer which was sent by email, printed in their Family Medicine Clerkship Information Package, and added to their electronic welcome package on the local LMS, and through an in-person presentation. During these presentations, informed consent forms were distributed to students for future participation in an interview after clerkship.

Sample Size

Of the 208 undergraduate medical students in the medical school clerkship, 160 students participated in the new innovative Family Medicine clerkship curriculum, dependent on the timing of their Family Medicine clerkship core. Scores from all participating clerks were analyzed for the first arm of the study. The UGME Program at McMaster University provided the exit exam scores of the previous class; exit exam scores of 119 undergraduate medical clerks from the 2019 class were analyzed for the second arm of the study. For the third, qualitative arm, the concept of “information power” was utilized to guide the determination of sample size, where interviews were conducted with consenting students who completed the new curriculum and until enough information was present to adequately achieve our objective [24]. Nine students and five maestros participated in this phase.

Data Collection

In order to achieve the first objective, a multiple-choice quiz (MCQ) comprised of 8 questions, derived from the previously described question bank, was administered through the LMS at three specific points throughout the clerkship rotation. The quizzes were randomized to ensure questions from pediatric, adult, and geriatric domains were represented in the same proportion at each offering. The MCQs were taken pre-, mid-, and post-rotation. The questions supporting these tests were derived from an existing formative assessment bank developed by the Canadian Undergraduate Family Medicine Education Directors (CUFMED) division of the College of Family Physicians of Canada (CFPC). Student scores were stored in the local LMS and were analyzed by the research team.

The MCQ quizzes were limited to a single attempt. There was a technical issue for the first group, which allowed a few students (n = 4) to complete more than one attempt on the pretest and mid-rotation quiz; however, the data from their first attempt was used in the analysis. This technical issue was fixed for all subsequent quizzes and did not affect any of the other groups.

The exit exam scores, the other component of student performance, were compared with the pre-intervention class.

To understand students’ experiences and perceptions of the learning pathway, as well as the impact of this curriculum on their clinical performance, semi-structured interviews were conducted across all clerkship sites among students with a variety of achievement levels on the in-progress quizzes. Maestros were also purposively sampled to include a variety of experience levels in order to explore the impact of the new curriculum on the educators’ role and on the perceived quality of educational experience provided. Interviews took place in person or via video conferencing platform and were transcribed verbatim. Interviews were audio recorded and lasted about 30–45 min each, with honesty and confidentiality emphasized. Quantitative and qualitative component results were integrated and compared to generate conclusions [21].

Data Analysis

All statistical analyses for the quantitative data were performed using Stata Software Package, version 14.2 (StataCorp LLC, Texas, USA). Significance was defined as having a p-value of < 0.05, all p-values were double sided, and 95% confidence intervals were computed.

For the first study arm, the results of the MCQ pre-, mid-, and post-rotation quizzes were compared in a one-way repeated measure analysis of variance (ANOVA) to understand whether there was a difference in test scores amongst students before, during, and after the innovative clerkship curriculum. In addition, the mean scores, and standard errors were computed for each group’s MCQ score. Tukey’s HSD post hoc methodology was utilized to indicate where the significant effects occurred between the three test groups.

To compare student performance in the innovation group to the previous class, exit exam scores were evaluated using a one-way ANOVA with Group (Pre-intervention clerks, Post-intervention clerks) as the only factor. These analyses determined whether there was a significant difference in mean exit exam scores between the two groups. Summary statistics of the exit exam scores for both groups were also estimated, providing mean scores and standard errors.

In order to determine and understand the perceptions of the new clerkship curriculum, semi-structured interviews were conducted with both students and maestros. Interviews were analyzed using an inductive content analysis approach [25]. The intent of this analysis was to develop a rich description of learner and maestro views, which offers both a summative and an integrative element across participants. Initial coding began by analysts coding independently, then meeting to discuss and create a refined coding schema, (Appendix 2) which was then applied to all interviews. N-Vivo 12 was used to manage qualitative data analysis.

Results

MCQ Tests Results

MCQ tests (pre-rotation, mid-rotation, and post-rotation) were completed and assembled for six groups; group seven did not participate due to the 2020 Covid-19 pandemic’s disruption of their clerkship rotation. A total of 142 of the 160 students involved in the clerkship completed the MCQs; therefore, MCQ scores were reviewed and analyzed for these individuals.

Results showed that the innovative clerkship curriculum elicited statistically significant differences in mean MCQ test scores over its time course (F (2, 423) = 9.10, p = 0.0001; Table 1). Furthermore, post hoc comparisons using the Tukey HSD test indicated that clerks’ MCQ performance at Pre-rotation (M = 61.54%, SE = 1.67) was significantly lower than at Mid-rotation (M = 69.39%, SE = 1.49) and Post-rotation (M = 69.20%, SE = 1.26). However, mean MCQ scores did not significantly differ between Mid-rotation and Post-rotation (Table 2).

Table 1

Summary statistics and one-way analysis of variance (ANOVA) of multiple-choice question test scores as a function of three different time points during Innovative Clerkship Program among Learners (N = 142)

Test groups (N = 142)	Mean score (%)	Standard deviation	Standard error	95% confidence interval for mean	ANOVA output
					F-statistic		Prob > F
Pre-clerkship	61.54	19.92	1.67	58.26, 64.83	9.10	0.0001
Mid-clerkship	69.39	17.70	1.49	66.47, 72.31
Post-clerkship	69.20	15.07	1.26	66.71, 71.68

Table 2

Tukey’s HSD post hoc analysis results demonstrating significant pairwise comparisons of mean multiple-choice question test scores between three different time points during Innovative Clerkship Program among learners (N = 142)

Comparison	Difference of means (%)	Standard error of difference	Tukey, t-value	Adjusted, P-value	Tukey (95% confidence interval)
Test
Mid vs. Pre	7.85	2.10	3.74	0.001	2.91, 12.78
Post vs. Pre	7.65	2.10	3.65	0.001	2.72, 12.59
Post vs. Mid	−0.19	2.10	−0.09	0.995	−5.12, 4.74

Exit Exam Cohort Results

A total of 119 students’ exit exam scores were analyzed from the previous class of Family Medicine clerks (class of 2019) against the students’ exit exam scores from the new clerkship curriculum (N = 160). The mean exit exam score was higher among the intervention group (M = 74.77%, SE = 0.56) than the controls (M = 73.96%, SE = 0.49); however, there was no significant difference between the exit exam scores of the intervention group compared to the previous group (F (1, 277) = 1.10, p = 0.2958, Table 3).

Table 3

Summary statistics and one-way analysis of variance (ANOVA) of exit exam scores by intervention status among learners in the Innovative Clerkship Program and the previous cohort of learners (N = 279)

Cohort	Sample size (n)	Mean score (%)	Standard deviation	Standard error	95% confidence interval for mean	ANOVA output
						F-statistic		Prob > F
Pre-intervention	119	73.96	5.39	0.49	72.98, 74.93	1.10	0.2958
Post-intervention	160	74.77	7.04	0.56	73.67, 75.87
Total	279	74.23	6.39
Mean Difference (Post–Pre)*		0.81		0.77	−0.71, 2.33

*The last row shows the mean difference, standard error, and the 95% CI around the mean difference

Qualitative Results

Overall, there were five maestros interviewed from the new Family Medicine clerkship, and a total of nine students from five groups, all of whom completed their clerkship between September-December 2019. Participants discussed their perceptions of content, format, tutorials and assessment.

Perceptions of Content

Students shared both positive and negative feedback about the content of the e-learning pathway. In many examples, the contribution of the tutorial discussion to positive appraisal of content was related to the presence of complex issues of care such as social determinants of health, de-prescribing, or appraising guidelines for application to a specific patient. Participants frequently remarked that they appreciated the presence of these issues in the e-modules and benefitted strongly from discussion about applying these principles in practice.

Participants were resounding in their emphasis on the need for content to be relevant to clinical care and outlined the importance of providing concrete strategies and practical skills. They provided many examples of using the content from the COMPASS curriculum in later clinical experiences, and although students appreciated the practical content that did exist, they wanted more:

I was also expecting to learn more practical things that you would really need to know as a family doctor, but I don’t think we got so much of that. … we didn’t really talk about if you see a patient like this, this is how you prescribe, this is how you fill out this form or refer someone to this other specialist. … (Student #5)

Maestros expressed their satisfaction with the material that was covered in the tutorials and appreciated students’ feedback on topics discussed. Several maestros had a positive perception on the complexity of pre-clerkship cases which were reviewed, mentioning that they stimulated students to think differently and enhanced their learning process.

Perceptions of Format

Asynchronous E-modules

Overall, students had more positive than negative opinions about the structure of e-modules, with emphasis placed on the helpful ways e-modules prepared them for tutorials. Participants also appreciated the practice questions and concept application exercises provided in the learning pathway, which were recognized as being useful in addressing gaps of knowledge by most students:

And, also, the concept, so the CAE (Concept Application Exercises) that you mentioned, like the exercises that we got to do, I think that was really, really helpful. I really liked that part of that, like that really forced me to look into the topics, to think like systematically and, yeah, I think that was really helpful. (Student #6)

Synchronous Web Conferenced Tutorials

Students provided considerable feedback about the format of tutorials which initially were plagued with technical issues relating to a remote technology delivery system that few students or faculty had experience with. At the time they encountered the learning pathway, most material was delivered in-person and, pre-pandemic, there was vastly less experience and expertise with web conferencing technologies.

In many examples, the negative perception of the tutorial format was related to gaps in communication between maestros and students. Participants frequently remarked that there were many awkward silences, leading to distractions and wasted time:

I think one was a technical issue, so the speaker would, let’s say, ask a question to Niagara [campus] and then having them, like needing the time to get the mic working properly if there was any issue, sometimes you could have a significant time delay. And then within that you would just lose focus and not really be paying attention anymore. And I think it just detracts from your own kind of experience. (Student #4)

Maestros also expressed the challenges they faced when communicating with students: “I still struggle. I don’t know what’s the right amount of talking and the right amount of texting. That is still a struggle and I haven’t figured out that balance.” (Maestro #3) Some maestros expressed their difficulty with using the features of the online platform to optimize communication; however, in most cases, maestros were able to gauge what worked best.

Participants appreciated when maestros facilitated discussions and structured their tutorials as being discussion-based as opposed to a didactic lecture format. For example, one participant stated:

One of the times that I found tutorials very helpful, when they were actually encouraging us to have a discussion. When they weren’t and it was very didactic, that was not helpful at all, because we all sort of tuned out at that point. (Student #5)

When discussing their opinions on the synchronous format of tutorials, a majority of students reported that they perceived them as being beneficial for their learning. Several participants stated that this format helped “dedicate time to spend learning about the topics” (Student #5) and enhanced their learning because of “pretty engaging discussions all around.” (Student #8) Many participants mentioned that their classmates’ contribution was important in increasing their knowledge about different aspects of Family Medicine.

Participants expressed frustration of having to connect remotely to a maestro, with the technical challenges often presented. However, participants also addressed the concept of ensuring equity of teaching for students from regional campuses, as they noted that pre-pandemic, almost all teaching sessions, either pre-clerkship or clerkship, were “broadcast” from Hamilton, creating an unequal experience for students at the other campuses. Maestros also commented on the advantages of the new tutorial system in providing uniform learning to all students:

From an equity perspective for the students who previously may have felt that they’re not quite getting the same kind of education experience as the ones who are physically in the room with the tutors, or you know, making sure that the different sites have the same education experience, because you know, even an in-person tutorial can be varied depending on who’s teaching it and how it’s organized. So, I think from that perspective, McMaster especially with having regional campuses, I think that’s a potential advantage there for the students. (Maestro #4)

Interestingly, both students and maestros commented on the potential for student-led tutorials remotely facilitated by a faculty preceptor as a way of promoting student engagement.

Students’ Perceptions of Maestros

Maestros’ delivery style appeared to impact students’ perception of tutorial. Aspects of maestros’ teaching styles that were praised by participants included prompting in-class debates, facilitating discussions, encouraging student-led presentations for the discussion of important topics or for answering questions, and providing feedback and key summaries at the end of tutorials. One participant provided a distinct example of a stimulating tutorial, in which they lauded the maestro’s teaching style:

So, one of the maestros did a really, really good job of having cases. And I'm trying to remember which tutorial it was, but they had really good case examples, that it was sort of you go through the case, they make us answer it in our groups, and then we review it, so that they call on us to review and they give us sort of like the main points to take away. I really liked that structure because it not only made us all work together to try to think of an answer, but then it also made sure everyone was engaged because it held us accountable and it asked us, so what was your answer. (Student #9)

Due to the online format of tutorials, participants acknowledged the challenges associated with approaching maestros to discuss personal experiences: “I didn’t feel that I could approach the tutorial leader at all because I was in a room full of other… because they didn’t really know me, I didn’t really know them, they were just kind of this face on the screen, so it’s a little depersonalized.” (Student #5) Participants expressed their appreciation of maestros who offered support and structured tutorials so students would feel comfortable talking about their personal challenges. Students reported that this support and guidance also helped them grow as a medical student, to “develop reasoning and judgement” (Student #8) and made them comfortable asking questions.

Perception of Assessment

Participants appreciated the formative multiple-choice questions (MCQ), stating that they were useful, “it makes me want to review the information that was tested.” (Student #9). Participants commonly commented on ways in which the MCQ enhanced their learning:

There were sort of like learning points, like there were like questions to test comprehension throughout, sprinkled throughout the e-modules, I found very helpful…By having those questions, I think were very, were the most important thing for me for my clinical rotation. (Student #1)

Positive appraisal about the assessment was frequently linked to the Concept Appraisal Exercises (CAEs). In many cases, participants mentioned how effective these exercises were in helping them think analytically and encouraged them to review topics on their own time:

But the post ones, the ones that were much longer were definitely helpful.…I just appreciated having those questions, was like a prompt for learning about that topic and because they were hard enough and I felt challenged, I found them interesting. (Student #9)

Integration/Limitations

This project aimed to create and evaluate an innovative Family Medicine clerkship curriculum, designed as an e-learning pathway with asynchronous e-modules using existing pre-clerkship case-based problems, interwoven with additional layers of complexity to encourage application of core foundational concepts into different clinical scenarios. The modules spanned the lifecycle; incorporated a series of knowledge check assessments that assisted with student identification of learning gaps and progress; engaged students with multimedia, interactivity, and concept appraisal exercises; and were bolstered by a series of scheduled synchronous tutorials facilitated by expert faculty.

The results show that the curriculum had a positive influence on student learning. Review of the MCQ quiz scores shows that students’ knowledge improved through engagement with the e-learning pathway and clinical exposures that occurred as part of the clerkship rotation. These data show that students had the largest gain in knowledge between pre-intervention and roughly the mid-way point of the rotation, with little apparent improvement from mid-to-end of the rotation. This might be explained by a few different factors. First, we must acknowledge that the quiz items may have been overly challenging, creating a plateau on performance improvement. Second, the mid-rotation quiz occurred slightly later than mid-rotation; with students having completed 3/5 of the rotation. Thus, they would have had more instructional interventions (curriculum) and clinical experiences between pre-rotation to mid-rotation than from mid- to end-rotation. With respect to learning relative to previous classes, we noted no significant difference between exit exam scores between students who underwent the innovative e-learning system compared with the previous class. This finding highlights, at least, a non-inferiority outcome for the new curriculum.

Our qualitative analyses of student and faculty perceptions highlighted the influence of greater curriculum integration on student readiness to learn, clinical capability, and provided the foundation for more robust assessment of student competency. Overall, both students and maestros perceived the e-learning pathway as being beneficial to student knowledge translation. Students perceived the CAE as the most helpful form of formative assessment; perhaps due to the written feedback provided, although the knowledge check quiz questions were also perceived as useful. The e-modules were generally found to provide helpful material for tutorials and useful to revisit. Negative perceptions of the synchronous web conferenced tutorials most commonly related to technical difficulties, and lack of opportunity for meaningful discussion due to dead space and the remote tutorial format. Despite this, students did appreciate the opportunity for meaningful discussion with their classmates and appreciated the equitable access to the maestro across all sites. Notably, at this time, we realize the technical challenges experienced are mostly moot; since the pandemic, videoconferencing platforms have become the norm rather than the exception, and both student and faculty comfort with remote technologies and virtual discussions has increased.

In integrating the quantitative and qualitative results, we are confident that learning occurred over the course of the Family Medicine clerkship; this is evidenced in faculty reports of student performance, the improved MCQ outcomes, and the non-inferiority of the exit exam scores compared to the pre-intervention cohort. The fact that we were not able to demonstrate statistically significant improvement in terms of our first objective may be influenced by several factors. With respect to the MCQ quizzes, the small number of test items (eight) was a pragmatic choice to avoid testing fatigue. Although the questions were derived from a reliable question bank reviewed and improved by two expert MCQ writers, the bank has not been the subject of rigorous psychometric review. As the questions for each quiz were randomly selected from question pools, it was impossible to ensure the same level of difficulty. The exit exam—while not the subject of reliability and variability studies—has been evaluated and subjected to robust in-house psychometrics. We should reasonably accept effective and non-inferior student learning as an outcome for this study, as it is relevant that students within the intervention group performed as well as their peers who were not. The interviews were all recorded near the end of project completion and subject to recall distortion and depending on timing of the clerkship rotation, may have been negatively skewed by the earliest group’s technological difficulties. The last group was not able to participate due to the pandemic, as medical undergraduate learners were abruptly removed from clinical learning environments March 15, 2020.

Conclusion/Insights

The novel Family Medicine clerkship afforded students more opportunities for knowledge translation and better met the intended goals of the revised COMPASS curriculum. The positive effects on learning and participant perceptions justify the success of our curricular intervention, which is equivalent across all campuses and easily scalable to other clerkship disciplines. Opportunities to provide high quality, educationally robust, asynchronous, technology-enhanced learning have become critical as medical schools rapidly pivoted away from in-person group instruction and limit direct clinical experience on account of the Covid-19 pandemic. Indeed, we realized additional benefits of this e-learning pathway when clinical learning environments were curtailed in 2020. There is room for further study to understand the impact of this pivot.

We envision the development of similar spirally integrated, e-learning pathways across clerkship disciplines. In doing so, we aim to offset some of the largest challenges with transitions to clerkship. We anticipate that this will have demonstrably positive impact on long-term knowledge retention, clinical experiences, and performance on instrumental licensing exams.

Learning path component	Configurations	Instructional methods	Objectives
Clerkship orientation	Asynchronous online course modules	• Video presentation • Rotation resource list	• Understand expectations of FM clerkship
	Synchronous web conference session	• Overview of web conferencing technology presentation • Q&A/discussion
Learning path pretext quiz	Asynchronous online assessment	• Formative, no-stakes quiz of 8 single-select multiple-choice questions randomly selected from 3 different question pools      ○ 3 of 33 questions from the pool of General Adult items      ○ 3 of 34 questions from the pool of Pediatrics items      ○ 2 of 16 questions from the Care of the Elderly items • Adapted from an existing formative assessment bank developed by the CUFMED (Canadian undergraduate family medicine education directors) division of the College of Family Physicians of Canada (CFPC). The bank was mapped to national core clerkship competencies	• Link to nationally approved FM Clerkship Competency Objectives approved July 2018 by CanMEDS-FMU entitled LearnFM* http://learnfm.ucalgary.ca/objectives/ • Link to Undergraduate program & FM clerkship objectives https://www.medportal.ca/public/objectives *Denotes link to LearnFM objectives  + Denotes link to FM clerkship objectives
Course 1: Pediatrics	Asynchronous pediatrics modules	• Pediatric case interactive module      ○ approximately 30-min multimedia, interactive module using scenarios and self-assessment questions; based on pre-clerkship PBL tutorial problem • Concept Application Exercise      ○ 2 scenario-based manually graded, free text response items with several question prompts	• Baby/Child Preventive Care + * • Describe the impact the social determinants of health have on health outcomes + * • Appreciate the role of the Family Physician as an advocate for health + * • Contraception-Patient centered decision making * +  • Common skin conditions-recognize & treat*
	Synchronous web conference tutorial	• Instructor-led tutorial by topic 'maestro' from an independent location • Students joining in small groups from 4 different locations • Interactive instructional design, using discussion tools as well as audience response systems (e.g., polling questions) • Extends scenarios and topics initially presented in the asynchronous module
Course 2: Adolescent medicine	Asynchronous adolescent medicine modules	• Adolescent case interactive module • Concept Application Exercise      ○ 3 scenario-based manually graded, free text response items	• Confidentiality: Understand issues around including:    ○ Child protection +     ○ Consent and capacity +  • STI Screening and treatment: Diagnose and treat, understand populations at risk + * • Prenatal Care: + *    ○ Understand how pregnancy changes commonly used medications    ○ Understand role of FM in prenatal care (screening, immunization etc.)    ○ Adolescent pts and pregnancy -heightened risks • Develop an approach to screening for substance use in pregnant patients & develop treatment plans* • Mood Disorders in Adolescents: + *    ○ Understand the use of validated tools for screening + *
	Synchronous web conference tutorial	• Same structure as web conference tutorial above
Course 3: Adults	Asynchronous adult modules	• Adult case interactive module • Concept Application Exercise    ○ 5 scenario-based manually graded, free text response items	• Management of chronic conditions in the family medicine setting: + *    ○ Prescribing/de-prescribing and non -pharmacological management + *    ○ Appreciate both modifiable and non- factors for chronic medical conditions like hypertension,  diabetes, CAD + * • Occupational Health +     ○ -Understand the impact of occupational stressors on patients’ health    ○ Take a full occupational work history • Understand the general approach to motivational counseling in order to help patients make lifestyle changes + * • Apply Preventive Health screening guidelines and indications for testing: + *    ○ Understand sensitivity and specificity of testing +     ○ Know Canadian evidence based guidelines for screening + *    ○ Develop an approach to counseling a patient who requests screening and testing for conditions for which it is not indicated +
	Synchronous web conference tutorial	• As above
Mid-rotation quiz	Asynchronous online assessment	• Formative, no-stakes quiz of 8 single-select multiple-choice questions randomly selected from 3 different question pools (as in Pretest above)
Course 4: Geriatrics	Asynchronous geriatrics modules	• Geriatric case interactive module • Concept application exercise    ○ 2 scenario-based manually graded, free text response items	• Describe an elderly patient's functional status using patient and collateral history. + * • Define “Frailty″ as it applies to the following common problems as they contribute to functional decline: +     ○ Incontinence, falls    ○ Polypharmacy    ○ Depression, Cognitive impairment • Identify important topics of discussion with the elderly patient and family/caregiver meetings including: + *    ○ Advance care planning and Goals of Care    ○ Caregiver burnout    ○ Driving    ○ Community resources    ○ Transition planning
	Synchronous web conference tutorial	• As above
Course 5: Palliative care	Asynchronous palliative care modules	• Palliative care case interactive module • Concept application exercise    ○ 7 scenario-based questions: 2 manually graded free text responses and 5 multiple choice items	• Describe the palliative care approach to care, and who may benefit from it + * • Assess and manage pain and other common symptoms in palliative care + * • Understand the interprofessional approach to providing palliative care +  • Organize care for the actively dying patient and family +  • Describe an approach to responding to grief, bereavement, and suffering +
	Synchronous web conference tutorial	• As above
Learning path posttest quiz	Asynchronous online assessment	• Formative, no-stakes quiz of 8 single-select multiple-choice questions randomly selected from 3 different question pools (as in Pretest and Mid-Rotation Quiz above)
End of rotation exit exam	Asynchronous online assessment	• Higher stakes, and delivered via secure online examination platform • 15 multiple choice questions, developed in-house • Focused on clinical decision making and mapped to core clerkship objectives

Name of code	Description
Assessment Evaluation MCQ (Pre, mid, post) CAE Exit Exam	Describes students’ perceptions on the non-formative evaluations and their usefulness in terms of self-assessment, skill development, and knowledge awareness
Comparison to tutorials in other specialties	Captures comparison and reflection between COMPASS or FM curriculum and that of other specialties
Reflections on Covid online clerkship	After experiencing the online Covid clerkship, many clerks may revise their thoughts about online discussions and learning. Some reflect upon how the ubiquity of online learning has changed norms and comfort related to participation
Expectations Clinical COMPASS	Captures responses to 2a, expectations for FM clerkship before they began. I have created sub-codes for expectations pertaining to different aspects of FM clerkship. Overall, I do not anticipate this will be a terribly useful code but worth capturing as we go through
Experiences with other aspects of FM curriculum Classroom Clinical and Preceptor Experience: Positive Negative	This code lets us capture experiences and opinions related to non-COMPASS aspects of FM learning which may influence or illuminate the utility or areas for improvement of compass. “Classroom” = academic half day and other non-COMPASS, non-clinical learning experiences
Negative perspectives or experiences with e-modules Format: Lack of structure/detail No feedback on formative MCQ questions Pre-clerkship cases not useful Difficulty staying engaged	I have separated reflections on e-modules and tutorials and am asking the analyst to categorize these as positive or negative. Some may be neutral, but try to see if the student is suggesting a change (put it in negative) or noting that the status quo is acceptable (put it in positive)
Positive feedback or experiences with e-modules Format: CAEs and Practice Questions helpful in learning Properly structured No technical problems with access Helpful to revisit modules and cases later Helpful for tutorial preparation
Negative feedback or experiences with tutorials Format: Discussion challenges Distractions or disruptive experiences Negative perceptions of having Maestro online Participation challenges Tri-campus integration Requirement to gather in person for online tutorial Technical challenges Overlap or discrepancy between module and discussion Transition to clinical challenges
Positive feedback or experiences with TUTORIALS Format: Discussion is valuable Experience with communication platform Level of engagement Positive perspective on breadth of knowledge covered Preparation time is adequate Positive perception of the transition of knowledge and skills to clinical Tutor facilitated well
Preparation for future FM work	Comments relating to how they felt COMPASS prepared them to proceed to future FM work (which may include clerkship or electives or residency training. It may also refer to FM-related work they may do in another specialty)
Regional campus perspectives	The issue of equity and participation across different campuses seems particularly important
Student traits and previous experiences Interest in FM (pre-clerkship and after clerkship) Placement of FM core within clerkship Previous clerkship experiences outside of FM Resources used for self-improvement	Students often talk about how FM fell within their clerkship year (first or last rotation so was or was not so sufficient) changed their experience of the curriculum (previous experiences outside of FM). This will also capture talk about how interest and experience with FM changed their experience of the curriculum
Suggestions for improvement	For concrete suggestions or requests for changes to the program
Synchronicity	The question of synchronous vs. asynchronous aspects seems important and not always tied to negative or positive feedback
Online v in person	The issue of being together in person or online (and varying combinations of this (e.g., students in one room, tutor online) can be captured here. Much of this talk will also probably get categorized as positive or negative feedback/experiences around the tutorial, but I think it’s useful to keep it in one spot
Topics covered Negative perspectives Positive perspectives	Since topics covered in tutorials and e-modules were relatively similar, pulled content/topic out as its own code. This includes comments related to topics that they would have liked to discuss more, and topics which they greatly enjoyed and perceived as the most useful for learning

Name of code	Description
Perception on content Positive Negative	Description of learners and maestros’ opinions on the concepts covered and discussed in e-modules and tutorials, also includes perceptions/comments on the transition of knowledge/content to clinical practice
Perception on format Positive Negative	Description of how learners and maestros perceived the content was delivered for e- modules and tutorials (including online vs. in person, synchronicity and regional campus perspectives)
Perception of maestros/tutorial leaders Positive Negative	Comments from learners relating to the positive and negative appraisal of maestros in terms of their teaching styles, level of engagement and involvement, effort in connecting with learners etc
Perception of assessment evaluations Positive Negative	Learners perception of assessment evaluations presented throughout e-modules and tutorials (e-modules MCQ, CAEs)
Relationship between COMPASS and interest in FM	Speaking to how well the COMPASS curriculum influenced learners’ perception and interest in the FM clerkship and discipline