Curriculum Development with the Implementation of an Open-Source Learning Management System for Training Early Clinical Students: An Educational Design Research Study.

BACKGROUND: Learning management systems (LMSs) have not been explored from an educational design research (EDR) perspective for developing clinical curricula and supporting novice clinical students with self-regulated learning during their early clinical rotations.
METHODS: An EDR approach was used to inform a de novo implementation of an LMS during an early clinical rotation of medical students. The EDR consisted of three phases: analysis and exploration; design and construction; and evaluation and reflection. Process and evaluation data (including academic years 2018 and 2019) from two student cohorts (total n = 190, 107 without and 83 with LMS exposure) at one academic teaching hospital were analyzed.
RESULTS: Learning theories and concepts of self-regulated learning were used to develop and implement an LMS clerkship prototype. For design and construction, the maturing prototype design included flipped-classroom elements, in-class activation, voluntary digital self-assessments, and clinical teaching videos. For evaluation and reflection, global satisfaction improvement was significant (from 3.9 to 4.4 on a 5-point Likert scale, p < 0.05). There was a positive evaluation trend for all evaluation items related to learning climate, self-regulated learning, and perceived usefulness of the LMS prototype; however, these changes were not statistically significant. The teaching hospital also improved its ranking after the introduction of the LMS prototype. Nearly all students (94%) used the LMS material. The average number of times the LMS course was accessed per student was 70 (range: 7-172), and the average duration students spent online was 58 minutes (range: 9-165).
CONCLUSION: Our data indicate that using an EDR approach was helpful for systematically introducing an LMS in a clerkship curriculum informed by learning theory. Our evidence-oriented curriculum reform was associated with higher student satisfaction and appeared to support self-regulated learning in the workplace. Further research should explore which elements of an LMS most effectively help to achieve educational outcomes.

Introduction

Novice clinical students are confronted with numerous challenges in terms of managing their learning progress in a hectic clinical environment.1 Difficulties can arise from becoming involved in the clinical work itself and actively participating in direct patient care, as well as from frequent (re-) integration processes into new healthcare teams and workplaces during their core clerkship year.2 Supporting effective and efficient self-regulated learning strategies in clinical workplaces is, therefore, a fundamental goal of clerkship curricula.3 While a myriad of aspects relevant for blending technology with clinical teaching have been researched at the level of single tools or discrete didactic formats,4–6 little attention has been paid to systematically exploring affordances of a learning management system (LMS) to support self-regulated learning in clerkships.

LMSs are web-based virtual environments that can accommodate a range of didactic strategies with regard to course material, stakeholder interaction and collaboration.7,8 A general review on the impact of information and communication technologies on the micro- and meso-level of medical education has shown that their use can have positive effects on student attitudes, learning outcomes, and learning efficiency.9 A review of blended learning interventions in clinical teaching has shown positive evidence for the improvement of clinical competencies such as history taking, clinical examination and clinical reasoning with technology-enhanced teaching.6

Working with an LMS has become common practice in undergraduate medical education,7 primarily because of convincing practical advantages rather than empirical evidence. Similarly, working with information technology in clinical care has become an expected core competency for graduating medical students.10–13 In addition, the COVID-19 pandemic has led to a high demand for flexible and creative use of virtual teaching and learning environments.14

In the context of clinical rotations, medical educators have begun to research the integration of different virtual tools into their clinical curricula. These methods range from personal digital assistants for clerkship students15 to web-based learning modules and virtual patients in clerkships across disciplines, including emergency medicine,16 family medicine,17 pediatrics,18,19 radiology,20 surgery,21,22 geriatrics,23 ophthalmology,24 and psychiatry.25 Learning outcomes up to the level of clinical competence have tended to be slightly better for curricula that include e-learning elements. However, these studies have primarily researched specific e-learning tools and do not report on the implementation process of an LMS to support the self-regulated learning strategies of medical students in the clinical workplace. Self-regulated learning encompasses those activities that learners use to achieve their cognitive, affective, and behavioral learning goals.26

In this study, we used an educational design research (EDR) approach to develop and implement an open-source LMS prototype as the main curricular change in the clinical learning context of early clinical students. We evaluated this evidence-oriented curriculum reform by comparing student perceptions before and after implementing the LMS.

Materials and Methods

We used an EDR approach27,28 to analyze (EDR Phase 1), design (EDR Phase 2) and evaluate (EDR Phase 3) the integration of the open-source LMS, ILIAS (), as the main curricular change in a traditional clerkship curriculum at the University Hospital of Psychiatry in Bern, Switzerland.29 The main educational project goal was to improve student workplace-based learning support with self-regulated learning strategies in a clinical context. A secondary goal was to evaluate ILIAS and its affordances for clerkship quality management purposes.

The total study sample included 190 undergraduate medical students who completed their clinical rotations (four weeks per student) in psychiatry at the same teaching hospital (Figure 1). Each academic-year cohort consisted of student subgroups (n = 8–10 students per rotation) that spent one month, full time, at the same teaching hospital. At our medical school, five teaching hospitals accommodate all clinical students in the psychiatry specialty. At the end of each academic year, these five teaching hospitals are ranked based on monthly student evaluations, which are administered through independent educational specialists. During the course of this study, our teaching hospital was the only one that implemented an LMS prototype.

Figure 1

Study design.

EDR Phase 1: Analysis and Exploration

We began with an analysis and exploration of a total of 107 paper-based evaluation forms with global satisfaction ratings and written feedback from clinical students in the 2018 academic year at our teaching hospital. The first phase of the EDR process involved an in-depth analysis and exploration of the target educational challenge, learner characteristics, and the learning context. Initially, we conducted a non-systematic search of the literature using different databases (PubMed, Google scholar, ERIC) to identify relevant theoretical frameworks and empirical studies for our EDR project. Next, we used multiple sources to extract information relevant to self-determination,30 self-directed learning,31 and self-regulated learning.32,33 Further data sources included pre-clerkship questionnaires, informal interviews with clinical residents involved in teaching clerkship students, and students’ clerkship evaluations from 2018. EDR phase 1 also involved building an informal educational project team, finding collaborators, and identifying financial and time resources. Two project members acquired technical knowledge through faculty development courses to work with the ILIAS learning environment.

EDR Phase 2: Design and Construction

Based on EDR phase 1 results, we identified tools in the LMS with the potential to stimulate and support self-regulated learning during the clerkship (Figures 1 and 2). We selected the following clerkship curriculum elements for piloting LMS support for self-regulated learning: the clerkship orientation seminar, formative knowledge self-assessments, clinical skill motivational interviewing, and a student research paper conference. We screened existing video material (published documentaries of our teaching hospital and television interviews of clinical experts from our hospital) for suitability and also developed new video material with teaching video experts and actors. We used the software Camtasia (TechSmith Corporation, Okemos, MI, USA) to create screencasts to introduce students to working with the electronic health record system in a training environment without real patient data.

Figure 2

Concept for integrating a learning management system into a clerkship curriculum.

EDR Phase 3: Evaluation and Reflection

For this study, we analyzed clerkship evaluations for the 2018 academic year (before LMS implementation) and 2019 (after LMS implementation) that concerned working with an open-source LMS at one academic teaching hospital (Table 1). We used monthly clerkship evaluations, perceptions of the relevance of the LMS material, formal and informal feedback rounds with students, and student exit interviews as iterative information sources to adapt how the LMS prototype could be continuously improved. These data also helped to adapt LMS content and to gain insight on whether students felt adequately supported with the self-regulated learning activities. The LMS prototype for the clerkship continuously matured based on new insights and reflections. We used user data to analyze the engagement of students with the LMS material and time spent online. A more in-depth reflection on all of the results and insights from the first year using the LMS served to define necessary adaptions for the ongoing clerkship year.

Table 1

Student Evaluations Before and After Learning Management System Prototype Implementation at One Academic Teaching Hospital

	2018 Cohort without LMS Prototype (n = 107)	2019 Cohort with LMS Prototype (n = 83)
Evaluation-Item Statements	Average (SD)	Average (SD)	Difference
“I was effectively introduced to the clerkship rotation”	5.14 (0.27)	5.49 (0.33)	+0.35*
“I knew the theoretical and practical learning objectives”	4.57 (0.43)	5.01 (0.35)	+0.44 (n.s.)
“The clerkship curriculum was well structured”	4.98 (0.51)	5.54 (0.41)	+0.56*
“I felt integrated in the team”	5.28 (0.36)	5.47 (0.43)	+0.19 (n.s.)
“In case I didn’t know something, I found the help I needed”	5.45 (0.34)	5.64 (0.17)	+0.19 (n.s.)
“I was inspired to learn during this clerkship”	4.65 (0.46)	4.94 (0.42)	+0.29 (n.s.)
“Overall, I learned a lot in this clerkship rotation”	4.82 (0.47)	5.21 (0.51)	+0.39 (n.s.)
“I would recommend this teaching hospital without reservation”	4.70 (0.57)	5.17 (0.56)	+0.47 (n.s.)
“The e-Learning material on the LMS was helpful”	n.a.	4.3 (0.88)	n.a.
Overall satisfaction	3.9 (0.26)	4.4 (0.47)	+0.5*
Teaching hospital ranking	3 out of 5	2 out of 5

Notes: *Mann–Whitney U-test. Statistically significant at the p < 0.05 level. Evaluation-items were answered on a 6-point-Likert scale and ranged from 1 = Completely disagree to 6 = Fully agree. The 5-point-Likert LMS item ranged from 1 = Disagree to 5 = Agree and for overall satisfaction from 1 = Unsatisfied to 5 = Highly satisfied.

Abbreviations: LMS, learning management system. SD, standard deviation. n.a., not applicable; n.s., not significant.

Data Collection and Analysis

Evaluation data were collected with paper-based, end-of-clerkship rotations for the 2018 cohort and through the LMS for the 2019 cohort. We used a 5-point Likert scale for the evaluation of student satisfaction with the overall clerkship curriculum and LMS use. Members of the research team took written notes from feedback rounds and informal interviews with teaching residents (n = 7) to iteratively inform LMS content development. All evaluation data were anonymized, organized in Excel (Microsoft Corporation, Redmond, WA, USA), and statistics were calculated using R (Version 1.2.1335, R Foundation for Statistical Computing, Vienna, Austria).

Participants and Ethics

For this study, research data from the medical students of the clerkship cohort of 2018 (n = 107) and 2019 (n = 83) were used. The LMS was officially provided by the University of Bern and only accessible to students and university staff. One author informed all participants verbally and in writing about the ongoing curriculum reform and study and received written consent from all participants for anonymized clerkship evaluation, LMS use and interview data. Regular participation in the clerkship was mandatory, but LMS use was optional. There was no coercion involved in making clerkship evaluation data available for this study. The Regional Ethics Committee of the Canton of Berne, Switzerland deemed this research exempt from additional ethical review. Confidentiality and anonymity with regard to electronic data were maintained throughout the study.

Results

The results of the analysis and exploration phase for developing an LMS prototype are shown in Figure 2 and Table 2. Clerkship evaluations of perceived learning support and the learning atmosphere for students with and without LMS exposure at the same teaching hospital are summarized in Table 1.

Table 2

Links Between Learning Theory and Learning Management System Pilot Implementation

Learning TheoriesΨ	Derived Opportunities for Digital Support of Clerkship Studentsτ	Pilot LMS* Implementation
Self-determination (“Why I learn”)
Relatedness	-Integration into a new clinical environment-Create the sense of a welcomingatmosphere	-LMS course invitation before starting the clerkship-Welcome video from the head of the department-Introduction video to the clinical workplace
Competence	-Scaffolding learning opportunities to develop a sense of competence	-Screencast video on working with electronic health records and finding relevant information
Autonomy	-Creating opportunities for students todefine individual learning goals andpriorities	-Online course forum to define and share learning goals
Self-directed learning (“How I learn”)
	-Create a learning environment that helps students control their learning process and breaks down complex clinical skills into manageable units-Make relevant resources easily accessible	-E-learning module on motivational interviewing with role-modelling of required communication strategies-Online repository with essential literature and material-Learner activation tools in an orientation seminar (live voting)
Self-regulated learning (“Am I learning?”)
	-Provide opportunities for students to engage with their progress in achieving learning goals and making adjustments to their learning process-Stimulate reflections on the need for supervision for core learning objectives	-Online self-assessment of the perceived need for supervision according to competency-based learning goals-Online formative self-assessment of knowledge (multiple-choicequestions)

Notes: *LMS, Learning management system. ΨAdapted from Schumacher et al (2013) and Artino et al (2015). τBased on work of Ryan and Deci (2000) on self-determination, Knowles (1975) on self-directed learning, and Schunk and Zimmerman (1998) on self-regulated learning.3,26,30–32

The clerkship curriculum in 2018 was based on four weeks full-time presence in the teaching hospital with work on the wards and face-to-face teaching (columns 1 and 3 in Figure 2) and did not include any e-Learning material. The overall satisfaction-rating average was 3.9 (5-point Likert scale, 1 = unsatisfied, 5 = highly satisfied) and the teaching hospital was ranked third out of the five academic teaching hospitals in psychiatry and accommodated the highest number of students per year per teaching hospital (Table 1). Although many students wrote that they had experienced high-quality clinical supervision and had profited from a rich learning environment, there were some who reported long periods of down-time, or simply shadowing clinical residents, and that they had not been able to do any work. Furthermore, we saw several written evaluations that illustrated how some students were highly dependent on residents to structure their learning experience. Other challenging factors identified through interviewing clinical residents included regular resident turnover due to residency training requirements, and a broad range of residents’ experiences in the clinical teaching of clerkship students.

The target educational challenge was defined as the support of self-regulated learning activities of novice clinical students using an LMS. The non-systematic literature search helped us to identify theories of self-determination (factors influencing the desire to learn), self-directed learning (control over the personal learning process), and self-regulated learning (monitoring learning achievements) as relevant frameworks (Table 2). In addition, we were able to identify several studies on blended learning possibilities in clerkships to inform the design of our LMS prototype.6,13,16,17,20,21,23,25,34 We found no LMS implementation studies in the context of clerkships. To implement the LMS and create clerkship-specific content, we assembled a team consisting of clinical experts, medical education specialists, researchers, and e-learning specialists. One researcher enrolled in a modular faculty development program and used the course material for the ongoing screening of the literature and LMS concept development (Figure 2). Participation in specialty-specific educational conferences helped to identify practice examples for working with e-learning material in clerkships.

The LMS prototype was first offered in March 2019, and content was created based on the EDR phase 1 results, informed by learning theory (Table 2), and continuously adjusted based on iterative clerkship evaluations and student feedback on the learning atmosphere and self-regulated learning behavior. For each pilot element, educational objectives, instructional activities and materials, and assessment and feedback procedures were specified in the clerkship syllabus. The clerkship orientation seminar was redesigned, and the LMS prototype was used to support a flipped-classroom approach to virtually introduce students to the clinical workplace using a pre-clerkship questionnaire and videos (a formal welcome video of the head of the department and annotated screencast videos for working with electronic health records). Furthermore, the LMS was used to activate students during the orientation seminar with a live-voting tool to explore their career interests and make their learning preferences visible. Students were asked to post their personal learning priorities in an online forum of the LMS during the first day of the clerkship.

We used the LMS to administer formative knowledge tests to (re-) activate relevant knowledge (a total of 40 multiple-choice-questions) at the start and end of the clerkships. To support students with transferring motivational interviewing skills to the clinical workplace, we developed a learning module embedded in the LMS that built on pre-clerkship communication training. This module contained four new videos with a clinical psychologist and an actress as a patient. Self-assessment questions were used for knowledge activation together with a short, exemplary, case vignette. Four video scenes showed different phases of motivational interviewing (engaging and establishing a therapeutic relationship, focusing, evoking, and planning). These videos were intended as “on-the-run” go-to examples for students to prepare for their patient interviews.

Relevant material, including textbook references, scientific articles, working documents, and links to additional web material were continuously collected and evaluated for use as part of the online student clerkship library. These were also intended for students’ preparation for the student research paper conference. For clerkship quality management, we developed an evaluation questionnaire and administered it through the LMS.

The average overall clerkship satisfaction rating for our teaching hospital increased after introducing the LMS prototype (Table 1) from 3.9 to 4.4 (5-point Likert scale, 1 = unsatisfied, 5 = highly satisfied). The teaching hospital improved its ranking to second out of the five academic teaching hospitals in psychiatry based on 30-item monthly evaluations in 2019. The LMS prototype received an overall rating of 4.3 (5-point Likert agreement scale to the statement “The E-Learning materials (ILIAS-course) were helpful” 1 = strongly disagree, 5 = strongly agree). The average evaluation rating was also higher for every other evaluation item, although only three items reached statistical significance in the student cohort exposed to the LMS prototype (introduction to the clerkship, curriculum structure and global satisfaction). Nearly all students (94%) used the LMS prototype. The average number of times the LMS course was accessed per student was 70 (range: 7–172) and the average duration students spent online in the course was 58 minutes (range: 9 −165). A third of all students (35%, n = 29) accessed the course material after their clerkship rotation had ended. The orientation seminar based on a flipped-classroom approach received an average school grade of 5.6 (range: 4–6, Swiss grading system, 1 = worst grade, 6 = top grade). The student research paper conference (with research paper repository and instructions on the LMS) received an average grade of 5.3 (range: 3–6). The perceived need for supervision for motivational interviewing interventions (supported with teaching videos in the LMS) decreased on average by one supervision level on a 6-point Likert supervision scale (from 2.8 to 3.6, 1 = student only observes, 6 = independent with indirect supervision). Students mentioned the helpful structure of the content on the LMS in written evaluations, and immediate graphical presentations of monthly evaluations in the LMS were perceived as helpful by clerkship administrators.

Discussion

Summary of Findings

We used an EDR approach to develop, implement, and evaluate an LMS prototype in the context of a traditional, core clerkship in psychiatry to support student self-regulated learning. Our LMS prototype included flipped-classroom elements, in-class activation, standardized clinical skill demonstrations, self-assessment questionnaires, and a key literature repository. Internal and external evaluation ratings improved after introducing the LMS prototype at our teaching hospital and indicated that students exposed to the LMS prototype reacted positively to the affordances of an LMS and that self-regulated learning activities were effectively supported.

Analyzing clerkship evaluations of a full clerkship year in combination with searching the literature led to a richer and more in-depth understanding of the target educational challenge of supporting students’ self-regulated learning with an LMS. Some of the identified challenges will differ according to the clerkship context. Complex clinical scenarios, stigma, and threshold concepts such as the biopsychosocial model might be more relevant for choosing LMS content in psychiatry clerkships35 as compared to surgical clerkships36 and requires further exploration. However, the method of linking educational challenges and empirical studies back to underlying theories of learning was an inspiring process for identifying opportunities for student support using an LMS.28 This approach might be particularly helpful for de novo implementations of LMS prototypes in clerkship curricula.

The identified theories and studies informed the iterative LMS construction and implementation processes, as well as the design of the digital clinical skill module. The LMS we used allowed for designing and using e-learning tools at different levels of curriculum planning. These included preparing the integration into a new clinical workplace,1 flipping elements of integration and welcoming students,22 activating students during a face-to-face seminar, activating prior knowledge with multiple-choice questions, and providing students with online demonstrations of clinical skills.6 However, these steps were the first attempts at designing LMS elements for a clerkship curriculum. More in-depth and granular cognitive task analyses will be necessary to fully leverage the potential of LMS-based learning support in clerkships.37

For this early phase of a long-term EDR project, we used student satisfaction ratings, LMS user data, and feedback and reflection rounds with students as indicators of implementation success and to address the target educational challenge of supporting self-regulated learning.32,33 We found higher overall student satisfaction in the cohort exposed to the LMS prototype and in almost all students engaged with the LMS content. These evaluation data were collected through an independent evaluation department and not designed by the LMS prototype developers. As this is the first phase of an EDR project, the data presented here show that introducing an LMS prototype does not appear to have any unintended negative effects. As a consequence of feedback rounds in the first months, we realized the importance of orienting students in regard to the learning process based on the LMS and its content, and reemphasizing self-regulated learning opportunities explicitly during the orientation seminar. The orientation seminar was, therefore, modified accordingly. To better understand the impact of an LMS prototype on self-regulated learning and to avoid potential overwork of learners, future studies should include additional data sources such as workplace-based assessments and standardized questionnaires for self-regulated learning.26 We did not find any LMS implementation studies in undergraduate clinical education to compare our findings to. However, research in business education has shown that user satisfaction was positively related to LMS effectiveness from the learner perspective.38

Strengths and Limitations

To our knowledge, this is one of the first studies that systematically explored the development and implementation of an LMS prototype in the undergraduate clinical education context using an EDR approach. Strengths of this study include the use of learning theory to inform the design and implementation process and multiple data sources to evaluate the outcome of the LMS prototype implementation from the learners’ perspective. As students only retrospectively evaluated their learning experience, we did not consider self-complacency bias as relevant in this study. One limiting factor of this study was the summarizing of early results from working with an LMS prototype in a core clerkship on the learners’ satisfaction level and prototype usage data. Therefore, these evaluation data are preliminary and follow-up studies on higher educational learning outcomes are needed.

Conclusion

Learning theories can be used to systematically develop and implement an LMS prototype in the context of undergraduate clinical education. Our preliminary exploration and implementation data of an evidence-oriented LMS prototype in a psychiatry core clerkship indicate that it might positively affect student satisfaction and support student self-regulated learning activities. Future research should explore which elements of an LMS most effectively help to achieve educational outcomes.