Self-regulated learning and critical reflection in an e-learning on patient safety for third-year medical students.

OBJECTIVES: To explore the influence of critical thinking, self-regulated learning and system usability on the acceptance of e-learning on patient safety.
METHODS: A cross-sectional study was conducted, using a 32-question online survey. One hundred ninety-three (n=193 medical students participated in the study and were asked to rate levels of reflective thinking, self-regulated learning and attitudes towards patient safety using scales from the Questionnaire for Reflective Thinking, the Attitudes to Patient Safety Questionnaire and the System Usability Scale. Differences between reflection levels were calculated using paired t-tests, associations between critical thinking and self-regulated learning were calculated using linear correlations. We performed linear multiple regression analysis to identify predictors for student acceptance of the e-learning.
RESULTS: Students (n=193) engaged in intermediate levels of reflection (5-point Likert, M=3.62, SD=0.73) and significantly (t(143)=15.15, p<0.001, d=1.57) lower levels (M=2.35, SD=0.87) of critical reflection. Most students showed high (≥ 4; 44.1%) or intermediate (<4 level > 2; 29.4 %) levels of self-regulated learning. A regression model indicated that 5 predictors (Reflection, critical reflection, self-regulated learning, relevance, usability) explained 65.3% of the variance (R²=0.653, F(5, 96)=39.02, p<0.01) of perceived total quality.
CONCLUSIONS: This study shows that reflection and learning skills are important factors for e-learning acceptance, but perceived relevance and system usability play a more important role.  From a didactic perspective, it is indispensable to provide the students with sufficient examples and links to professional practice to enhance the perception of relevance and to improve system usability permanently.

Introduction

E-learning has become a standard teaching approach in medical education. As part of the educational process, e-learning possesses characteristics that have the potential to improve upon traditional teaching methods: E-learning courses typically offer increased flexibility, adaptability, and student centeredness compared with traditional lectures, and provide far-reaching opportunities for collaboration and discussion. Thus, e-learning programs offer students the opportunity to engage in self-regulated and interactive learning at their own pace.

Increasingly, e-learning is being used to teach and improve patient safety, which is a primary goal of healthcare systems worldwide. To enhance patient safety, both researchers and practitioners recommend (among a diversified set of actions) a change in the organizational culture towards a culture of safety. Such cultural changes require individuals’ willingness and ability to deeply reflect on actions, values, and beliefs to develop new mental models in which safety is central. An important foundation for developing a culture of safety involves acquiring specific knowledge about elements of patient safety. Given the significance of this knowledge, and of individual reflection, values, and beliefs, e-learning programs that provide ample opportunities for self-regulated learning may indeed be especially suited to prepare students for subsequent practice-based learning experiences related to patient safety (e.g., simulations). The abilities to reflect on a situation, the self and the learning process are not only essential to self-regulated learning, but are also fundamental for successful experiential learning through simulation exercises or bedside learning.

Self-regulated learning and the perception of relevance

Self-regulated learning strategies can be considered key assets in guiding the individual learning process, as they show positive correlations with academic success in e-learning settings. For self-regulated learning, the learner must regulate a set of three layers: 1) the choice of cognitive strategies; 2) the use of metacognitive knowledge; and, 3) the choice of goals and resources deployed for learning. The first layer (choice of cognitive strategies) is largely influenced by motivational factors and the perception of the relevance of the topic. In this regard, technical aspects of system usability and content-related aspects of the course will have a great influence on this layer of self-regulated learning, as they both influence the learners’ motivation to complete an online course.  The use of metacognitive knowledge requires reflection on one’s own learning process, and thus critical thinking. Critical thinking helps self-regulated learners to structure their learning process (third layer: regulation of the self) and thus facilitate deep processing of new knowledge.

E-Learning Patient Safety (ELPAS)

Against the background of self-directed learning, we developed an interactive e-learning course for third-year medical students to acquire basic competencies in patient safety. Based on the learning objectives that were developed by the World Health Organization4 we created a course titled E-Learning Patient Safety (ELPAS), which focused on two major aspects: teamwork and error management. The course consisted of two modules. Module 1 focused on teamwork, and included both scientific background (e.g., Big Five in Teamwork) and practical application via video case studies. Module 2 addressed the field of error management, grounded in the work of Reason.,, In the second module, students had to apply theoretical knowledge in case studies using the London Protocol. Our course used differentiated content (e.g., journal articles, videos, interactive quizzes, and podcasts) to provide patient safety knowledge. Unlike a traditional lecture (or recorded lecture), learning with ELPAS focused on self-regulation: Users could decide individually on learning time, learning speed, and which content they used to study. Students worked in small groups and used Web 2.0 technology (e.g., etherpads and discussion boards) for discussions with peers and tutors. Online quizzes enriched the learning material to allow frequent self-assessment. Additionally, students were required to complete a set of five assignments. Two of the assignments were classical multiple-choice tests, while the other three were specifically designed to foster reflection. In two of these three assignments, students collaborated in groups of six and developed solutions for case studies in which they had to reflect on real cases presented to them as a report or short video. The third of these three assignments required critical peer feedback on their recensions of a scientific paper on patient safety. Thus, to achieve the learning goals, students had to engage in self-regulated learning to master the basic knowledge, which they had then to apply interactively to a number of case studies. Both online modules were available on the learning management system of the university, which is used throughout the school, and thus well known to the students. ELPAS was mandatory for all third-year medical students at our university.

Aim and research hypotheses

The study aimed to identify educational and technical variables that help to explain students’ acceptance of the ELPAS course. Based on self-regulated learning theory, we hypothesized that: a) students who showed higher levels of critical thinking (reflection and critical reflection) would engage more in self-regulated learning, and b) students who showed more self-regulated learning behaviors would feel more comfortable with the course and thus rate ELPAS better than those who engaged less in self-regulated learning. Furthermore, we expected that: c) there would be an association between the perceived relevance of the topic and evaluation results. Following these hypotheses, we explored which predictors were relevant for e-learning acceptance by medical students. Referring to Davids and colleagues, we also expected that: d) evaluation results would be influenced by the perceived usability of the e-learning system. Clarification of these hypotheses may help to guide future developments in the field of e-learning.

Methods

Study design and participants

This was a cross-sectional survey study, using a 32-question online survey. A total of 193 medical students (57% of invited participants) participated in the study, 68% of whom were female. Most participants were between 20 and 29 years old (89%), and only some of them (21%) had prior experience in a healthcare profession such as nursing or pre-hospital care. Age and sex distributions of the sample were comparable to the total population in this cohort (338 students). As participation was voluntary, and no incentive was offered for participation, the response rate was within the expected range.

The study was planned and conducted in compliance with the Declaration of Helsinki on ethical principles for research involving human subjects. Participants were informed about the goals of the study and were told that participation was completely voluntary. Participants were informed that they could terminate their participation at any time without any negative consequences. The survey was completely anonymous (i.e., no personally identifiable data was collected, retrieved, or reviewed at any stage of our analysis). The Ethics Committee of Freiburg University approved the protocol.

Data collection method

The survey was administered to 338 third-year medical students through personalized links. We used a third-party survey tool to guarantee anonymity. The survey contained scales for self-regulated learning, reflective thinking, perceived relevance, and system usability. The self-regulated learning scale contained four items (e.g., “Through ELPAS, I could monitor my learning progress easily”) on a 5-point Likert scale ranging from 1 (“do not agree”) to 5 (“fully agree”) and was adapted from Reinhard, there were no prior psychometrics available for this scale. The reflective thinking scale measured both reflection (four items; e.g., “I sometimes question the way others do something and try to think of a better way”) and critical reflection (four items; e.g., “This course has challenged some of my firmly held ideas”) on a 5-point Likert scale ranging from 1 (“do not agree”) to 5 (“fully agree”). The scale was developed and tested by Kember and colleagues, and showed acceptable factor structure in a confirmatory factor analysis (c²=179.3, df=100, CFI = 0.903) and moderately strong reliability (reflection: ɑ=0.63, critical reflection: ɑ=0.68). To measure perceived relevance (two items: “Teaching students about patient safety should be an important priority in medical students training”; “Learning about patient safety issues before I qualify will enable me to become a more effective doctor”), a subscale from the German Attitudes to Patient Safety Questionnaire (GAPSQ) was used; its reliability was tested by the original authors (ɑ = 0.74). To measure system usability, we used the robust System Usability Scale (SUS)., Students also rated the quality of nine elements of the e-learning on a 5-point Likert scale ranging from 1 (“I think it is very bad”) to 5 (“I think it is very good”). As we were interested in the perception of the overall quality of the ELPAS course, we summed the single ratings and calculated their mean as an overall rating of the course. Prior to calculating the means for self-regulated learning, reflective thinking, perceived relevance, system usability, and total quality of the e-learning course, we analyzed the reliability of the scales using Cronbach’s alpha. All relevant scales (see Table 1) showed acceptable to good reliability.

Table 1

Descriptive results on the item-level and Cronbach’s Alpha for the scales

Scale	Item	Mean	SD	α
Self-regulated learning: 5-point Likert25		2.93	1.04	0.81
	ELPAS enabled me to arrange my learning time flexible.	3.76	1.23
	Due to the introduction of ELPAS, I dealt with the content more intensively than in a traditional lecture.	2.66	1.38
	Due to the introduction of ELPAS, I learned more independently than in a traditional lecture.	2.77	1.39
	Through ELPAS, I could monitor my learning progress easily.	2.51	1.21
Reflection: 5-point Likert18		3.62	0.73	0.71
	I sometimes question the way others do something and try to think of a better way.	2.68	1.21
	I like to think over what I have been doing and consider alternative ways of doing it.	3.96	0.92
	I often reflect on my actions to see whether I could have improved on what I did.	3.87	0.93
	I often reappraise my experience, so I can learn from it and improve for my next performance.	3.95	0.89
Critical reflection: 5-point Likert18		2.35	0.87	0.77
	As a result of this course, I have changed the way I look at myself.	2.34	1.17
	ELPAS has challenged some of my firmly held ideas.	2.23	1.14
	As a result of ELPAS, I have changed my normal way of doing things.	2.72	1.17
	During this course, I discovered faults in what I had previously believed to be right.	2.23	1.03
Relevance: 7-point Likert26		5.41	1.42	0.90
	Teaching students about patient safety should be an important priority in medical students training.	5.47	1.42
	Learning about patient safety issues before I qualify will enable me to become a more effective doctor.	5.36	1.54

Note: Figures in bold represent aggregated results of each scale

Statistical analysis

SPSS version 24 was used for statistical analysis. Prior to statistical analysis, the items were checked for plausibility, and missing data analysis was performed using the expectation-maximization algorithm (p>0.05). No item was eliminated from the analysis because of missing data. Differences between participant-reported reflection and critical reflection were tested for significance using paired t-tests.  Associations between critical thinking and self-regulated learning were calculated using linear correlations. We expected that perceived relevance, critical thinking, and self-regulated learning would explain large amounts of the total variance on overall rating and accordingly established a linear multiple regression model. Suspecting that system usability might be an important additional predictor, we included the usability results in the model. Significance for all tests was set at p<0.05. Effect sizes were computed according to Cohen’s d.

Results

In total, students rated the online course as “average” (M= 2.93, SD = 0.8) on the 5-point Likert scale for overall rating (1 = “I think it’s very bad”; 5 = “I think it’s very good”) and perceived the objectives as “relevant” (M = 5.41, SD = 1.42) on the 7-point Likert scale (1 = “don’t agree at all”, 7 = “fully agree”). While we did not evaluate the pre-existing computer skills of the participants, we did check whether the participants were overtaxed by the technical requirements or the content-wise requirements of the e-learning by using a 5-point Likert scale (1= “requirements are too low for me”; 3 = “fits very well”, 5 =“requirements are too high for me”). Students reported that the course fit well with their abilities with respect to content (M=3.13, SD= 0.63) and technical requirements (M=3.30, SD=0.64).

The 5-point Likert scales measured the overall levels of reflective thinking for reflection and critical reflection (5 = highest level). While most of the students showed intermediate to high levels of reflection when doing the online course (M=3.65, SD=0.73), the levels of critical reflection were significantly (t(143) = 15.15, p<0.001, d=1.57) lower (M= 2.38, SD = 0.88). Most students reported high (≥ 4; 44.1%) or intermediate (< 4 level > 2; 29.4 %) levels of self-regulated learning, while 26.6 % showed low self-regulated learning levels (≤ 2.0 on the 5-point scale). The results show a moderate correlation between reflective thinking and self-regulated learning (r=0.49, p<0.001), and a strong correlation between self-regulated learning and overall rating (r=0.61, p<0.001). See Table 1 for an overview of scale items and measures.

Reflection, critical reflection, self-regulated learning, relevance perception, and system usability were entered in a multiple regression analysis to predict overall rating. The results of the regression indicated that the five predictors explained 65.3% of the variance (R²=0.653, F(5,96)=39.02, p<0.01). Tests, if data met the assumption of collinearity, indicated that multicollinearity was not a relevant concern (relevance, reflection, critical reflection, self-regulated learning, system usability tolerance= 0.58–0.85, VIF = 1.5–1.73). Overall rating was primarily predicted by system usability (ß=.35, t(102)=4.56, p<0.001), followed by perceived relevance (ß=.25, t(102)=3.27,p=0.001) and critical reflection (ß=.22, t(102)=2.94, p=0.004). To a lesser extent, self-regulated learning (ß=.17, t(102)=2.17, p=0.032) also predicted overall rating. Reflection as the remaining predictor showed only small, statistically non-significant beta coefficients (ß=0.10, t(102)=1.60, p=0.113). The raw and standardized regression coefficients of the predictors are shown in Table 2.

Table 2

Regression analysis

Model	B	SEB	ß	t	Sig.
Constant	0.279	0.259		1.077	0.284
Relevance	0.133	0.041	0.251	3.273	0.001
Reflection	0.113	0.070	0.103	1.603	0.112
Critical reflection	0.192	0.065	0.215	2.944	0.004
Self-regulated learning	0.126	0.058	0.174	2.174	0.032
System Usability	0.013	0.003	0.353	4.556	0.000

Note: Independent variable was overall rating “total quality of the E-Learning course”. R2=.670, adjusted R2=.653

Hypotheses a, b, c, and d are supported by our study: students with high levels of reflection or critical reflection reported higher self-regulated learning behavior, and students with high self-regulation rated the course better than those with low self-regulation.

The perceived relevance of the topic of patient safety was an important predictor for overall rating. Additionally, as hypothesized, perceived system usability influenced ratings of the total quality of the e-learning course.

Discussion

With this study, we tried to understand better, how educational variables predict students’ overall rating of the ELPAS course. Our analysis shows that most students engage in reflective thinking while taking the ELPAS course, whereas critical reflection is employed to a much lower extent. Reflection itself can be considered a self-regulated learning activity, and is associated with self-regulated learning in our data, however the associations remain rather weak. This might be explained by individual motivation: Even if students can reflect and do reflect, they might not be motivated to engage deeply in self-regulated learning. Although we did not measure motivation, we did measure the level of perceived relevance, which correlates significantly with reflection, and, therefore, arguably relates to the level of self-regulated learning.

Concerning Mezirow’s transformative learning theory, students will adopt significant changes of perspectives and thus engage in deeper knowledge processing strategies only when their level of reflection deepens towards critical reflection. Our analysis showed a pronounced association between critical reflection and self-regulated learning, indicating a medium-to-large effect size. In Boekaerts’ three-layered model of self-regulated learning, critical reflection is essential when regulating the self and defining individual goals (i.e., the outer layer of the model).With hypotheses b, c, and d, we aimed to understand predictors for total quality outcomes of the e-learning course, from an educational and instructional design perspective. The influence of self-regulated learning in the model was less pronounced than expected, but was still detectable and statistically significant, suggesting that self-regulation capabilities are important for succeeding in e-learning environments. The design of the e-learning system used in this study might help explain this: it facilitated self-regulated learning and metacognitive strategies by providing immediate feedback (monitoring strategies), linking relevant literature (resource strategies) and allowing discussions with peers (cooperation strategies). Hence, it supported those students who were unused to self-regulate their learning process independently. For students unable or unwilling to use cognitive and metacognitive strategies to improve their self-regulation, such online courses will have limited effects for either acquiring knowledge or develop beneficial attitudes towards patient safety.

Whether students perceive the specific content of a didactic session as relevant for their individual learning goals depends on various factors. Particularly with novice medical students, prior experience in the field of practice, and thus better ability to rate the practical importance of content, might influence the perception of relevance. However, in our study, students’ perception of topical relevance did not differ significantly between students irrespective of prior experience in healthcare professions. While the perception of relevance did not differ between the two groups, our analysis showed that those students who considered the content relevant for their professional development rated the course better than those who found the content less relevant. Therefore, we argue from a didactic perspective that it is essential to provide students with sufficient examples and links to professional practice to enhance their perception of relevance.

Besides educational predictors, we expected system usability to be a major predictor for overall quality rating. Indeed, system usability was the most pronounced predictor for overall quality rating but also showed strong correlations with self-regulated learning. These results suggest that improving the usability of an e-learning course will not only satisfy more students but will also support learning.-

We evaluated a novel comprehensive e-learning course on patient safety. While several reports exist for evaluations of online courses that were conducted to teach specific skills around patient safety (e.g., safety in blood transfusions and dosage calculation),,, only very few exist in which the topic of patient safety was approached holistically by an online course. Typically, patient safety education uses more face-to-face teaching methods such as simulation, role-playing, discussions, or games. Our study showed that patient safety education using a distance-learning methodology was acceptable to a significant proportion of the population in question and that improving usability might increase students’ acceptance of such courses. Thus, e-learning courses may become a valuable part of patient safety education and may be considered as complementary to more traditional training methods such as simulation or role-playing.

Limitations

This study measured the influences of self-regulated learning and various levels of reflection on students’ overall ratings of the quality of an e-learning course. Because the data collection was anonymized, objective indicators generated by the learning management system, including time logs or test results, were not available. Therefore, the outcome parameter was highly subjective and may have been influenced by several factors which are not accounted for in the study, and thus could not be controlled (e.g., individual learning preferences, prior experience with e-learning tools, and computer literacy). The generalizability of the results is limited because we only included medical students from Freiburg University, and the response rate of 57% was not optimal.

Moreover, the voluntary participation of the students may have introduced selection effects; it is also possible that motivated students were more likely to participate. The medium-to-strong correlations of the scales could be the result of overlapping constructs and common-method bias. We also relied on self-reported data, which may have given rise to single-source bias. Furthermore, the study’s design is cross-sectional and hence does not allow for a causal interpretation of the relationships found in the predictive model. Longitudinal or intervention studies should be conducted to examine the causality of the proposed relationships.

Conclusions

Our results suggest that students engage in critical thinking when they study in our e-learning environment on patient safety. Thus, we believe that through a combination of different learning tasks, reflection and critical reflection can be fostered through e-learning. This could be used to prepare students for subsequent interactive face-to-face sessions. However, students will need a distinct set of self-regulated learning skills to maximize the benefit of such learning environments. While reflection and learning skills are important for successful student engagement with e-learning environments, perceived relevance and system usability also play an important role with respect to students’ acceptance of an online course. When designing novel e-learning environments, medical educators should set standards towards system usability, as this will significantly affect students learning experience with the e-learning course.

Conflict of Interest

The authors declare that they have no conflict of interest.