Digital Learning to Improve Safe and Effective Prescribing: A Systematic Review.

With the aim to modernize and harmonize prescribing education, the European Association for Clinical Pharmacology and Therapeutics (EACPT) Working Group on education recommended the extensive use and distribution of digital learning resources (DLRs). However, it is unclear whether the complex task of prescribing medicine can be taught digitally. Therefore, the aim of this review was to investigate the effect of diverse DLRs in clinical pharmacology and therapeutics education. Databases PubMed, EMBASE, CINAHL, ERIC, and CENTRAL were systematically searched. Sixty-five articles were included in the analyses. Direct effects on patients were studied, but not detected, in six articles. Skills and behavior were studied in 11 articles, 8 of which reported positive effects. Knowledge acquisition was investigated in 19 articles, all with positive effects. Qualitative analyses yielded 10 recommendations for the future development of DLRs. Digital learning is effective in teaching knowledge, attitudes, and skills associated with safe and effective prescribing.

There is ample evidence that medical students are insufficiently trained in safe and effective prescribing.1 A recent European multicenter study showed that only one in four final‐year medical students chose the most appropriate therapy in a case‐based examination and that roughly half of their prescriptions contained one or more errors.2 Moreover, recently graduated doctors make more potentially hazardous prescribing errors than consultants.3 Fortunately, the need for urgent changes in international clinical pharmacology and therapeutics (CPT) education is becoming increasingly apparent, and the first steps toward improvement have been taken. Key learning outcomes were identified in an international multicenter Delphi study, and the European Association for Clinical Pharmacology and Therapeutics (EACPT) Working Group on education has published a list of recommendations to harmonize and modernize CPT education.4, 5, 6 One of these recommendations is for CPT teachers to use more (online) digital learning resources (DLRs) and preferably share these at a national or international level.5 These resources are not limited to (online or offline) e‐learning programs, but may also include more innovative ways of teaching, such as podcasts, simulations, serious games, and virtual or augmented reality. Indeed, the possibility to distribute high‐quality content and to reuse this extensively, after only a single investment of time and money, is one of the advantages of DLRs over traditional teaching methods. Other potential advantages for teachers include the possibility to rapidly update, revise, and standardize the content and the use of learning management systems to track individual learner progress. DLRs also have potential advantages for learners, such as the possibility to alter study pace and revise content extensively, to use multimedia and interactive elements, and to study anytime and anywhere. On the other hand, first generation DLRs may also have significant drawbacks because of the lack of interaction with teachers and peers.7 Studies of healthcare education have shown that DLRs increase knowledge compared with no education and are at least equal to traditional learning methods in terms of knowledge acquisition and learner satisfaction.8, 9 However, safe and effective prescribing is a more complex process, requiring the integration of cognitive skills (e.g., knowledge, problem‐solving, and decision making) and attitudes in a busy and potentially stressful workplace.10 Whether DLRs are suitable to teach these components in an effective manner has been questioned.7 Unfortunately, the effect of the diverse DLRs on safe prescribing remains largely unclear, and none of the existing review articles on digital health education have focused on prescribing. DLRs are often evaluated in terms of subjective satisfaction and knowledge acquisition only, and studies evaluating more clinically relevant outcomes are scarce because they are costly, time‐consuming, and limited by numerous confounding factors.7, 11 In order to help implement the EACPT working group recommendation on internationally distributed DLRs, more information about their effects on prescribing is required. Therefore, the aims of this review are to identify published forms of DLR used to teach rational prescribing and to assess their effect on learner‐related and clinically relevant outcomes, with a view to determining which elements are effective in teaching prescribing.

Literature Review

The database search yielded 2,110 records, 125 of which were eligible on the basis of title and abstract. Fifty‐nine full‐text articles were eligible for inclusion.12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70 Six additional articles were identified from the previously used database,71, 72, 73, 74, 75, 76 making a total of 65 articles. An overview of all included articles (with effect‐estimates when available) is presented in Table S1. The complete selection process, as per the Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) guidelines, is shown in Figure 1.

Figure 1

Preferred Reporting Items for Systematic Reviews and Meta‐Analyses (PRISMA) flow diagram for study selection. Search strategies used for both databases are described in Table S2. CINAHL, Cumulative Index to Nursing and Allied Health Literature; ERIC, Education Resources Information Centre.

Digital learning resource characteristics

Table 1 shows the characteristics of the digital teaching interventions. Most (N = 55; 85%) articles described the use of (online) e‐learning,12, 13, 14, 15, 16, 17, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 36, 37, 38, 40, 41, 43, 44, 48, 49, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 76 11(20%) of which described e‐learning combined with traditional classroom teaching, making the intervention blended learning.15, 19, 21, 26, 33, 38, 41, 62, 65, 66, 72, 73 Eight articles (15%) combined e‐learning with some sort of digital communication channel, such as Web forums or chatrooms for interaction with peers and teachers.17, 21, 28, 36, 40, 51, 63, 66 Of the six articles (9%) describing a digital assessment, only one was not combined with e‐learning.45 Nine articles (14%) described more unique learning interventions: gamified e‐learning,18 a text‐based virtual patient,35, 64 an augmented reality virtual patient,50 e‐learning with email case discussions,75 a wiki‐like student drug formulary,46 a drug‐dosage simulator,39 a digital drug repository,42 and informative podcasts.47 Fifty‐one percent of interventions were aimed at postgraduate physicians,12, 13, 15, 18, 19, 20, 23, 24, 27, 28, 29, 30, 31, 33, 34, 36, 40, 42, 50, 54, 55, 58, 59, 60, 62, 64, 65, 67, 68, 71, 72, 73, 74, 75 25% were aimed at nonmedical prescribers (all of whom were nurses),15, 17, 18, 21, 22, 29, 35, 44, 47, 58, 61, 62, 63, 64, 66 20% at medical students,14, 25, 32, 37, 39, 43, 45, 46, 49, 53, 56, 57, 76 and 11% at pharmacy students.14, 38, 48, 51, 52, 69, 70 Nine postgraduate courses (14%) were aimed at prescribers from more than one background;15, 18, 29, 58, 61, 62, 63, 64, 71 one was an undergraduate course (the Australian National Prescribing Curriculum).14 Elements such as anywhere, anytime availability and interaction were mostly present, but many interventions were not described in detail.

Table 1

Characteristics of the digital learning resources

Intervention characteristics	N (%)
Type of learning intervention
E‐learning	55 (84.6)
Blended learning	12 (18.5)
Digital assessment	6 (9.2)
Social learning	9 (13.8)
Other	9 (13.8)
Target population
Physicians	49 (75.4)
Nurse prescribers	16 (24.6)
Pharmacists	12 (18.5)
Dentists	4 (6.2)
Education level
Postgraduate	44 (67.7)
Final year	4 (6.2)
Undergraduate	17 (26.2)
Characteristics
Compulsory
Yes	16 (24.6)
No	26 (40.0)
Not reported	23 (35.4)
Anytime–anywhere availability
Yes	42 (64.6)
No	4 (6.2)
Not reported	19 (29.2)
Type of assessment
Formative	20 (30.8)
Summative	12 (18.5)
Not reported	33 (50.8)
Interactivity
Yes	41 (63.1)
No	3 (4.6)
Not reported	21 (32.3)
Case oriented
Yes	34 (52.3)
No	2 (3.1)
Not reported	29 (44.6)

Multiple interventions and populations possible, percentages add up to >100%. Others may not add up to 100% due to rounding of percentages.

Research designs and study quality

Ten articles (15%) had a descriptive design,14, 25, 32, 39, 41, 49, 50, 58, 63, 76 11 (17%) a qualitative design,13, 15, 19, 20, 21, 30, 35, 46, 47, 57, 66 and 44 (68%) a quantitative design.12, 16, 17, 18, 22, 23, 24, 26, 27, 28, 29, 31, 33, 34, 36, 37, 38, 40, 42, 43, 44, 45, 48, 51, 52, 53, 54, 55, 56, 59, 60, 61, 62, 64, 65, 67, 68, 69, 70, 71, 72, 73, 74, 75 Twenty‐six of the quantitative articles (59%) used a randomized23, 24, 28, 29, 31, 34, 37, 40, 52, 56, 59, 64, 65, 67, 72, 73, 74, 75 or nonrandomized22, 27, 38, 54, 60, 62, 69, 70 trial design; the remaining 18 articles (41%) either used no control or compared one group before and after the intervention.12, 16, 17, 18, 26, 33, 36, 42, 43, 44, 45, 48, 51, 53, 55, 61, 68, 71 The nature of the control group was diverse: five articles (11% of quantitative) compared the use of clinical practice guidelines to DLRs,27, 28, 37, 59, 64 and six (14%) compared traditional face‐to‐face teaching with e‐learning.22, 34, 38, 54, 60, 65 Blended learning was compared with e‐learning in only two articles (5%).38, 62 There were also numerous less relevant control groups: eight articles (18%) compared the DLR to no education at all,23, 24, 29, 31, 40, 69, 72, 73 and four articles (9%) compared standard education to standard education with extra e‐learning.52, 56, 70, 75 The scores for the separate Medical Education Research Study Quality Instrument (MERSQI) domains are shown in Table 2. The mean MERSQI was 11.6 ± 3.0; there were 8, 22, and 14, articles of low, medium, and high quality, respectively.

Table 2

Study designs and quality

Study characteristics	N (%)
Research design
Descriptive	10 (15.4)
Qualitative	11 (16.9)
Quantitative	44 (67.7)
MERSQI (N = 44 quantitative)
Overall (mean ± SD)	11.6 ± 3.0
Low (5.0–8.5)	8 (18.2)
Medium (9.0–13.0)	22 (50.0)
High (13.5–18.0)	14 (31.8)
Study design (MERSQI score)
Single group, cross‐sectional (1)	11 (25.0)
Single group, before and after intervention (1.5)	7 (15.9)
Prospective cohort (2)	8 (18.2)
Randomized controlled trial (3)	18 (40.9)
Study outcomes – highest Kirkpatrick level (MERSQI score)
Learner attitude – Kirkpatrick level I (1)	11 (25.0)
Knowledge acquisition – Kirkpatrick level II (1.5)	17 (38.6)
Learner behavior – Kirkpatrick level III (2)	10 (22.7)
Patient outcomes – Kirkpatrick level IV (3)	6 (13.6)
Sampling: Institutions (MERSQI score)
Single institution (0.5)	22 (50.0)
Two institutions (1)	5 (11.4)
Three or more institutions (1.5)	17 (38.6)
Sampling: Response rate (MERSQI score)
<50% or not reported (0.5)	13 (29.5)
50–74% (1)	6 (13.6)
≥75% (1.5)	25 (56.8)
Type of data (MERSQI score)
Assessment by study participant (1)	9 (20.5)
Objective (3)	35 (79.5)
Instrument validity (MERSQI score)
Not applicable	1 (2.3)
Content validity reported (1)	12 (27.3)
Internal structure reported (1)	6 (13.6)
Relationship with other variables reported (1)	7 (15.9)
Data analysis: Sophistication (MERSQI score)
Descriptive statistics only (1)	8 (18.2)
Beyond descriptive statistics (2)	36 (81.8)
Data analysis: Appropriate
Yes (1)	34 (77.3)
No (0)	10 (22.7)

Instrument validity consists of three independently scored items (reported or not reported) and, therefore, does not add up to 100%. Others may not add up to 100% due to rounding of percentages.

MERSQI, Medical Education Research Study Quality Instrument.

Head‐to‐head comparisons

Four of five studies comparing DLRs to the use of clinical practice guidelines found no between‐group difference. The other article reported a significant increase in knowledge (8.4 ± 0.8 vs. 6.1 ± 1.3 on a 9‐question scale) and self‐reported competence (3.8 ± 0.6 vs. 3.5 ± 0.7 on a 10‐point Likert scale from incompetent to competent) in opioid prescribing after interactive web‐based training as compared with guideline only.59 Articles directly comparing traditional teaching methods to DLRs reported equal effects in two,38, 54 higher knowledge test scores after interventions with e‐learning in two,60, 65 and a better performance on knowledge tests after a traditional lecture in one.22 One article reported a 5% increase of prescribing errors made by physicians after e‐learning, but a decrease from 58% to 37% after pharmacist‐led feedback and targeted education.34 The two articles comparing blended learning with nonblended DLRs found the intervention to be beneficial, but no relevant between‐group differences were found.

Patient outcomes (Kirkpatrick level IV)

Six articles reported outcomes at a patient level.12, 28, 42, 55, 73, 74 One found a relative risk of antibiotics being preferred by the patient of 0.48 (95% confidence interval 0.34–0.68) after an online training on shared decision making.73 Another evaluated the effect of online training in the use of a point‐of‐care CRP test and online enhanced communications training, when these interventions were combined a relative risk of 0.38 (95% CI 0.25–0.55) was observed.74 Both articles report no harm to patients. Three articles found no effects on prescribing errors of online trainings on the management of diabetes, elderly, and chronic opioid use.12, 28, 55 One found no effect of a series of interventions, one of which was the introduction of a digital drug repository and e‐learning.42

Behavioral outcomes (Kirkpatrick level III)

Eleven studies investigated behavioral change in learners.24, 29, 34, 56, 60, 62, 65, 68, 72, 74, 75 Three found no clear benefit on prescribing errors as measured with medication chart reviews,34, 60, 65 whereas the other eight reported positive effects, such as a reduction in antibiotic prescriptions,24, 72, 74 the correct use of a Broselow pediatric dosing tape,29 and improved legibility and completeness of written prescriptions.62

Knowledge outcomes (Kirkpatrick level II)

All 19 articles that used knowledge acquisition as outcome reported the intervention to increase test scores as compared with preintervention testing or no education.16, 18, 23, 31, 37, 38, 45, 48, 51, 52, 54, 56, 59, 60, 64, 65, 69, 70, 71 Three studies reported sustained effects on knowledge lasting up to 6 months.23, 31, 59

Learner satisfaction (Kirkpatrick level I) and self‐reported outcomes

Twenty‐one articles reported the reaction of participants, mostly assessed using 5‐point Likert scales or open questions.17, 18, 19, 20, 21, 22, 27, 30, 31, 33, 35, 38, 40, 43, 51, 53, 54, 61, 65, 66, 67 All results were positive, but no structural differences between DLRs and traditional teaching methods were identified. Thirteen studies found an increase in self‐reported confidence and knowledge on different domains of safe prescribing.12, 15, 23, 26, 31, 40, 48, 53, 55, 57, 59, 60, 67

Recommendations for future DLRs

Recommendations for elements to include in future DLRs for safe prescribing were compiled on the basis of recurrent themes mentioned in qualitative reports and are shown in Table 3.

Table 3

Recommendations for future digital learning resources on safe and effective prescribing

#	Element to include in DLR	Mentioned in
1	Anytime–anywhere availability. By far the most appreciated feature of DLRs is the possibility to study anytime and anywhere. It promotes just‐in‐time learning, puts the student in charge of his/her learning process, and enhances time efficiency. This is especially beneficial for postgraduates, because they often have little study time available because of their clinical duties and have external (e.g., continuous education points) or internal (e.g., interest in the topic) motivation to start learning.	13 articles15, 17, 20, 21, 26, 33, 35, 40, 43, 46, 47, 51, 70
2	Time demand. The anytime–anywhere availability impinges on time scheduled for other activities, including social life. This may lead to motivational problems, and DLRs are often perceived as too long and time‐consuming. Openness about the required time investment and offering sufficient “protected” study time may help overcome these problems. Additionally, the use of bite‐size chunks of information is advocated, in accordance with the cognitive load theory.	8 articles17, 20, 21, 22, 44, 53, 66, 70
3	Learning pace and revision. Associated with the anywhere−anytime availability is the possibility for students to alter pace, skip sections, and revise parts of the DLR extensively. This can be facilitated by structuring the DLR and offering clear navigational menus.	6 articles 22, 26, 33, 44, 47, 51
4	The use of quizzes and game design principles. Interactive elements, such as quizzes and other game design principles (e.g., competition with peers), are stimulating. Moreover, they help students to gauge their learning needs and may direct them to parts of the DLR that require revision.	4 articles21, 44, 47, 69
5	Feedback. The possibility to offer direct feedback on (multiple choice) questions is found to be helpful, but is limited to generic pregenerated content that may sometimes be insufficient.	6 articles22, 35, 43, 53, 57, 69
6	Contact with peers and teachers. The lack of social interaction with peers and teachers is not easily overcome by using digital communication channels, such as Web forums and email, as these features remained largely unused.	5 articles15, 22, 36, 51, 66
7	Content. Generic, irrelevant, or unauthentic content should be avoided. Instead, there should be a clear connection with predefined and clearly communicated learning goals. The DLR is appreciated if acquired skills and knowledge are directly applicable.	12 articles13, 15, 17, 18, 19, 20, 21, 22, 44, 51, 57, 69
8	Accessibility and design. Limited accessibility due to low internet connectivity, other IT problems, or insufficient computer knowledge of the users is highly demotivating, as is an outdated or visually unattractive design. Some accessibility issues may be limited by offering on‐demand computer support.	10 articles13, 15, 17, 18, 22, 44, 47, 51, 66, 69
9	Multimedia and learning styles. The use of multimedia, such as audio podcasts, videos, or animations, can be very helpful in explaining challenging content. However, different users prefer different learning styles, and it advisable to present content in various ways.	4 articles18, 20, 22, 44
10	Web links. Links to other resources (e.g., scientific articles, online drug formularies) may aid learning and are easily incorporated in the DLR. They can be distracting if too numerous or not entirely relevant.	5 articles17, 21, 35, 46, 66

DLR, digital learning resource; IT, information technology.

Discussion

To the best of our knowledge, this article is the first to systematically assess the effects of different types of digital teaching in the context of prescribing education. E‐learning, blended learning, and digital assessment are commonly used in prescribing education, and the effects on knowledge acquisition are comparable with those of traditional teaching methods. Additionally, DLRs seem to be effective in teaching skills associated with safe and effective prescribing, but whether this directly benefits the patient remains questionable. The most advantageous and disadvantageous elements, as identified from qualitative reports, are compiled into recommendations for best practice.

Types of DLR

Novel techniques, such as augmented reality and serious games, are increasingly used for training healthcare providers. Perhaps unsurprisingly, these techniques focus primarily on teaching the visuospatial aspects of subjects, such as anatomy and surgery.77, 78 While Nifakos and Zary50 describe the potential to combine virtual patients with augmented reality for teaching safe and effective prescribing, only two studies actually used a text‐based (not augmented) virtual patient for this purpose.35, 50, 64 Bond et al.18 reported the use of serious game design principles in their e‐learning program, but no serious games were identified. The use of an antibiotic dose‐response simulator for teaching dose regimens was described but was not assessed for effectiveness.39 Most DLRs were produced and used locally. Only Australia (National Prescribing Curriculum79), England (SCRIPT and Prescribe80, 81), and the Netherlands (Pscribe82) have nationally available prescribing e‐learning modules. An example of a more international DLR is the Teaching Resource Center (TRC), produced by the Centre of Human Drug Resource (CHDR) and Leiden University Medical Centre in the Netherlands.83, 84 Although this longitudinal program is used for teaching the six‐stepmodels of rational prescribing from the World Health Organization's Guide to good prescribing, it was not included in the current analyses because published articles on the TRC focused on basic pharmacology rather than on prescribing training. It is, however, a good example of how the free distribution of resources could lead to widespread and international adoption. Higher numbers of users will increase cost‐effectiveness, and it is known that sharing DLRs as open educational resources (free to use, modify, and distribute derivatives) enhances their timely improvement and overall quality.85 Unfortunately, only one of the included articles mentioned sharing the DLR in a truly open manner.43

Effects of DLRs on safe and effective prescribing

A recent overview of review articles on medical education found that DLRs are at least as efficacious as traditional teaching methods when it comes to knowledge acquisition and student satisfaction.8 However, these results were found for healthcare education in general and might not be applicable to prescribing education, because prescribing is a complex skill that is affected by the prescriber's attitude and by the workplace environment.10 Moreover, the effects of DLRs on skills and patient outcomes are less evident and often reported for specific skills (e.g., intubation) only.86, 87 This review identified several effects of DLRs on diverse skills and attitudes associated with safe and effective prescribing, such as moderation of antibiotic use. Therefore, it seems that DLRs are efficacious in teaching safe and effective prescribing skills and attitudes. However, we did not find DLR use to be accompanied by a reduction in prescribing errors or by a benefit to patients. This is probably because the process of prescribing medication is multifactorial and aspects, such as workplace environment and workload, may influence the results.88 Most studies assessing effects on knowledge acquisition only compared outcomes before and after the intervention or did not use a control. Unsurprisingly, these studies found positive effects of DLRs. Of the five studies directly comparing pharmacotherapeutic DLRs with traditional teaching methods, two favored DLRs, two had equivocal findings, and one favored traditional teaching methods. However, the effect of DLRs relative to guideline use and for other healthcare disciplines is well established, and, overall, DLRs are found to be no better or worse than traditional teaching methods.8, 9 Sikkens et al.56 reported a mixture of knowledge and behavioral outcomes by examining students by means of a simulated pharmacotherapeutic consultation with a patient actor. This type of objective structured clinical examination (OSCE) may be the best surrogate we have to real‐life prescribing for undergraduate students. Interestingly, they found that students performed significantly better on the OSCE 6 months after the e‐learning intervention than did students who did not follow the e‐learning intervention. Several other articles reported sustained effects for up to 3 months, suggesting that DLRs have a long‐term effect on knowledge. Overall, learners are well satisfied with DLRs for safe and effective prescribing. However, a positive feeling about DLRs is no guarantee for their quality, and it is more interesting to find which elements of DLRs improve quality.7

Recommendations for the production of DLRs for safe prescribing

The Association for Medical Education in Europe (AMEE) guide #32 on e‐learning in medical education highlights the potential advantages of DLRs, but provides relatively little information about which elements to include or avoid when producing a DLR.89 In addition to providing a practical framework to create postgraduate medical DLRs, de Leeuw et al.90, 91, 92 identified relevant elements on the basis of focus interviews and an international Delphi study. Although their research focused on postgraduate continuous medical education in general, the recommendations are likely to be relevant for (undergraduate and postgraduate) prescribing education as well. Unfortunately, owing to heterogeneous study designs, outcomes, and insufficient reporting of DLR characteristics (e.g., presence of cases and interactive quizzes), quantitative analyses were not feasible. However, on the basis of qualitative reports, we were able to compile a list of 10 recommendations for the future development of DLRs for safe and effective prescribing (Table 3). These recommendations are in accordance with those previously identified for (postgraduate) medical education in general.8, 90 The most appreciated element is anytime−anywhere availability. Not only may this save the learner precious (travel) time, but the autonomy associated with this is a pillar of adult learning theory and stimulates just‐in‐time learning, improving the relevance and applicability of the content. On the other hand, the anytime−anywhere availability affects the learner's time scheduled for other (e.g., social) activities. This is one of the reasons why DLRs are often perceived as requiring too much time. Practical solutions exist (e.g., offering protected study time), but this does not address the underlying problem: a lack of motivation. Unsurprisingly, many of the other recommended elements (game design principles, multimedia, and directly applicable content) aim to increase the learner's motivation.90 Many articles report the lack of peer and teacher interaction to be disadvantageous or mention the presence of such contact (with blended learning) as beneficial. Several DLRs tried to accommodate this by making digital communication possible. Unfortunately, many of these techniques were insufficiently used by the students. Therefore, it seems that while DLRs are complementary to traditional prescribing education, it remains questionable whether they will ever be able to replace face‐to‐face prescribing education.

Limitations

Undoubtedly, useful digital teaching tools are being used but not scientifically studied or published. This is evidenced by the numerous relevant conference abstracts that have never appeared as full‐text articles. Therefore, it seems that underreporting (and perhaps publication bias) may have limited the integrity of this review article. In order to thoroughly assess the types of DLR used for prescribing education, other research designs, such as (international) surveys among prescribing educators, may be valuable. The mean quality of the articles included in this review was low‐to‐moderate, but surprisingly slightly higher (difference 1.1, P = 0.03 compared with Brinkman et al.1) than that found in previous review studies.1, 93 This is most likely due to the nature of DLRs, making it relatively easy to perform multicenter trials with adequate response rates. Moreover, a high proportion of studies used a randomized controlled trial design, leading to high MERSQI scores.94 However, the relevance of the control groups is not scored in the MERSQI, and an alarming number of studies compared the DLR with no education at all. The obvious findings of these articles are of limited value and outcomes may be easily biased due to the Hawthorne effect.95 Although the search was compiled and performed with the assistance of an experienced medical information specialist, it is possible that we missed relevant literature because our search criteria were too strict. This is supported by the finding that a previously used database, not focused on prescribing education, contained six previously unidentified relevant articles. During the selection process, it seemed that our predefined eligibility criteria left some gray area as to what is considered “teaching safe and effective prescribing.” However, consensus was reached by including up to three experts in the decision process. For example, although relevant to optimal medication use, we chose to exclude DLRs that teach drug dosage calculations to (unspecialized) nurses, because in many countries nurses are not licensed to prescribe medicine. These decisions limit the repeatability of our findings.

Conclusions

E‐learning is the most‐used type of digital learning for safe and effective prescribing education and is efficacious in teaching undergraduate and postgraduate prescribers the required knowledge, skills, and attitudes. Although this may ultimately benefit patient care, direct effects on patient outcomes have not yet been established. Ten recommendations for the future development of pharmacotherapeutic DLRs are provided (Table 3).

Implications for future research

Now that we know DLRs can be used to teach safe and effective prescribing, future research should focus on their broad implementation in European (and perhaps worldwide) medical curricula. Because the included articles rarely reported DLRs to be openly shared between universities, future research should focus on improving the sharing of educational resources and on understanding why educators are hesitant to do so. Obviously, patient care is influenced by many factors other than educational interventions for prescribers. Therefore, it remains difficult to show the effects of such interventions on “hard” outcomes. However, our data show that it is feasible to achieve effects directly influencing the patient, and that this results in the most relevant articles. Therefore, we recommend future DLRs to be evaluated as high as possible on the Kirkpatrick pyramid of educational outcomes, preferably on the level of patient outcomes. In addition, future articles evaluating the effects of DLRs should compare these interventions with relevant control interventions rather than no education.

Search Strategy and Data Analysis

Databases PubMed, EMBASE, Education Resources Information Centre (ERIC), the Cumulative Index to Nursing and Allied Health Literature (CINAHL), and the Cochrane database (CENTRAL) were searched for articles on digital methods (e‐learning, online course, virtual reality, etc.) for teaching CPT or prescribing (prescribing, drug therapy, pharmacovigilance, etc.) to graduate or undergraduate prescribers (physicians, advance nurse practitioners, medical students, etc.). A complete list of the search terms used, as well as the complete searches, can be found in Table S2. The search in the first four databases was performed on July 30, 2018; the search in CENTRAL was added on March 6, 2019. No other date restrictions were used. Additionally, colleagues from the Centre for Population Health Sciences (CePHaS), Singapore, provided a previously compiled database (unpublished data) of the effects of e‐learning on healthcare education. This database was manually searched for articles relevant to prescribing education. The search details are presented in Table S2.

Study selection

Articles were included if they described or evaluated at least one DLR for teaching safe and effective prescribing to graduate and undergraduate medical and nonmedical prescribers (doctors, nurse specialists, dentists, and pharmacists). Exclusion criteria were: (i) articles not written in English or Dutch, (ii) not an original research article (e.g., conference abstracts) or unavailability of the fulltext, (iii) learning resources not using digital technologies, and (iv) DLRs aimed at prescribing other therapies than medications (e.g., dietary and transfusion). Articles were selected independently by two reviewers (M.B. and A.W.); discrepancies were discussed, and when no agreement was reached a third reviewer (J.T.) was consulted.

Definitions, data extraction, and study quality assessment

Table 4 lists the definitions used to classify the DLRs. These classifications and other data were extracted into a spreadsheet independently by two reviewers (M.B. and A.W.) and cross‐checked between them. Differences were resolved by discussion. Research designs were categorized as descriptive (articles describing the production of a DLR without evaluation), qualitative (open‐ended survey questions or interviews, but no quantitative outcomes), or quantitative (studies using any kind of quantitative measurement, including Likert‐type surveys). The qualitative and quantitative results of mixed‐methods articles are presented separately, but the research design was scored on the quantitative part. The quality of the quantitative studies was assessed using the MERSQI.94 The MERSQI consists of six domains. Per domain, a maximum of three points may be scored; the minimum varies, leading to a possible score of 5–18 points. Some domains may be scored nonapplicable, leading to a lower maximum score; this was corrected using the formula: MERSQI = (total score/maximum possible)*18. Although no official cutoff values exist, scores of <8.5, 8.5–13.5, and >13.5 were used as cutoff for low, intermediate, and high quality, respectively, in line with previous articles.93, 96 The outcomes were grouped according to Kirkpatrick's pyramid model of educational outcomes.97

Table 4

Definitions of digital learning resources

Type of digital learning resource	Definition used for this review
E‐learning	Web‐based (online) or offline course offering a mixture of text, multimedia, and interactive elements
Digital assessment	Formative or summative examination, exceeding the use of simple interactive questions in e‐learning
Social component	(Mostly online) learning environment including any type of digital interaction with peers or teachers, such as Web forums, chatrooms, or email support
Podcasts	Informative digital audio files
Blended learning	Any type of digital learning resource combined with face‐to‐face teaching
Virtual patient	Patient simulator (may vary from simple text‐based to virtual/augmented reality)
Virtual reality/patient	Interactive computer‐generated experience in a virtual environment
Augmented reality	Virtual elements projected on real‐world environment
Serious/educational game	A video game with the purpose of training an individual

The definitions used for this review are adapted from van den Berg et al. and Piovesan et al.98, 99

Data analysis

Data were analyzed using SPSS (IBM SPSS Statistics for Windows, version 22.0; IBM, Armonk, NY) and Microsoft Excel 2016 (Redmond, WA). Owing to heterogeneity in study design and outcomes, only descriptive statistics were used.

Funding

No funding was received for this work.

Conflict of Interest

The authors declared no competing interests for this work.

Table S1. Included articles.

Click here for additional data file.

Table S2. Search strategies.

Click here for additional data file.