Better performance of medical students on pharmacotherapy knowledge and skills tests is associated with practising with e-learning program P-scribe.

AIMS: Junior doctors write most hospital prescriptions, yet are more than twice as likely to make an error in their prescriptions compared to senior doctors. A possibility to enhance pharmacotherapy education is through the use of e-learning modules. The aim of this study was to determine whether P-scribe, as the chosen e-learning resource, helps students in passing their pharmacotherapy assessments.
METHODS: This retrospective study was undertaken in the Erasmus Medical Center, the Netherlands. All 270 medical students who started their master's curriculum in the academic session of 2017-2018 were included. Data were analysed to identify the frequency of student's use per e-learning module, total time students spent on e-learning modules and timing of the use of e-learning modules in relation to their assessments. The results of the assessments were analysed to identify possible correlations between the time students spent using P-scribe, their timing of use and their assessment results.
RESULTS: Students who passed their knowledge-based assessment first time had a mean practice time of five more hours than students who did not pass first time (P < .05, 95% CI: 3.4-6.6). These students practised on average six e-learning modules more (P < .05, 95% CI: 4.1-7.0) than students who failed their first attempt. Students who passed their skill-based prescription test first time, practised on average five more e-learning modules (P = .006, 95% CI: 1.4-8.3) than students who failed their first attempt.
CONCLUSION: Students who passed their pharmacotherapy assessments first time spent more time, and practised more frequently, with e-learning modules.

What is already known about this subject?

Previous studies showed that junior doctors make the majority of hospital prescription errors.

It is known that pharmacotherapy is a difficult subject to teach, since it should incorporate knowledge and skill.

It is important to investigate whether e‐learning modules can improve both components of the education on pharmacotherapy.

What this study adds?

Students who practise more, and more frequently with the e‐learning program P‐scribe, pass their prescribing assessments more often on the first attempt.

There is a relation between the amount students practise with P‐scribe and their ability to pass pharmacotherapy knowledge‐based assessments, but most importantly, there is a relation between the use of P‐scribe and their ability to pass their skill‐based assessments.

INTRODUCTION

The literature is unanimous about the need for improvement of knowledge and skills in prescribing medication by junior doctors. , , , , , In their first two years of training, doctors are more than twice as likely to make an error in their prescription compared to senior doctors. This is especially alarming considering the fact that junior doctors write 68% of all hospital prescriptions. Errors in prescriptions can lead to patient dissatisfaction, unnecessary side‐effects, hospital admissions, disability and even death. , The majority of prescription errors stem from a lack of knowledge and experience. Although 61% of European medical students feel confident about finding relevant drug information, only 29% are thought to be adequately prepared by their medical curriculum to prescribe responsibly.

Prescribing is a difficult skill; doctors not only need knowledge on clinical pharmacotherapy, rational prescribing also requires a feeling for pharmacotherapeutical reasoning and the skill of writing a prescription. A modern way to enhance competency is through the use of digital learning resources. The use of digital learning resources has been found at least as effective as traditional teaching methods in acquiring knowledge; however, it might be less preferable when teaching the practical aspect of prescribing.

There are several e‐learning resources available for the teaching of pharmacotherapy. An example of an e‐learning resource is P‐scribe. P‐scribe is a Dutch web‐based e‐learning platform for pharmacy and medical students, which is used at all Dutch university medical centres and at several medical centres in Belgium. In P‐scribe, e‐learning modules can be built that are adapted to the needs of the curriculum (for an example, see Appendix). It is based around the World Health Organization's (WHO) 6‐step model, which is an educational tool to encourage rational pharmacotherapy. At the Erasmus MC, a University Medical Centre (Rotterdam, the Netherlands), P‐scribe is used in the medical master's curriculum as part of flipped classroom teaching. The e‐learning modules used at the Erasmus MC are developed by medical doctors and pharmacists who work as clinical pharmacotherapy teachers. In the e‐learning modules students are guided through different life‐like cases in which a change in pharmacotherapy is needed. Each e‐learning module focuses on a different aspect of rational pharmacotherapy, for example side‐effects or polypharmacy.

The aim of our study was to determine whether the use of P‐scribe helps students in passing their pharmacotherapy knowledge‐ and skills‐based assessments. Information on the effect of e‐learning modules could help instruct future students, since improvement in pharmacotherapy education is crucial. The hypothesis was that students who spend more time using P‐scribe and who use it more frequently will receive better results on their assessments, and thus hopefully improve their knowledge and skills. A study by Keijsers et al. already showed students' satisfaction with the e‐learning platform P‐scribe. However, to our knowledge, the effect of practising with the e‐learning module P‐scribe on knowledge and skills assessments has not yet been analysed.

METHODS

At the Erasmus Medical Centre, pharmacotherapy is taught through a combination of classes and self‐study. For self‐study, students can use e‐learning modules and educational videos. After three years of bachelor's (undergraduate) curriculum, students start with their three‐year master's (graduate) curriculum. During this master's curriculum students have several internships, which are each preceded by educational weeks. In the master's curriculum, knowledge and skills in pharmacotherapy are tested by two summative assessments, as shown in Figure 1. The first summative assessment in the master's curriculum is the Dutch National Pharmacotherapy Assessment, , which has to be done at the start of the second year of the master's curriculum. This assessment is a knowledge‐based test consisting of 60 multiple choice questions on pharmacotherapy. For this assessment, students can prepare by studying a reader, taking a practice test in P‐scribe and by using several apps to study. The second summative assessment has to be done at the start of the third year of the master's curriculum. This assessment is a skill‐based prescription test in P‐scribe. The assessment consists of three case‐based prescriptions and filling out one complete WHO‐6‐step model (for examples, see Appendix). These cases are developed and corrected by clinical pharmacotherapy teachers from the hospital pharmacy. For each assessment there are cases chosen from the categories opioids, children, adjustment for kidney function or general. During this skill‐based assessment, students are allowed to use online pharmacotherapeutic guides. This assessment is corrected using a rubric form, which is evaluated annually. For this skill‐based prescription test students prepare by taking a formative test in the beginning of their master's curriculum, on which they get feedback, but which is not graded.

FIGURE 1

master's program Erasmus Medical Centre. The blue arrows mark the summative pharmacotherapy assessments

This retrospective study was undertaken in the Erasmus Medical Centre, Rotterdam, the Netherlands. In this study we included all 270 medical students who started their master's curriculum in one of the five starting moments of the academic year of 2017–2018 (between 1 September 2017 and 31 August 2018). Each student had a personal account in the e‐learning program P‐scribe prior to this study. On registering in P‐scribe, students agreed to have their data stored and used for research. We coded student data to ensure anonymity of data. We collected data on results generated from September 2017 to April 2020. We included all data through data tracking in P‐scribe, which showed the time students spent on particular e‐learning modules. If an e‐learning module had been used for less than one minute, this session was excluded.

Data were analysed to identify the frequency of student's use per e‐learning module, the total time students spent on e‐learning modules and the timing of when the e‐learning modules are used in relation to the assessments. The results on both the summative assessments were analysed to identify possible correlations between the time students spent using P‐scribe, their timing of use and their assessment results. To check for errors in the database, a random portion of 10% of the entered data were checked by a colleague. These 10% were entered once more by the colleague in a separate dataset and by using the function ‘compare data sets’ in SPSS, we showed there were no significant errors in this database check.

The research proposal was reviewed by the Medical Ethics Committee Erasmus MC and they determined that the Medical Research Involving Human Subjects Act was not applicable to this research. Data were analysed with the statistical package IBM SPSS statistics 25.0.

RESULTS

As shown in Figure 2, during the investigated period students used P‐scribe for 2484 hours in total; 473 hours were spent on practice tests, the other 2011 hours were spent on e‐learning modules. E‐learning modules used in the master's program can be divided in three groups: e‐learning modules which were preparation for a face‐to‐face class, e‐learning modules which were included in the weekly schedules but were not preparation for a class and extracurricular e‐learning modules. Extracurricular e‐learning modules are, for example, e‐learning modules used in the bachelor's curriculum which can be repeated.

FIGURE 2

Amount of time spent on the e‐learning program P‐scribe

There were 270 students included in the study, of which 71% were female. The mean age was 24.8 years (SD 2.1). Of the total of 270 students included in the study, 233 students completed the Dutch National Pharmacotherapy Assessment, as shown in Table 1. For students to pass this assessment, 85% of all questions needed to be answered correctly. Of the students who completed the assessment, 42.2% passed first time, 37% needed one resit to pass and 7% had to take the assessment a third time before they passed. The remaining 37 students, who have not yet completed the assessment, chose to postpone the assessment or were on a (temporary) break from their study.

TABLE 1

Test characteristics

	Dutch National Pharmacotherapy Assessment (n = 233)	Skills‐based prescription test (n = 255)
Results
Passed first time	114 (42.2%)	241 (89.2%)
One re‐sit	100 (37%)	6 (2.3%)
Practice tests frequency
0	137 (50.7%)	48 (17.8%)
1	95 (35.2)	62 (23%)
2 or more	38 (14.2%)	160 (59.2%)
Timing of use of practice test
Day of assessment	22 (16.5%)	150 (67.6%)
1 day before	62 (46.6%)	51 (23%)
2 days before	21 (15.8%)	8 (3.6%)
>2 days before	27 (20.3%)	11 (4.9%)

To prepare for the assessment 49.3% of all students used a practice test in P‐scribe. The majority of the students used the practice tests in the last 24 hours before the assessment.

The skill‐based prescription test was completed by 255 students. Students could get a maximum of nine points per prescription, one point for each step of the WHO 6‐step and three points for the final prescription in the WHO 6‐step. To pass this assessment, students needed to score more than 21 points out of the possible 36. Of the students who completed the assessment, 89.2% passed first time. The 15 students who have not yet completed the assessment chose to postpone the assessment or were on a (temporary) break from their study.

To prepare for the assessment, 82.3% of all students used practice tests in P‐scribe. The majority of all students used the practice tests on the day of the assessment due to the test being in the afternoon.

As shown in Table 2, students who passed their Dutch National Pharmacotherapy Assessment first time had a mean practice time of five more hours (300 minutes) than students who did not pass this assessment first time (P < .05; 95% CI of difference [3.4–6.6]). In the same trend, students who passed the skills‐based prescription test first time had a mean practice time of 3.3 more hours (198 minutes) than students who did not pass this assessment first time; however, this difference is not statistically significant (P = .072; 95% CI of difference [−0.3–6.9]). Students did not spend more time on practice tests.

TABLE 2

Use of P‐scribe in students who pass or fail their assessments during their first attempt.

	Dutch National Pharmacotherapy Assessment				Skills‐based prescription test
	Pass	Fail	Mean difference	95% CI of difference	Pass	Fail	Mean difference	95% CI of difference
Mean total practice time (min)	732	432	300	[203–397]	574	377	198	[−18–413]
Mean total practice frequency	14.8	9.2	5.6	[4.1–7.0]	11.9	7.1	4.8	[1.4–8.3]
Mean practice test time (min)	28	24	4	[−6–14]	85	77	8	[−39–55]
Mean practice test frequency	0.7	0.7	0.1	[−0.1–0.3]	1.6	1.5	0.1	[−0.8–1.0]
Mean e‐learning time (min)	608	340	267	[177–358]	463	275	188	[−10–386]
For class	296	205	91	[53–129]	249	167	82	[−2–165]
Scheduled – Not for class	288	126	161	[103–220]	200	82	118	[−10–245]
Extracurricular	24	9	15	[6–24]	15	26	−12	[−30–7]
Mean e‐learning frequency	12.2	7.2	5.1	[3.7–6.4]	9.6	5.0	4.6	[1.5–7.7]
Mean days between practice test and assessment	2.4	2.8	−0.4	[−3.2–2.3]	1.0	11.9	−10.9	[−37.2–15.6]

Students passing the Dutch National Pharmacotherapy Assessment first time, practised on average six e‐learning modules more (P < .05; 95% CI of difference [4.1–7.0]) than students who failed their first attempt. For example, students repeated an e‐learning module, or did an extracurricular e‐learning module. Students who passed their skill‐based prescription test first time, practised on average five more e‐learning modules (P = .006; 95% CI of difference [1.4–8.3]) than students who failed their first attempt.

DISCUSSION

This is the first retrospective study to analyse the effect of the use of the e‐learning program P‐scribe. The aim of our research has been to determine whether the use of P‐scribe is related to the passing percentages on pharmacotherapy knowledge and skills assessments in medical students. Because we used a retrospective study design, we have been able to extract a great amount of detailed data on the complete academic session of 2017–2018.

Even though study time has proven to be an unreliable predictor of academic performance, in our study students who passed their assessments first time did spend more time on e‐learning modules than students who did not pass their assessments first time. , , Surprisingly this was true for both the knowledge‐based assessment (+5 h) and the skill‐based assessment (+3.3 h). Although the statement cannot be made that the passing of these assessments was solely due to use of P‐scribe, there is a relationship between the usage and the results on the assessment. Especially noticeable is how big the difference of usage of P‐scribe is in preparing for the Dutch National Pharmacotherapy Assessment, since there is much more study material besides P‐scribe for this assessment in comparison to the skill‐based prescription test. The skill‐based prescription test only has practice material in P‐scribe. Therefore, it is remarkable that the difference in study time in P‐scribe between students who pass their exams the first time, and those who do not, is especially visible in the Dutch National Pharmacotherapy Assessment. This might be an indicator that students who pass their Dutch National Pharmacotherapy Assessment the first time also use other practice material more, but that cannot be determined by this study.

Based on the summary of research by Phelps, we hypothesized that students would benefit from the use of practice tests. However, contrary to our hypothesis, the positive effect on the results of the assessment seems to be particularly achieved by the use of e‐learning modules and barely by the practice tests. Between the students who passed or failed their assessment, there was no statistically significant difference in the frequency and time spent on practice tests. This smaller positive impact of practice tests on the scoring of the assessments could be attributed to the way the practice tests were used by the students. The practice tests were largely used shortly before the test was taken, while the e‐learning modules were used in a dispersed manner throughout the curriculum.

Not only did students spend more time on e‐learning modules in P‐scribe, students who passed their assessments first time also practised more frequently in P‐scribe. This finding supports the recommendation of the Delphi study of Brinkman et al. which advocates a curriculum in which clinical pharmacology and therapeutics are integrated longitudinally, from the beginning of the curriculum and repeated frequently.

Prescribing is a complex task which, according to Tobaiqy et al., consists of eight different competencies. In a study by McQueen et al., it has already been shown that an online tool can help build better dose calculations. Furthermore, the review by Chumley‐Jones et al. showed that web‐based learning interventions can result in knowledge gain in medical students. It is notable that, in our study, there is not only a relation between the P‐scribe e‐learning modules and the knowledge‐based assessment, but also with the skill‐based assessment, where all eight competencies come together. This is in contrast to the described disadvantage of e‐learning modules being less able to teach the practical aspect of prescribing as discussed by Maxwell and Mucklow.

A possible explanation for the relation between the e‐learning modules and the assessments is the way the e‐learning modules are built. In P‐scribe the modules can be built according to the needs of the curriculum. It is equipped to guide students through the WHO‐6‐step model, and to therefore teach them to prescribe rationally in life‐like situations. In this way, students are taught practical guidelines to use during prescribing. In the e‐learning modules used in the Erasmus MC curriculum, students are stimulated to write realistic prescriptions, with the possibility to use the references they are going to need to use as junior doctors.

Our results show the importance of the frequent use of e‐learning modules in P‐scribe throughout the curriculum. In the Erasmus MC, several compulsory and non‐compulsory e‐learning modules have already been embodied in the whole medical curriculum. Our data substantiates this recurrent structure of pharmacotherapy education and it is possible to use this as a foundation for future curricula and curricula elsewhere. Since P‐scribe is used in all Dutch and several Belgian Medical University centres, and the Dutch National Pharmacotherapy Assessment in all Dutch Medical University centres, we think the results are generalizable. These results are not only generalizable to the Netherlands and Belgium, but evidence on the use of e‐learning modules on prescribing skills could be used by any medical curriculum.

The results can also be used to motivate students by stressing the importance of frequently using e‐learning modules, not only for better results on assessments, but hopefully also for better prescribing skills. The practice tests as currently used by students seem not to contribute to the results of the assessments. Perhaps if used, like the e‐learning modules, in a more frequent manner throughout the curriculum, they will contribute more to the results on pharmacotherapy assessments.

There are some important potential drawbacks associated with our study. For example, a limitation of this study is the fact that the data are based on the data tracking in P‐scribe. P‐scribe tracks the amount of time which the module was open on the computer, not how much time the student actually spent on it. Also, through data tracking, it is not possible to distinguish between an active or a passive level of using the e‐learning modules. However, this potential overestimation of hours spent on study will most likely not affect our firm and significant results and it could be assumed that this potential overestimation is equally distributed throughout the whole study population. Also, the data tracking function in P‐scribe has been studied previously and was found to be a valid instrument to register and map data. Another potential limitation of this study is the structure of inflows in the curriculum studied. Students started at five different points in the academic year, which means that they could have been taught by different teachers and that there is a time difference of about 40 weeks between the first inflow and the last. However, the structure of their curriculum, e‐learning modules and order of classes did not change in this period.

Future studies should investigate the actual time spent on the e‐learning modules. It could provide more detailed data on how students use the e‐learning modules to study pharmacotherapy. This could strengthen the foundation for pharmacotherapy education in medical curricula. We have demonstrated the relation between e‐learning modules in P‐scribe and the pharmacotherapy assessments. It is hoped that spending more time on P‐scribe not only will have a relation with passing pharmacotherapy assessments, but also, most importantly, will eventually result in doctors who are able to prescribe more accurately, safely and effectively, which should be researched more closely.

CONCLUSION

The need for improvement of pharmacotherapy education is undeniable. One way to enhance this competency is through the use of e‐learning modules in the e‐learning platform P‐scribe. Students who passed their pharmacotherapy assessments first time spent more time, and practised more frequently, with e‐learning modules in P‐scribe. This is true for the knowledge‐based assessment and for the skill‐based assessment. E‐learning modules should therefore offer students the possibility to practise frequently throughout the whole curriculum to support the teaching of pharmacotherapy.

COMPETING INTERESTS

The authors declare no conflicts of interest.

CONTRIBUTORS

L.K., F.R. and J.V. designed the study. L.K. processed the data, performed the analysis and drafted the manuscript. F.R. and J.V. aided in interpreting the results. All authors discussed the results and commented on the manuscript.