Evaluation of online clinical pharmacology curriculum resources for medical students.

AIMS: To identify and evaluate clinical pharmacology (CP) online curricular (e-Learning) resources that are internationally available for medical students.
METHODS: Literature searches of Medline, EMBASE and ERIC databases and an online survey of faculty members of international English language medical schools, were used to identify CP e-Learning resources. Resources that were accessible online in English and aimed to improve the quality of prescribing specific medications were then evaluated using a summary percentage score for comprehensiveness, usability and quality, and for content suitability.
RESULTS: Our literature searches and survey of 252 faculty (40.7% response rate) in 219 medical schools identified 22 and 59 resources respectively. After screening and removing duplicates, 8 eligible resources remained for evaluation. Mean total score was 53% (standard deviation = 13). The Australian National Prescribing Curriculum, ranked highest with a score of 77%, based primarily on very good ratings for usability, quality and suitable content.
CONCLUSION: Using a novel method and evaluation metric to identify, classify, and rate English language CP e-Learning resources, the National Prescribing Curriculum was the highest ranked open access resource. Future work is required to implement and evaluate its effectiveness on prescribing competence.

What is already known about this subject

Medical schools have been moving away from dedicated clinical pharmacology (CP) teaching, contributing to students feeling that their CP knowledge and prescribing skills are lacking

Physician trainees and newly licensed physicians are more susceptible to making mistakes when prescribing

e‐Learning resources may be on par with traditional classroom learning.

What this study adds

A comprehensive, framework‐based methodology to evaluate e‐Learning resources

A comprehensive search for open access international CP e‐Learning resources revealed only 8 eligible English‐language resources

The National Prescribing Curriculum of Australia scored the highest based on comprehensiveness, quality, usability and content suitability for medical students.

INTRODUCTION

In every type of clinical practice, physicians must know how to prescribe safely, effectively, and within regulatory parameters; however, suboptimal prescribing remains 1 of the most serious issues that healthcare providers and patients face.1, 2, 3 A systematic review of the prevalence, incidence and nature of prescribing errors in adult and paediatric inpatients across 63 studies, found that medication errors, of varying clinical importance, occurred at a median rate of 52 (interquartile range 8–227) per 100 admitted patients or in 7% (interquartile range 2–14%) of all medication orders.4 Similarly, the UK EQUIP (Enhancing the Quality of User Involved Care Planning) study reviewed over 124 260 prescriptions from 19 hospitals and found 11 077 errors with a mean error rate of 8.9%.5

The EQUIP study further noted that physician trainees may be particularly susceptible to making errors as 4190 (38%) of the 11 077 were committed by first‐year residents (new graduates in their first year of speciality training).5 They documented that residents have insufficient support to make competent decisions about drugs that they are not familiar with, and do not adequately have the information to address all of the issues with common drugs.

A survey of 2413 UK medical students and recent graduates found that 74% felt that they were inadequately trained in clinical pharmacology (CP), citing a lack of learning opportunities.6 A similar theme arose in a survey conducted with final year McMaster University medical students (n = 143) where 73.3% of the survey participants felt that CP was not sufficiently covered in their medical education even though 73.1% of them agreed that it was a priority topic.7

In Canada, CP and toxicology (CPT) is the specialty of the Royal College of Physicians and Surgeons of Canada that focuses on optimal drug therapy and appropriate prescribing.8 Many Canadian medical schools lack a dedicated CPT or rational therapeutics course, instead attempt to integrate it with other subjects.9 A report by Maxwell and Webb in 2006 also documented the disappearance of dedicated courses and assessments related to CPT in the UK, which in turn decreased the available time devoted to improving prescribing competence.10 The European Association of Clinical Pharmacology and Therapeutics has also reported a need to restructure their current CP education to deal with deficiencies in CP knowledge and practical prescribing skills.11

In the UK, the mandatory Prescribing Safety Assessment was developed by the Medical Schools Council and British Pharmacological Society to ensure that graduating medical students met a threshold of CP knowledge and prescribing skills. However, British students had previously noted that there were insufficient opportunities to learn about medications during medical school potentially making it difficult to meet the expectations of the Prescribing Safety Assessment.6, 12

In collaboration with British Pharmacological Society, our group developed an Ontario Prescribing Competency Assessment for Medical Trainees, with the aim of highlighting the importance of learning this discipline and improving prescribing quality.13, 14 This pilot demonstrated that there are major gaps in prescribing competency that require more concentrated training in medical school to address. In addition, a survey of leadership from all 17 Canadian medical schools (n = 372) found that faculty country‐wide believe that there is a need to improve the threshold of prescribing competency and would welcome a standardized assessment to improve CPT learning.15

Although medical schools still predominantly educate through traditional means (e.g. lectures, tutorials, bedside learning), they have also adopted e‐Learning in the form of blended learning to enhance education.16, 17 This report defines e‐Learning resources as structured online websites or modules created for learning purposes. The proposed benefits of online educational resources include improved standardization of curricula, ability to select and extend the reach of excellent teachers, flexibility of delivery, pace, interactivity, and inclusion of formative assessment to suit individual learner preferences.2

A meta‐analysis conducted in 2008 studied e‐Learning interventions for students or practitioners in the health professions (e.g. physicians, veterinarians, nurses) in a wide variety of countries. Based on 63 studies including 24 randomized trials, e‐Learning interventions compared with non‐Internet‐based interventions had a small effect on knowledge gain (0.12, 95% CI, 0.003–0.24; P = .045). These findings suggest that the effectiveness of e‐Learning may be on par with traditional methods of teaching.18 A subsequent systematic review of 51 studies including 30 RCTs, found that interactivity, practice exercise, repetition, and feedback were the features associated with improvements in knowledge, skills, and behaviours.19 These findings plus those advocated in the Medical Education Framework20 emphasize ease‐of‐use, clarity of communication, intuitive and appealing design, interactive features, ability to engage in discussion, practice exercises, and repetition in the creation of e‐Learning Resources.18, 19, 20, 21, 22, 23

The objectives in this study were: (i) to compile an inventory of online English language curriculum resources for teaching CP to medical students; and (ii) to evaluate their comprehensiveness, usability, quality and content suitability for medical students.

METHODS

We performed primary literature reviews of ERIC, Medline and EMBASE to identify international e‐Learning resources related to CP which would be available to medical students. Search terms included (Clinical pharmacology OR prescribing OR pharmaceutical education OR prescription competency) AND (e‐Learning OR distance education OR learning modules OR distance learning OR interactive learning OR computer‐assisted instruction OR computer assisted instruction OR computer‐based instruction OR computer based instruction OR web‐based training OR web based training) AND (medical student OR medical education OR medical school). We restricted our search to the previous 10 years as of December 2017.

Our protocol was approved by the Hamilton Integrated Research Ethics Board (ID: #4167). A short, self‐administered online survey was developed for medical school faculty using the survey tool LimeSurvey (Version 2.71.1) and according to published guidelines.24, 25 The survey asked respondents if they knew of any online curriculum resources related to CP. It was pre‐tested internally for clarity and completeness.

A CP faculty contact list was created using lists provided by national professional societies related to CP education in predominantly English‐speaking countries (Canada, UK, USA, Australia, New Zealand and South Africa). We also searched for relevant faculty using individual medical school faculty lists. If no CP department was identified, the school dean or department chair was included with the expectation that they would recommend relevant individuals.

The survey was launched in February 2017 with 3 reminders sent over the span of a month.

Resources identified by either our literature search or by the faculty survey were screened by the investigators to determine if it available online in English and open to everyone with no need for payment or institutional account. Duplicate, irrelevant resources, traditional textbooks and online textbooks were also excluded from further analysis. Remaining resources were then evaluated for comprehensiveness. This was based on coverage of the 15 therapeutic groups (e.g. cardiovascular, psychiatry, endocrine) found in the McMaster Essential Drug List for Medical Students (Appendix S1). If none of the recommended drug families within a group were addressed, the group score was zero, 1–4 drug families addressed yielded a score of 1, and more than 5 families addressed, received a score of 2. The maximum possible score for comprehensiveness was 30.

Resources were then evaluated by medical student volunteers (n = 7) for usability and quality and by CP faculty (n = 3) for content suitability using 5‐point Likert scales (0–4). Usability evaluated the overall organization of the resource, if it had an intuitive layout, if navigation through the modules was easy or difficult, the user interface, the clarity in which the resource presents itself, and if it adequately communicated with the user. Quality evaluated levels of user interaction, ability to collaborate, level of feedback, use of clinical case, if key concepts were highlighted, and the level of engagement with quizzes and assessments. Faculty rated suitability based on whether the resource was targeted to an appropriate level of difficulty for final year medical students nearing graduation (a common threshold for evaluation of all levels of medical school experience), and whether it could be incorporated into the curriculum.

Inter‐rater agreement was measured with intra‐class correlations (ICC) and the final ratings were obtained by consensus.26 ICC was calculated using IBM SPSS Statistics (Ver. 25.0), 2‐way random consistent ICC analysis for medical student raters and 2‐way mixed consistency ICC analysis for faculty.

Total scores for each online resource were calculated by summing percentage scores for each of the 4 sections (comprehensiveness, usability, quality and content suitability), each weighted equally.

RESULTS

A total of 236 schools in Canada, USA, UK, Australia, South Africa and New Zealand were identified, 17 of which were omitted due to lack of available online directories. 619 invitations to faculty were sent, with 252 responses (40.7%) received. A total of 81 resources were identified, 59 from the faculty survey, 22 from the literature review.

Of the 81 resources, 73 were excluded for various reasons (details in Figure 1. Flow Chart) leaving 8 potential CP e‐Learning resources available for evaluation (Table 1). A report of all excluded resources with rationale can be found at Appendix S2.

Figure 1

Flow diagram of clinical pharmacology e‐Learning resource selection

Table 1

Final list of clinical pharmacology e‐learning resourcesa

Name	Country of origin	Link
Teaching Resource Center (TRC)	The Netherlands	https://coo.lumc.nl/TRC/default.aspx?direct=true
Pharmacology Education Project (IUPHAR)	UK	https://www.pharmacologyeducation.org/
National Prescribing Curriculum (NPC)	Australia	https://learn.nps.org.au/mod/page/view.php?id=5546
Drugs Resource at Barts & the London School of Medicine (BRATS)	UK	http://www.drugs.smd.qmul.ac.uk/
Medical Education (ME)	The Netherlands	https://www.medischonderwijs.nl/lrs.net/Student/LRS.aspx
WikiTox (TOX)	International	http://curriculum.toxicology.wikispaces.net/
Life in the Fastlane (FAST)	Not disclosed	https://lifeinthefastlane.com/tox-library/
Navigating Medical Emergencies (RCPSC)	Canada	http://navme.royalcollege.ca/ebook-e-prod/EN/

As of end‐March 2018.

The evaluations for comprehensiveness, quality, usability and content suitability are summarized in Table 2.

Table 2

Scores for the clinical pharmacology e‐learning resources

Resource	Investigators	Medical students		Faculty	Average
	Comprehensive	Usability	Quality	Suitability
NPC	67	83	73	85	77
TRC	100	63	50	48	65
BRATS	33	58	60	83	58
IUPHAR	93	53	33	35	54
TOX	40	70	50	30	48
FAST	60	48	45	30	45
RCPSC	23	58	48	38	41
ME	17	53	38	43	37
Mean (SD)	54 (29)	60 (11)	49 (12)	59 (21)	53 (12)

Summarized scores of the reviewed resources in this study arranged from highest to lowest average score.

BRATS, Drugs Resource at Barts & the London School of Medicine; FAST, Life in the Fastlane; IUPHAR, Pharmacology Education Project; ME, Medical Education; NPC, National Prescribing Curriculum; RCPSC, Navigating Medical Emergencies; TOX, WikiTox; TRC Teaching Resource Center.

The National Prescribing Curriculum (NPC) developed in Australia was the highest rated resource, with a total score of 77. The NPC was developed to help undergraduate and postgraduate medical trainees in developing prescribing competency. The modules focused on the clinical aspect of prescribing, requiring users to interact by answering questions, going through the steps of prescribing, as well as developing a personal formulary and completing an end‐module summary quiz. Usability, quality and content suitability scores were high, noting the augmented interactivity, practice material and generally applicable cross‐national prescribing norms as a major benefit.

The Teaching Resource Center produced in the Netherlands for medical students was the second highest rated resource with a score of 65. This resource scored highly for its comprehensive drug coverage and above average for usability and quality. However, it was complicated to navigate, had minimal opportunities for self‐testing, and provided limited feedback.

The Drugs Resource at Barts & The London School of Medicine, developed in the UK for medical students, ranked third overall with a score of 58. Even though it ranked highly for both usability and interactivity, its limited scope resulted in a low comprehensiveness score. However, faculty rated the content as suitable for medical students.

Similar to Teaching Resource Center, the Pharmacology Education Project included content covering the majority of the therapeutic groups, and multiple drug classes within each therapeutic groups. However, Pharmacology Education Project along with WikiTox and Life in the Fastlane rated low for quality because they lacked interaction, practice material and provided minimal feedback. Life in the Fastlane also had the lowest score in usability due to lack of a strong organizational structure. Medical Education and Navigating Medical Emergencies both had issues with a lack of comprehensiveness and quality suitability resulting in the 2 lowest scores (37 and 41 respectively).

Inter‐rater reliability was generally good with ICC values for medical students >0.7, except for the Medical Education resource (ICC = 0.506) and >0.8 for all faculty ratings between all the resources.

DISCUSSION

This project is the first to create an evaluation metric, based on e‐Learning effectiveness and guidance literature, and use it to evaluate readily available English‐language e‐Learning resources for medical student education in CP. Given the pressures on medical school teaching time and budgets, it is important to identify existing high quality, relevant online curriculum resources as a means to supplement essential curriculum.19 The NPC was found to be the most valuable for medical students due to its combination of accessibility, comprehensiveness, usability, e‐Learning quality principles and content suitability. The limited number of available resources, with only 1 originating in North America, is disappointing but aligns with previous studies suggesting a reduced focus on clinical pharmacology in medical schools.10

Our study has several limitations, including a focus only on English‐language resources, and exclusion of resources requiring payment. Additionally, all raters were Canadian which may skew our comprehensiveness and suitability results towards a Canadian perspective; however, the themes identified in usability and quality should be universally applicable. Most importantly, this study did not seek to determine if the resources actually improved user knowledge or prescribing competence. This is an important future study.

Adoption of international CP e‐Learning resources may have challenges, given the varying drug names, prescribing norms and drug regulatory structures across countries. However, these are remediable issues. As e‐Learning and the technology behind it continues to improve in quality and reliability to better engage learners, the prevalence of e‐Learning's role in the classroom will be very likely to continue to grow in the future.

Next steps in research include evaluating the effectiveness of the e‐Learning resources for medical student knowledge and prescribing skills, and to use similar evaluation methods to rate some of the less accessible resources.

CONCLUSIONS

There is a dearth of high quality, free, readily available, English‐language e‐Learning resources focused on improving CP knowledge and prescribing competence. Given the lack of resources and funding to develop objective, evidence‐based material, medical schools may wish to adapt the highly rated resources reviewed in this study.

Appendix S1. McMaster essential/core drugs list for medicalstudents

Click here for additional data file.

Appendix S1. Excluded clinical pharmacology and toxicology (CPT) e‐Learning resources

Click here for additional data file.