The outcomes of recent patient safety education interventions for trainee physicians and medical students: a systematic review.

OBJECTIVE: To systematically review the latest evidence for patient safety education for physicians in training and medical students, updating, extending and improving on a previous systematic review on this topic.
DESIGN: A systematic review. DATA SOURCES: Embase, Ovid Medline and PsycINFO databases. STUDY SELECTION: Studies including an evaluation of patient safety training interventions delivered to trainees/residents and medical students published between January 2009 and May 2014. DATA EXTRACTION: The review was performed using a structured data capture tool. Thematic analysis also identified factors influencing successful implementation of interventions.
RESULTS: We identified 26 studies reporting patient safety interventions: 11 involving students and 15 involving trainees/residents. Common educational content included a general overview of patient safety, root cause/systems-based analysis, communication and teamwork skills, and quality improvement principles and methodologies. The majority of courses were well received by learners, and improved patient safety knowledge, skills and attitudes. Moreover, some interventions were shown to result in positive behaviours, notably subsequent engagement in quality improvement projects. No studies demonstrated patient benefit. Availability of expert faculty, competing curricular/service demands and institutional culture were important factors affecting implementation.
CONCLUSIONS: There is an increasing trend for developing educational interventions in patient safety delivered to trainees/residents and medical students. However, significant methodological shortcomings remain and additional evidence of impact on patient outcomes is needed. While there is some evidence of enhanced efforts to promote sustainability of such interventions, further work is needed to encourage their wider adoption and spread. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

This systematic review provides an update of the evidence on courses teaching core concepts of patient safety to medical students and trainees/residents.

The results confirm an increasing trend for developing educational interventions in patient safety delivered to trainees/residents and medical students.

However, we found that significant methodological shortcomings in studies reporting such interventions remain and additional evidence of impact on patient outcomes is needed.

While there is some evidence of enhanced efforts to promote sustainability of such interventions, further work is needed to encourage their wider adoption and spread.

The main limitations of this systematic review relate to the quality of the included studies, and to only including articles published in the English language.

Introduction

Educational interventions for quality and safety improvement have garnered increasing interest over recent years. The importance of such interventions is acknowledged by the development and integration of dedicated patient safety and quality improvement curricula and frameworks into medical education at all levels. For example, the Association of American Medical Colleges (AAMC) endorses the introduction of formal quality improvement education from medical school through to postgraduate training and continuing medical education.1 2 The Accreditation Council for Graduate Medical Education (ACGME)3 and CanMEDS4 5 competency frameworks incorporate essential competencies relating to quality and safety for medical professionals. The WHO has developed a Patient Safety Curriculum Guide for Medical Schools6 and, recently, a multiprofessional edition.7 Such curricula aim to guide and support educators in developing and implementing educational programmes in patient safety.

There has been a significant increase in the number of publications relating to patient safety courses, particularly those aimed at residents. A systematic review on teaching patient safety and quality improvement to medical students and residents was published in 2010,8 identifying 41 studies published between January 2000 and January 2009, of which 27 included an evaluation of the described intervention. This review identified significant methodological limitations in most studies, including low response rates, single centre recruitment and small sample sizes (median=41 participants per study, IQR 20–106).8 Although most interventions were well received by participants, and resulted in improvements in safety and quality knowledge scores, few studies were able to demonstrate changes in learners’ behaviour or potential patient benefit.8 The reviewed articles also identified multiple barriers to sustainable integration of the courses, which spanned learner, faculty and institutional factors.8

Patient safety education is a rapidly emerging field and it is likely that, in part due to the recent development and implementation of patient safety curricula and frameworks highlighted above, an increasing number of articles have been published since this last systematic review, perhaps addressing some of the aforementioned methodological limitations of the older studies. The aim of this study was thus to perform a focused systematic review of research reporting courses that teach core concepts in patient safety, and that target medical students and junior physicians, published since 1 January 2009. We describe the educational content and teaching methods employed, evaluate the learning outcomes achieved and explore factors influencing implementation of these patient safety courses.

Methods

Data sources and search strategy

We prespecified the methods utilised in this systematic review and present them in accordance with PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) guidelines.9 A literature search was performed using the electronic databases of Embase (1996 to 2014 Week 18), Ovid MEDLINE (1996 to April Week 5 2014) and PsycINFO (2002 to May Week 1 2014); although the focus of this systematic review was to identify papers published since the last systematic review covering this topic, that is, from January 2009 onwards, we used a search strategy incorporating earlier start dates. This allowed us to perform an evaluation of the sensitivity of our search strategy by ensuring five reference papers that we identified as being highly relevant studies before performing the literature review10–14 were identified by our search strategy. All five reference papers were identified, and thus we were able to begin our search from the end of the data collection period of the previous systematic review covering this topic.8

Our search strategy (see online supplementary appendix 1) incorporated the two broad themes of ‘medical education’ and ‘patient safety’, and the content areas were combined using the Boolean operator ‘and’; a pilot search revealed that ‘medical education’ successfully encompassed both ‘education’ as the intervention and ‘medical students and/or trainees and/or residents’ as the population of interest. Search terms were generated with the assistance of key words from core reference texts15 and relevant articles,8 and a combination of MeSH terms and free text words (truncated wherever appropriate) were used to maximise the sensitivity of the search. We limited the search to human studies published in English language, and removed duplicates. Additional articles were sought through hand searching of reference lists of included studies.

As our data comprised studies that were previously published and publicly available, this study did not require ethical approval.

Eligibility criteria

We included articles that described and evaluated an educational intervention that explicitly exposed medical students and/or trainees/residents to core concepts of patient safety. Articles that included medical students and/or trainees/residents in addition to other population groups were not excluded. To be included, reviewed articles were required to have sufficient empirical data for analysis (eg, conference proceedings were excluded), the educational intervention was required to include patient safety as core content and the study had to include an evaluation of the educational intervention. Detailed eligibility criteria can be found in online supplementary appendix 2.

Article review process

Titles of the initial 4027 articles identified by the search strategy outlined above were reviewed by an academic physician with expertise in patient safety and medical education (MA). After excluding articles with titles that were clearly irrelevant to the topic at hand, the remaining abstracts were reviewed for inclusion independently by MA and a second physician with expertise in medical education (MAK). Disagreements were resolved through consensus, involving a third reviewer with expertise in patient safety and medical education (NS), as necessary.

Data extraction and quality assessment

Consistent with Best Evidence Medical Education (BEME) recommendations,16 administrative data (including publication details and country of origin), topic-related data (including details of the educational intervention and number and type of participants) and research-related data (including methodology and results) were extracted from the studies that were identified as relevant. Factors influencing curricular implementation of the intervention were categorised under four broad headings (learner factors, faculty factors, curricular factors and learning environment factors) devised by the authors of the previous systematic review on this topic.8 Only factors that were explicitly described by the authors of the papers included in this systematic review were counted and categorised in this manner.

Assessing the quality of interventions is a well-documented challenge facing systematic reviews of educational interventions.17 The BEME review protocol recommends a system for assessing the quality of studies based on grading,16 but as no specific guidance as to how to apply these grades is provided, we assessed quality by extracting information on both stated and perceived limitations of the study as assessed by study design, sample size, completeness of data and overall coherence between study aims, methods and conclusions.

Analysis

Given the anticipated heterogeneity in study designs and outcomes as per the previous systematic review on this topic,8 quantitative synthesis of the data (ie, meta-analysis) was not performed. Simple quantitative statistics were used to report on educational content, methodologies used, study populations and learning outcomes (where reported).

Studies were categorised by the learning outcomes reported by the authors, using the modified version of Kirkpatrick's levels of evaluation adopted by the BEME collaboration as a grading standard for systematic reviews.16 This assesses impact on learners’ satisfaction (level 1), changes in learners’ attitudes (level 2a), measures of learners’ knowledge and skills (level 2b), change in learners’ behaviour (level 3), changes to clinical processes/organisational practice (level 4a) and benefits to patients (level 4b). Correspondingly, the results of this systematic review are presented according to the Kirkpatrick learning outcome assessed.

Results

Selected articles

The initial yield of the review was 4027 articles retrieved by the search strategy. The subsequent title screen of articles identified 304 potentially relevant titles for the abstract review stage. Independent review of abstracts against the eligibility criteria by two reviewers (MAK, MA) followed by consensus resulted in 61 papers for review. The agreement between the reviewers was excellent (κ=0.917, 95% CI 0.871 to 0.963). Review of the full text identified 25 papers that fully met the eligibility criteria for inclusion. An additional eligible paper was identified from hand searching of relevant reference lists, resulting in 26 papers for analysis. This process is summarised in figure 1.

Figure 1

Flow diagram illustrating our search strategy.

Characteristics of included studies and study settings

Table 1 summarises the main characteristics of the included studies, including study design, participant number and type, and course structure and content. The majority of the 26 studies were conducted in the USA (n=17, 65%). Of the remaining studies, five (19%) came from the UK,18–22 two (8%) from the Netherlands,23 24 one from China25 and one from the Republic of Korea.26 Participants comprised trainees in 15 (58%) studies (often resident or specialty trainee/registrar grade), and medical students in the remainder. No studies recruited students and trainees/residents simultaneously. Participants learned in interdisciplinary groups in six of the studies: four involved students,18 27–29 one included junior and senior physicians,19 and another comprised residents and faculty.30 One study involved senior physicians (attending or consultant grade level) participating as part of faculty development activities, although their learning outcomes were not directly assessed.31

Table 1

Study characteristics, course structure and content

Lead authorReferenceYearCountry	Study type	Participant number and specialty	Course structure	Course content
Aboumatar412012USA	Before and after study	120 third-year medical students. Recruited from a single institution	3-day clinically oriented patient safety intersession using role-play and simulation, skills demonstrations, small-group exercises and case-based learning	Medical error understanding and prevention, teamwork and communication, systems thinking
Ahmed202014UK	Before and after study	1169 junior physicians across a region (16 institutions)	Monthly 60 min sessions led by junior physicians between January and July 2011. Sessions comprised case-based discussion and analysis of patient safety incidents encountered in practice, facilitated by trained faculty	Key patient safety concepts, RCA/systems-based analysis, communication and teamwork, incident reporting
Anderson182009UK	Before and after study	199 students including 58 final year medical students learning in uni-professional groups and 36 learning in interprofessional groups as part of a regional programme	1-day workshop involving DVD and small-group facilitated discussion to analyse key safety issues using the National Patient Safety Agency RCA tool. Supporting handbook containing additional relevant materials. Nine events held over 2 years	DVD of patient journey to focus on learning themes of situational awareness, communication, leadership and empowerment. RCA
Arora212012UK	Before and after study	27 surgical residents. Recruited from across 19 hospitals	3 h training programme comprising lectures, video demonstrations and small-group discussions	Patient safety overview, adverse events, human factors, systems-based analysis, communication and teamwork in surgery
Cox282009USA	Before and after study	Over 787 interprofessional teams of medical, nursing, health administration and respiratory therapy students. Recruited from across 3 sites	4-week curriculum comprising lectures, problem-based learning, small-group work, simulation. Participants given cases describing a medical error. Team-based simulation of RCA and use of performance improvement tools. Presentation on completion	Patient safety overview, RCA, QI overview, teamwork
Cox312011USA	Prospective cohort study	12 faculty members and 46 internal medicine residents.Recruited from a single institution	3 h long faculty development session including videos, role-play and mock facilitation sessions. Plus manual of key safety education topics. Implementation of an alternative reporting system for anonymous narratives of ‘care that did not go as intended’. Monthly ‘Safety Story’ sessions of 4–6 residents with faculty member to discuss contributing factors and propose potential solutions	Faculty training included patient safety overview, RCA and teamwork
Dudas342011USA	Retrospective pre–post study	108 medical students (second-year, third-year and fourth-year students as part of paediatric clerkship). Recruited from a single institution	During course of 9-week clerkship, 25 min online video on systems-based analysis of medical errors. 60 min large-group faculty demonstration of Learning From Defects tool. Subsequent self-directed small-group identification and analysis of medication errors in practice. Group presentation at closing 60 min session	Systems-based analysis
Gupta432014USA	Retrospective pre–post study	26 neonatology fellows. Recruited from a single institution	Workshops, web-based modules, completion of a quality and safety project, presentation at departmental conference, participation in departmental morbidity and mortality conference. Optional selected readings and web-based modules	Core patient safety concepts, QI, human factors, communication and teamwork, error disclosure, incident reporting and systems thinking
Hall322010USA	Before and after study and comparison with historical control	146 third-year medical students undertaking a medicine clerkship. 65 in intervention group, 81 in control group. Recruited from a single institution	2 mandatory 1 h patient safety ‘booster’ conferences. First conference involved RCA brainstorming exercise of an adverse event. Assignment to identify and summarise an actual patient safety event or concern. During second conference case presentation including proposed system modifications to improve patient safety	RCA including proposed system modifications for improvement
Holland422010USA	Before and after study	26 PGY-3 internal medical residents. Recruited from a single institution	4-week rotation comprising web-based patient safety and QI curriculum including interactive modules, and self-directed reading and assignments. Completion of QI proposal and presentation at end of rotation	Patient safety overview, QI overview including PDSA, medical error, RCA, human factors engineering, safety interventions
Jansma232010The Netherlands	Before and after study with 6-month follow-up	33 specialty registrars (GP, anaesthesiology, dermatology, internal medicine). Recruited from a single institution	2-day course comprising plenaries, group discussions and role-play	Patient safety overview, human error, disclosure, medicolegal aspects of critical incidents, RCA, tips and tools to improve safety in practice
Jansma242010The Netherlands	Prospective cohort study	71 residents (surgical and non-surgical). Recruited from 5 hospitals	Multispecialty 2-day patient safety course including plenaries and small-group sessions. At end of course participants asked to formulate 1 action point to improve patient safety	Patient safety overview, human factors, teamwork, contribution to safer care (including RCA), medicolegal aspects
Jericho332010USA	Before and after study	Anaesthesiology residents (approximately 51—number not clearly stated). Recruited from a single institution	90 min interactive case-based lecture coupled with an expectation of adverse event reporting. Supplemented with education manual. Quarterly conferences to discuss reports and near-immediate feedback from Department of Safety and Risk Management	Patient safety definitions, adverse event reporting, investigation/process improvements, communication, and apology and remedy
Jha222013UK	Before and after study with control group and follow-up	263 junior physicians across a region (155 in intervention group, 108 in control group)	3 h teaching session. Intervention group: patients shared their stories about their experience of safety incidents. Non-intervention group: teaching delivered using “standard methods of teaching”, including presentations and small-group work	Error analysis. Teaching session covered: prescribing, teamwork and communication
Leung252010China	Before and after study	130 third-year medical students.Recruited from a single institution	Two 60 min whole-class lectures using contemporary medical incidents as illustrative cases	Based on WHO curriculum: patient safety overview, human factors, systems thinking, team working, understanding and learning from error, introduction to QI, medication safety
Miller272014USA	Before and after study	110 medical and allied health students. Recruited from a single institution	1 h introductory lecture discussing general patient safety and QI topics followed by 2 courses (‘Introduction to the Culture of Safety’ and ‘Teamwork and Communication’) including group discussions	Patient safety, QI, teamwork, communication
Myung262012Republic of Korea	Before and after study	156 second-year medical students. Recruited from a single institution	1-week course composed of interactive lecture, discussion and small-group debriefing	Based on WHO curriculum: patient safety overview, human factors, systems thinking, team working, understanding and learning from error, introduction to QI, medication safety; in addition: RCA
Paxton372010USA	Before and after study with control group and follow-up	51 surgical clerkship students including 46 medical and 5 physician assistant students.Recruited from a single institution	2 h small-group discussion incorporating slide presentation	Patient safety overview, RCA, epidemiology, error theory, error disclosure and legal considerations
Rodrigue302013USA	Before and after study	42 residents and 36 faculty members. Recruited from a single institution	5 online modules that residents and faculty members completed together in pairs (duration of each module unreported)	Performance improvement, QI, patient safety, teaching and learning
Scott402011USA	Prospective cohort study	680 residents across medical and surgical specialties. Recruited from a single institution	Economic incentive comprised retirement benefit of 1.5% of residents’ annual salaries. Multifaceted educational campaign including monthly email notifications, audience presentation at major conferences (exact frequency not stated) and one-on-one discussion	Presentation covered mechanics of incident reporting, discussing barriers and dispelling myths
Shaw392012USA	Randomised controlled trial	371 interns across medical and surgical specialties. Recruited from across 2 hospitals	2 interventions compared: Online Spaced Education programme consisting of cases and questions that reinforce over time, and SQ programme comprising online slide-show followed by quiz	Covered all 9 2009 NPSGs including handover, patient identification, hand hygiene and medication safety
Slater192012UK	Before and after study	11 multiprofessional teams comprising 55 health professionals (including 16 junior physicians and 12 senior physicians). Recruited from across 5 sites	20-week ‘TAPS’ programme. 2 h online learning module; multiprofessional workshops to conduct QI project, executive-group discussion for organisational learning	Human error, QI tools (process mapping, fishbone diagrams and measurement for improvement)
Smith352012USA	Prospective cohort study	280 internal medicine residents over 2 years. Recruited from a single institution	Monthly noontime QIC. RCA of selected real-life safety events (selected by seniors, analysed by residents not associated with the case). Limited RCA with online resources and mentorship. Presentation to fellow residents and seniors. Intervention proposed and followed through where possible	RCA and QI
Stahl382011USA	Before and after study with control group	110 third-year medical students on surgical clerkship (67 in intervention group, 43 in control group). Recruited from a single institution	Two-part patient safety curriculum: all students attended 1-day lecture on introductory theories, video and small-group discussion (first year). Intervention group attended additional 1.5–2 h clinically oriented classroom discussion, videos, simulation and role-play (third year)	Patient safety principles, crew resource management, team skills, task management and situational awareness
Tess442009USA	Retrospective pre–post study	74 internal medicine residents. Recruited from a single institution	Educational intervention coupled with reorganisation of clinical services to integrate patient safety and QI into daily clinical practice. The educational intervention incorporated an online module in year 1, and a 3-week rotation in QI in year 2. Faculty-led workshops on RCA, performance improvement and the institutional approach to QI	Patient safety overview, QI and RCA
Wilson292012USA	Prospective cohort study	23 graduate level students (including 7 medical students).Recruited from a single institution	Weekly 3 h sessions held over a 15-week period. Each session comprised a presentation by a visiting expert, discussion on assigned reading material and small-group patient safety project work	Patient safety overview, human factors analysis, systems approach to error analysis, crew resource management, law, and policy, and team building

GP, general practitioner; NPSGs, National Patient Safety Goals; PDSA, plan, do, study, act; PGY, postgraduate year; QI, quality improvement; QIC, QI conference; RCA, root cause analysis; SQ, safety questionnaire; TAPS, Training and Action for Patient Safety.

Characteristics of the courses

Features of the courses including the teaching modalities employed and the core content covered are summarised in table 2. The majority of courses employed a mixture of didactic and experiential teaching methods. Small-group discussions/workshops and lectures were commonly used approaches: n=14 (54%) and n=12 (46%) of courses, respectively. Multimedia approaches including web-based content, videos and/or DVDs were also employed in 10 studies (38%), mostly as an adjunct to other approaches and less so as a central feature of the course. Case-based learning utilising real-life examples of adverse events identified by either participants themselves20 31–35 or presented by patients,22 was used as a core feature in seven (27%) courses. Project work (quality or safety improvement) was used in six studies (23%), and role-play and simulation were used in only four studies (15%). The latter is in contrast to studies of non-technical skills training (such as team training), which typically rely on resource-intensive simulation-based teaching modalities.36

Table 2

Core features of the courses studied, and Kirkpatrick's levels of evaluation

Characteristic	Studies involving students (n=11)	Studies involving trainees/residents (n=15)	All studies (n=26); number (%)
Educational modality
Small-group discussion/workshop	8	6	14 (54)
Lecture	7	5	12 (46)
Multimedia (web, DVD)	3	7	10 (38)
Case-based learning	2	5	7 (27)
Project/presentation requirement	2	4	6 (23)
Simulation/role-play	3	1	4 (15)
Core content
Patient safety overview (includes key terminology, emergence of safety)	7	10	17 (65)
Root cause/systems-based analysis	6	10	16 (62)
Communication and teamwork	6	7	13 (50)
Quality improvement	4	8	12 (46)
‘Human factors’	2	6	8 (31)
‘Systems thinking’	3	2	5 (19)
Medication safety	2	2	4 (15)
Error disclosure	1	3	4 (15)
Incident reporting (methods, barriers)	0	3	3 (12)
Kirkpatrick's level of evaluation
1: Participation	7	12	19 (73)
2a: Attitudes/perceptions	9	11	20 (77)
2b: Knowledge/skills	7	7	14 (54)
3: Behavioural change	3	13	16 (62)
4a: Organisational change	0	6	6 (23)
4b: Patient benefit	0	0	0

The most common content of the courses included a general overview of patient safety (including key terminology and the emergence of patient safety) and root cause analysis and/or systems-based analysis, featured in 17 (65%) and 16 (62%) studies, respectively. Communication and teamwork skills (both core ‘non-technical skills’) education was included in 13 (50%) studies, and quality improvement principles and methodologies in 12 (46%) studies. ‘Human factors (engineering)’ and ‘systems thinking’ were also covered in some studies, although these phrases were typically ill-defined by authors. Other less frequently covered content included medication safety, error disclosure, and incident reporting methods and barriers. Only three studies (12%) explicitly based their curricular content on the WHO's Patient Safety Curriculum Guide for Medical Schools; interestingly, all three were studies conducted outside of the USA.21 25 26 Of studies conducted in the USA, nine (53%) cited regulatory standards in education as the rationale to their work. This included reference to the AAMC Medical Schools Objective Project report, which recommends that medical schools deliver patient safety education to undergraduates,1 and the ACGME,3 which lists common competencies in practice-based learning and systems-based practice.

Study design and quality assessment

The majority of studies employed a before-and-after study design (n=18, 69%); four of these included a control group: two involved a contemporaneous control,22 37 one a historical control,32 and one a randomised contemporaneous control group.38 Only three (12%) studies included additional long-term follow-up, at 6 weeks,22 6 months,23 or ‘between 1 and 12 months’.37 Five (19%) studies involved a postintervention evaluation only. One study was a randomised controlled trial, however, due to logistical constraints, the control group did not undergo matched assessment of behavioural outcome measures.39

The median sample size across studies was 109 participants (IQR 52–188), and one outlier study had 1169 participants;20 some studies did not clearly indicate the exact number of participants. For example, one study was described as involving ‘over 787’ participants pooled over several years.28 The majority of studies were conducted within a single institution (n=18, 69%). Other common methodological limitations included poor response rates,19 22 35 40 41 inadequate description of the course19 39 and/or inadequate reporting of results.22 28 29 33 Limitations relating to the assessment tools employed are described in the following section.

Study evaluation and main findings

Table 2 displays the levels of evaluation assessed across the studies categorised by participant type (medical student or trainee/resident). Studies involving students primarily focused on participant satisfaction, attitudes and knowledge/skill acquisition, with less emphasis on behavioural change. In contrast, nearly all (n=13 of 15, 87%) studies involving trainees/residents examined behavioural change as a learning outcome, with six (23%) studies examining organisational impact through participant engagement in quality improvement work.19 20 35 42–44 None of the studies explored patient benefit (level 4b) as a result of the course.

The outcome measures, main findings and level(s) of evaluation reported in each study are displayed in table 3. Assessment tools used and main findings are discussed further under the respective Kirkpatrick's level headings below.

Table 3

Study outcome measures and main findings

Lead authorReferenceYear	Outcome measures	Main findings	Level of evaluation
Aboumatar412012	Primary outcome measures: preintervention and postintervention safety knowledge (19-item bespoke test), self-efficacy in safety skills (9-item bespoke survey), system-based thinking (using validated STS). Secondary outcome measures: Postintervention student satisfaction and safety intentions (2-item survey)	High participant satisfaction—intersession quality rated as excellent or very good by 92%. Significant improvement in composite systems thinking scores (61.15–67.56, p<0.001). Significant improvement in self-efficacy for all taught communication and safety skills (p<0.001). Significant improvement in safety knowledge scores pre–post (64% vs 83%, p<0.001). High self-reported safety behavioural intentions—85% reported they would speak up about safety concerns	1, 2a, 2b, 3
Ahmed202014	Participants’ satisfaction postcourse. Patient safety knowledge (MCQs), skills (bespoke questionnaire) and safety attitudes (modified validated questionnaire) pre–post. Behavioural change via questionnaire and review of ‘quality improvement databases’	High participant satisfaction. Significant improvement in 2 of 4 safety attitudes domains (ability to influence safety and behavioural intentions). Significant improvement in objective safety knowledge (51.1–57.6%, p<0.001). Trainees reported significantly more patient safety incidents in the 6 months following introduction of the intervention (p<0.001). 32 QI projects in various stages of implementation	1, 2a, 2b, 3, 4a
Anderson182009	Multimethod evaluation. Pre–post questionnaire assessing safety knowledge and perceptions of course (hopes, concerns and expectations). Additional postcourse satisfaction questionnaire and focus groups	Majority (>50%) satisfied with course; however, low scores on perceived preparation for the course. Postcourse medical student concerns emerged as being unfounded, and hopes and expectations in both the uni-professional and interprofessional groups were met. Focus group revealed consensus of added value in working interprofessionally. Significant improvement in students’ knowledge whether working uni-professionally or interprofessionally (p=0.001)	1, 2a, 2b
Arora212012	Participant satisfaction postcourse. Patient safety knowledge (MCQs) and safety attitudes (modified validated questionnaire) pre–post. Safety event identification and reporting 6 months postcourse via proforma	High participant satisfaction—overall satisfaction mean 4.63/5. Significant improvement in 2 of 4 safety attitude domains (attitudes to error analysis and improving safety, and ability to influence safety). Significant improvement in objective safety knowledge (45.3–70.6%, p<0.01) and subjective safety knowledge (p<0.01). Postcourse, participants recorded a higher number of observations associated with greater understanding, recognition and analysis of patient safety issues	1, 2a, 2b, 3
Cox282009	Professional group differences in attitudes and skills on 6 subscales (human fallibility, disclosure of medical errors, teamwork/communication, event reporting, systems of care, curricular time spent with other professionals). Assessed by bespoke survey pre–post intervention	Significant professional group differences preintervention in all 6 subscales. Postintervention differences in 4 subscales were resolved with the exception of human fallibility (p<0.001) and curricular time spent together (p<0.001). Medical students scored significantly worse on all subscales apart from human fallibility	2a, 2b
Cox312011	Satisfaction via simple survey. Qualitative analysis of narratives using constant comparative method	High participant satisfaction—85% rated it as a positive learning experience. 44% self-reported improvement in safety attitudes. High participant engagement—78% of residents submitted a story and 87% attended at least 1 safety session. 79 narratives submitted by residents over 3 months. Majority of stories involved errors (86%)	1, 2a
Dudas342011	Participant satisfaction. Patient safety attitudes (modified items derived from Safety Attitudes Questionnaire)	High participant satisfaction—76% recommended that the session continue. Significant improvements in patient safety attitudes pre–post in 9 of 10 items (p<0.01)	1, 2a
Gupta432014	Participant satisfaction postcourse (survey). Self-assessment and knowledge assessment about quality and safety principles precourse using a bespoke tool	High participant satisfaction. Experiential components were felt to be of most value.Almost half (49%) of items in the knowledge assessment were answered correctly preintervention (but no postintervention comparison data were reported). 75% of participants had ongoing formal or informal roles in QI or patient safety within their current practice environment following the course (specific time postintervention unreported by authors)	1, 2b, 4a
Hall322010	Patient safety attitudes and self-reported safety skills (previously published tool). Comparison preintervention and 1 year postintervention and with historical control. Analysis of student-submitted reports compared with contemporaneous reports from patient safety reporting system (PSN)	At baseline, no differences in any patient safety attitudes or safety skills between intervention and control. At 1 year postcourse, intervention group expressed significantly higher comfort level in identifying the cause for an error postintervention (3.72 vs 3.27, p<0.05). No significant difference in PSN worthy reports or in blame tone between participants and PSN reporters. Significantly higher robustness of proposed solutions by participants compared with PSN reporters (3 vs 0, p<0.001)	2a, 2b
Holland422010	Curriculum evaluation. Objective knowledge assessed via MCQs and true/false items precourse and immediately postcourse. Reflection on learning assessment at year-end including knowledge, skills, abilities and beliefs items	High satisfaction with curriculum (mean 3.53/4). Residents perceived significant improvements in knowledge, skills, abilities, beliefs and commitment to improve quality of care (all p<0.001). Significant improvement in knowledge (19.50–23.00, p<0.05). 20 QI projects proposed, 50% at various stages of implementation	1, 2a, 2b, 3, 4a
Jansma232010	11-item questionnaire exploring attitudes, intentions and behaviour towards reporting incidents (using vignettes and modified previously published tool). Assessed at baseline, immediately postcourse and 6 months postcourse	Attitudes towards incident reporting significantly improved (5 of 6 vignettes), p<0.001. Intentions towards incident reporting significantly improved between baseline and 6-month follow-up (p<0.05). No significant improvement in reporting behaviour	2a, 3
Jansma242010	Satisfaction and patient safety behaviours (via semistructured interview) 3 months postintervention to assess whether action implemented and the barriers and promoters to action(s)	High participant satisfaction—mainly positive reaction by 67%. 91 action points formulated by 68 participants. 62 (90%) residents reported taking action at 3 months; 50 (55%) actions were carried out fully. Barriers to implementing actions mentioned more than twice as frequently as compared with promoters. Barriers mostly related to work pressures and rotations	1, 3
Jericho332010	Attitudes towards adverse event reporting assessed preintervention and postintervention using a bespoke questionnaire (12 months). Quarterly adverse event reports submitted by residents	Significant improvement in attitudes towards reporting (no p value). Number of reports increased from 0 per quarter in the 2 years preintervention to 28 per quarter for the 7 quarters postintervention, with no sign of decay	2a, 3
Jha222013	Acceptability of the intervention by participants postintervention. Preintervention and postintervention administration of the APSQ, assessing attitudes and knowledge.52 Follow-up at 6 weeks: repeat APSQ, in-depth interviews, and an online survey about success in implementing learning points	Response to patient involvement in teaching was largely positive. Mean attitude and knowledge scores on the APSQ increased postintervention compared with preintervention (no p values reported). Response rate to 6-week follow-up APSQ was poor (38%). Only 6 participants participated in follow-up in-depth interviews; 3 provided evidence of implementation of learning in practice	1, 2a, 2b, 3
Leung252010	Patient safety attitudes and self-report knowledge (adapted previously published questionnaire) assessed precourse and 3 months postcourse	Participants supportive of inclusion of patient safety in curriculum and in professional examinations. Significant improvement in 8 of 15 items on patient safety attitudes. Significant improvements in all 5 items on self-reported patient safety knowledge; however mean scores still perceived as ‘fair’ or ‘poor’	2a, 2b
Miller272014	Postintervention questions exploring perceptions of the intervention. Patient safety attitudes (16-item bespoke questionnaire) preintervention and postintervention	Overall positive feedback about the course content. 69% of medical students preferred taking the course individually (the remainder preferring a groupwork format). Significant improvement in all items of the survey (p<0.05) assessing patient safety attitudes among medical students	1, 2a
Myung262012	Participant satisfaction (method not described). Patient safety awareness (40-item bespoke questionnaire) pre–post	Student and faculty commented on repetition of some material and desire for more interactive educational methods. Significant improvement in patient safety awareness in 36 of 40 items (p<0.05)	1, 2a
Paxton372010	Patient safety knowledge assessed via MCQ precourse and postcourse, and again at between 1 and 12 months postcourse. Application of learning assessed on long-term follow-up. Control group compared precourse and 6 months postcourse	Significant improvement in knowledge score at short-term (29.3–73.7%, p<0.001) and long-term follow-up (49.1%, p<0.001). 57.1% said they had applied the information learned in practice. No significant difference in knowledge found in control group	2b, 3
Rodrigue302013	Perceptions of experience with faculty development opportunities, performance and QI tools and training (bespoke survey). Resident participation in performance improvement, QI and patient safety programmes	Non-significant increase in number of residents that felt their training programme provided tools and training in QI. Postintervention, residents reported a non-significant increase (12.1%) in participation in departmental/institutional QI or safety projects, with faculty reporting a significant increase (38.2%, p=0.001)	2a, 3
Scott402011	Satisfaction with reporting mechanism. Participant attitudes and motivation regarding reporting and intervention (bespoke survey). Percentage of all adverse event reports submitted by residents via electronic reporting system	83% felt the system was burdensome. Monthly average number of adverse events reported by residents significantly increased by 5.5 times (6 (1.6%) to 33 (9%), p<0.001). Significant improvement in relative proportion of near-miss reports (0.3 (6%) to 9 (27%), p<0.001). Main motivators for reporting were patient wellness (87%) and financial incentive (64%)	1, 2a, 3
Shaw392012	Programme satisfaction using 7-item survey postintervention and focus group to explore experiences. NPSG-knowledge improvement using MCQ test preintervention and postintervention. NPSG-compliant behaviours in a simulation scenario. Self-reported confidence in safety and quality (bespoke survey)	Spaced Education participants found cases authentic, engaging and memorable. Significantly higher proportion of Spaced Education interns responded positively to satisfaction and self-reported confidence items (4 of 7 items, p<0.05). Both online programmes significantly improved knowledge (p<0.001). No significant difference in knowledge in control group. Higher proportion of Spaced Education participants with improved NPSG-behaviours (mean 4.79/13 vs 4.17/13 in SQ group; significant for surgical participants: 5.67 Spaced Education group vs 2.33 SQ group, p<0.05)	1, 2a, 2b, 3
Slater192012	Satisfaction questionnaire to evaluate online module and each workshop. Patient safety culture assessed using modified ‘Hospital Survey on Patient Safety Culture’ precourse and postcourse. Knowledge assessed using MCQs pre–post. Project outcomes using run charts. Interviews to explore experiences with TAPS	High rates of satisfaction for workshops (mean score 4.1/5), less so for online module (3.3). No change in safety culture scores for most dimensions apart from significant improvement in ‘communication/openness’ (p<0.01). Improved multiprofessional communication and teamwork reported via interview. Of the 5 participants who completed pre–post knowledge test, all but 1 improved score. 8 of 11 teams demonstrated improvements in patient safety practices/outcomes via run charts	1, 2a, 2b, 3, 4a
Smith352012	Satisfaction questionnaire to cohorts across the 2 years. Qualitative analysis of cases presented, interventions proposed and success of follow-through	High participant satisfaction—overall quality of QI conference mean 4.49/5. 46 interventions suggested; attempt to initiate 25 (54%) and of these 18 (72%) deemed successful: 8 led to objective permanent system-wide change and 10 resulted in subjective behavioural change	1, 3, 4a
Stahl382011	Participant satisfaction. Participant knowledge pre–post (24-item questionnaire based on previous studies). Participant behaviour postcourse (number of times observed and intervened in a patient safety risk)	Significantly greater satisfaction in intervention vs control group (75% vs 54%, p<0.05). Significantly greater improvement in patient safety knowledge in intervention vs control group (83% vs 75%, p<0.001). Significantly greater proportion of intervention group self-reported intervening to avoid error compared with control group (77% vs 61%, p<0.05)	1, 2b, 3
Tess442009	Programme evaluation, survey of participant attitudes (bespoke survey), and participation in patient safety and QI work	High participant satisfaction including significant improvement in quantity of teaching, and overall value of clinical rotations postintervention. Significant postintervention improvement in 6 of 12 questions addressing attitudes about culture of safety and 3 of 11 items on residents’ perception of educational goals during the residency programme (all p<0.05). All participants completed an adverse event review. Significant improvement in engagement with departmental QI meeting (>66% postintervention vs 10%)	1, 2a, 3, 4a
Wilson292012	Course satisfaction. Evaluation based on class participation (30%), peer evaluation (15%) and group project paper and presentation (total 55%)	The attendance score for medical students was the lowest (8.59 of 10). Peer evaluation of all students was high; medical students were the ‘low outlier’ in 8 of 10 categories. Students rated assigned reading material as extremely helpful. Learners’ perceived that analysing the case studies in multidisciplinary groups gave more insight into understanding the problems and proposing solutions	1

APSQ, Attitudes to Patient Safety Questionnaire; MCQ, multiple choice question; NPSG, National Patient Safety Goal; PSN, Patient Safety Network System; QI, quality improvement; SQ, safety questionnaire; STS, System Thinking Scale; TAPS, Training and Action for Patient Safety.

Level 1: participation/satisfaction

This was assessed in 19 (73%) studies. Satisfaction was mostly assessed using questionnaires postintervention requiring responses on a Likert scale. Three studies supplemented satisfaction questionnaires with either focus groups18 39 or interviews with participants.19 Satisfaction with the courses was generally high, although response rates were poor in some studies.22 35 41 Two studies evaluating courses that included web-based content reported poor uptake34 or lower satisfaction rates19 with the web-based learning component.

Level 2a: attitudes/perceptions

Patient safety attitudes/perceptions were assessed using a variety of tools in 20 (77%) studies. Bespoke questionnaires comprising items mapped to course learning objectives were used in 11 studies.18 26–28 30 31 33 39 40 42 44 Two studies used modified versions of validated tools,21 34 and a further four studies used modified versions of previously published questionnaires.20 23 25 32 One study used the previously published ‘Attitudes to Patient Safety Questionnaire’.22 One study assessed systems-based thinking using a validated scale (‘System Thinking Scale’, STS),41 and one study assessed perceived patient safety culture using the modified ‘Hospital Survey on Patient Safety Culture’.19 Of studies evaluating patient safety attitudes preintervention and postintervention, the majority of studies reported significant improvement in at least some domains. The study assessing systems-based thinking reported significant improvement in STS scale scores postintervention,41 while the study evaluating perceived patient safety culture reported no change postintervention.19

Level 2b: knowledge/skill acquisition

Fourteen (54%) studies evaluated knowledge acquisition using objective and/or self-report measures. Objective tests were used in 12 studies;18–22 31 37–39 41–43 these comprised multiple choice or true/false questions mapped to course learning objectives. One of these studies used knowledge questions from the ‘Attitudes to Patient Safety Questionnaire’.22 Most studies demonstrated significant improvements in knowledge acquisition, although in one study a poor response rate precluded statistical testing,19 in another no comparison between preintervention and postintervention scores was reported,43 and in yet another, increases in performance were observed postintervention but no statistical analyses were reported.22

Learners’ patient safety skills were assessed in seven (27%) studies,20 28 32 39 41–43 all of which employed self-reported measures. Six of these studies demonstrated significant improvement in scores for most or all items, with the remaining study not reporting a comparison between preintervention and postintervention scores.43

Level 3: behavioural change

Changes in safety-related behaviours were assessed in 16 (62%) studies, in a number of ways: behavioural intentions assessed via questionnaire;23 41 42 self-reported safety-related actions (eg, incident reporting);19 20 22–24 30 37 38 or by safety-related actions determined objectively.20 21 33 35 39 40 44 Of these latter studies, objective assessment included qualitative assessment of patient safety observations,21 National Patient Safety Goal (NPSG)-related behaviours assessed via simulation,39 engagement in quality improvement work20 35 44 and incident reporting assessed via submissions to formal hospital reporting systems.33 40 All studies reported favourable changes in safety-related behaviours, with the exception of one study, which found that whereas learners’ intentions to report significantly improved postcourse, actual (self-reported) incident reporting did not increase following the course.23 Notably, all but 3 of the 16 studies that evaluated change in participant behaviour were conducted on trainees/residents as opposed to on medical students.

Level 4a: organisational change

Six (23%) studies evaluated organisational change as an outcome measure of their course. Each of these studies involved learner engagement in quality improvement work,19 20 35 42–44 and all these studies reported subsequent positive impact at organisational level, including through the initiation/continuation of quality improvement projects/roles.20 35 42–44 Three quarters of the participants in one study indicated they had a formal or informal role in patient safety or quality improvement within their current practice environment.43 The team-based ‘Training and Action for Patient Safety’ (TAPS) programme found that 8 of the 11 interdisciplinary teams were able to demonstrate improvements in patient safety outcomes and/or practices through the use of weekly data plotted on run charts.19

Factors influencing curricular implementation

Table 4 displays the key factors influencing curricular implementation that we identified, with selected illustrative quotes and categorised under previously designed framework headings.8 In terms of learner factors, many studies identified the need to ensure personal/clinical relevance of the material to learners, with opportunities to apply the learning in order to enhance engagement.20 39 For studies involving physicians, competing clinical commitments were identified as barriers to engagement.21 In studies employing interprofessional modalities, improved teamwork and communication were welcome additional benefits of the course.19 However, difficulties in delivering such interprofessional learning were highlighted.28 Most studies identified the need for adequate faculty, with protected time to support delivery of the course and competing clinical commitments of faculty being barriers to faculty engagement.20 Some commented on the newfound maturity of the faculty infrastructure,41 while others aspired to broaden their faculty infrastructure to ensure sustainability of the course.42 Faculty role-modelling and clinical credibility were noted to be important influencing factors.25

Table 4

Factors influencing implementation of patient safety courses

Factors	Illustrative quotes from published articles
Learner factors
Enhancing learner engagement by ensuring clinical relevance	“The cases, exploring incidents that were largely based on events that had in fact happened, were felt to be realistic and directly applicable to the context of the interns.”39“We believe that using authentic clinical scenarios brought forward by trainees as opposed to hypothetical scenarios as in previous studies ensured relevance to trainees and furthermore, stimulated trainee engagement in QI work.”20
Empowering learners through application of learning	“Our program challenges residents to apply their skills in systems-based practice to a resident-driven, hospital-based project in an effort to solidify their commitment to QI beyond the structured rotation.”42
Competing clinical/service delivery commitments	“Although all general surgical residents were invited, just more than one half actually attended, citing scheduling conflicts and service delivery pressures as reasons for not doing so.”21
Learning interprofessionally improved teamwork and communication	“The programme promoted better multi-professional communication and teamwork.”19
Faculty factors
Investment in faculty development is essential	“Successful implementation of this curriculum, however, requires attention to faculty development. It took several years at our institution to achieve this and some schools may not have similar resources.”41
Faculty role-models and importance of clinical credibility	“Faculty had clinical background and we feel that our students can relate to them more readily.”25
Protected faculty time	“The residency program further invested in quality by naming both an assistant and associate program director for quality amounting to roughly 0.1 full-time equivalent spent working on the QICs and subsequent project implementation.”35“The main barrier to faculty engagement….was competing clinical commitments.”20
Curricular factors
Promoting patient safety as a science	“The topic of safety was approached as a ‘science’ with a defined set of principles and theories, and supported with published literature.”41
Competing curricular demands	“Whole-class lectures are by no means the best way to teach patient safety but we find it the easiest format to integrate into a busy curriculum.”25“Making the course mandatory would have been one way of overcoming this but this would require curricular change at regional level.”21
Balance between didactic and experiential learning	“The students want to increase small-group discussions and simulation sessions, which would be more effective than didactics.”26
Balance between reinforcement of learning and repetition of teaching material	“The rapid decline in long-term post-test scores indicates that…students would benefit from frequent reinforcement of the application of this material.”37“Only half of the students elected to view it (online video)…this may be due to the perceived redundancy of the information presented.”34
Central administrative support necessary for sustainability	“We were able to arrange small-group sessions for the randomised, decentralized project for three months, but a core educational activity that includes all residents and is managed centrally would be more sustainable.”31
Creating interprofessional learning opportunities is challenging	“It is complicated and time-intensive to plan and deliver meaningful and satisfying inter-professional learning experiences.”28
Learning environment factors
Institutional culture as key to implementation	“It is important to focus not only on individual attitudes and intentions, but also on a stimulating environment, including hospital culture and patient safety policies.”23
Ensuring a safe learning environment	“Several residents commented that they felt safe with the reporting methodologies and follow-up.”33“We believe that few of these reports of safety concerns would have been brought forward without providing a structured forum for discussion in a trusted and collegial environment.”32
Forging improved links between training programmes and hospital improvement activities	“To foster engagement and sustainability, we are now working to more deliberately and consistently integrate patient safety education with the hospitals’ systems improvements.”31“The chair of the department and the program director were very supportive of this endeavour.”35
Financial support to fund the programme	“VA hospital's willingness to financially support 2 residents per month in this intensive patient safety and quality improvement rotation…Dedicated faculty rotation leaders supported by the VA with protected time to teach and mentor residents.”42

QI, quality improvement; QICs, QI conferences; VA, Veterans Affairs.

Competing curricular demands were commonly cited as barriers to sustainability of the courses, with some suggesting instituting the course as a mandatory requirement to ensure protected time for learning.21 Promoting patient safety as a science was felt to be a key factor for successful implementation by the authors of one study.41 The majority of studies appreciated the need to strike a balance between didactic and experiential teaching modalities, and of the need for sufficient reinforcement while avoiding repetition and duplication of material. The authors of one study recognised that delivering a centrally administered intervention to the whole trainee population may ensure greater sustainability of the course than delivering it to a sample of the cohort.31

In terms of institutional/learning environment factors, many studies recognised institutional patient safety culture as a key determinant of successful implementation.23 Ensuring a safe learning environment to allow open discussion of sensitive material (eg, relating to adverse events) was recognised as being of particular importance when delivering education on patient safety. Forging improved links between the service provider (hospital) and the training providers was recognised as key to ensuring sustainability, particularly for courses that aimed for engagement in quality improvement work as a follow-on to the course.35

Sustainability

Six (23%) studies identified in this review reported data from courses that had been sustained over at least 2 years,18 20 27 31 35 43 two studies reported ‘booster’ courses designed to enhance/reinforce established safety educational interventions delivered earlier in the course of training,32 38 and one study described an educational intervention coupled with reorganisation of clinical services to facilitate quality and safety improvement efforts.44

Discussion

This systematic review provides an update of the evidence on courses teaching core concepts of patient safety to medical students and trainees/residents. We identified 26 studies published between January 2009 and May 2014. This is in contrast to a previous systematic review addressing the same topic but with a wider remit and time period (January 2000 to January 2009), which found 27 studies published incorporating evaluation of the interventions.8 This suggests that there is increasing interest in developing, delivering and evaluating courses teaching patient safety.

In the previously published systematic review,8 the interventions were mostly well received by participants and resulted in improvements in safety and quality knowledge scores. However, few studies were able to demonstrate changes in learners’ behaviour (Kirkpatrick's level 3) or potential patient benefit (level 4b). Moreover, thematic analysis of the articles identified multiple barriers to sustainable integration of the courses, which spanned learner, faculty and institutional factors. Our systematic review has also found the included interventions to be mostly well received by participants, with improvements in safety knowledge and attitudes. Whereas more studies in our review were able to demonstrate positive changes in participant behaviour relative to the previous review, this was mainly for interventions targeted at trainees/residents rather than medical students, and most of these data on participant behaviour were self-reported. None of our identified studies demonstrated patient benefit (level 4b) from the interventions, although measurement of changes in clinical outcomes following educational interventions is notably difficult, in part due to the complexities in establishing true cause and effect.

Assessment of organisational change (level 4a) resulting from the intervention was also infrequent in our identified studies, particularly in those involving medical students. Furthermore, in the studies we reviewed, barriers to sustainable integration of the courses also spanning learner, faculty and institutional factors, were identified. Such factors included poor learner engagement, lack of expert faculty, competing educational priorities and an unsupportive institutional culture. There is no clear relationship between the length of the patient safety course and effect on learning outcomes, although a meaningful analysis of this is confounded by differences in course content and study design, quality and reporting.

Despite increasing evidence for the efficacy of educational interventions in patient safety, the wider implementation and adoption of successful interventions has been slow.45 46 As a result, recommendations to promote curricular integration of patient safety education aim to address the barriers outlined above—for example, through investing in faculty development, promoting patient safety as a science, and integrating patient safety competencies into accreditation standards and certification examinations, to ensure protected time and incentives for medical engagement.46 47

As in the earlier systematic review by Wong and colleagues,8 the majority of studies we identified in this systematic review were conducted in the US and preferentially targeted residents over medical students. The dominance of US studies in this systematic review may reflect the explicit integration of competencies in patient safety and quality improvement within national curricular statements and guidance.1 3 The majority of studies we identified in our review had small participant numbers, relied on single centre recruitment, and were designed as before-and-after studies with no control group or follow-up. Therefore, overall, the methodological quality of studies of patient safety interventions in medical students and trainees/residents has not changed significantly between this systematic review and the previously published one.8 This is despite recent years being characterised by the development of curricula and frameworks specifically targeting patient safety.1 2

Our systematic review does, however, provide some positive evidence of developments in the literature. Many of the studies we identified used previously published and/or validated assessment tools, demonstrating a knowledge and appreciation of the emergent evidence base in patient safety education. In line with good educational practice, the majority of studies employed experiential learning modalities (such as group discussion and project work), although one study relied solely on didactic lectures to facilitate integration into a ‘busy curriculum’.25 Interestingly, case-based learning of real-life adverse events was used in few studies, despite the recognised value of reflecting and learning from error and adverse events,48 and their popularity among trainees.49 50 It is particularly encouraging to note that we found an increase in studies explicitly commenting on sustainability of the described interventions, and their integration into the wider institution, in comparison to the previous systematic review.8 This may reflect a trend to more consideration of the longer term sustainability of patient safety interventions.

In the previous systematic review,8 the core content most commonly comprised of root cause analysis, systems thinking, general patient safety concepts and error incident reporting (all identified in over 30% of courses). In contrast, we found content to most commonly cover root cause/systems-based analysis, general patient safety concepts, communication and teamwork, quality improvement and human factors (all identified in 30% or more of published courses). Importantly, there was a marked increase in the proportion of studies covering general patient safety concepts between the previous systematic review and this one, from 34% to 65%. Coverage of root cause/system-based analysis also increased from 41% to 62% of studies. In addition, between the two systematic reviews there was a decrease in the number of studies covering error/incident reporting, from 32% to 12% of studies. This discrepancy between the two systematic reviews may reflect the different search strategies used. However, it may also relate to, for example, the increasing recognition of the importance of communication and teamwork in patient safety,51 and the importance of a foundation in basic patient safety knowledge and concepts. Without sufficient studies with long-term follow-up data on patient outcomes, it is difficult to ascertain the true implications of these changes in core content. This is clearly an area for future research.

The main limitations of this systematic review relate to the quality of the included studies and the narrower focus when compared with the previous systematic review. We only included manuscripts published in the English language. We may have missed some relevant studies, although no systematic review can truly claim to find all relevant studies. There was significant heterogeneity across the studies in terms of number and type of participants targeted, the educational content of the course, the teaching methods employed, assessment tools used and the outcomes measured, which prevented a quantitative synthesis of the results. Moreover, the identification of factors influencing implementation of the courses was wholly dependent on the quality of reporting of such factors by the authors, many of whom did not stipulate identifying such factors as the primary aim of their study. It may be that important barriers and enablers to the sustainable integration of patient safety courses remain unreported, although it is important to note that we identified similar barriers and enablers to those identified in the previous systematic review.8 In box 1 we offer some recommendations for a minimum description of content that could be used in future studies evaluating patient safety courses. Adhering to these should improve study reporting and the comparison of the relative effectiveness of patient safety training interventions.

Study design (eg, prospective, retrospective, before and after design, control groups)

Study setting (eg, single centre, multicentre)

Participants including inclusion and exclusion criteria

Intervention

Delivery method of all aspects of the intervention (eg, online, didactic lecture, group setting)

Thorough and explicit description of course content

Description of those delivering the intervention (faculty), their training and their qualification

Educational theory/theories underpinning the intervention

Method(s) of evaluation and detailed description of exactly when these were conducted

Specific outcomes assessed (eg, knowledge, attitudes, patient outcomes)

Length and type of follow-up

Data analysis methods

Factors influencing course implementation (barriers and enablers)

Limitations of the intervention

Areas for further work

In addition to the need for future studies to address the aforementioned limitations in the evidence base, the relationship between approaches to teaching (including underpinning educational theory) and the different types of learning outcomes, should also be explored. So, too, should the relationship between implementation approaches and the impact on sustainability of an educational intervention. Such knowledge should optimise the quality of the evidence base and facilitate the development of robust evidence-based guidelines on factors that can improve outcomes at multiple levels following educational interventions for patient safety.

For those involved in medical education, there are recommendations aimed at addressing barriers to the implementation of patient safety courses. These can be classified into recommendations related to the learner, faculty, curriculum and learning environment. Learner-relevant recommendations include: ensure courses have personal and/or clinical relevance, and offer the opportunity to apply learning to enhance engagement; ensure freedom from competing clinical/service delivery commitments; and make learning interprofessional. Faculty recommendations include: invest in faculty development; establish role models with clinical credibility; and ensure protected faculty time to deliver the patient safety course free from other commitments. Curricular recommendations include: promote patient safety as a science; avoid competing curricular demands; ensure an adequate balance between didactic and experiential learning, and between reinforcement of learning and repetition of teaching material; and provide adequate central administrative support to ensure sustainability. Finally, recommendations for the learning environment include: recognise the institutional culture as key to implementation; ensure a safe learning environment; foster links between training programmes and hospital improvement activities; and provide adequate financial support to fund the programme.

Conclusions

There is an increasing trend for the development of educational interventions in patient safety delivered to trainees/residents and medical students. The majority of such courses are well accepted by learners, and improve patient safety knowledge, skills and attitudes. Moreover, some interventions have been shown to result in positive behaviours, particularly through the subsequent engagement of trainees/residents in quality and safety improvement projects. However, no studies in the current systematic review demonstrated patient benefit. Significant methodological shortcomings in current studies exist, and additional evidence of the impact of such interventions on patient outcomes is needed. In addition, although the evidence appears to suggest some maturation in the approach and infrastructure required to support on-going delivery, significant barriers to the implementation of patient safety education remain. Further work is needed to successfully address the challenges and promote the sustainable integration of education and training in patient safety.