Strategies to evaluate healthcare provider trainings in shared decision-making (SDM): a systematic review of evaluation studies.

DESIGN AND
OBJECTIVES: We performed a systematic review of studies evaluating healthcare provider (HCP) trainings in shared decision-making (SDM) to analyse their evaluation strategies. SETTING AND PARTICIPANTS: HCP trainings in SDM from all healthcare settings.
METHODS: We searched scientific databases (Medline, PsycInfo, CINAHL), performed reference and citation tracking, contacted experts in the field and scanned the Canadian inventory of SDM training programmes for healthcare professionals. We included articles reporting data of summative evaluations of HCP trainings in SDM. Two reviewers screened records, assessed full-text articles, performed data extraction and assessed study quality with the integrated quality criteria for review of multiple study designs (ICROMS) tool. Analysis of evaluation strategies included data source use, use of unpublished or published measures and coverage of Kirkpatrick's evaluation levels. An evaluation framework based on Kirkpatrick's evaluation levels and the Quadruple Aim framework was used to categorise identified evaluation outcomes.
RESULTS: Out of 7234 records, we included 41 articles reporting on 30 studies: cluster-randomised (n=8) and randomised (n=9) controlled trials, controlled (n=1) and non-controlled (n=7) before-after studies, mixed-methods (n=1), qualitative (n=1) and post-test (n=3) studies. Most studies were conducted in the USA (n=9), Germany (n=8) or Canada (n=7) and evaluated physician trainings (n=25). Eleven articles met ICROMS quality criteria. Almost all studies (n=27) employed HCP-reported outcomes for training evaluation and most (n=19) additionally used patient-reported (n=12), observer-rated (n=10), standardised patient-reported (n=2) outcomes or training process and healthcare data (n=10). Most studies employed a mix of unpublished and published measures (n=17) and covered two (n=12) or three (n=10) Kirkpatrick's levels. Identified evaluation outcomes covered all categories of the proposed framework.
CONCLUSIONS: Strategies to evaluate HCP trainings in SDM varied largely. The proposed evaluation framework maybe useful to structure future evaluation studies, but international agreement on a core set of outcomes is needed to improve evidence. PROSPERO REGISTRATION NUMBER: CRD42016041623. © Author(s) (or their employer(s)) 2019. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

A strength of this study is the fact that we sought all types of evaluation strategies for healthcare provider trainings in shared decision-making and included all types of study designs from post-test studies to qualitative and cluster-randomised controlled studies.

A strength of this study is the fact that we developed an evaluation framework, which may be a first step towards agreement on a core set of evaluation outcomes and measures to improve evidence on healthcare provider trainings in shared decision-making.

A limitation of this study is the fact that we did not analyse which measures are useful to evaluate healthcare provider trainings in shared decision-making.

A limitation of this study is the fact that we focused on the analysis of evaluation outcomes, but did not analyse evaluation strategies with regard to appropriate measurement points or the match between training contents and use of evaluation outcomes.

A limitation of the proposed evaluation framework is that it focuses on evaluation outcomes, but does not take into account aspects like appropriate study designs.

Introduction

Healthcare policies, clinical guidelines and a growing body of research strongly advocate for the implementation of shared decision-making (SDM) as a central element of patient-centred care.1 Policy makers are interested in SDM, because it tackles overuse, underuse and misuse of healthcare interventions all at the same time.2 In SDM, the patient and at least one clinician share information and values, deliberate the next step and arrive at a jointly made decision.3 Patients who experienced SDM reported less decisional conflict and improved satisfaction,4 but evidence regarding health-related outcomes is limited.4–6 To date, the most conclusive argument for SDM is ethical. Patients have the right to learn about available treatment options and their implications, and to participate in decision-making regarding their health.4 7 Despite multiple implementation initiatives8 and widespread support, SDM is not yet implemented in routine care.7 9

Interventions to foster the implementation of SDM usually target healthcare providers (HCPs), patients or both.10 They may include the distribution of written educational material or patient decision aids, patient coaching, audit and feedback for HCPs or HCP trainings in SDM.11 HCP trainings in SDM are group or online courses that address HCP SDM attitudes, knowledge or skills. They include the use of lectures, case studies, role play, group discussion or didactic materials.12 HCP trainings in SDM are considered key to implement SDM in healthcare, but it is unclear what kind of trainings are most effective and which outcomes they affect.10–13 The lack of consensus on an evaluation framework for HCP trainings in SDM partly accounts for this lack of evidence.14

Evaluation frameworks support practitioners and researchers in the design of coherent evaluation strategies.15 Kirkpatrick’s four-level training evaluation model16 is the most established and feasible model for training evaluation and can be applied to the context of HCP professional development.17 Kirkpatrick’s four levels are: 1) reaction, 2) learning, 3) behaviour and 4) results. The reaction level includes participant reactions to the training and can be assessed with attendance levels or subjective training impressions. The learning level covers participant changes in attitudes, knowledge or skills after the training. The behaviour level covers changes in participant behaviours or transference of training content to the workplace. The results level describes more tangible trainings results, for example, system effects or patient health outcomes.4 17 18

Elwyn et al 9 argue that SDM research has neglected investigation of diverse long-term consequences on the results level. They postulate that widespread implementation of SDM leads to safer and more cost-effective decisions, to reduce utilisation rates and to improve patient health outcomes, but evidence is lacking.9 The influential Quadruple Aim framework aims to improve the experience of care, the health of populations, the per capita cost of healthcare and the work life of HCPs,19 and may be useful to structure evaluation of HCP trainings in SDM on the results level.

In this review, we aimed to analyse how the diversity of evaluation strategies and the quality of published evaluations contributes to the current lack of evidence on HCP trainings in SDM. Thus, we aimed to investigate the quality of published evaluations of HCP trainings in SDM, and to analyse their evaluation strategies. We aimed to analyse evaluation strategies regarding 1) use of data sources, 2) use of unpublished or self-developed and published or psychometrically tested measures and 3) coverage of Kirkpatrick’s four levels. We aimed to categorise identified outcomes in an evaluation framework for HCP trainings in SDM based on Kirkpatrick’s four-level evaluation model16 and the Quadruple Aim framework19 to guide future research and to initiate discussion about a core set of evaluation outcomes for this purpose.

Methods

Registration and search strategy

This study followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines for systematic reviews20 in most parts (see online supplementary file S1). We made the following changes to the protocol: we adapted the PICOS (P: patient, problem or population, I: intervention, C: comparison, control or comparator, O: outcomes, S: study type) criteria to meet our research purpose, we did not remove duplicates in the secondary search, we did not assess risk of bias across studies and we did not use any summary measures or additional analyses as we limited our work to qualitative synthesis only. We performed an electronic database search employing Medline, CINAHL and PsycInfo databases (via OVID) on 26 June 2016. For this purpose, we developed a detailed search strategy for each database. We adapted the PICOS criteria20 and considered a combination of the following aspects appropriate: population AND intervention AND construct AND outcome OR study design. Terms and keywords were adapted for each database and searches in Medline and PsycInfo were limited to publications concerning humans. We updated the electronic database search on 30 January 2019. Full insight in the electronic database search strategy is attainable in online supplementary file S2. Moreover, we performed a secondary search including reference and citation tracking of included full-text articles, consultation of experts in the field of research via a shared decision-making facebook group and a screening of the Canadian inventory of SDM training programmes for healthcare professionals (http://www.decision.chaire.fmed.ulaval.ca/en/list-of-sdm-programs). Additionally, we screened references of two reviews on SDM interventions for HCPs.11 12 We registered details of the protocol for this systematic review on PROSPERO website accessible via www.crd.york.ac.uk/PROSPERO/display_record.asp?ID=CRD42016041623.

Article selection

We aimed to include articles reporting on summative evaluations (outcome or study design) of HCP (population) trainings (intervention) in SDM (construct) and developed inclusion and exclusion criteria, accordingly. We aimed to exclude articles reporting on formative evaluations or interventions that do not have the main aim to teach SDM skills to HCPs (Table 1). Following the first database search (26 June 2016), two reviewers independently screened titles and abstracts of a random sample of 300 (>5%) records identified in the electronic database to ensure sufficient inter-rater reliability. We discussed any differences until we reached consensus. Records identified in the electronic database search were then split in half to be assessed for possible inclusion in the study by one of two reviewers. Following the update of the database search (30January 2019), two reviewers independently screened all identified records and discussed any differences until consensus was reached. Two reviewers independently assessed full-text articles for eligibility by applying inclusion and exclusion criteria (box 1). We resolved differences by discussion until we reached consensus. If consensus could not be reached, the final decision was made by discussion with two other reviewers.

Table 1

Descriptive data of included study articles

Study	Country of origin	Study design	Healthcare provider sample*	Patient sample*
Bernhard et al 54	AUS, NZ, CHE, GER, AUT	RCT	Medical, surgical, radiation and gynaecological oncologists (n=62)	n=769
Bieber et al 23	GER	RCT	Specialists in internal medicine (n=13)	n=111
Bieber et al 22	GER	RCT	Specialists in internal medicine (n=10)	n=85
Bieber et al 24	GER	NCBA	Physicians with direct patient contact (n=123)	n/a
Bieber et al 30	GER	RCT	Physicians treating patients with cancer (n=27)	n=107
Butow et al 55	AUS	RCT	Medical, surgical, radiation and gynaecological oncologists (n=62)	n=158
Cohen et al 50	UK	CRCT	General practitioners (n=20)	n=n/r
Davis et al 51	UK	QUAL	General practitioners (n=21)	n=38
Dion et al 41	CAN	NCBA	Family medicine residents (n=247)	n/a
Edwards et al 52	UK	RCT	General practitioners (n=20)	n=n/r
Edwards et al 52	UK	CRCT	General practitioners (n=21)	n=747
Edwards et al 53	UK	QUAL	General practitioners (n=18)	n/a
Edwards et al 52	UK	CRCT	General practitioners (n=20)	n=352
Feng et al 42	USA	RCT	Primary care physicians (n=118)	n=n/r
Geiger et al 31	GER	RCT	Physicians (n=38)	n=152
Härter et al 25	GER	CRCT	Physicians treating patients with cancer (n=33)	n=160
Jo and An59	KOR	CBA	Female intensive care unit nurses (n=41)	n/a
Kasper et al 32	GER	NCBA	Physicians working in outpatient clinics (n=10)	n=40
Körner et al 26	GER	CRCT	Healthcare provider executives of different occupational backgrounds (n=74)	n/a
Körner et al 27	GER	CRCT	Healthcare provider executives of different occupational backgrounds (n=74)	n=n/r
LeBlanc et al 33	CAN	CRCT	Family medicine group physicians (n=39)	n=544
Légaré et al 36	CAN	CRCT	Family medicine group physicians (n=33)	n=459
Légaré et al 35	CAN	CRCT	Family physicians (n=306)	n=449
Légaré et al 34	CAN	CRCT	Family physicians, residents and nurse practitioners (n=250)	n=250
Loh et al 28	GER	Post	General practitioners (n=20)	n/a
McCallister et al 43	USA	NCBA	Pulmonary and critical care medical fellows (n=16)	n/a
Metcalfe et al 56	AUS	NCBA	General practitioners (n=63)	n/a
Murray et al 37	CAN	RCT	Oncology or palliative care nursing and allied healthcare providers (n=88)	n/a
Price-Haywood et al 44	USA	CRCT	Primary care physicians (n=18)	n=161
Sanders et al 57	NL	CRCT	General practitioners (n=42)	n=175
Sanders et al 58	NL	CRCT	General practitioners (n=47)	n=226
Simmons et al 60	USA	NCBA	Internal medicine residents (n=98)	n/a
Stacey et al 38	CAN	RCT	Call centre nurses (n=39)	n/a
Stacey et al 39	CAN	NCBA	Oncology medical residents (n=11)	n/a
Sullivan et al 46	USA	RCT	Internal medical residents, attendings (n=45)	n/a
Sullivan et al 45	USA	RCT	Internal medical residents (n=213)	n/a
Tinsel et al 29	GER	CRCT	General practitioners (n=36)	n=1120
Towle et al 40	CAN	QUAL	Family physicians (n=6)	n=198
Volk et al 47	USA	Post	Clinicians from diverse specialties (n=49)	n/a
Wilkes et al 48	USA	CRCT	Primary care physicians (n=120)	n=712
Yuen et al 49	USA	Post	Internal medicine residents (n=29)	n/a

*Participants who provided data for analysis.

†Articles report data from one study.

AUS, Australia; AUT, Austria; CAN, Canada; CBA, controlled before-after study; CHE, Switzerland; CRCT, cluster-randomised controlled trial; GER, Germany, KOR, Korea; n/a, not applicable; NCBA, non-controlled before-after study; NZ, New Zealand; NL, The Netherlands; n/r, not reported; QUAL, qualitative study; Post, post-test only study; RCT, randomised controlled trial.

Inclusion criteria*

Article reports data on a health care provider training in SDM

Article reports data on summative evaluation of a SDM training

Article has the aim to evaluate a SDM training

Exclusion criteria

Article is a study protocol or clinical trial report

Article reports on a study in which participants were medical or health care students

Article only reports on formative evaluation of a SDM training

Article reports on an intervention that is limited to or has the main aim to teach the use of a decision-making tool

Article reports on an intervention that does not have the main aim to teach SDM communication skills

Article reports on a complex intervention, in which health care provider training is only a component and no separate summative evaluation data on the training are reported

Article is written in a language other than English, German, French or Dutch (languages spoken by teams members)

Full text of the article is not available

*Articles were excluded if inclusion criteria were not met.

Data extraction, quality assessment and analysis of evaluation strategies

We used data extraction sheets to collect descriptive data of included articles, for example, country of origin of the study, study design, characteristics of HCP and patient samples. Furthermore, we extracted data on evaluation outcomes reported in included articles and all data relevant to assess study quality of included articles. Data extraction sheets were pilot-tested and adjusted accordingly. We assessed study quality of included articles with the integrated quality criteria for review of multiple study designs (ICROMS) tool.21 Two reviewers independently performed data extraction and quality evaluation and discussed any differences until consensus was reached. One reviewer performed analysis of evaluation strategies in discussion with the team. As study results are repeatedly published in more than one article, we will present results on two levels: study and article, if applicable.

Quality assessment with the ICROMS tool

The ICROMS tool appraises the quality of multiple study designs and stems from an iterative process over 2 years that included review of existing quality criteria, pilot testing and expert consensus. It aims to establish criteria critically appraising the quality of multiple study designs, in order to broaden the database for systematic reviews and to inspire rigorous research.21 The ICROMS tool comprises 7 dimensions and defines 33 specific criteria for these dimensions applicable only to some study designs. ICROMS dimensions are 1) clear aims and justification, 2) managing bias in sampling between groups, 3) managing bias in outcome measurements and blinding, 4) managing bias in follow-up, 5) managing bias in other study aspects, 6) analytical rigour, 7) managing bias in reporting/ethical considerations. The ICROMS tool is applicable for cluster-randomised and randomised controlled trials, controlled and non-controlled before-after studies, controlled and non-controlled interrupted times series, cohort studies and qualitative studies. As the ICROMS tool is not applicable to post-test studies, we did not assess study quality for articles reporting on this study type. ICROMS-specific criteria are answerable with yes (2 points), no (0 points) or unclear (1 point). The ICROMS tool defines mandatory criteria and minimum scores for different study types to distinguish if studies are fit for inclusion in a systematic review. Minimum scores vary per study type and range from 16 for qualitative studies over 18 for controlled before-after studies to 22 for non-controlled before-after studies or cluster-randomised and randomised controlled trials. Detailed information on the ICROMS tool is attainable in the original publication.21 We analysed quality assessment results on article level.

Analysis of evaluation strategies

One reviewer analysed evaluation strategies regarding use of data sources (HCPs, patients, standardised patients, observers, training process and healthcare data), use of unpublished or self-developed and published or psychometrically tested measures and coverage of Kirkpatrick’s four levels of reaction, learning, behaviour and results.16 One reviewer categorised identified evaluation outcomes in the proposed evaluation framework for HCP trainings in SDM (Figure 1) that is based on the Kirkpatrick’s four-level evaluation model16 and the Quadruple Aim framework.19 One reviewer developed comprehensive subcategories of evaluation outcomes based on the measures identified in the review and categorised evaluation outcomes accordingly. The study team supervised this process and provided feedback in team discussions. As study results are repeatedly published in more than one article, we will present results on two levels: study and article, if applicable.

Figure 1

Evaluation framework for healthcare provider trainings in shared decision-making (SDM).

Patient and public involvement

We did not involve patients in the conduction of this study.

Results

Literature search and article selection

The electronic database search on 26 June 2016 identified 5317 records. After removal of duplicates, 4543 records remained. We found an additional number of 1636 records through the secondary search. The electronic database search on 30 January 2019 identified additional 1222 records. After removal of duplicates, 1055 records remained. We finally screened 7234 records, of which some are likely to be unidentified duplicates due to our complex search strategy. We excluded 7137 records based on title and abstract screening and assessed 97 full-text articles for eligibility. Of the remaining 97 full-text articles, we excluded 56 full-text articles by applying inclusion and exclusion criteria (Table 1). We excluded the majority of full-text articles because they did not meet the first inclusion criterion and did not report data on an SDM training for HCPs. We included 41 articles in this review. Figure 2 shows the process of article selection.

Figure 2

Preferred Reporting Items for Systematic Reviews and Meta-Analyses flow chart. *Due to multiple search strategies, duplicates were not removed.

Descriptive data of included studies and articles

Identified articles (n=41) report on studies (n=30) conducted in a limited number of countries (n=10). Most studies were conducted in the USA (n=9), Germany (n=8) and Canada (n=7). Eleven articles report on studies from Germany,22–32 nine articles report on studies from Canada33–41 and eight articles report on studies from the USA.42–49 Six articles depict one study from the UK50–53 and four articles present studies conducted either multinationally,54 in Australia,55 56 the Netherlands57 58 or Korea.59 The majority of included articles (n=27) report on cluster-randomised25–27 29–31 33–36 44 48 50 52 57 58 and randomised22 23 30 31 37 38 42 45 46 50 54 55 controlled trials. Further articles report on one controlled59 and seven non-controlled24 32 39 41 43 56 60 before-after studies, three qualitative40 51 53 and three post-test28 47 49 studies. Most articles (n=34) report on the evaluation of physician trainings,22–25 28–33 35 36 39–46 48–58 60 two articles report on trainings for nurses38 59 and five articles report on trainings for diverse HCPs.26 27 34 37 47 Overall, identified articles report on HCP samples ranging from 6 to 306, and n=25 articles22 23 25 27 29–36 40 42 44 48 50–52 54 55 57 58 report on the use of patient samples ranging from 38 to 1120. Table 1 illustrates descriptive data of included studies and articles.

Quality results of the ICROMS tool

Assessment of the quality of included articles with the ICROMS tool was applicable to 38 of the included articles (Table 2). Three articles were post-test studies,28 47 49 which could not be assessed with the ICROMS tool. Of the 22 articles that met the minimum score, 7 reported on randomised controlled trials,22 30 31 37 38 42 55 12 on cluster-randomised controlled trials25 29 33 35 36 44 48 50 52 57 58 and 3 reported on qualitative studies.40 51 53 Looking in detail at the 16 articles23 24 26 27 32 34 39 41 43 45 46 50 54 56 59 60 that did not meet the minimum score, most of them failed to meet criterion 3E (blinded assessment of primary outcome), 3F (reliable primary outcome measures) and 7D (free of other bias). For detailed results regarding ICROMS criteria, see online supplementary file S3.

Table 2

Quality results of the ICROMS tool

Study	Study design	ICROMS score	Minimum score* met	Mandatory criteria met	Recommendation for inclusion
Bernhard et al 54	RCT	20	No	No	No
Bieber et al 23†	RCT	20	No	No	No
Bieber et al 22†	RCT	24	Yes	Yes	Yes
Bieber et al 24	NCBA	19	No	No	No
Bieber et al 30†	RCT	23	Yes	Yes	Yes
Butow et al 55	RCT	22	Yes	No	No
Cohen et al 50†	CRCT	26	Yes	No	No
Davis et al 51†	QUAL	21	Yes	Yes	Yes
Dion et al 41	NCBA	18	No	No	No
Edwards et al 52†	RCT	18	No	No	No
Edwards et al 52†	CRCT	24	Yes	No	No
Edwards et al 53†	QUAL	20	Yes	No	No
Edwards et al 52†	CRCT	26	Yes	Yes	Yes
Feng et al 42	RCT	22	Yes	No	No
Geiger et al 31	RCT	23	Yes	Yes	Yes
Härter et al 25†	CRCT	23	Yes	Yes	Yes
Jo and An59	CBA	16	No	No	No
Kasper et al 32	NCBA	19	No	No	No
Körner et al 26†	CRCT	20	No	No	No
Körner et al 27†	CRCT	18	No	No	No
LeBlanc et al 33†	CRCT	26	Yes	No	No
Légaré et al 36†	CRCT	24	Yes	No	No
Légaré et al 35†	CRCT	23	Yes	No	No
Légaré et al 34†	CRCT	19	No	No	No
Loh et al 28	Post-test	n/a	n/a	n/a	n/a
McCallister et al 43	NCBA	19	No	No	No
Metcalfe et al 56	NCBA	8	No	No	No
Murray et al 37	RCT	22	Yes	Yes	Yes
Price-Haywood et al 44	CRCT	22	Yes	No	No
Sanders et al 57†	CRCT	22	Yes	No	No
Sanders et al 58	CRCT	23	Yes	Yes	Yes
Simmons et al 60	NCBA	11	No	No	No
Stacey et al 38	RCT	23	Yes	Yes	Yes
Stacey et al 39	NCBA	18	No	No	No
Sullivan et al 46	RCT	17	No	No	No
Sullivan et al 45	RCT	19	No	No	No
Tinsel et al 29	CRCT	25	Yes	Yes	Yes
Towle et al 40	QUAL	18	Yes	No	No
Volk et al 47	Post-test	n/a	n/a	n/a	n/a
Wilkes et al 48	CRCT	22	Yes	Yes	Yes
Yuen et al 49	Post-test	n/a	n/a	n/a	n/a
No. of articles		38	22	11	11

*ICROMS minimum score for study type: CRCT and RCT: 22, CBA: 18, NCBA: 22, QUAL: 16, for further details see original publication of the ICROMS tool.21

†Articles report data from one study.

CBA, controlled before-after study; CRCT, cluster-randomised controlled trial; ICROMS, integrated quality criteria for review of multiple study designs; NCBA, non-controlled before-after study; Post-test, post-test only study; n/a, not applicable; QUAL, qualitative study; RCT, randomised controlled trial.

Most of the included studies (n=30) and articles (n=41) report use of more than one type of data source to evaluate training effects. Of the studies employing HCP-reported data (n=27), eight studies24 28 45–47 49 56 59 relied solely on HCPs for training evaluation. The remaining 19 studies additionally employed other types of outcomes, for example, patient-reported,22 23 27 31–36 40 44 48 51 54 55 observer-rated,31 32 37–40 42 43 52 60 standardised patient-reported outcomes44 48 or training process and healthcare data.32 33 36–39 41 44 60 The three studies not relying on HCPs as data source25 29 30 57 58 combined patient-reported data with observer-rated measures25 57 or training process and healthcare data.29 57 Table 3 presents an overview of the data sources used for training evaluation in identified studies and articles.

Table 3

Analysis of data source use

Data source	Healthcare providers	Patients	Observers	Standardised patients	Training process and healthcare data
Bernhard et al 54	▲	▲
Bieber et al 23*	▲	▲
Bieber et al 22*	▲	▲
Bieber et al 24	▲
Bieber et al 30*		▲
Butow et al 55	▲	▲
Cohen et al 50*					▲
Davis et al 51*	▲	▲
Dion et al 41	▲				▲
Edwards et al 52*	▲
Edwards et al 52*		▲
Edwards et al 53*	▲
Edwards et al 52*	▲		▲
Feng et al 42	▲		▲
Geiger et al 31	▲	▲	▲
Härter et al 25*		▲	▲
Jo and An59	▲
Kasper et al 32	▲	▲	▲		▲
Körner et al.26*	▲
Koerner et al 27*	▲	▲
LeBlanc et al 33*	▲	▲			▲
Légaré et al 36*	▲	▲			▲
Légaré et al 35*	▲	▲
Légaré et al 34*	▲	▲
Loh et al 28	▲
McCallister et al 43	▲		▲
Metcalfe et al 56	▲
Murray et al 37	▲		▲		▲
Price-Haywood et al 44	▲	▲		▲	▲
Sanders et al 57*			▲		▲
Sanders et al 58*		▲
Simmons et al 60	▲		▲		▲
Stacey et al 38	▲		▲		▲
Stacey et al 39	▲		▲		▲
Sullivan et al 46	▲
Sullivan et al 45	▲
Tinsel et al 29		▲			▲
Towle et al 40	▲	▲	▲
Volk et al 47	▲
Wilkes et al 48	▲	▲		▲
Yuen et al 49	▲
No. of articles (n=41)	34	20	12	2	12
No. of studies (n=30)	27	15	12	2	11

*Articles report data from one study.

▲, the article reports the use of this type of data source for training evaluation.

All but one study40 employed quantitative evaluation strategies. Of the 29 remaining studies, articles of 17 studies22 23 26 27 29 32–39 44–46 48 52 55 57–59 reported use of a combination of unpublished or self-developed and published or psychometrically tested measures. Nine studies24 28 41–43 47 49 56 60 employed only unpublished or self-developed measures and three studies25 30 31 54 applied only published or psychometrically tested outcomes.

Looking at studies’ evaluation strategies with regard to coverage of Kirkpatrick’s four levels, most studies covered two (n=12) or three (n=10) levels. One study54 measured only on the results level and seven studies covered all four evaluation levels (table 4). All but one32 of the studies measuring on the reaction level also used outcomes on the learning level.24 26 28 33 37–39 41 45–48 53 55 56 60 Articles of five studies33 36 49 54 55 59 indicate measurement on the results level without covering the evaluation level of behaviour.

Table 4

Coverage of Kirkpatrick’s evaluation levels

Study	Reaction	Learning	Behaviour	Results
Bernhard et al 54				▲
Bieber et al 23*			▲	▲
Bieber et al 22*				▲
Bieber et al 24	▲	▲
Bieber et al 30*				▲
Butow et al 55	▲	▲		▲
Cohen et al 50*				▲
Davis et al 51*			▲	▲
Dion et al 41	▲	▲
Edwards et al 52*		▲	▲
Edwards et al 52 *			▲	▲
Edwards et al 53	▲	▲	▲	▲
Edwards et al 52*			▲	▲
Feng et al 42		▲	▲	▲
Geiger et al 31			▲	▲
Härter et al 25			▲	▲
Jo and An59		▲		▲
Kasper et al 32	▲		▲	▲
Körner et al 26*	▲	▲	▲
Körner et al 27*			▲
LeBlanc et al 33*	▲	▲		▲
Légaré et al 36 *		▲		▲
Légaré et al 35*		▲		▲
Légaré et al 34*		▲	▲	▲
Loh et al 28	▲	▲
McCallister et al 43		▲	▲
Metcalfe et al 56	▲	▲	▲
Murray et al 37	▲	▲	▲	▲
Price-Haywood et al 44(2014)		▲	▲	▲
Sanders et al 57*			▲	▲
Sanders et al 58*			▲	▲
Simmons et al 60	▲	▲	▲	▲
Stacey et al 38	▲	▲	▲	▲
Stacey et al 39	▲	▲	▲
Sullivan et al 46	▲	▲	▲	▲
Sullivan et al 45	▲	▲	▲	▲
Tinsel et al 29			▲	▲
Towle et al 40		▲	▲	▲
Volk et al 47	▲	▲
Wilkes et al 48	▲	▲	▲	▲
Yuen et al 49		▲		▲
No. of articles (n=41)	17	26	27	31
No. of studies (n=30)	17	23	21	22

*Articles report data from one study.

▲, the article reports the use of evaluation outcomes on this level for training evaluation.

Evaluation outcomes identified in the articles of this review were categorised in comprehensive subcategories of the proposed evaluation framework (table 5). The first level of HCPs' reactions to the training includes provider-reported appraisal of the training and objective acceptability and feasibility data. The second level of HCPs’ learning includes provider-reported learning like subjective knowledge gain, attitudes and intentions to engage in SDM, confidence in SDM communication skills and medical competencies. Objective learning measures are knowledge tests like multiple-choice or open questions on contents covered in the training. The third level of HCPs’ behaviour includes provider-reported or (standardised) patient-reported and observer-rated SDM performance and assessment of the patient-provider interaction. The fourth-level HCP training in SDM results reflects the Quadruple Aim framework including the work life of HCPs, patient population health, patient experience of care and healthcare system costs. Table 5 presents the subcategories of evaluation outcomes and how frequently they were used in included articles. Detailed information on the outcomes used in respective articles can be found in the online supplementary file S4.

Table 5

Categories of evaluation outcomes integrated in the evaluation framework

	No. of articles
Healthcare providers’ reactions	17
Provider-reported training appraisal	16
Overall training appraisal and satisfaction	11
Appraisal of training content	5
Appraisal of training materials	3
Appraisal of training didactics	2
Appraisal of training organisation and delivery	4
Appraisal of training impact	6
Ideas for training improvement	1
Objective training feasibility and acceptability data	4
Healthcare providers’ learning	27
Provider-reported learning	23
Subjective knowledge gain	3
Attitude to SDM	8
Attitude to care	3
Intention to engage in SDM	7
Confidence in SDM and communication skills	10
Confidence in medical competence	1
Objective learning measures	7
Healthcare providers’ behaviour	26
Provider-reported SDM and provider-patient interaction	14
Patient-reported SDM and provider-patient interaction	11
Standardised patient-reported SDM and provider-patient interaction	2
Observer-rated SDM and provider-patient interaction	12
Healthcare provider training in SDM results	31
Work life of healthcare providers	12
Provider-reported stress and burnout	2
Provider reaction to the decision	6
Provider satisfaction with care	4
Provider-reported provider-patient relationship	2
Patient population health	11
Patient-reported health literacy	2
Patient-reported intention to treatment adherence	3
Patient-reported adherence	2
Patient-reported health	10
Medical records	2
Patient experience of care	18
Patient-reported reaction to the decision	11
Patient-reported satisfaction with care	4
Patient-reported attitude to SDM and care	8
Patient-reported provider-patient relationship	3
Provider-reported patient reaction to care	4
Healthcare system costs	13
Provider-reported medical practice	4
Patient-reported decisional outcome	3
Standardised patient-reported physician’s final recommendation	1
Observer-recorded provider recommendation or decision	1
Healthcare resource use	2
Training costs	1
Medical record review of decision-making	1
Duration of provider-patient interaction	4

Detailed information on evaluation outcomes is attainable in online supplementary file S4.

Discussion

Our review aimed to investigate how the diversity of evaluation strategies and the quality of published evaluations contributes to the current lack of evidence on HCP trainings in SDM. Thus, we analysed the quality of published articles on HCP trainings in SDM, and analysed their evaluation strategies regarding 1) use of data sources, 2) use of unpublished or self-developed and published or psychometrically tested measures and 3) coverage of Kirkpatrick’s four levels. We found 41 articles reporting on 30 studies that met our inclusion criteria. Most of these studies were cluster-randomised and randomised controlled trials that evaluated SDM trainings for physicians and were conducted in high-income countries like Canada, the USA, the UK or Germany. Sample sizes varied largely. Of the 38 articles eligible for assessment with the ICROMS tool, only 11 articles met ICROMS quality criteria. Diverse strategies were used to evaluate HCP trainings in SDM, but most studies relied on provider-reported outcomes, covered two or three of Kirkpatrick’s levels and combined published and unpublished measures. The proposed evaluation framework based on Kirkpatrick’s four-level evaluation model16 and the Quadruple Aim framework19 appears useful for the design or analysis of strategies to evaluate HCP trainings in SDM.

The poor quality of identified publications indicates that researchers should aim to design more methodologically sound studies to evaluate HCP trainings in SDM. The ICROMS tool is a decision matrix to evaluate the robustness of studies for inclusion in a review21 and present results could inspire researchers to be more rigorous in their study. Since measurement bias was a common problem of many included studies, it would be good to use more objective training acceptability and feasibility data, more objective learning and observer-rated measures and healthcare data for evaluation. However, assessment of specific learning objectives may require application of self-developed measures. Combined with psychometrically sound primary outcomes, this may be the ideal evaluation approach.

Although HCPs were the main data source in included studies, reaction to the training was the least studied evaluation level. Training participants’ favourable reactions are substantial for the training to be effective as participants’ positive appraisal determines their motivations to learn from the training.16 Following the reaction level, researchers should assess HCP learning using objective learning measures for knowledge gain. Provider-reported learning measures are useful to establish training effects on HCP attitudes, intentions and confidence regarding SDM-related behaviour, which are the predecessors of actual behaviour.17 According to the theory of planned behaviour, a positive attitude, acquirement of relevant knowledge and improvement of skills determine HCP behavioural intentions, and thus behaviour change.17

Measurement of behaviour change is central, as change of SDM-related behaviours is usually the main aim of HCP trainings in SDM. Since there is no gold standard for measuring SDM61 and measurement from different viewpoints is inconsistent,62 multiperspective assessment from the viewpoints of HCPs (standardised), patients and observers appears the best approach.5 Ideally, validated measures should be used to ensure quality and comparability of results, but a lack of psychometrically tested SDM measures61 poses a problem. It is also difficult to assess behaviour change in clinical practice, because it is unclear when changes manifest themselves.16 However, it is critical to establish behaviour change, before measuring training effects on the results level4 16 to avoid the risk of interpreting random effects independent from the training.

To establish training effects on the results level relevant to multiple stakeholders,9 we recommend reference to the Quadruple Aim framework.19 Beneficial training effects on the work life of HCPs may increase their motivation to implement SDM in practice. Currently, HCPs often experience SDM as another burden and demand on their time, and are therefore often reluctant to implement SDM in routine practice.63 Although effects of SDM on affective-cognitive aspects of patient experience of care are well established, evidence regarding patient population health is sparse.4 If studies showed beneficial SDM training effects on healthcare system costs, policy makers could be encouraged to initiate system changes to foster the implementation of SDM.9

In sum, the poor study quality and the multitude of evaluation strategies used in identified studies limit conclusive evidence on HCP trainings in SDM. The heterogeneous use of SDM and other outcome measures compromises the interpretation and integration of research results.4 10 11 64 65 To achieve solid empirical evidence, we need consensus on a core set of evaluation outcomes and validated measures on all levels of the proposed framework for HCP trainings in SDM. In the design of evaluation studies, researchers should aim to cover all four levels of the framework and include outcomes on the results level that relate to the Quadruple Aim framework. Researchers should aim to use outcomes that are valued by multiple stakeholders like patients, HCPs as well as healthcare managers, executives and policy makers. They should also aim to use validated observer-rated measures and objective data to limit bias, whenever feasible. If researchers applied these recommendations, evaluation studies could have more impact and better support the implementation of SDM in routine practice.9 65

This review has some limitations. First, our primary search included only three databases and inclusion criteria were limited to studies aiming to evaluate HCP trainings in SDM. Consequently, we may have missed some studies, but we assume that our broad secondary search strategy made up for this limitation. Second, we did not analyse evaluation strategies regarding a match of training contents and evaluation outcomes. Additionally, we did not analyse which evaluation outcomes previously showed SDM training effects, which could be valuable information for the design of an evaluation study. However, previous studies investigated the relation between SDM and patient outcomes4 5 65 and interested researchers may obtain valuable information there. Third, our quality assessment with the ICROMS tool can be seen as a limitation as well as a strength of this review. On the one hand, the ICROMS tool is not applicable to post-test studies and considers patient-reported and provider-reported outcomes as unreliable, which introduces a negative bias to our quality results. On the other hand, we provided an overview of the quality of studies in the field, demonstrating a lack of robust evaluation studies. This review has further strengths. First, this review comprises multiple study designs from post-test studies to qualitative and cluster-randomised controlled studies, which reflect the diversity of studies in the field. Second, this review provides an analysis of current strategies to evaluate HCP trainings in SDM and how their diversity functions as a barrier to conclusive evidence. Third, this review proposes an evaluation framework for HCP trainings in SDM that is based on the well-established Kirkpatrick’s evaluation model and the Quadruple Aim framework. The framework may provide guidance in the design of coherence evaluation strategies for HCP trainings in SDM. Fourth, the proposed framework may initiate discussion and hopefully agreement on a core set of validated outcome measures useful for the purpose and meaningful to stakeholders.