Impact analysis studies of clinical prediction rules relevant to primary care: a systematic review.

OBJECTIVES: Following appropriate validation, clinical prediction rules (CPRs) should undergo impact analysis to evaluate their effect on patient care. The aim of this systematic review is to narratively review and critically appraise CPR impact analysis studies relevant to primary care.
SETTING: Primary care. PARTICIPANTS: Adults and children. INTERVENTION: Studies that implemented the CPR compared to usual care were included. STUDY
DESIGN: Randomised controlled trial (RCT), controlled before-after, and interrupted time series. PRIMARY OUTCOME: Physician behaviour and/or patient outcomes.
RESULTS: A total of 18 studies, incorporating 14 unique CPRs, were included. The main study design was RCT (n=13). Overall, 10 studies reported an improvement in primary outcome with CPR implementation. Of 6 musculoskeletal studies, 5 were effective in altering targeted physician behaviour in ordering imaging for patients presenting with ankle, knee and neck musculoskeletal injuries. Of 6 cardiovascular studies, 4 implemented cardiovascular risk scores, and 3 reported no impact on physician behaviour outcomes, such as prescribing and referral, or patient outcomes, such as reduction in serum lipid levels. 2 studies examined CPRs in decision-making for patients presenting with chest pain and reduced inappropriate admissions. Of 5 respiratory studies, 2 were effective in reducing antibiotic prescribing for sore throat following CPR implementation. Overall, study methodological quality was often unclear due to incomplete reporting.
CONCLUSIONS: Despite increasing interest in developing and validating CPRs relevant to primary care, relatively few have gone through impact analysis. To date, research has focused on a small number of CPRs across few clinical domains only. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://www.bmj.com/company/products-services/rights-and-licensing/

Clinical prediction rules (CPRs) are increasingly developed and advocated for use in clinical practice. However, little is known regarding the effectiveness of these tools versus usual care for relevant clinical outcomes.

This is the first systematic review of CPRs relevant to primary care that have gone through impact analysis.

The main limitation of this review is that the electronic search was limited to 30 prespecified journals, which may mean that some relevant studies were not retrieved. This search was supplemented with key author searches and reviewing other resources known to publish CPRs.

Nevertheless, this is the first study to examine, in detail, impact analysis studies of CPRs relevant to primary care.

Introduction

Clinical prediction rules (CPRs) are clinical tools that quantify the individual contributions that various components of the history, physical examination and investigations contribute towards diagnosis, prognosis or likely response to treatment in a patient.1 These tools simplify, standardise and attempt to increase the accuracy and consistency of clinicians’ diagnostic and prognostic assessments and management decisions.1 2 Well-recognised examples of CPRs include the Framingham cardiovascular risk score, the Ottawa ankle rule and the Centor score for sore throat.

Developing and validating a CPR requires reference to specific methodological standards.1 3 4 Conventionally, these tools go through three distinct stages prior to full implementation in a clinical setting.1 3 4 The first stage is derivation, where the independent and combined effects of explanatory variables such as symptoms, signs and/or investigations, are established. The next stage is validation, where the final derived CPR is evaluated first in a similar clinical setting (internal validation), followed by different clinical settings (external validation). If in following these stages predictive accuracy is established, then the final stage of evaluation is to test the impact of using the CPR in clinical practice, ideally in a randomised controlled trial (RCT) for relevant clinical outcomes.1 5 6 Impact analysis aims to investigate if the implementation of a CPR in clinical practice is better than usual care for the patient, process of care and/or cost outcomes.7 8

Our research group recently published two studies detailing the development and content of an international register of CPRs relevant to primary care.2 9 With increasing interest in CPRs, large numbers have been derived but fewer have been validated or tested in an impact analysis study.2 If CPRs are to truly improve the quality of patient care then evaluation of these tools is crucial.

The aim of this systematic review is to present a narrative and critical analysis of CPRs relevant to primary care which have gone through impact analysis.

Methods

The methods used for identifying CPRs from the literature and in developing a register of these tools relevant to primary care have been published in detail previously.2 9 These methods are summarised below.

Search strategy

An electronic search string for PubMed was developed to retrieve CPRs relevant to primary care from 30 preselected medical journals (see online supplementary appendix 1 for search string and journals included).9 No restriction was placed on language. Original electronic searches were conducted from 1980 to 2009, which for the purposes of this review were updated to the end of 2013.2 In addition, secondary sources of CPRs were searched including the Journal of the American Medical Association (JAMA) Rational Clinical Examination series, a handbook of CPRs and personal resources.2 9 10 Author searches for key experts in the field were also conducted for additional relevant articles. Furthermore, reference lists of each relevant impact analysis study were searched to identify possible additional studies.

Inclusion criteria

Studies were eligible for inclusion if they met the following criteria.

(1) Population: Relevant to primary care defined as ‘normally the point of first medical contact within the healthcare system, providing open and unlimited access to its users, dealing with all health problems regardless of the age, sex or any characteristic of the person concerned’.11 Although studies may not have been conducted in a primary care setting, they were eligible for inclusion providing they were relevant to primary care (eg, implementation of the Canadian head CT rule with the aim of reducing imaging for patient presenting with minor head injury). This inclusion criterion was designed to be broad to acknowledge variation in the same-day diagnostic tests that are available across different countries and the international variation in the role of primary care clinicians. Studies set in the emergency department were considered relevant to primary care if following application of the CPR the patient could be discharged home following application of the CPR.

(2) Intervention: CPR defined as ‘a clinical tool that quantifies the individual contributions that various components of the history, physical examination and investigations make towards the diagnosis, prognosis or likely response to treatment in a patient’.1 Diagnostic, prognostic and management CPRs were included and screening questionnaires (ie, applied to apparently healthy people who may be at increased risk of a disease or condition) were excluded. A requirement for inclusion was that the CPR comprised the entire intervention. Studies where the CPR was implemented as part of a broader guideline, protocol or decision aid were excluded. Studies that used a CPR to determine eligibility for trial inclusion but were not part of the intervention were also excluded.

The following study designs were included: (cluster) RCT, controlled before–after or interrupted time series studies. Uncontrolled study designs were excluded as the aim of this review was to examine the effectiveness of CPR implementation, rather than the performance of the CPR which would be captured in validation studies using observational study designs.

(3) Comparison: Usual care.

(4) Primary Outcome: Physician behaviour, for example, ordering of diagnostic tests; process of care, for example, number of inpatient bed days; and/or patient outcomes, for example, duration of symptoms.1

Data extraction

All articles were initially screened for inclusion according to title and abstract by one reviewer. Potentially relevant articles were then reviewed by a second reviewer with any disagreements resolved by a third independent reviewer. For each relevant article, the following data was extracted: (1) name of CPR; (2) type of CPR: prediction rule, decision rule or both; (3) clinical domain: using International Classification of Primary Care—second edition (ICPC-2)12; (4) clinical setting; (5) study population; (6) primary outcome of interest; (7) predictive accuracy of the CPR (measured by sensitivity/specificity (95% CIs) where reported, otherwise, the model's c-statistic was recorded) and (8) impact on primary outcome of interest.

Data analysis

Critical analysis of CPR impact analysis

Each article was critically appraised, using a published framework for impact analysis of CPRs.7 Developed in 2011 by an expert panel, this four-phase framework provides guidance for impact analysis studies (see figure 1). The phases are as follows: (1) Exploratory phase; evaluate the level of evidence and predictive accuracy of the CPR; (2) Preparation for impact analysis; consider potential barriers, assess acceptability of the CPR to clinicians and local stakeholders and conduct a pilot study; (3) Experimental phase; evaluation of the CPR with monitoring of the use of the CPR in a clinical setting; (4) Long-term implementation phase; examine if a CPR with reported positive impact on relevant clinical outcomes is implemented long-term and how this was achieved.7

Figure 1

Framework for the impact analysis and implementation of clinical prediction rules (CPRs).7

Summary of effect on process and outcome of care

Meta-analysis was not possible due to heterogeneity in CPRs and outcomes of interest, so a narrative analysis was conducted. In this section, wherever appropriate and where data was available, crude ORs and absolute risk reductions (ARR) were calculated.

Methodological quality assessment

The methodological quality of each impact analysis study was independently evaluated by two reviewers (MU and BC), and by a third reviewer if consensus was not reached (EW). For each study design, an appropriate quality assessment checklist was used. RCTs and cluster RCTs were assessed through the Cochrane risk of bias tool.13 Controlled before–after studies, and interrupted time series studies, were evaluated through Cochrane criteria for these study designs.14

Results

Overview of studies

Study identification

A flow diagram of the search strategy is presented in figure 2. The PubMed search (1980–2013) and supplementary sources searches retrieved a total of 86 158 studies, of which 1111 CPR studies were identified following review of title and abstract. A total of 18 studies met the inclusion criteria for the systematic review.

Figure 2

Flow diagram of search strategy.

Description of included impact analysis studies

A total of 14 unique CPRs were tested in 18 impact analysis studies (table 1 and online supplementary appendix 2). According to ICPC-2, these studies were classified into four broad clinical domains, namely, musculoskeletal, most commonly the Ottawa ankle rule15–20 (n=6); cardiovascular21–27 (n=6); respiratory28–32 (n=5); and neurological33 (n=1). The majority of studies were conducted in North America (Canada n=10, the USA n=4), with the remainder in the UK (n=2) and France (n=2). Most studies were set in the emergency department (ER) (n=9) and primary care (n=7). The remainder were carried out in the outpatient department (n=2).

Table 1

Summary of impact analysis studies of CPRs relevant to primary care

Author, year, country	CPR name, CPR predictive accuracy (95% CI), study design	Population and study setting	Intervention and comparison	Primary outcome(s)	Results: primary outcome (95% CI)
Auleley, 1997, France15	Ottawa ankle rulesensitivity 100% (95% to 100%), specificity 50% (46% to 55%)cluster RCT	4980, ≥18 years, emergency departments of 5 Paris university teaching hospitals	Intervention: educational intervention to encourage CPR use (ie, posters, pocket cards and data forms).Postintervention: only posters used to sustain the intervention effect.Comparison: Usual care	Physician behaviour: Referral for radiography (ankle/foot)	Relative reduction intervention site: 22.4% (95% CI 19.8% to 24.9%), control group increase of 0.5% (95% CI 0% to 1.4%).
Cameron, 1999, Canada19	Ottawa anklerulesensitivity 100% (95% to 100), specificity 50% (46% to 55%)controlled before–after	1648, ≥18 years, male 885, female 763mean age 38 (18–91), emergency departments in 10 hospitals	Group A: little or no prior use of the CPR and educational intervention (educational meeting, posters, pocket cards and patient information leaflets).Group B: some prior use of the CPR and educational intervention.Group C: active local implementation of the CPR and no educational intervention.	Physician behaviour: Referral for ankle X-ray	No reduction referral for ankle X-rays: intervention before 73%, after 78%, p=0.11, control: before 75%, after 65%, p=0.022
Stiell, 1994, Canada, ER16	Ottawa ankle rulecontrolled before–aftersensitivity 100% (95% to 100), specificity 50% (46% to 55%)	2342, ≥18 years, emergency departments of 2 hospitals	Intervention: educational intervention to encourage CPR use (ie, lecture, pocket cards, and posters).Postintervention: posters remained in ER.Comparison: Usual care	Physician behaviour: referral for radiography (ankle/foot)	Ankle X-ray: relative reduction 28% in intervention group, increase of 2% in control group (p<0.001).Foot X-ray: relative reduction of 14% intervention group, increase of 13% in control group (p<0.05).
Boutis, 2013, Canada, ER20	Low-Risk Ankle Rulesensitivity 100% (93.3% to 100)specificity NRITS	2151, children aged 3–16, emergency departments of six hospitals	Phase 1: no interventionPhase 2: educational interventions to encourage CPR use (ie, physician education, pocket cards, posters) and CDSSPhase 3: CDSS onlyComparison: usual care	Physician behaviour: referral for ankle X-ray	Relative reduction in ankle X-rays in intervention sites compared to control sites.RR: 21.9% (95% CI 15.2% to 28.6%)
Stiell, 1997, Canada, ER17	Ottawa Knee Rule;sensitivity 100% (94% to 100), specificity49% (46% to 52%)Controlled before-after	3907, ≥18 years, emergency departments of 4 hospitals (2 community and 2 teaching)	Intervention: educational interventions to encourage CPR use (ie, lecture, pocket cards and posters).Comparison: Usual care	Physician behaviour: referral for knee radiography	Relative reduction of 26.4% of patients referred for knee X-ray in intervention group (77.6% vs 57.1% (p<0.001), vs relative reduction of 1.3% in control group (76.9% vs 75.9%, p=0.6)
Stiell, 2009, Canada, ER18	Canadian C-spineRulesensitivity 99% (96% to 100%), specificity 45% (44% to 46%)Cluster RCT	11 824, ≥16 years, emergency departments of 6 hospitals	Intervention: educational interventions to encourage CPR use (ie, lecture, pocket cards and posters) and CDSS at point of requesting imagingComparison: usual care	Physician behaviour: diagnostic imaging rate of cervical spine	Relative reduction of 12.8% for cervical spine imaging (95% CI 9% to 16%) intervention group. Control group showed a relative increase of 12.5% (95% CI 7% to 18%)
McIsaac, 2002, Canada, Primary care29	McIsaacSensitivity83% (no CIs), specificity 94% (no CIs)RCT	621, ≥3 years, general practice, 97 participating GPs	Intervention: mailed educational intervention (published score with summary explanation with pocket card). Physicians were provided with a sticker to apply to the encounter form that listed the score and management approach.Comparison: physicians only received the education material.	Physician behaviour: unnecessary antibiotic prescriptions (negative throat swab)	Non-significant difference intervention vs control groups in unnecessary antibiotic prescription (20.4% vs 16.1%, p=0.29)
McIsaac, 1998, Canada, Primary care28	Centor scoresensitivity 90% (no CIs), specificity 92% (no CIs)RCT	396, ≥15 years, general practice, 450 participating GPs	Intervention: mailed CPR with summary explanation and patient information.Physicians asked to complete an encounter form.Comparison: mailed educational intervention and a control form with no score or management actions.	Physician behaviour: antibiotic prescription	Non-significant reduction in antibiotic prescription in intervention group (27.8%) vs control (35.7%) (p=0.09)
McGinn, 2013, USA32	Walsh rule for streptococcal pharyngitis Heckerling rule for pneumonia Walsh rule: c-statistic: 0.71 (95% CI 0.67 to 0.74) Heckerling rule: c-statistic 0.82 (0.74 to 0.9) RCT	168 Primary care providers, 2 large academic ambulatory care centres in New York	Intervention: education session and computerised CDSS with CPRs embedded promoting physician to calculate scores of both CPRs and receive management recommendations.Comparison: usual care with background information on CPRs	Physician behaviour: change in antibiotic prescription	Intervention group significantly less likely to order antibiotics than control (age-adjusted RR, 0.74; 95% CI 0.60 to 0.92).Absolute risk difference 9.2%.
Worrall, 2007, Canada30	Modified Centor scoresensitivity 90% (no CIs), specificity 92% (no CIs)RCT	533, ≥19 years, 37 practices in eastern NewfoundlandCPR:170RADT: 120RADT+CPR:102Control:141	CPR group: decision rules onlyRADT group: rapid antigen test onlyRADT+CPR group: decision rules and antigen test combinedComparison: usual care	Physician behaviour: prescribing rate of antibiotics	Prescription rates: CPR alone—55% RADT—27% (NS)RADT+CPR—38% (p<0.001)Control: 58%
Little, 2013, UK31	Fever PAINc-statistic: 0.71RCT	631, ≥3 years, general practice (48 UK practices)	CPR group: CPR was applied and antibiotic prescribed according to the score.CPR+RADT group: CPR was applied and antibiotic prescribed or RADT carried out according to the score.Comparison: Delayed prescribing	Patient behaviour: patient-reported symptom severity days 2–4 after consultation on a 7-point Likert scale	Greater improvements in symptom severity for CPR group compared to control (−0.33, 95% CI −0.64 to −0.02)
Pozen, 1984, USA, ER21	Pozen score for chest painsensitivity 94% (no CIs), specificity 78% (no CIs)ITS	2320, aged ≥30 male and ≥40 female, emergency departments of 6 US hospitals	Intervention: research assistant presented physicians with the CPR probability score.Comparison: usual care, the CPR probability was calculated but not presented to the physicians.	Physician behaviour: appropriate admission/discharge	30% relative reduction in patients admitted to CCU who did not have acute coronary syndrome
Kline, 2009, USA, ER22	Kline chest pain CPRc-statistic 0.74 (0.65 to 0.82)RCT	369 adults presenting with chest pain, one emergency room in an academic urban US hospital	Intervention: clinicians and patients received a printout of CPR result displayed numerically and graphically.Comparison: usual care, no printout was provided to clinicians or patients.	Physician behaviour: hospital admission with no significant cardiovascular diagnosis	No significant decrease for patients admitted with no CVD diagnosis: 11% vs 5% (95% CI −0.2% to 11%), p=0.059
Persell, 2012, primary care27	Framingham risk estimate and global cardiovascular risk scoreCluster RCT	N=14 physicians, n=218 adult patients randomised to intervention, n=15 physicians, n=217 adults patients randomised to control, US primary care	Intervention: individualised CVD risk estimate posted to high-risk patients and their physicians alerted by secure emailControl: usual care	Patient: reduction in LDL-cholesterol level by 30 mg/dL	No difference in the primary outcome (11% vs 11.1% OR 0.99, 95% CI 0.56 to 1.74, p=0.96) but intervention patients were more likely to receive a prescription for a statin (11.9% vs 6%, OR 2.13, 95% CI 1.05 to 4.32, p=0.038)
Grover 2007 and 2008, primary care25 26	Framingham risk scoreRCT	N=3053 adults mean age 56.4, male 66.9%, n=230 primary care physicians, 10 provinces in Canada primary care	Intervention: patients identified as high risk and randomised to intervention had their individualised coronary risk profile discussedControl: usual care, coronary risk profile withheld	Patient outcomes: Reduction in LDL-cholesterol level Reduction in BP	Statistically significant reduction in LDL and total cholesterol-HDL ratio in intervention vs control and patients were more likely to reach lipid targets Patients in intervention group were more likely to receive appropriate antihypertensive treatment and more likely to start or modify treatment
Hall, 2003, UK23	New Zealand cardiovascular risk scoreNRPilot RCT	323, aged 35–75 years, patients with no history of cardiovascular or renal disease, one UK hospital outpatient department (OPD) clinic	Intervention: risk scores were clearly documented at the front of the notes of patients.Comparison: usual care	Physician behaviour: Prescription of risk-modifying drugs Management of CVD risk factors	No significant between-group differences: change in diabetes treatment 42% (95% CI 34% to 50%) vs 58 (95% CI 29% to 45%), change in antihypertensive drugs 26 (95% CI 10% to 22%) vs 10% (95% CI 5% to 16%), change in lipid lowering drugs: 12% (7% to 17%) vs 9% (95% CI 4% to 14%) Referral to dietician 10% (95% CI 6% to 15%) vs 13% (95% CI 7% to 19%)
Hanon, 2000, France24	Framingham risk scoreNRRCT	1243, aged 18–75 years with hypertension attending a general physician	Intervention: physicians knowledge of the calculated risk score.Comparison: usual care	Patient and Physician behaviour: change in BP, patients prescribed dual therapy	No difference in BP (patients with BP <140/90 mm Hg intervention: 64%, control 62%) or % patients on dual therapy (41% intervention vs 46% control)
Stiell, 2010, Canada, ER33	CT head rulesensitivity 100% (96% to 100%), specificity 51% (48% to 53%)Cluster RCT	4531, alert and stable adults with minor head injury aged ≥16 years, 12 emergency departments in three provinces of Canada (6 teaching sites, 6 community sites)	Intervention: educational interventions to encourage CPR use (ie, lecture, pocket cards and posters) and real-time reminder at point of requesting imagingComparison: usual care	Physician behaviour: proportion of patients referred for CT imaging	Increased proportion of patients referred for CT imaging intervention: before: 62.8%, after: 76.2% (difference: 13.3% (95% CI 9.7% to 17.0%)Control: before: 67.5%, after: 74.1% (difference: 6.7% (95% CI 2.6% to 10.8%)

CPR, clinical prediction rule; NA, non-applicable; NR, not reported; NS, non-significant.

Regarding study design, there were four cluster RCTs,15 18 27 33 eight RCTs,22 24–26 28–32 one pilot RCT,23 three controlled before–after studies,16 17 19 and two interrupted time series.20 21 In a total of 16 studies, the intervention was the impact of the CPR alone,15–29 32 33 and two studies used different trial arms to test the CPR alone versus CPR and protocol versus usual care.30 31 Two studies integrated the CPR into a computerised clinical decision support system (CDSS).20 32 Two studies used real-time CPR reminders at the point of test ordering.18 33

Critical analysis of CPR impact analysis studies

Preparation for impact analysis: level of evidence of CPR, consideration of potential barriers and assessment of CPR acceptability

Fifteen of 18 studies implemented a CPR that was externally validated, while three studies tested a CPR that had been derived or internally validated only.21 30 31 Ten studies reported the CPR's sensitivity from validation studies in identifying the target outcome which ranged from 85% to 100%.15–20 28 29 31 33 Five studies identified and addressed potential barriers for implementation before impact analysis, most frequently through barriers analysis.18 20 22 32 33 Six studies assessed the acceptability of the CPR to clinicians prior to the implementation phase of the study, usually through training sessions and engagement with local stakeholders.15 19 22 27 32 33 Seven studies reported that a pilot or simulation phase was conducted or there was a previous impact analysis on the same CPR by the same authors.16–19 21 32 33

Impact analysis phase: adherence with CPR use and reasons for non-adherence

Twelve studies tracked the use of the CPR during implementation, usually with standardised data collection forms or computerised tools.15–18 20 22 25 26 29 31–34 Overall, adherence with CPR use varied between studies ranging from 57.5% to 100%, with reported reasons for non-adherence including fear of missing the diagnosis, preference for own clinical judgement and patient request.20 33 Of the 12 studies that tracked CPR adherence, seven reported adherence of ≥80%, four reported adherence of 60–80%, and one study reported adherence of ≤60%. Nine of 12 studies reported a positive impact on primary study outcome following CPR intervention, but there was no clear link between level of adherence and successful CPR implementation. Clinicians’ acceptability of CPR use during the intervention phase was evaluated in four studies,16 17 20 33 of which two assessed the reported rate of comfort using a five-point Likert scale.16 17

Postimplementation phase: maintaining use of CPR

Of 10 studies with a positive impact on primary outcome, 4 evaluated the effect of the CPR in a post-intervention phase ranging from 5 to 12 months.15 16 18 20 To maintain CPR use, two studies used a passive strategy of posters, one retained computerised clinical decision support, and one did not employ any particular strategy. In all four studies, the effect of CPR use was maintained.

Importance of study design in assessing the impact of CPRs

There were five uncontrolled before–after studies retrieved during the initial search which were excluded from data analysis based on their uncontrolled study design (see online supplementary appendix 3).35–39 These studies tested the impact of the Ottawa ankle rule (n=2), the Canadian C-spine rule (n=1), the CT head rule (n=1), and the Glasgow Blatchford bleeding score (n=1). All five studies demonstrated a positive impact on primary outcome, usually physician behaviour in ordering imaging.

Effect on the process and outcome of care

Overall, 10 studies reported that CPR implementation resulted in a positive impact on primary outcome, while eight studies reported no impact versus usual care. There were no clinically important adverse outcomes reported. Studies are presented according to clinical domain. Table 2 presents a summary of the estimated effect sizes for the impact analysis studies.

Table 2

Table of estimated effect sizes for impact analysis studies

Author, year CPR name	Study design (n)	Sample size calculation reported (n)	Primary outcome	Effect size: crude OR of improvement in primary outcome in intervention vs control (95% CI)	Absolute risk reductions (95% CI)
Auleley, 199715 Ottawa ankle rule	Cluster RCT (4980)	Yes (900)	Physician behaviour: referral for radiography (ankle/foot)	Crude OR 0.03 (0.01 to 0.06)	22.8% (20.0% to 25.7%)
Stiell, 199416 Ottawa ankle rule	Controlled before–after (2342)	NA	Physician behaviour: referral for radiography (ankle/foot)	Ankle X-ray: crude OR 0.11 (0.08 to 0.16)Foot X-ray: crude OR 0.73 (0.57 to 0.94)	33.4% (28.9% to 37.9%)6.6% (1.1% to 11.7%)
Cameron, 199919 Ottawa ankle rule	Controlled before–after (1648)	NA	Physician behaviour: referral for ankle X-ray	Crude OR 0.96 (0.60 to 1.55)	0.8% (−8.5% to 9.8%)
Boutis, 201320 Low-risk ankle rule	ITS (2151)	NA	Physician behaviour: referral for ankle X-ray	NA	NA
Stiell, 199717 Ottawa knee rule	Controlled before–after (3907)	NA	Physician behaviour: referral for knee radiography	Crude OR 0.42 (0.35 to 0.51)	18.8% (14.7% to 22.9%)
Stiell, 200918 Canadian C-spine rule	Cluster RCT (11 824)	Yes (9600)	Physician behaviour: diagnostic imaging rate of cervical spine	Crude OR 0.82 (0.74 to 0.90)	5% (2.5% to 7.5%)
Pozen, 198421 Pozen score for chest pain	ITS (2320)	NA	Physician behaviour: appropriate admission	NA	NA
Kline, 200922 Kline chest pain CPR	RCT (369)	Yes (400)	Physician behaviour: admission with no significant cardiovascular diagnosis	Crude OR 0.47 (0.22 to 1.04)	5.4% (−0.2% to 10.9%)
Persell, 201227 Framingham risk score	Cluster RCT (425)	Yes (406)	Patient:proportion of patients with a reduction in LDL-cholesterol level by 30 mg/dL	Crude OR 0.99 (0.55 to 1.81)	0.1% (−0.0% to 0.0%)
Grover, 2007 and 200825 26 Framingham risk score	RCT (3053)	Yes (3000)	Patient: reduction in LDL-cholesterol level	Reduction in LDL-cholesterol level: mean difference −0.33 mg/dL (−0.5.4 to −1.1; p=0.02)	NA
Hall, 200323 New Zealand cardiovascular risk score	Pilot RCT (323)	NA	Physician behaviour: prescription of risk modifying drugs, management of CVD risk factors	Diabetes treatment: crude OR 1.28 (0.82 to 2.01)Antihypertensive drugs: crude OR 1.62 (0.84 to 3.12)Lipid lowering drugs: crude OR 1.48 (0.72 to 3.04)Referral to dietician: crude OR 0.78 (0.40 to 1.54)	−6.0% (−16.6% to 4.7%)−5.5% (−12.9% to 1.9%)−3.7% (−10.3% to 3.0%)2.5% (−4.47% to 9.57%)
Hanon, 200024 Framingham risk score	RCT (1243)	No	Patient and physician behaviour: BP, patients prescribed dual therapy	Normal BP: crude OR 1.09 (0.87 to 1.38)Dual therapy: crude OR 0.82 (0.65 to 1.02)	−2.1% (−7.4% to 3.3%)4.9% (−0.6% to 10.4%)
McIsaac, 200229 McIsaac	RCT (621 patients, 97 physicians)	Yes (850 patients, 85 physicians)	Physician behaviour: unnecessary antibiotic prescriptions (negative throat swab)	Crude OR 0.71 (0.47 to 1.08)	4.9% (−1.1% to 10.9%)
McIsaac, 199828 Centor	RCT (396)	Yes (800)	Physician behaviour: antibiotic prescription	Crude OR 0.69 (0.45 to 1.05)	8.1% (−1.0% to 17.3%)
McGinn, 201332 (1) Walsh rule (streptococcal pharyngitis)(2) Heckerling rule (pneumonia)	RCT (168)	No	Physician behaviour: change in antibiotic prescription	Crude OR 0.66 (0.50 to 0.86)	9.3% (3.2% to 15.3%)
Worrall, 200730 Modified Centor score	RCT (533)	Yes (196)	Physician behaviour: prescribing rate of antibiotics	Crude OR 0.89 (0.57 to 1.40)	2.9% (−8.2% to 13.9%)
Little, 201331 FeverPAIN	RCT (6131)	Yes (909)	Patient behaviour: patient-reported symptom severity	Adjusted mean difference−0.33 (−0.64 to −0.02; p=0.04)	NA
Stiell, 201033 CT head rule	Cluster RCT (4531)	Yes (4800)	Physician behaviour: proportion of patients referred for CT imaging	Crude OR 0.81 (0.69 to 0.96)	4.7% (1.0% to 8.4%)

CPR, clinical prediction rule.

Musculoskeletal (cluster RCTs n=2, controlled before–after studies n=3, interrupted time series n=1)

All six musculoskeletal studies focused on the implementation of CPRs for deciding on further imaging for patients presenting with ankle, knee or neck injury.15–20 All the included musculoskeletal CPRs had reported sensitivities of 100% in their validation studies, and they all focused on physician behaviour in deciding to order imaging. Of these six studies, five reported a positive effect on reducing imaging with crude ORs ranging from 0.03 to 0.9615–18 20 (see table 2). All studies adopted an educational approach to encourage CPR use among clinicians through use of educational meetings, posters and pocket cards. Of note, three studies tested the impact of the Ottawa ankle rule; two controlled before–after trials in Canada, and one cluster RCT in France.15 16 19

Cardiovascular (cluster RCT n=1, RCTs n=4, interrupted time series n=1)

Of six cardiovascular studies, two implemented chest pain CPRs to assess the impact on physician decision-making regarding emergency admission for patients with suspected myocardial infarction.21 22 One of these studies reported a 30% relative reduction in patients admitted inappropriately.21 The remaining four studies implemented cardiovascular risk scores in general practice. Three of these studies reported no impact on physician behaviour such as prescribing and referral to dieticians, or on patient outcomes such as reduction in lipid levels.23 24 27 However, in one large-scale RCT (n=3053) that published its findings in two separate articles, both patient lipid levels and physician antihypertensive prescribing were improved.25 26

Respiratory (RCTs n=5)

Of five respiratory studies, four focused on physician behaviour in terms of antibiotic prescribing for sore throat in general practice.28–30 32 Of these four studies, only one reported significantly reduced antibiotic prescription rates in the intervention group (age-adjusted relative risk 0.74, 95% CIs 0.60 to 0.92) versus usual care.32 The primary outcome in the fifth study was reported symptom severity in patients presenting with sore throat, and antibiotic prescribing was included as a secondary outcome.31 This study found that use of the CPR alone or CPR in combination with a rapid antigen detection test improved patient-reported symptom severity and duration, and reduced antibiotic use by 29% (adjusted risk ratio 0.71, 95% CI 0.50 to 0.95).31

Neurological (cluster RCT n=1)

One study implemented the Canadian CT head rule, which guides the ordering of brain imaging in patients presenting following minor head injury.33 Despite this CPR having 100% sensitivity in validation studies, it did not reduce imaging rates. In process evaluation, clinicians’ reported unease with certain components of the rule and fear of missing a high-stakes diagnosis as reasons for not adopting the CPR.33

Methodological quality assessment of included studies

Studies were heterogeneous with regard to risk of bias. For the RCT designs (n=13), five studies were considered low risk of bias for random sequence generation, and five were considered low risk in relation to allocation concealment (figure 3A). The remaining RCT studies had an unclear risk in these domains. Owing to the nature of many of the interventions, it was not always possible to blind participants and research personnel, therefore, performance bias was judged to be unclear or high in over half these studies. In the non-randomised study designs, the risk of selection bias was high in all studies while the risk of blinding and contamination was low in all studies (see figure 3B). Overall, six studies tested the impact of a CPR in which the authors were involved in developing.17 18 22 29 31 33 The impact that this may have in terms of bias is unclear.

Figure 3

(A): Methodological quality assessment of impact analysis studies with RCT study design. (B) Methodological quality assessment of impact analysis studies with controlled before-after study design.

Discussion

Summary of main findings

This review indicates that despite the increasing research interest in developing and validating CPRs, relatively few of these tools relevant to primary care have gone through impact analysis. Implementation has been restricted to a few clinical domains mainly musculoskeletal, cardiovascular and respiratory, and certain CPRs have undergone multiple evaluations, for example, the Ottawa ankle rule. Of 18 studies meeting inclusion criteria, 10 demonstrated an improvement in primary outcome with CPR use when compared to usual care. Approximately half these successful studies focused on changing physician behaviour in ordering imaging for patients presenting with ankle, knee and neck musculoskeletal injuries.15–18 20 Four studies with a positive impact on the study's primary outcome successfully implemented post-RCT measures to maintain the impact through both passive (posters) and active strategies (retention of computerised CDSS).15 16 18 20

Studies which aimed to alter physician behaviour regarding prescribing were less successful with three of six such studies successful in reducing prescription rates.23 31 32 Studies that reduced antibiotic prescription rates invested significant time before CPR implementation in assessing acceptability to clinicians, and also integrated the CPR into the clinical work flow through computerised clinical decision support or point-of-care reminders.31 32 The importance of this type of impact analysis preparation in adequately addressing barriers to implementation, and in integrating the CPR into the clinical workflow has been highlighted.5 7 40 In this review, 12 studies considered barriers to implementation and/or gauged the acceptability of the CPR to clinicians prior to impact analysis. However, only four studies integrated the CPR into clinical work flow using either computerised CDSS or point-of-care reminders.18 20 32 33

The perceived seriousness of the target condition may also affect implementation of CPR. For instance, the impact of the Canadian CT head rule was evaluated in the diagnostic pathway of intracranial bleeding following minor head injury.33 This CPR has 100% sensitivity, and though implemented by an experienced CPR research group, this CPR did not impact on CT imaging rates.33 In a parallel process evaluation, clinicians’ reported unease with certain components of the rule, and fear of missing a high-stakes diagnosis as reasons for not adopting the CPR.33

Overall, adherence with CPR use during implementation varied considerably between studies ranging from 57% to 100%. Reasons for non-adherence, established through process evaluation, related to fear of missing the diagnosis, preference for own clinical judgement, and patient request for further investigation or management.

Comparison with existing literature

Previous CPR reviews relevant to inpatient and paediatric settings reported issues around the variability of methodological quality in conducting CPR studies and a paucity of impact analysis studies.3 6 41 The issue of methodological quality has recently been addressed with the publication of two standardised reporting guidelines for CPR derivation and validation studies, and systematic reviews of CPRs.42 43 These publications will have an important role to play in standardising CPR research and in promoting robust validation of CPRs which should then be prioritised for evaluation in future impact analysis studies.

CPRs, which demonstrate improvements in the process of care and/or patient outcomes, should then be considered for inclusion in relevant clinical guidelines to facilitate dissemination into clinical practice. A recent survey which examined the use of CPRs in clinical practice by GPs in the UK reported that GPs most often used cardiovascular, depression, fracture and atrial fibrillation CPRs.44 CPR use was dictated by perceived clinical utility, familiarity and local policy requirements. In a supplementary review of clinical guidelines, very little interguideline consistency was found to guide clinicians in terms of which, if any, of these tools to use in practice.44 Prioritising the evaluation of a few adequately validated CPRs with proven predictive accuracy in relevant clinical settings would add significantly to this evidence base and facilitate, if appropriate, the inclusion of certain CPRs into future clinical guidelines.

Implications for clinical practice and research

CPR research is a relatively new methodological discipline and a challenging area of research.2 In the conduct of this review, several uncontrolled before–after impact analysis studies were retrieved. While these studies have a role in contributing to the overall evidence base, they are not a substitute for carefully conducted RCTs in determining the effectiveness of CPRs on clinically relevant outcomes.6 In this review, the majority of included RCTs focused on physician behaviour or process of care as the primary outcome. This is not surprising considering how challenging it is to demonstrate differences in patient outcomes, requiring much larger sample sizes which significantly increase running costs.10 In addition, contextual issues which exist between countries, due to differences in healthcare delivery, healthcare systems and incentives, render process of care outcomes difficult to generalise.5 6

Certainly, CPR impact needs to be considered early in the development phase of any new CPR. For instance, Irish research shows high levels of GP referrals to symptomatic breast units.45 Recent research efforts have focused on the development of a breast cancer CPR for use in primary care to aid these referral decisions.46 However, although this CPR underwent methodologically robust development and demonstrates good predictive accuracy it is unlikely its use will impact on referral rates. This is due largely to the existence of a low-risk threshold for referral driven by a combination of factors including patient expectation, media interest and fears of medicolegal ramifications for clinicians if a diagnosis is missed. So when considering an impact analysis RCT in this clinical domain, these contextual issues would need to be addressed in tandem with validation and impact analysis studies.

Certain clinical domains have seen a proliferation of CPR research, particularly musculoskeletal and cardiovascular conditions. The publication of several carefully conducted impact analysis trials for CPRs relating to knee, ankle and neck injuries is largely due to one Canadian research group, while historically, the availability of large UK population data sets facilitated the development of cardiovascular prognostic CPRs.15–18 47 48 In this review, five impact analysis studies (two were uncontrolled before–after studies detailed in online supplementary appendix 3) focused on the impact of the Ottawa ankle rule in emergency room settings, three of which were conducted in the same country.15 16 19 35 36 Ideally, CPR development and impact analysis should be aligned with clinical need rather than developing or testing the effectiveness of CPRs when accurate tools already exist.44

The relatively small number of impact analysis studies retrieved means it is not possible to make firm conclusions about the overall effectiveness of these tools in primary care. However, certain CPRs, such as the Ottawa ankle and knee rules, are appropriate for use in clinical practice, and have a role in reducing unnecessary imaging rates. Future research should focus on conducting RCTs of broadly validated CPRs with consideration of contextual and local implementation factors.7 Pertinent issues include how best to integrate the CPR into clinical workflow, and the potential benefits of embedding CPRs as part of computerised clinical decision support.

Study limitations

Although this review was conducted systematically, and multiple resources searched to retrieve relevant articles, electronic searches were limited to 30 preselected journals, and as a result it is possible that relevant studies were not retrieved. In addition, this search was last updated in December 2013. However, to the best of our knowledge, this review is the first to analyse, in detail, CPR impact analysis studies relevant to primary care. The broad definition of primary care used for this review led to the inclusion of impact analysis studies conducted in the emergency room setting. This was necessary to account for the variation in primary care services and access internationally. Studies that implemented CPRs as part of a broader guideline, protocol or decision aid were excluded. Finally, due to the heterogeneous nature of the included studies, meta-analysis was not possible.

Conclusion

Impact analysis of CPRs in primary care has, to date, focused on a small number of CPRs in a limited number of clinical domains. Future research should focus on prioritising well-validated and accurate CPRs for impact analysis to determine if these tools impact on the process of clinical care and patient outcomes.