Healthcare provider perceptions of clinical prediction rules.

OBJECTIVES: To examine internal medicine and emergency medicine healthcare provider perceptions of usefulness of specific clinical prediction rules.
SETTING: The study took place in two academic medical centres. A web-based survey was distributed and completed by participants between 1 January and 31 May 2013. PARTICIPANTS: Medical doctors, doctors of osteopathy or nurse practitioners employed in the internal medicine or emergency medicine departments at either institution. PRIMARY AND SECONDARY OUTCOME MEASURES: The primary outcome was to identify the clinical prediction rules perceived as most useful by healthcare providers specialising in internal medicine and emergency medicine. Secondary outcomes included comparing usefulness scores of specific clinical prediction rules based on provider specialty, and evaluating associations between usefulness scores and perceived characteristics of these clinical prediction rules.
RESULTS: Of the 401 healthcare providers asked to participate, a total of 263 (66%), completed the survey. The CHADS2 score was chosen by most internal medicine providers (72%), and Pulmonary Embolism Rule-Out Criteria (PERC) score by most emergency medicine providers (45%), as one of the top three most useful from a list of 24 clinical prediction rules. Emergency medicine providers rated their top three significantly more positively, compared with internal medicine providers, as having a better fit into their workflow (p=0.004), helping more with decision-making (p=0.037), better fitting into their thought process when diagnosing patients (p=0.001) and overall, on a 10-point scale, more useful (p=0.009). For all providers, the perceived qualities of useful at point of care, helps with decision making, saves time diagnosing, fits into thought process, and should be the standard of clinical care correlated highly (≥0.65) with overall 10-point usefulness scores.
CONCLUSIONS: Healthcare providers describe clear preferences for certain clinical prediction rules, based on medical specialty. Published by the BMJ Publishing Group Limited. For permission to use (where not already granted under a licence) please go to http://group.bmj.com/group/rights-licensing/permissions.

This is the first study to examine healthcare provider perceptions of usefulness of CPRs in the hospital setting.

Providers consistently rated CHADS2, the Thrombolysis in Myocardial Infarction (TIMI) Score (NSTEMI), Wells Score for Pulmonary Embolism, Alcohol Abuse CAGE and the Ottawa Ankle Rule highly. These CPRs would be ideal candidates for integration into an electronic health record (EHR).

Emergency medicine providers consistently rated CPRs more positively and may serve as early adapters to CPRs integrated into electronic health records.

For all providers, the qualities of CPRs being useful at point of care and that these help with decision-making, save time diagnosing, fit well with one's thought process, and should be the standard of clinical care correlated highly (≥0.65) with usefulness scores.

A significant limitation of the results of this study is that mean ratings for CPR characteristics reflect only the opinion of healthcare providers who selected the CPR as one of the top three most useful.

Introduction

Evidence-based medicine was announced as a fundamental paradigm shift in medicine in the early 1990s and predicted to de-emphasise intuition, clinical experience and pathophysiological rationale in favour of hard scientific evidence.1 Decades later, the accessible body of clinical research has grown exponentially, but translation into common clinical practice has been protracted and inconsistent.

The seamless integration of clinical prediction rules (CPRs) into the point of care will aid in transferring evidence-based medicine into daily clinical practice. CPRs can be defined as validated tools that quantify the individual contributions that components of history, physical and laboratory results make towards a diagnosis, prognosis or treatment response.2 A few commonly used CPRs include the CENTOR criteria, which predicts the likelihood of Streptococcal pharyngitis;3 the CAGE score, which serves as a screening test for alcoholism;4 and the CHADS2 score, which predicts the risk of stroke in patients with atrial fibrillation.5

CPRs integrated into electronic clinical decision support tools have demonstrated the ability to shape healthcare provider behaviour towards more evidence-based clinical practice.6 However, provider adoption continues to be a significant barrier to widespread use of clinical decision support as a whole, which is reported at 10–20%.7 Efficiency, usefulness, information content, user interface and workflow have been reported by clinicians to be the keys to effective decision support.7 These are likely to be large determinants of clinician adoption rates.

In light of the growing interest in integrated clinical decision support, and CPRs in particular, this study sought to help address the biggest challenge of implementation, poor provider adoption. The study focuses on provider perceptions of usefulness of CPRs in an effort to illuminate preferences, attitudes and thoughts that might be relevant to all types of clinical decision support. We examine healthcare provider perceptions of usefulness based on specialty and level of training with the ultimate goal of discovering which CPRs might be better adopted by these providers.

Methods

A web-based survey platform was distributed to 401 healthcare providers between 1 January and 31 May 2013 in two academic medical centres, Hofstra North Shore—LIJ School of Medicine and Boston University, in the USA.

The survey content and structure were informed by qualitative interviews with physicians, a literature review and feedback received after pilot testing. The survey was piloted for approximately 1 month and after minor modifications, for instruction clarity and reduced length, distributed via email to attending physicians, nurse practitioners and residents training in the fields of internal medicine (IM) and emergency medicine (EM).

Providers were included in this study if they were credentialed as medical doctors, doctors of osteopathy or nurse practitioners, and were currently employed in either the IM or EM departments at each institution. Providers were excluded if they were currently involved in the study.

Participants were recruited, consented and asked to complete the survey via email. Additionally, providers were approached during grand rounds and resident afternoon conferences to encourage them to complete the survey. Laptops with the survey preloaded were placed at meetings to encourage completion. In addition, providers were sent reminder emails twice a month throughout the study period.

The survey consisted of three distinct sections. In the first section, participants were asked for demographic information, including hospital affiliation, professional degree, current position (attending vs resident), percentage of time devoted to clinical responsibilities, primarily outpatient versus inpatient practice, years of practice, medical specialty, race, gender and age. Demographic information, including race and gender, was assessed to determine the extent to which findings could be generalised to other medical communities.

In the second section, providers were asked to pick from a list of 24 CPRs: National Emergency X-Radiography Utilization Study (NEXUS) C-Spine Rule,8 Canadian C-Spine Rule,9 Ottawa Knee Rule,10 Walsh,11 Lee Index,12 The Thrombolysis in Myocardial Infarction (TIMI) Risk Score (NSTEMI),13 CHADS2,5 4T Score for Heparin-Induced Thrombocytopenia (HIT),14 Ottawa Ankle Rule,15 Pulmonary Embolism Rule-Out Criteria (PERC),16 Wells Score for deep venous thrombosis (DVT).17 Wells Score for Pulmonary Embolism (PE),18 Alcohol Abuse CAGE,4 Model for End-Stage Liver Disease (MELD) Score,19 San Francisco Rule for Syncope,20 Modified Early Warning System (MEWS),21 CURB 65,22 Ranson's Criteria,23 Pittsburgh Knee Rule,24 Predicting Tuberculosis (TB) in Patients,25 Pneumonia Severity Index (PSI)/Pneumonia Patient Outcomes Research Team (PORT) Score,26 Acute Physiology and Chronic Health Evaluation (APACHE II),27 Mortality in Emergency Department Sepsis (MEDS)28 and Ventilator Associated Pneumonia (VAP).29 They were asked to select all of the CPRs that were familiar to them. Of those CPRs, participants were then asked to select three that they found most useful.

The last section of the survey applied only to those three CPRs. They were asked questions about their perception of the utility and favourability of the CPRs. Statements such as “The 4T score for Heparin-Induced Thrombocytopenia is easy to use” were rated on a Likert scale from 1 (strongly disagree) to 5 (strongly agree). The last question in this section asked the provider to rate the CPR on a 10-point scale in terms of overall usefulness.

Statistical methods

Descriptive statistics, such as means and SDs for continuous variables, and frequencies and proportions for categorical variables, were used to describe the respondent characteristics.

The χ2 test or Fisher's exact test, as appropriate, was used to explore the association between each of the categorical questionnaire items and the key variables of interest (eg, IM vs EM). The Mann-Whitney test was used to compare the target groups on the ordinal and continuous variables. Finally, the Spearman correlation was used to measure the correlation between selected ordinal variables and the usefulness of the CPR.

Results

Of the 401 healthcare providers distributed the web-based survey, 22 individuals declined participation, 111 respondents agreed to participate but did not finish the survey, 1 individual completed the survey but left the agreement field blank, and 4 individuals left the agreement field blank and did not finish the survey. A total of 263 individuals, 66% of those asked to participate, agreed to respond, provided written informed consent and completed the survey. No stipend was provided.

Demographic characteristics

The IM and EM groups were compared on a number of demographic characteristics (table 1). There were significant differences between the two groups in terms of the institution they represented. Whereas a greater proportion of IM respondents were from Hofstra North Shore-LIJ School of Medicine (77% vs 63%), a greater proportion of the EM respondents were from Boston University (37% vs 22%; p<0.01). Significant differences were also noted when comparing the IM and EM groups on position occupied. Attending physicians in EM were over-represented compared with IM attending physicians (54.2% vs 28%, respectively; p<0.001).

Table 1

Demographics of survey participants

	TotalN=298	Internal medicineN=215 (72%)	Emergency medicineN=83 (28%)	p Value
Institution				0.01
Hofstra North Shore-LIJ School of Medicine	176 (59%)	137	39
Boston University	119 (40%)	75	44
Other	3 (1%)	3	0
Degree				0.22
Medical degree	274 (92%)	199	75
Doctor of osteopathy	20 (7%)	12	8
Nurse practitioner	4 (1%)	4	0
Role				<0.0001
Attending	105 (35%)	60	45
Hospitalist	16 (5%)	16	0
House staff	167 (56%)	129	38
Nurse practitioner	5 (2%)	5	0
Other	5 (2%)	5	0
Practice location				<0.0001
All outpatient	69 (23%)	38	31
Mostly outpatient	31 (11%)	22	9
Equal	15 (5%)	11	4
Mostly inpatient	124 (42%)	117	7
All inpatient	57 (19%)	26	31
Years of practice				0.06
1–4	183 (61.4%)	140	43
5–9	43 (14.4%)	23	20
10–14	22 (7.4%)	16	6
15–20	21 (7%)	15	6
>20	29 (9.7%)	21	8
Age (years)				0.34
25–29	108 (36%)	85	23
30–39	116 (39%)	76	40
40–49	44 (15%)	31	13
50–59	20 (7%)	16	4
60–69	6 (2%)	4	2
70+	4 (1%)	3	1
Race (may select >1)				NA
Caucasian	185 (62%)	119	66
African-American	10 (3.3%)	8	2
Asian	80 (27%)	71	9
Hispanic	11 (3.8%)	7	4
Native American	1 (0.3%)	1	0
Other	11 (3.7%)	8	3
Gender				0.38
Female	117 (39%)	88	29
Male	180 (61%)	127	53

*Attending—physician who has completed postgraduate medical training. House Staff—physician who is undergoing postgraduate medical training. Hospitalist—internal medicine physician who works only in an inpatient setting. Doctor of osteopathy—medical doctor who completed osteopathic medical school.

NA, not available.

Participants were diverse in terms of age, race and total years of practice. There was a male predominance (61%), which paralleled that seen in national US physician data where only about one-third of medical doctors are women.30 Compared with national US physician data, our sample included slightly less Caucasians, 62% vs 75%; and less African-Americans, 3.3% vs 6%; and more Asian, 27% vs 12.8% physicians. The majority (75%) of the physicians were between 25 and 39 years of age, and had nine or fewer years of practice.

Most familiar and most useful CPRs

Participants were asked to select an unlimited number of CPRs that they were familiar with and of those choose three they felt were the most useful (table 2). The Alcohol Abuse CAGE,4 CHADS2,5 TIMI Score (NSTEMI)13 and Wells Score for PE18 were in the top five most frequently chosen as familiar and useful. Ranson's criteria23 was one of the top five most selected as familiar but not as useful and vice versa for the MELD score.19 The CHADS25 score was chosen as most useful by most participants (63%).

Table 2

All 24 CPRs, frequency of selection as familiar and top 3 most useful, ordered by mean 10-point usefulness score

All 24 CPRs	FamiliarN (%)	UsefulN (%)	Useful score mean
NEXUS C-Spine Rule8Imaging in patients at risk for c-spine fracture	14 (4.6)	33 (11)	8.54
Canadian C-Spine Rule9Imaging in patients at risk for c-spine fracture	85 (29)	28 (9)	8.5
Ottawa Knee Rule10Imaging in patients with knee trauma	77 (26)	9 (3)	8.5
Walsh11Likelihood of Streptococcal pharyngitis	110 (37)	27 (9)	8.39
Lee Index12Perioperative cardiovascular risk	30 (10)	10 (3)	8.38
TIMI Score (NSTEMI)13Mortality in patients with NSTEMI	253 (85)	89 (30)	8.12
CHADS25Stroke risk in patients with atrial fibrillation	255 (86)	184 (62)	8.01
4T Score for HIT14Likelihood of HIT	76 (26)	19 (6)	7.91
Ottawa Ankle Rule15Imaging in patients with ankle trauma	170 (57)	55 (18)	7.84
PERC16Rules out pulmonary embolism	78 (26)	38 (13)	7.84
Wells Score for DVT17Estimates likelihood of DVT	212 (71)	43 (14)	7.48
Wells Score for PE18Calculates risk of pulmonary embolism	232 (78)	82 (28)	7.29
Alcohol Abuse CAGE4Screen for alcohol abuse	271 (91)	64 (21)	7.27
MELD19Estimates mortality in end-stage liver disease	211 (71)	56 (19)	7.26
San Francisco Rule for Syncope20Risk stratification of patients with syncope	62 (21)	10 (3)	7.22
MEWS21Identifies clinically deteriorating patients	96 (32)	7 (2)	7
CURB 6522Mortality in patients with pneumonia	192 (64)	41 (14)	6.88
Ranson's Criteria23Mortality in patients with pancreatitis	262 (88)	33 (11)	6.53
Pittsburgh Knee Rule24Imaging in patients with knee trauma	17 (6)	2 (1)	6.5
Other (please list)	19 (6)	6 (2)	6.33
Predicting TB in Patients25Predicts likelihood of tuberculosis	15 (5)	1 (0)	6
PSI/PORT Score26Mortality in patients with pneumonia	148 (50)	18 (6)	5.83
APACHE II27Estimates mortality in ICU patients	193 (65)	12 (4)	5.8
MEDS28Estimates mortality in septic ED patients	100 (34)	6 (2)	NA
VAP29Predicts risk of VAP	48 (16)	2 (1)	NA

APACHE II, Acute Physiology and Chronic Health Evaluation; CPR, clinical prediction rule; ED, emergency department; HIT, Heparin-Induced Thrombocytopenia; ICU, intensive care unit; MEDS, Mortality in Emergency Department Sepsis; MELD, Model for End-Stage Liver Disease; MEWS, Modified Early Warning System; NA, not available; NEXUS, National Emergency X-Radiography Utilization Study; PE, pulmonary embolism; PERC, Pulmonary Embolism Rule-Out Criteria; PORT, Pneumonia Patient Outcomes Research Team; PSI, Pneumonia Severity Index; TIMI, Thrombolysis in Myocardial Infarction; VAP, Ventilator Associated Pneumonia.

When the list of CPRs most frequently selected as most useful is evaluated by specialty, the rankings diverge. EM providers were more likely to choose CPRs commonly used in emergency departments like PERC,16 NEXUS C-Spine Rule,8 Ottawa Ankle Rule15 and Canadian C-Spine.9 IM providers were more likely to choose CPRs commonly used on inpatient services like CHADS2,5 TIMI score (NSTEMI),13 Alcohol CAGE4 and MELD.19 Of note, both lists for CPRs rated as most useful included the Wells Score for PE.18

EM versus IM healthcare provider perceptions of CPRs

Providers were then asked specific questions about each of the three CPRs they rated as most useful. EM providers, compared with IM providers, rated their CPRs significantly more positively as having a better fit into their workflow (p=0.004), helping more with decision-making (p=0.037) and better fitting into their thought process when diagnosing patients (p=0.001) (table 3). There was a trend observed, although not meeting statistical significance, where EM providers consistently reported higher Likert scores for positive CPR qualities, such as easy to use, and IM providers consistently reported higher Likert scores for negative CPR qualities, such as limits independent decision. Lastly, compared with IM providers, EM providers rated their CPRs on a 10-point scale as overall significantly more useful (p=0.009).

Table 3

Mean ratings of all chosen CPRs by EM versus IM

CPR characteristic	EMmean (SD)	IMmean (SD)	p Value
Easy to use	3.93 (1.04)	3.77 (1.00)	0.112
Useful at point of care	3.94 (1.03)	3.78 (1.01)	0.141
Currently look-up electronically	2.98 (1.21)	2.91 (1.17)	0.583
Would use if electronic	3.49 (1.18)	3.57 (1.13)	0.659
Fits into workflow	3.92 (1.06)	3.65 (0.99)	0.004
Helps with decision-making	3.96 (1.07)	3.79 (0.98)	0.037
Saves time diagnosing	3.50 (1.05)	3.33 (1.01)	0.088
Limits independent decision	1.96 (0.82)	2.12 (0.88)	0.242
Patient too complex to use CPR	2.05 (0.77)	2.25 (0.83)	0.118
Fits into thought process	3.85 (1.03)	3.63 (0.94)	0.001
Many colleagues use	3.61 (1.01)	3.54 (0.96)	0.572
Should be standard clinical care	3.52 (1.02)	3.57 (0.97)	0.588
Overall usefulness scale	7.43 (1.87)	6.84 (2.03)	0.009

CPR, clinical prediction rule; EM, emergency medicine; IM, internal medicine.

Specific CPR overall usefulness score by provider type

The overall usefulness score was considered to be the ultimate indicator of strength of provider preference for the CPR. Scores for each CPR's usefulness were compared across provider specialty, resident versus attending position, and primary outpatient versus inpatient practice.

The only CPR with a significant difference between usefulness scores between specialties was the Ottawa Ankle Rule15 and the Wells Score for PE,18 both preferred by providers in EM. Of note, many of the 24 CPRs could not be compared by specialty because these were not selected by any EM providers as one of the top three most useful, including the 4T Score for HIT,14 APACHE II,27 Lee Index,12 MELD,19 MEWS,21 Predicting TB,25 Ranson's,23 Ventilator Associated Pneumonia29 and MEDS.28

Two differences were observed between usefulness scores of providers working in mostly or all inpatient versus outpatient settings. Inpatient providers rated the 4T Score for HIT14 as significantly more useful, while providers working in mostly or all outpatient settings rated the Walsh score11 as significantly more useful. There were no differences between resident versus attending ratings of overall usefulness for any CPR.

CPR characteristics and overall usefulness score

Ratings for perceived qualities of each CPR were analysed in terms of their correlation with usefulness score (table 4). For all providers, EM and IM, the perceived qualities of being useful at point of care and helps with decision-making, saves time in diagnosing, fits into one's thought process, and should be the standard of clinical care correlated highly (≥0.65) with usefulness scores.

Table 4

Correlations between CPR characteristics and the overall usefulness rating

CPR characteristic	Emergency medicine	Internal medicine	p Value
Easy to use	0.734	0.581	0.07
Useful at POC	0.767	0.681	0.219
Currently look-up electronically	0.267	0.383	0.379
Would use if electronic	0.480	0.656	0.077
Fits into workflow	0.768	0.634	0.072
Helps with decision-making	0.763	0.677	0.222
Saves time diagnosing	0.704	0.660	0.569
Limits independent decision	0.200	0.188	0.936
Patient too complex to use CPR	0.074	0.165	0.535
Fits into thought process	0.725	0.668	0.453
Many colleagues use	0.630	0.556	0.435
Should be standard clinical care	0.778	0.748	0.631

CPR, clinical prediction rule; POC, point of care.

Discussion

Perceived utility of clinical decision support tools and clinical guidelines have been previously studied;31 32 however, this is the first study to examine healthcare provider perception of usefulness of CPRs in the hospital setting. Providers surveyed in this study reported clear preferences for certain CPRs. Participants consistently rated CHADS2,5 TIMI Score (NSTEMI),13 Wells Score for PE,18 Alcohol Abuse CAGE5 and the Ottawa Ankle Rule15 highly. These CPRs would be ideal candidates for integration into an electronic health record (EHR).

Interestingly, EM providers consistently rated their chosen CPRs more positively. We found as well that qualities like ease of use, saves time, helps with decision-making, and should be standard of clinical care had a strong relationship to providers’ perception of utility. These qualities should be considered as requirements for a CPR considered for integration into an electronic health record.

Improved clinical care as well as decreased costs and decreased waste are potential results of provider preferred integrated CPRs. Although the USA spends nearly double the average, $3923, of all of the Organisation for Economic Co-operation and Development (OECD) countries33 on healthcare, American patients receive about 55% of recommended clinical care.34 Overtreatment and failures in execution of care processes are partially responsible for waste in healthcare spending, estimated as exceeding 20%.35 Meta-analysis of the effect of clinical decision support has shown that providers with decision support were more likely to provide preventive care services and order appropriate treatments.7

Limitations

A significant limitation of the results of this study is that mean ratings for CPR characteristics reflect only the opinion of healthcare providers who selected the CPR as one of the top three most useful. However, the structure of the survey also ensures that CPR characteristic ratings were made only by providers who were likely to use the CPR in daily practice.

Additionally, participants were recruited during academic conferences, including grand rounds as well as afternoon conferences for residents. This may have increased the number of participants who attend academic conferences, and who are more familiar with CPRs.

Implications for clinical practice and research

Meaningful clinical decision support requires not just understanding healthcare provider perceptions, but also choosing tools that are strongly evidence-based and have been tested for their effectiveness. Future trials should focus on evaluating the clinical impact of healthcare provider preferred CPRs.

Conclusion

Healthcare providers describe clear preferences for certain characteristics and disease-specific CPRs. EM providers consistently rated CPRs more positively and may serve as early adapters for CPRs integrated into EHRs. Understanding provider perceptions may help to address limiting factors in meaningful integration of clinical decision support into our electronic health systems.