Improving Clinical Practice Using a Novel Engagement Approach: Measurement, Benchmarking and Feedback, A Longitudinal Study.

BACKGROUND: Poor clinical outcomes are caused by multiple factors such as disease progression, patient behavior, and structural elements of care. One other important factor that affects outcome is the quality of care delivered by a provider at the bedside. Guidelines and pathways have been developed with the promise of advancing evidence-based practice. Yet, these alone have shown mixed results or fallen short in increasing adherence to quality of care. Thus, effective, novel tools are required for sustainable practice change and raising the quality of care.
METHODS: The study focused on benchmarking and measuring variation and improving care quality for common types of breast cancer at four sites across the United States, using a set of 12 Clinical Performance and Value(®) (CPV(®)) vignettes per site. The vignettes simulated online cases that replicate a typical visit by a patient as the tool to engage breast cancer providers and to identify and assess variation in adherence to evidence-based practice guidelines and pathways.
RESULTS: Following multiple rounds of CPV measurement, benchmarking and feedback, we found that scores had increased significantly between the baseline round and the final round (P < 0.001) overall and for all domains. By round 4 of the study, the overall score increased by 14% (P < 0.001), and the diagnosis with treatment plan domain had an increase of 12% (P < 0.001) versus baseline.
CONCLUSION: We found that serially engaging breast cancer providers with a validated clinical practice engagement and measurement tool, the CPVs, markedly increased quality scores and adherence to clinical guidelines in the simulated patients. CPVs were able to measure differences in clinical skill improvement and detect how fast improvements were made.

Introduction

Poor clinical outcomes are caused by multiple factors such as disease progression, patient behavior, and structural elements of care including misaligned payment incentives. One other important factor is the quality of care delivered by a provider at the bedside [1-3]. Improving clinical care quality leads to better outcomes and lower costs [4]. Consequently, reducing unwarranted variation and addressing poor quality provider practice represent significant opportunities to improve patient outcomes. Effective interventions that make a positive impact in these areas are urgently needed, but traditional quality improvement tools have not worked well and practice change is difficult to implement [5].

Providers, health care systems and payers are looking for novel, more effective ways to raise the quality of care [6]. Metrics, dashboards, educational outreach, payment incentives, and other efforts to increase awareness and provide accountability have been tried but with only modest effects on reducing variation and raising quality [7-9]. More recently, guidelines and pathways promise advancing evidence-based practice to standardize care and reduce variation [10] but by themselves have shown mixed results [11].

Disappointments with guideline and pathway adherence, as well as earlier efforts to change clinician practice, may largely stem from the inability to meaningfully engage the physician (or other provider) and integrate learning improvement opportunities with practice change, patient outcomes, and costs of care [12, 13]. Effective provider engagement is linked to improved learning in a well-described process, a construct called “adult learning theory” [5]. Among the principles forming this construct are: 1) the ability to self-direct; 2) active learning; 3) work relevance; and 4) motivation to learn [14]. Successful learning interventions for providers thus requires clinically relevant tools using multi-modal content to actively engage users, taking into account different learning needs to deliver “just in time” information with feedback [15]. Put simply, traditional lectures or written guidelines and pathways are not enough.

Effective, novel tools are required for sustainable practice change and raising the quality of care. These tools should be tailored around familiar clinical encounters and test an individual clinician’s gaps in knowledge (work relevant). Ideally, the tool would provide users the flexibility to make clinical decisions as in real-world practice (self-direction), leverage peer performance comparisons (motivation to learn) and deliver individualized feedback of their performance (active learning) [5]. The “practice-learning environment” in this new concept of quality improvement is not distinct from patient care, but is driven by clinical context enriched by a variety of activities and using a tool that respects individual learning styles.

This is a multi-site, multi-program initiative, using practice-learning principles to educate and improve adherence to clinical pathways and evidence-based medicine in breast cancer practice. We used Clinical Performance and Value® (CPV®) vignettes as a novel educational tool to engage providers in a multi-modal educational experience that addressed individual gaps in knowledge and leveraged peer-to-peer benchmarking. The primary goal of the initiative was to determine if this approach engaged and motivated providers to be responsible for their own clinical education and see if they responded with better performance over time.

Materials and Methods

Study overview

The study was conducted at four sites across the United States: two National Cancer Institute designated sites, one multi-hospital health system, and one community-based oncology practice, and focused on measuring variation and improving care quality in breast cancer. Members of the four multi-disciplinary breast cancer teams included physicians, advance practice professionals, and other clinicians. Participants were asked to care for simulated patients, using CPV® vignettes via online interactive sessions [16], with serial engagements of measurement and feedback over four rounds of data collection. These rounds of measurement and feedback occurred every 4 months. The start and end dates varied across all sites, with all data being collected between June 2013 and November 2015.

Participants

The four sites included in the study were self-selected by site leadership to commit to having a platform for their clinical providers to give higher quality care. Individual providers were considered eligible for participation if they were employees or contractors at their respective sites who worked in the oncology service and had experience in the clinical care of breast cancer patients. In total, there were 87 providers who met the initial eligibility criteria. Providers were introduced to the study through onsite meetings at each of the four sites and were notified of their inclusion in the study. Providers could opt out by providing written communication to the authors. Neither their peers nor site leadership was notified of this action. Those who remained in the study were contacted via email and telephone to ensure they completed their simulated patients each round.

Engagement and measurement tool

The CPV simulated patient is cared for online and requires the clinician to respond to open-ended questions about the care they would deliver for the CPV patient. Each CPV takes approximately 20 - 30 min to complete and guides the participant through a typical patient encounter comprised of four standard domains of care: taking history, conducting the physical examination, ordering diagnostic workup, and diagnosing with a treatment plan (DxTx). Cases have explicit evidence-based scoring criteria and each participant’s responses are compared against those criteria by trained physician scorers.

CPVs measure a provider’s ability to evaluate, diagnose, and treat specific diseases and conditions in an actual care setting. They have been extensively validated against actual clinical care and have been shown to efficiently and accurately measure actual practice, not just clinician knowledge [16]. This measurement and feedback method has been used in a variety of care settings and multi-country comparisons [17] to measure variation, improve quality and outcomes, evaluate policies and lower costs by standardizing practice [18].

Cases

In this study, the CPV vignettes represented a range of typical breast cancer patients to the providers and were created to identify and assess variation in adherence to evidence-based practice guidelines and pathways. In all, two sets of 12 vignettes were created (Table 1). Cases ranged from ruling out cancerous breast lesions to treating early-stage breast cancer to managing more advanced disease. Determining which case set to use was made by hospital and practice leadership.

Table 1

List of CPV Cases

Set A	Set B
Atypical ductal hyperplasia	Atypical ductal hyperplasia
Invasive ductal CA, stage I	Invasive ductal CA, stage I
Invasive lobular CA, stage II CHF	Invasive lobular CA, stage II CHF
Invasive ductal CA, stage II	Invasive ductal CA, stage IIB in pregnancy
Invasive ductal CA, stage III	Invasive ductal CA, stage III
Recurrent stage IIIB with bone metastasis	Recurrent stage IIIB with solitary CNS metastasis
Recurrent stage II invasive ductal CA	Isolated axillary recurrence, invasive ductal CA
Stage IIIB with recurrent disease to brain, liver	Local recurrence of stage IIIB invasive ductal CA
Multifocal breast cancer, stage II	Bilateral breast CA, stage IIIA (left), stage I (right)
Diffuse large B cell NHL, stage IIIAE. No breast CA	Ductal carcinoma in situ
Hyperparathyroidism from adenoma. No progression of breast CA	Inflammatory breast CA stage IIIB
Dense breasts	Primary stage IV breast CA with bone metastasis

The scoring criteria for the cases were based on a number of sources, including National Comprehensive Cancer Network (NCCN) clinical practice guidelines, recommendations from oncology societies, and established relevant medical guidelines, as well as internal protocols and pathways designed by participating institutions. Trained physician abstractors, blinded to the vignette-taker’s identity, scored each vignette to capture on- and off-pathway/guideline care in history-taking, physical exam, workup studies ordered, and DxTx.

Prior to the start of the first round, each provider was asked to complete a short demographic survey. In each round, providers completed two CPV cases and received confidential feedback on each patient vignette they cared for via a personalized feedback report (Fig. 1). Individual feedback included an overall CPV quality score and specific scores for each domain of care, as well as personalized recommendations for improvement with links to clinical guidelines and the medical literature. In the feedback, individual scores were benchmarked against their colleagues.

Figure 1

Example feedback form.

In addition to individual-level feedback, providers at each site were engaged in group feedback sessions (either in-person or virtual) held after every round. Only aggregate group-level performance was shared with the participants at each site, and the group’s trends regarding adherence to pathways and guidelines were communicated. Importantly, the group-level feedback provided a structure and forum for providers to interact with each other concerning specific care pathways, areas of variation, and opportunities for collective improvement. In these hour-long sessions, quality of care themes were identified by leadership and used to create points of emphasis for additional physician engagement and improvement.

Data analysis

Do serial measurement, benchmarking and feedback (at individual and group levels) increase the CPV quality of care scores and adherence to evidence-based pathways over time? To examine this question, data were gathered to track clinician scores across rounds. The main outcomes of interest are the 1) overall vignette score, corresponding to the percentage of items correctly addressed by the participants according to the guideline-based scoring criteria; 2) domain scores: history, physical, workup, and DxTx; and 3) specific areas of care in the management of breast cancer (surgery, axillary management, radiation therapy, chemotherapy, and hormonal therapy). A two-sample variance-comparison test looked at whether use of CPVs decreased the variation in domain and overall scores between rounds. Analyses were conducted on specific subgroups, including provider type, percentage of breast cancer patients in the provider’s panel, and providers with high teaching loads.

Analyses were performed using Stata 13.1.

Results

In total, of the 87 total eligible providers, 81 participated in multiple rounds of the study and completed the introductory survey (Table 2). All six non-participating providers were excluded at the provider’s request. There was no significant difference found in provider characteristics between participators and non-participators.

Table 2

Provider Characteristics, Overall and by Site

	Overall	Site 1	Site 2	Site 3	Site 4
N	81	31	22	18	10
Average age (years)	46.4	47.5	45.0	45.6	47.6
Provider type
Medical oncologist	34	7	12	7	8
Radiation oncologist	11	3	6	-	2
Surgeon	13	6	2	5	-
Other (other MD, PharmD)	10	10	-	-	-
APP (PA, APRN)	13	5	2	6	-
Breast cancer patients (%)	46.4	47.0	29.1	83.4	16.0
Time spent teaching (%)	14.2	18.2	12.0	17.2	1.5
Average years of experience	11.6	10.8	11.4	12.0	13.7
Average case load per week (all)	20.6	21.4	13.2	33.6	10.8

Among sites, there was no significant difference in provider age, percentage of participants who were physicians, or years of experience. The average total case load per week per practitioner was 20.6, the average percentage of breast cancer patients seen (as a percentage of total case load) was 46.4%, and the average percentage of time spent on teaching activities was 14.2%. Using a Bonferroni correction, a one-way comparison of case load showed a significant difference only between sites 2 and 3 (P = 0.03), but was otherwise not significant. A similar comparison of the percentage of breast cancer patients seen showed a significant difference between all sites (P < 0.01, overall), except for sites 2 and 4 (P = 1.00). Time spent teaching was similar across all sites (P > 0.05), except sites 1 and 4 (P = 0.01) and sites 3 and 4 (P = 0.03).

There was a large amount of variability in overall CPV quality scores (Fig. 2). At baseline, the overall mean across all sites was 56.6% with a wide level of variation (range 22-84%). Average scores in workup (40.0%) and DxTx (45.6%) were substantially lower than the other two domains, similar to findings in other CPV studies [19]. DxTx scores also showed tremendous variation in practice with a range of 2.5-100%. Comparisons between the four sites at baseline showed no statistically significant difference in overall (P = 0.86) or DxTx (P = 0.94) scores.

Figure 2

Round 1 overall CPV score for all provider sites.

Following multiple rounds of CPV measurement, benchmarking and feedback, we compared performance in the baseline round with follow-up rounds (Table 3). Between baseline and round 4, the overall mean scores increased by nearly 14-70.5% (P < 0.01). Looking at the domain scores, significant increases were seen across all domains (P < 0.01 for each domain). Scores in the workup domain increased over 23% by round 4 (P < 0.01). Similarly, DxTx scores increased 12% versus baseline to 57.6% (P < 0.01).

Table 3

Average Overall and Domain CPV Scores and Standard Deviation (SD) at Baseline and Subsequent Rounds

	Baseline	Round			P-value
		2	3	4	Round 4 vs. baseline
Number of CPVs	149	151	149	130
Overall score	56.6 (12.6)	60.0 (14.4)	69.2 (13.2)	70.5 (16.8)	< 0.01
Domain score
History	69.5 (13.1)	70.9 (15.7)	80.5 (16.8)	85.4 (15.0)	< 0.01
Physical	83.1 (20.6)	81.5 (21.8)	89.4 (18.9)	84.8 (19.9)	< 0.01
Workup	40.0 (20.8)	48.9 (22.3)	57.7 (24.1)	63.4 (20.6)	< 0.01
Diagnosis-treatment	45.6 (20.8)	51.4 (20.7)	61.9 (19.5)	57.6 (18.6)	< 0.01

We examined variation across rounds using a one-tail test of variance to compare changes in the standard deviation of the overall and domain scores between baseline and round four. The overall score showed a statistically significant increase in variation from 12.6% vs. 16.8% (P < 0.01), while the domain of history demonstrated a borderline significant increase in variation between baseline and round four (P = 0.06). Physical and workup domains had minor decreases in variations (P = 0.66 and 0.54, respectively), whereas DxTx variability had a much larger decrease from 20.8% to 18.6%, although this change was also not significant (P = 0.09).

Disaggregating the participants by percentage of time spent teaching, we found that variability decreased among those who reported spending less than 20% of their time teaching. At baseline, no significant difference was found in scores or variability between these two groups. For those who spent less time teaching, the overall score was an average of 3.6% higher (P = 0.06), and the related variability was 1.4% lower (P = 0.18). By round four those who reported more of their time seeing patients averaged 4.7% higher overall scores (P = 0.10) and with 5.7% less variability (P < 0.01) compared to those who reported more time teaching.

We next looked at specific areas of care in the management of breast cancer where adherence to the evidence base has proven problematic, including workup of the axilla and chemotherapy (Table 4). Comparing the baseline round and round 4, site 3 showed a marked increase in all five tracked areas, with percentage relative improvements ranging from 25% in hormonal therapy to over 100% in surgery. While the other sites did show improvement, they did not do so as uniformly as site 3. As shown in Table 4, two out of the three remaining sites showed improvements in surgery, axillary management, and radiation therapy. Chemotherapy scores showed large improvements in site 1 as well as site 3. Site 4 showed marginal improvement while site 2 showed a large decrease. Hormonal therapy, which started at relatively high baseline scores, proved difficult to improve, with only sites 1 and 3 showing any large increase.

Table 4

Comparison of Baseline to Latest Round of CPV Scores by Areas of Care

Area of care	Site 1		Site 2		Site 3		Site 4
	Baseline	Round 4	Baseline	Round 4	Baseline	Round 4	Baseline	Round 4
Surgery	85%	81%	52%	74%	40%	83%	69%	80%
Axillary management	62%	81%	39%	32%	50%	67%	31%	80%
Radiation therapy	58%	65%	57%	73%	50%	100%	83%	50%
Chemotherapy	38%	77%	50%	31%	50%	75%	40%	42%
Hormonal therapy	60%	81%	71%	72%	60%	75%	79%	67%

Discussion

Isolating practice variability from patient variability to untangle the causes of poor clinical care is a challenge for all quality improvement projects. In this study of four separate oncology systems, standardized virtual CPV patients removed patient variability to focus on provider practice variability and highlight opportunities to improve clinical decision making.

Serially engaging breast cancer providers with a validated clinical practice engagement and measurement tool markedly increased quality scores and adherence to clinical guidelines in the simulated patients. CPVs measured differences in clinical skill improvement and detected how fast improvements were made. The results indicate that CPV measurement was sensitive enough to accurately measure provider adherence to clinical pathways and were also responsive enough to detect meaningful improvements (or lack of improvement) over time. CPV measurement found a high degree of variability between and within sites that declined over time for DxTx and among certain subgroups. These results support the idea that direct, practice-based engagement of providers with proper learning tools can improve quality of practice and use of pathways.

When analyzing performance by domain, physical examination was consistently the highest score, presumably because it is the most straightforward area. The workup and DxTx domains demonstrated the lowest scores and the widest variability at baseline and also the greatest improvement over 9 - 18 months of CPV engagement. These are highly complex domains with multiple decision points, and most directly impact patient outcomes with important cost consequences for health systems, payers and, ultimately, patients. Notably, improvements in adherence to treatment guidelines has historically been difficult to effect [20]; that CPVs were able to identify these domains as opportunities for greatest change and drive the strongest improvements has important implications for improving both quality and cost.

Those who spent more time seeing patients had both higher quality scores and less variability in performance at baseline and by the end of the study. The expectation (unproven) was that providers who taught more might score higher (a marker for greater familiarity to guidelines) and/or have less variability (a marker for standardization within each site). That this was not demonstrated may be driven by a number of factors. First, providers who spend more time in clinic and see a consistent stream of patients could be more comfortable with simulated patient encounters, more familiar with evidence-based care, or more likely to recognize similar patients they have recently seen. Alternatively, those who spend more time teaching may be less likely to respond to feedback. The second supposition is somewhat worrying, as those with more teaching responsibilities are ones mentoring and modeling behavior for the next generation of clinicians. Further investigation of this is warranted.

This study looked at specific clinical areas of breast cancer care (surgical decisions, axillary management, radiation therapy, chemotherapy, and hormonal therapy) and found varied levels of improvement between sites. Chemotherapy is a complex clinical decision point in breast cancer care because multiple drug options, combinations and dosing regimens are available and decisions are made based on patient-, disease- and cost-related factors. The marked improvements seen at sites 1 and 3 show that, even in complex areas, measurement and feedback can improve decision making. Differences in baseline scores and improvement rates by site suggest that CPV quality scores can be used to direct performance improvement initiatives tailored to each organization.

To support practice change, physicians need the right tools to systematically examine their own practice performance [21]. Continuing medical education (CME) has been tasked with doing this, but a growing body of research has found that traditional CME in the quality improvement space can be irrelevant and ineffective at changing physician behavior [22], and results seldom involved reflection by the physician on their real learning needs and are often designed as passive add-ons to quality improvement activities [20, 23]. A 2010 Institute of Medicine report based on the analysis of 62 studies and 20 systematic reviews found that the “continuing education “system”, as it is structured today, is so deeply flawed that it cannot properly support the development of health professionals” [24].

Researchers have found physicians are unable to accurately self-assess their skills, and suggest external assessment, scoring and feedback would drive more effective professional development [25]. Others have found that physicians are often “not trained” to evaluate or use published guidelines and best practices [26]. Passive dissemination of information (e.g., publishing guidelines, reading peer review articles) is generally ineffective at changing practice and is unlikely to result in group-wide practice change when used alone [27]. Multifaceted interventions based on assessment of potential barriers to change are more likely to be effective than single interventions [26].

These frustrations about changing clinical practice have led to calls for better ways to engage clinicians and support high quality, high value care [5]. Physician engagement [28] is widely recognized as key to the success of nearly any health system initiative, and this extends naturally to any successful quality improvement programs [29]. Physicians identified as engaged in hospital quality improvement initiatives are much more likely to report successful experiences implementing quality improvement programs [21]. Methods that require active physician learning (e.g., one-on-one meetings, small-group workshops and programs tailored for a specific clinic) have been shown to most effectively change physician practice patterns to align with new clinical guidelines [5]. These, however, can be difficult and costly to scale.

Limitations to this study do exist. The sites that participated in this study were self-selected, and it may be that the providers at these sites were already primed for a quality improvement program. However, determination of how much of the leadership’s commitment was instilled in the providers prior to the study is difficult to measure, and the results give here may instead have been driven by the actual engagement process, which as noted above is necessary for successful implementation and adoption of a quality improvement program.

The importance of engagement in meaningful quality improvement work cannot be overemphasized, and the adult learning theory construct captures what is occurring in this area with CPVs. Disease-specific simulated patient vignettes (work relevant) that are open-ended and evidence-based (self-direction) and actively engage providers over multiple rounds of measurement, comparison, and feedback (motivation) are an effective strategy to improve clinical decision making and adherence guidelines (active learning). This multi-modal approach to changing physician behavior has the additional benefit of being efficiently scalable and applicable to multiple sites and organizations. Results from other studies show an increase in overall quality scores by the third round of intervention and permanent improvements after six rounds [30]. Documenting the impact of increased CPV quality scores on actual clinical practice and utilization is a priority for further study but evidence already exists that CPV use translates into better patient outcomes [2]. Beyond measuring quality (low) and variation (high), we found that engaging providers with their individual quality scores and delivering targeted feedback on opportunities to improve led to significant increases in overall quality and in the diagnosis and treatment among breast cancer providers.