Importance of an alternative approach to measuring quality in a volume-to-value world: a case study of diabetes care.

BACKGROUND: To develop a statistical tool that allows practitioners and/or their practice managers to easily select the relevant range in which volume and value are maximised.
METHODS: Data for the study were based on 55 primary care practices that participated in the Colorado Improving Performance in Practice programme in 2014. We used two composite variables including the volume of processes of care variables listed in Diabetes Practice Guidelines and value (quality) as measured by changes in the intermediate outcomes. We assessed volume/value trade-offs using a multilevel model with a time-varying covariate partitioned into a between-practice and within-practice effect.
RESULTS: The study revealed a strong linear relationship between volume and value (P<0.0001). Specifically, practices with an above-average volume of care as measured by their process of care scores also had above-average quality outcomes (expected value 57; average volume 49.48) as quantified by their intermediate outcome scores. Additionally, in those months when practices provided a volume of care that exceeded their average process of care score, further improvements occurred in quality as measured by intermediate outcome scores (P<0.0001).
CONCLUSION: Such findings suggest an inherent linkage between volume of care and quality. This statistical approach, if provided as an app containing an easy-to-use statistical calculator, will allow practice managers and clinicians to systematically identify volume/quality trade-offs, thereby reducing undertreatment and/or overtreatment among patients with chronicities.

Background

Despite the near-consensus that volume-to-value approaches will be explicit to the reimbursement models that will dominate the healthcare system now and into the future, the empirical transformation of this concept is wrought with complexities. The existence of an underlying tension and competition of interests between providers and patients suggests the need for a statistical tool that will allow the empirical verification of whether patients, insurers or the society as a whole are the beneficiary of the proposed value outcomes.1 As one author states, “… physicians and other health providers respond rationally to existing financial incentives (translation: they do what they get paid to do and generally try to, or have to, minimize those activities and services for which they are not paid).” 2 However, strong proponents of the volume-to-value model make a highly relevant point that has been underdiscussed by analysts during the translation of this conceptual model from theory to healthcare system praxis.3 4 This question is, ‘how can clinicians know with some degree of precision when volume trade-offs are decreasing patient outcomes?’

Consistent with this theme, the entirety of the January 2017 issue of The Lancet implicitly embraces the issue of the measurement of healthcare system overuse and underuse by offering strategies for the prevention of either extreme.5 However, while the centrality of quality measurement and evaluation approaches has been alluded to in the volume-to-value literature and the use of big data recommended as a necessary tool, a paucity of research exists regarding statistical methodologies that may be useful in supporting clinicians and/or their financial managers in ensuring that processes of care services are managed in such a way as to deliver balance within the emerging volume-to-value environment.6

This study introduces a statistical methodology that will allow physicians, their financial managers and/or others to optimise both volume of care and intermediate outcomes. As a result, the risks of financial loss that the volume-to-value paradigm poses for clinicians who treat persons with chronic health conditions can be reduced. Real-world data on providers who treat patients with diabetes are used to demonstrate this approach.

Measuring the relationship between volume of diabetes care and quality: current approaches

The level and volume of care delivered by clinicians plays a major role in the control of diabetes and resultantly in the patient’s outcomes as measured by the prevalence and incidence of adverse diabetes-related medical outcomes.7 Numerous national and local efforts have been launched that seek to improve the care of patients with diabetes.8 Donabedian’s paradigm is often used as the conceptual framework to determine the effectiveness of such interventions.9 10 This paradigm, a model which serves as a dominant one in the area of healthcare management and administration, suggests that a causal pathway exists in which structural characteristics of the environment such as use of the chronic care model affect processes of care such as diabetes-related screening, which in turn affect patient outcomes.11

The approach adopted by most studies that use Donabedian’s paradigm within the context of diabetes is one which uses structural change as the input, while treating both processes and outcomes as outputs. Donabedian’s model implies that structure entails process which, in turn, generates outcomes. Thus, the assumption is that it necessarily follows that structure is also related to outcomes. Moreover, many studies, based on this approach, report significant direct relationships between structures, processes and outcomes.12–18

As one analyses this approach, it is clear that this model also implies that the relationship between structure and outcome may be weaker than originally assumed. Thus, it is less than surprising that another group of studies report significant relationships between changes in structure and changes in care. Yet, empirical research does not confirm significant relationships between changes in structure and improved patient outcomes.19–29 This suggests that a structural shift from a volume-oriented clinical delivery system may not automatically generate adverse quality changes if decreases occur in the volume of processes of care as value-based reimbursement approaches become dominant. Accordingly, this study focuses on the direct relationship between changes in the volume of diabetes processes of care and quality changes as measured by diabetes intermediate outcomes. Few studies have addressed such a relationship and even fewer have corroborated the relationship.

Indeed, a review of the literature revealed no studies that have sought to export findings from a chronicity such as diabetes care into a framework that examines the implications of the findings for the volume-to-value paradigm. Moreover, only a small body of research considers both processes of care and intermediate outcomes as predictors of diabetes long-term value measures such as the rates of death, blindness, cardiovascular disease, amputation and patient’s overall physical health per dollar spent. For example, Kahn found that a change in a health-related quality-of-life score was significantly associated with the process of care composite (eg, exams, lab tests, diagnostic procedures).30 This finding therefore alluded to a relationship between diabetes care and diabetes long-term outcomes. However, the inclusion of diabetes with other chronic diseases, non-diabetic medications and counselling in the definition of care introduces ambiguity into the results.

Rather than using a combination of care and outcomes variable, Harman used separate patient care and intermediate outcome variables to predict the individual Health Outcome Survey physical and mental health scores of Medicare plan enrollees.31 The intermediate outcome measure was associated with changes in both physical and mental health while the process of care measure was only associated with a change in mental health.31 This study also suggests a relationship between volume of care and quality as measured by outcomes. However, the non-significant association between care and physical health weakened this connection.

The medical model of health, which is defined as an absence of disease and/or illness, implicitly hypothesises that intermediate outcomes literally mediate rather than moderate the relationship between process of care and outcomes. However, these studies may suffer from the proximity effect rather than providing an empirically confirmed relationship between volume of care and patient outcomes. Yet, very few studies have tested for a direct relationship between the volume of diabetes process of care variables and diabetes intermediate outcomes (eg, A1C, blood pressure and low-density lipoprotein (LDL) levels). Kirk et al. 32 used a bivariate process of care composite variable equalling ‘yes’ if the patients had received two A1C tests, one blood pressure reading and one LDL measurement in the past year, and ‘no’ if one or more processes of care could not be confirmed. This study makes one of the strongest cases for a volume-to-value transition because it resulted in a non-significant relationship between process of care and the bivariate A1C outcome measures, thereby suggesting that the volume of the processes of care can be decreased without impacting quality.32 Other research also found no significant correlations and suggested that cost savings can be achieved in diabetes care through volume decreases without an impact on quality.21

Indeed, only one single study that modelled the relationship between a diabetes process of care variable and intermediate outcomes found a significant relationship.33 Accordingly, this study suggests that, with an appropriately specified, easy-to-use statistical tool, providers or their financial managers can accurately define volume of care decreases that can be made without quality of care penalties. Such a tool can simultaneously reduce financial risk to providers and health risks to consumers within a volume-to-value healthcare environment.

The purpose of this study was to determine how changes in the ‘volume’ of processes of care services, that is, asking about tobacco use, developing self-management goals and timely testing of feet, eyes, A1C, LDL and nephropathy, affect intermediate outcomes such as controlled blood pressure, A1C and LDL at the practice level in a diabetes patient population. It also explores the implications of the findings for the volume-to-value transition process. Specifically, the study sought to answer the two following questions:

Do practices with a higher volume of processes of care, on average, have ‘better’ patient clinical outcomes, on average, using a chronicity such as diabetes as the case study?

Do practices exceeding or dropping below their average processes of care volume during a particular month also exceed or drop below their average ‘quality’ as measured by patient clinical outcomes in a month?

Methods

Data or study sample

This study is based on an analysis of data collected for the evaluation of Improving Performance in Practice, a Colorado-based programme aimed at transforming the delivery of healthcare by giving doctors the tools, systems and support they need to provide consistently high-quality care to all patients. All data were collected in 2014. Practices electing to sign up were provided with access to a Quality Improvement Coach, disease registry assistance, participation in a learning collaborative and healthcare administration consultation that was designed to elevate work flow, enhance the use of the chronic care model, increase open access scheduling, generate practice culture change and facilitate electronic medical record conversion. Monthly reports of process of care and clinical outcomes data obtained from the registries of diabetic patients at the participating practices were submitted to the study team by the practice managers. However, evaluations of this programme have revealed significant growth over time in the use of process of care measures (volume), but no growth in patient’s intermediate outcomes (quality).34 Thus, the study sought to disaggregate the larger pool of data so that the relationship between the volume of processes of care measures and value as measured by patient’s intermediate outcomes could be ascertained.

Measures

Monthly reports of process of care and clinical outcomes were obtained from registries of diabetic patients at the practices and were submitted to the study team by trained staff in each practice setting. Two composite measures were used: (1) the volume of process of care and services provided and (2) intermediate outcomes that indicated value or quality. The process of care measure was calculated as the monthly average of the per cent of patients with (1) dilated eye exams, (2) foot exams, (3) influenza vaccinations, (4) nephropathy tests, (5) smoking cessation counselling, (6) queries about tobacco use, (7) self-management goals, (8) A1C haemoglobin tests and (9) an LDL measurement in the past year. In contrast, the intermediate outcomes composite was calculated as the (1) monthly average of the per cent of patients with haemoglobin A1C measures <9, (2) per cent of patients with both systolic blood pressure measurements <130 and diastolic blood pressure measurements <80, and (3) the per cent of patients with LDL measurements <100 mg/dL.

Statistical analysis

Descriptive statistics (means, SD, maximums and minimums) were computed for practice characteristics. The date at which the practice diabetes registry was fully operational and stable was determined from the coaches and constituted time zero. A multilevel model with a time-varying covariate partitioned into a between-practice and within-practice effect35 was estimated using SAS Proc Mixed V.9.2. The equations used were as follows:

Level 1: Intermediate Outcome Compositeti=π0i+π1i(Process of Care Compositeti - Process of Care Composite.i)+eti

Level 2: π0i=β00+β01 (Process of Care Composite.i-57) +r0i

π1i=β10+r1i

Combined: Intermediate Outcome Composite ti=β00+β01 (Process of Care Composite.i - 57) +β10 (Process of Care Composite ti- Process of Care Composite.i) + roi+ r1i (Process of Care Compositeti- Process of Care Composite.i)+eti

The intermediate outcome composite measure was treated as the response variable, and the process of care composite variable was treated as a time-varying covariate partitioned into a between-subject and within-subject effect. The between-subject effect (Process of Care Composite.i) was an individual practice’s average across all time points on the process of care measures. The within-subject effect was the difference between a practice’s average process of care and the practice’s process of care measure in a particular month (Patient Process of Care Composite ti- Patient Process of Care Composite.i).

Conceptually, the between-practice effect can be explained as the effect of particular practices being generally higher or lower than other practices with regards to patient care. In comparison, the within-practice effect can be explained as the effect of a particular practice having a better or worse volume of patient care than usual in a particular month. In econometrics, these deviations from the average are referred to as shocks, innovations or errors. These terms are applicable in this context as well since the deviations may be the result of innovations such as quality improvement efforts or shocks such as loss of key personnel in the practice, and/or theoretically, efforts to reduce volume in order to prevent financial losses.

The overall patient care mean, 57, was subtracted from the between-subject effect to provide a meaningful zero (a value in the range of data) to aid in the interpretation of the intercept. Time was not included in the model as a fixed effect because previous analyses of these data have not shown a significant trend over time relative to intermediate outcomes.21 Consequently, the focus of this analysis was within practice fluctuations in intermediate outcomes rather than systematic change in intermediate outcomes.36

The intercept and the effect of within practice patient care were allowed to vary randomly. An autoregressive-moving average (ARMA (1,1)) structure was selected as the best fitting variance covariance structure in the unconditional model and served as the base model to which the process of care effects was added.37

Results

As table 1 reveals, data from 55 practices were included in the study. As this table reveals, the average number of patients with diabetes seen per month varied from a low of 4 patients to a high of 1206. However, because the patient outcome composite represents weighted rather than simple means as well as the means of means, the accompanying SD, and minimum and maximum percentages across the 55 practices do not reflect traditional horizontal and/or vertical totals.

Table 1

Practices weighted averages of their individual and composite processes of care and intermediate outcomes

Variable	n	Mean	SD	Min	Max
Patient intermediate outcomes
% patients with A1c>9.0%	55	33.2	15.2	8.4	74.9
% patients with blood pressure <130/80	55	38.6	12.0	13.1	62.1
% patients with <100 mg/dL	55	42.3	12.9	14.3	68.9
Patient outcome composite	55	49.2	10.9	21.5	69.4
Processes of care
% patients who received dilated eye exam in the past year	55	37.1	20.2	4.6	75.1
% patients with foot exam documented in the past year	55	55.0	25.0	0.0	93.9
% patients with tested for nephropathy (or exclusion)	55	58.6	21.2	11.9	96.7
% patients with self-management goal	54	49.8	32.7	0.0	96.7
% patients with queried about tobacco use	55	79.6	22.5	12.8	100.0
% patients with AIC documented in the past year	55	76.4	19.6	0.0	98.5
% patients with low-density lipoprotein measured in the past year	55	63.4	19.1	0.0	94.0
% patients with influenza vaccination documented in the past year	54	31.7	21.3	0.0	78.1
Process of care composite	55	57.2	16.5	23.3	85.6

In addition, the percentages are based on differentials in the number of data collection points per composite measure. This is because while based on a simple average the 55 practices participated in the programme for 13.9 months, the range for the period of participation spanned a period of 1–42 months. Additionally, tremendous variability characterised adherence to the recommended processes of care across the 55 practices. Stated in the language of the volume-to-value model, some of the individual practices in the sample primarily reflected a volume of diabetes services that was, on average, far below the levels for each measure that are traditionally associated with high quality in diabetes care.

For example, table 1 reveals that some of the participating practices reported that 0% of diabetes patients had received foot exams, A1C tests, LDL screening, influenza vaccinations or self-management goal-setting support during the period of the study. However, the aggregated data were not separated by period of participation in the programme. Thus, these low-service volumes may reflect the administrative fact that the period of participation in the initiative was lower for some practices than for others. Again, it is important to restate that the means, SD and ranges reported in the tables are not simple means. Rather, they are averages that are weighted by the number of patients served by each practice. Additionally, they are means that reflect the aggregate of monthly weighted means.

As mentioned, 55 practices with an average of 13.9 months and a range of 1–42 months of data were included in the analysis. When aggregated, the data consisted of 775 data points per composite measure. The average number of diabetes patients (age 18–75 years) seen per month across all practices was 195.1. However, tremendous variability existed in the size of these practices. For example, the smallest practice saw only 4 patients per month while the largest practice treated 1206 patients per month. It is for this reason that the data described in table 1 are based on weighted means, SD, and minima and maxima of the practices’ average individual and composite volumes of process of care, as well as patients’ intermediate outcome variables weighted by the number of patients served. Table 1 reveals the wide range of variation in processes of care and intermediate outcomes between the participating practices. For example, the practices had a weighted average of 49.2% on the intermediate outcome composite variable, but generated a range that extended from 21.5% to 69.4% relative to the patient outcome composite. The sheer variation in service volume and patient outcomes revealed by the descriptive data in table 1 suggests a need to assess the relationship between the volume of care delivered by these 55 practices and quality as measured by the patient’s intermediate outcomes.

Greater insight can be gained, of course, by disaggregating the process of care and intermediate outcomes data and assessing the degree of variability within the individual participating practices. Table 2 describes the findings from this aspect of the analysis. Specifically, there were approximately 775 data points for the 55 participating medical practices for each of the intermediate outcome variables. Thus, the entirety of the practices used maintained data on the intermediate outcomes. This was not the case with the processes of care variables. While the total data points equalled 775 across the samples for five of the eight processes of care variables, the data reveal that the provision of services involving self-management goals and influenza vaccinations were less likely to have been reported in any given month. While some practices fell below their own measures relative to A1C and LDL, testing influenza vaccinations and self-management goals were even more likely to exhibit variance from the mean.

Table 2

Range of practice deviations from the practice mean on individual and composite intermediate outcomes and processes of care

Variable	n	Min	Max
Intermediate outcomes
% patients with A1c>9.0%	775	−30.7	43.3
% patients with blood pressure <130/80	775	−37.1	30.8
% patients with <100 mg/dL	775	−37.9	28.6
Intermediate outcomes composite	775	−37.4	24.2
Processes of care
% patients who received dilated eye exam in the past year	775	−63.3	51.1
% patients with foot exam documented in the past year	775	−61.9	38.1
% patients with tested for nephropathy (or exclusion)	775	−85.1	33.1
% patients with self-management goal	747	−79.5	51.2
% patients with queried about tobacco use	775	−69.1	61.9
% patients with AIC documented in the past year	764	−85.8	57.0
% patients with low-density lipoprotein measured in the past year	764	−59.8	59.5
% patients with influenza vaccination documented in the past year	726	−49.0	57.3
Process of care composite	775	−47.7	22.4

AIC, haemoglobin A1C.

Even more importantly, however, table 2 summarises the range of within-practice deviations from the practice mean in intermediate outcomes and processes of care. For instance, in a given month, an individual practice could be, on average, as much as 37.4 percentage points below their usual average or as much as 24.2 percentage points above their average on the intermediate outcomes composite. Similarly, a practice could be as much as 47.7 percentage points below their process of care norm and as much as 22.4 percentage points above it. Unravelling the rationale underlying such variations becomes highly critical in a volume-to-value world. Do these variations represent patient-driven effects or are they provider-driven changes in service volume that occur as a response to changes in the shift in payment methodology? Even more importantly, are these variations in the volume of process of care services reflected in the intermediate outcomes of the patients with diabetes served by each practice?

Stated differently, ‘do volume of services decreases adversely affect quality as measured by intermediate outcomes in diabetes care?’ Table 3 reports these findings.

Table 3

Results of multilevel model predicting intermediate outcome composite

Fixed effects	Coefficient (SE)	T (df)	P value
Model for mean patient outcome (π0i)
Intercept (β00)	49.48 (1.07)	46.36 (55.8)	<0.0001
Practice mean patient care −57 (β01)	0.45 (0.06)	7.42 (53.8)	<0.0001
Model for practice mean-centred patient care (π1i)
Intercept (β10)	0.49 (0.05)	9.6444 (43.8)	<0.0001
Var in intercept	33.04 (17.20)	1.92	0.0274
Var in effect of within-person practice care	0.08 (0.03)	3.31	0.0005
Autoregressive order 1 term	0.91 (0.03)	30.28	<0.0001
Moving average  order 1 term	0.91 (0.04)	25.14	<0.0001
Var within practice	47.35 (16.51)	2.87	0.0021

As described in the ‘Methods’ section, a unique multilevel analytical approach was applied in order to more explicitly analyse the relationship between volume variations and quality outcomes as reflected in the intermediate outcomes. The model intercept of 49.48 (β00) can be interpreted as the patient outcome score for an average practice during an average month or more specifically the expected value for a practice with a patient care mean of 57, during a month when they are performing at their average. The coefficient of the practice mean patient care variable (β01) answers the question, ‘do practices with better processes of care on average have better intermediate outcomes on average?’According to this model, on average, for every additional unit or higher volume of processes of care provided relative to other practices by each practice across time, the intermediate outcomes, on average, increased by 0.45. More specifically, for every unit above the overall practice average process of care composite score of 57, the intermediate outcome score is 0.45 units higher. Accordingly, the degree of linearity between the volume of care and the intermediate outcomes in diabetes care suggests that volume to quality payment methodologies must be very carefully crafted in order to simultaneously achieve volume reductions without quality of care decreases.

This finding is further reinforced by the within-practice analysis. The coefficient of the practices’ mean process of care composite variable (β10) answers the question, ‘do practices exceeding or dropping below their average volume of processes of care during a particular month also exceed or go below their average “quality” of intermediate outcomes in that month?’ The coefficient of this term reveals that in a particular month, for each unit that a practice exceeds their average ‘service’ volume as measured by the process of care score, their intermediate outcome composite score increases by 0.48. In other words, the volume of care offered by each of the participating practices directly improved their quality as measured by the intermediate outcome composite scores.

Discussion

Principal findings

In the past decade, diabetes processes of care and intermediate outcomes improved in the USA. However, improvements in intermediate outcomes were weaker and were often not statistically significant.38 The results of this study reveal an approximately half a percentage point increase in outcomes per unit increase in care. This finding implies that slower growth in intermediate outcomes relative to processes of care is to be expected.

Yet, improvements in care do appear to affect outcomes. However, on average, two units of care in this study was the volume of care that was required to produce a single-unit change in intermediate outcomes. Stated differently, diminishing returns in outcomes suggest that higher volumes of care may be required in order to sustain the same level of quality. This suggests that pricing strategies that are independent of volume or which seek to reduce the volume of healthcare services while improving value may be inconsistent with the health outcomes production function for chronic disease areas such as diabetes.

The findings of this study are relevant to many audiences. In a non-volume-to-value environment, the implications are quite simple. The methods of this study could be adopted by researchers and programme evaluators who compare health plan performance and present it to the public. Consumers who use information of this type will benefit from knowing that providers with better care also have better outcomes. Therefore, report cards that identify providers with better care will become more helpful to patients in selecting care providers. For practitioners, this statistical approach can quickly confirm that their efforts to improve care are effective. As a result, it may also provide evidence that can motivate them to make even better improvements in care in expectation of even greater benefits to patients in terms of outcomes.

But these findings are even more relevant to volume-to-value policy advocates. This study demonstrates that while the volume of processes of care are very mutable they have direct effects on relatively immutable patient outcomes. This suggests that as the volume-to-value movement progresses, analyses such as this one are needed in virtually every single chronic disease area so that the guesswork is removed from volume-to-value payment methodologies and any volume/quality trade-offs made will be quantitatively explicit. If constructed as a calculator app, this approach will allow practice managers to quantify the volume-to-value trade-offs that are being made at any single point in time.

Limitation of the study

This research has a number of limitations. At the first level, while 55 practices were included in the study, these practices were selected based on a convenience sample. That is, a number of primary care practices were contacted and those who voluntarily decided to participate in the study were selected. Greater precision could have been introduced into the study by using practice samples that were matched in terms of practice size, patient characteristics and provider characteristics which only differed in terms of diabetes processes of care measures. However, the process of selecting and matching in primary care practice in this way was cost prohibitive.

A second limitation is that the study does not address phase number issues. That is, given the developments of an app that includes a value-to-volume statistical calculations, how difficult could it be to train practice managers in its use? How much individualisation would be needed to allow the tool to be used on a patient-by-patient basis? Thus, this article only addresses stage 1 of the process of introducing tools to support greater precision in marketing volume-to-value trade-offs.

Implications for future research

As is known, the treatment of persons with chronicities is now a dominant proportion of the services offered by primary care physicians. Thus, other studies are needed that test this statistical approach in other disease areas. In addition, studies are needed that assess the degree to which findings from the use of this volume-to-value apps would actually drive change in the behaviour of providers and their peer staffs. In addition, case study research is needed that focuses on the intricacies involved in generating volume/value trade-offs in a single medical practice. Therefore, while this study was based on data from an actual diabetes intervention with the 55 medical practices, the focus of the intervention was not volume/value trade-offs. Thus, more research is needed that explicitly addresses this matter. The future studies can also be stratified by age and other variables in order to determine subgroup differences (eg, healthcare disparities) in volume/value trade-offs.

Conclusions

Despite the limitations of this study and the intensity of the need for additional research, this research, nevertheless, is important. It demonstrates the criticality of exporting the focus of healthcare administration into clinical practices. The need for such integration will be even greater in the future as policies continuously get implemented that have direct impacts on medical practices.