Engineering practice variation through provider agreement: a cluster-randomized feasibility trial.

PURPOSE: Minimal-risk randomized trials that can be embedded in practice could facilitate learning health-care systems. A cluster-randomized design was proposed to compare treatment strategies by assigning clusters (eg, providers) to "favor" a particular drug, with providers retaining autonomy for specific patients. Patient informed consent might be waived, broadening inclusion. However, it is not known if providers will adhere to the assignment or whether institutional review boards will waive consent. We evaluated the feasibility of this trial design. SUBJECTS AND METHODS: Agreeable providers were randomized to "favor" either hydrochlorothiazide or chlorthalidone when starting patients on thiazide-type therapy for hypertension. The assignment applied when the provider had already decided to start a thiazide, and providers could deviate from the strategy as needed. Prescriptions were aggregated to produce a provider strategy-adherence rate.
RESULTS: All four institutional review boards waived documentation of patient consent. Providers (n=18) followed their assigned strategy for most of their new thiazide prescriptions (n=138 patients). In the "favor hydrochlorothiazide" group, there was 99% adherence to that strategy. In the "favor chlorthalidone" group, chlorthalidone comprised 77% of new thiazide starts, up from 1% in the pre-study period. When the assigned strategy was followed, dosing in the recommended range was 48% for hydrochlorothiazide (25-50 mg/day) and 100% for chlorthalidone (12.5-25.0 mg/day). Providers were motivated to participate by a desire to contribute to a comparative effectiveness study. A study promotional mug, provider information letter, and interactions with the site investigator were identified as most helpful in reminding providers of their study drug strategy.
CONCLUSION: Providers prescribed according to an assigned drug-choice strategy most of the time for the purpose of a comparative effectiveness study. This simple design could facilitate research participation and behavior change in non-research clinicians. Waiver of patient consent can broaden the representation of patients, providers, and settings.

RCT Entities:   Population Interventions Outcomes

Introduction

The US Congressional Budget Office called for comparative effectiveness research (CER) and reviewed current methodology,1 including systematic reviews, analysis of secondary data using observational methods, and generation of primary data (eg, in randomized controlled trials [RCTs]). While randomized designs are recognized as the gold standard for reducing the potential for bias, the traditional RCT has limitations in timeliness, feasibility, and generalizability.2–5 Alternatives are sought that incorporate the strengths of randomization but minimize the weaknesses of the traditional RCT.3,4,6 In particular, designs that can be embedded into health-care operations3,7 and that can involve non-research clinicians in the trial3 are expected to be more sustainable3,4,6 than traditional trials. For example, Fröbert and colleagues used registry data to capture outcomes in a RCT.8 D’Avoli and colleagues designed a RCT that gives providers the option of the trial for their patients at the point of care and relies on observational data.9 Although these novel designs represent important advances, the attendant consent and randomization of individuals necessitates infrastructure to accomplish this and limits generalizability to consenters.

Policy trials10 randomize clusters of patients – for example, by physicians10 or facilities.11,12 These can be structured to be of sufficiently low risk and complexity that individual patient consent may be waived.13 Thus, cluster-randomized trials (CRTs) represent another possibility for incorporating CER into routine practice. Furthermore, if simple enough, they can serve to broaden the scope to include settings, providers, and patients that might otherwise not be represented in comparative effectiveness randomized trials. Sabin and coworkers believe that ethical concerns for patient protection have limited the use of CRTs for comparing drug effectiveness, so they specifically examined the ethics of CRTs for this purpose.14 Because the process of obtaining informed consent may result in a study sample that does not reflect the general population, Sabin’s group focused on the possibility of waiving patient informed consent. They concluded that for a study that does not require research-specific procedures or visits, and for which the provider retains autonomy in therapeutic decision making, patient consent could be waived. They proposed a design in which clusters (eg, providers) are randomized to “favor” treatment A or treatment B when individual providers themselves see no reason to prefer one treatment over the other. If the randomized groups subsequently differed substantially in their use of A and B, inferences could be made, at a population level, on the relative value of the “favor A” strategy versus “favor B”.

The design proposed by Sabin and colleagues is intriguing but untested. Will institutional review boards (IRBs) waive patient informed consent? Will participating clusters actually favor the strategy to which they were assigned? We conducted a feasibility study to address these questions, using a comparison of chlorthalidone versus hydrochlorothiazide (HCTZ) as the candidate CER question. Interest in formally comparing these drugs is evidenced by the decision from the US Veterans Affairs Cooperative Studies Program to fund the planning of such a RCT (personal communication, Grant Huang) while our study was ongoing. There is good evidence that chlorthalidone reduces the morbidity and mortality associated with hypertension, yet strong evidence is lacking for HCTZ.15,16 Despite this, chlorthalidone is used much less often than HCTZ.16,17 We invited providers to participate in a pilot study to assess the feasibility of the CRT design for investigating this clinical question.

Methods

Design overview

The study was coordinated by the Pharmacy Benefits Management (PBM) Services of the US Veterans Health Administration (VHA). Agreeable providers at three facilities were randomized to either “favor chlorthalidone” or “favor hydrochlorothiazide”. This assignment was to apply when the provider had already decided to start a patient on a thiazide-type drug and thought that both options were equally good for the patient. Prescription patterns were followed using administrative data. We use the term “thiazide” in this report to indicate either chlorthalidone or HCTZ as a single-drug product.

Study setting

The VHA is an integrated health-care system that uses a team-based model for care.18 In 2004, new HCTZ prescriptions outnumbered new chlorthalidone prescriptions by over 50-fold in the VHA.17

Recruitment and eligibility

Site investigators at participating facilities were asked to recruit a local opinion leader (influential peer to champion the need for the study) and providers who prescribe thiazides for hypertension. There were no specific criteria for recruitment, other than provider willingness and an anticipated prescribing rate for thiazides of at least one new start per month for a provider. The coordinating center then contacted each provider by email to confirm their desire to participate.

Randomization

Randomization was stratified by facility. In facilities with at least two physicians and two non-physicians, randomization was further stratified by provider category. For strata with an even number of providers, the ratio of chlorthalidone to HCTZ was 1:1. For strata with an uneven number, the number randomized to chlorthalidone was increased by one. Proc Plan (SAS; v 9.2, SAS Institute Inc., Cary, NC, USA) was used to generate assignments for different strata sizes.

Intervention

The providers were informed of their assigned strategy by the site investigator. The investigator delivered to the provider a study mug with the assignment and recommended dose range imprinted (“Favor Chlorthalidone 12.5–25 mg/day” or “Favor Hydrochlorothiazide 25–50 mg/day”). The doses are those recommended by the VHA Pharmacy Benefits Management Services, based on clinical trial evidence.19 The day the provider received the mug was the start date for the 9-month intervention period for that provider/subject. Patient information letters were made available for provider use, as desired, and if permitted by the local IRB. It was left to the provider’s discretion whether to inform patients about the study. After mug delivery to the provider, the coordinating center emailed each provider to confirm the strategy assignment and to inform the provider of his/her confidential study identifier (StudyID) that would be used in the monthly reports. The monthly reports were sent to each provider in the body of an email. The report was formatted as a table with all providers included but identified by StudyID instead of by name. Each provider’s thiazide prescription activity in the prior month was enumerated, along with the percent of new thiazide starts that were for the assigned cluster drug.

Patient-prescription eligibility

Providers were not asked to apply eligibility criteria to patients. The criteria were applied during the analytic phase.

Inclusion criteria:

Receipt of a new outpatient VHA prescription for chlorthalidone or HCTZ (single-medication product) by a participating provider in the intervention period. The date of this prescription was the index date.

An inpatient or outpatient diagnosis of hypertension at a VHA visit within the year prior to, or including, the index date.

Exclusion criterion:

Any prescription for chlorthalidone or HCTZ in the 2 years prior to the index date.

These criteria were prespecified, except that the exclusion for prior use was changed from 1 year to 2 years. This change was made to better capture prior use in the face of inconsistent refill patterns. We screened for eligible patients using the administrative databases (see the “Data sources and variable definitions” section). Because it is known that prescriptions filled outside the VHA are incompletely represented in VHA databases, we also examined the electronic medical records (pharmacy file and progress notes) for mention of thiazide use in the historical period. For this review, we selected patients with no VHA hypertension prescriptions at the time of the index thiazide prescription. We reasoned that this sub-cohort might include patients transferring from non-VHA care and now obtaining prescriptions from the VHA. They would represent new users of VHA medications but not necessarily new users of thiazides. We conducted sensitivity analyses with and without patients identified through this review.

Data sources and variable definitions

Prescription, diagnostic, and identifier data for all patients were obtained from the VHA electronic administrative databases.20 The sources were the national patient files (inpatient and outpatient visits, and demographic information) and the prescription data from the PBM database (PBM v 3.0). Data from these databases were linked via unique patient identifiers. Medical-record review was conducted via centralized access to the electronic medical record.20

Patient-level outcomes were followed for 6 months after the start of thiazide therapy to capture dosing changes and drug nonpersistence.17 At 6 months after the patient started the thiazide, if there was a gap in medication possession that exceeded more than two times the days-supply value of the most recent fill, the patient was considered nonpersistent. For example, if the days-supply was 30 days, nonpersistence would be coded if 60 days or more had elapsed without a fill. A change to a combination product that contained the starting drug was not considered nonpersistence. Possible adverse drug reactions were identified using International Classification of Diseases, Ninth Revision, Clinical Modification21 visit diagnosis codes. These codes included: E942 (adverse effects associated with therapeutic use of cardiovascular drugs), 995.20 (adverse effect not otherwise specified [NOS]), 276.7 (hyperpotassemia), 276.8 (hypopotassemia), 275.2 (disorders of magnesium metabolism), 275.42 (hypercalcemia), and 790.29 (other abnormal glucose). A code for an adverse event was considered associated with the starting drug if the event date was during an active fill period or within 30 days of the end of the fill period and there were no further fills after the event date.

At the conclusion of the intervention, each site investigator completed a survey on provider motivation for participation and the value of study reminders about the cluster drug and dose. Site investigators based their responses on their interactions and conversations with providers. Respondents were also asked to list three barriers and facilitators the providers faced in following their cluster assignment. The initial survey was followed by a conference call with all site investigators to better define and explore these text responses. The themes from this call were then included in a second survey to determine those that the majority agreed were barriers/facilitators.

Pubmed22 was searched for provider publications in the years 2002–2012 with the following criteria: species: human, language: English, with an abstract, excluding reviews and letters. Matches of authors to study providers were made based on location (State), and last name and first initial.

Analysis

Estimation, not hypothesis testing, was the study goal. Therefore, point estimates and measures of variability are presented. The provider was the unit of analysis for the primary outcome (ie, percent of a provider’s prescriptions that were for the cluster drug), which accounts for the clustering. SAS (v 9.2) was used for statistical analysis.

Regulatory

The study was approved by the IRBs of the three participating sites and the coordinating center. All four granted waivers of patient consent and Health Insurance Portability and Accountability Act of 1996 authorization for use of patient data. At two of the three sites enrolling clusters, IRBs approved distribution of the patient information letter at the provider’s discretion. The third site did not want the patient letter distributed. The procedure for enrollment of providers was approved as a provider consent waiver at three sites and as an alteration of consent procedures at one. The trial was registered with Clinicaltrials.gov (NCT 01235377).23

Results

Three facilities (two medical centers, and one outpatient clinic) enrolled 20 providers/clusters, and two facilities identified local opinion leaders. Because of the pragmatic design, site investigators were not asked to record the number of providers screened and reasons for nonparticipation. The flow of clusters from randomization to analysis is shown in Figure 1. The final number of providers was 18, because two did not start any patients on a thiazide.

Figure 1

Flow of study subjects.

Abbreviation: VHA, US Veterans Health Administration.

Table 1 includes descriptions of the facilities, providers, and patients. Facility size is represented by the number of unique patients filling at least one outpatient prescription (for any medication) in a year. None of the 18 providers had published a clinical study in the 10 years prior to the trial. In the 2 years prior to the trial, providers largely chose HCTZ over chlorthalidone. On average, the historical percentage of provider new starts on chlorthalidone was 1% or less. Patient demographic characteristics appear well balanced. Approximately half of the patients did not have medical treatment for their hypertension when starting their study thiazide.

Table 1

Baseline characteristics of study sample

Characteristic	Cluster assignment
	Favor chlorthalidone	Favor HCTZ
Facilities: patients with at least one outpatient prescriptiona (count range)	2,818–31,511	2,818–31,511
Providers: number participating, with at least one prescription	10	8
Publication count 2002–2012 (median, IQR)	0 (0–0)	0 (0–0)
Prior chlorthalidone useb (mean, SD)	1.0 (2.5)	0.5 (1.0)
Patients: number totaled over clusters	77	61
Age, years (mean, SD)	61 (10.8)	60 (13.1)
Male (%)	99	98
Race: white (%)	69	69
Antihypertensive drug count categories (%)c
0	47	49
1	25	25
≥2	29	26

Notes:

For any medication, in 2012;

provider summary: % of new starts in 2010–2011;

active Veterans Health Administration prescriptions in 30 days prior to thiazide start.

Abbreviations: HCTZ, hydrochlorothiazide; IQR, interquartile range; SD, standard deviation.

Provider-level outcomes are shown in Table 2. Over the full 9-month intervention period, those randomized to favor chlorthalidone actually used that drug for most (80%) of their new starts. However, the monthly reports that were part of the intervention revealed an abrupt change in prescribing in some providers during follow-up. This is shown by the provider summary measures during the first and second halves of the intervention period. Adherence to the assigned strategy was high during the first half but dropped during the second half. This was due largely to a change in prescribing in three providers at one facility. Based on the prescription dates, it appeared that one provider led the change in behavior at this site. When site investigators debriefed their providers after the intervention, the site investigator from this facility indicated one barrier was:

Table 2

Prescribing outcomes of providers (n=18) over 9 months intervention

Outcomes	Cluster assignment
	Favor chlorthalidone	Favor HCTZ
	n=10	n=8
	Median (interquartile range)	Median (interquartile range)
New thiazide starts by provider (count)	7 (4.0–11.0)	7 (2.5–11.5)
Use of cluster drug (% of provider’s new starts)	80 (54.5–100)	100 (100–100)
During first half of intervention period	100 (83.3–100)	100 (100–100)
During second half of intervention period	63 (10.0–100.0)	100 (100–100)
Within physicians (n=108 prescriptions)	65 (50.0–85.7)	100 (100–100)
Within non-physicians (n=30 prescriptions)	100 (83.3–100.0)	100 (88.9–100.0)
Starting dose in recommended rangea	100 (100–100)	28 (0.0–66.7)
Ever dosed in recommended rangeb	100 (100–100)	45 (22.5–75.0)

Notes:

Percentage of new starts on cluster drug;

percentage of new starts on cluster drug, within 6 months of starting drug.

Abbreviation: HCTZ, hydrochlorothiazide.

Not getting good BP [blood pressure] control with the chlorthalidone (provider switched back to hctz half way thru study – for each new rx [prescription]).

As expected, those randomized to HCTZ used almost exclusively that drug. In the sensitivity analysis, we found that adherence to the assigned strategy was 73% for “favor chlorthalidone” and 100% for “favor HCTZ” when patients identified in the medical-record review as having prior thiazide therapy (n=15) were not excluded.

The pattern regarding use of the recommended dose was reversed. Providers in the chlorthalidone group always used recommended doses when starting a patient on the cluster drug. Providers in the HCTZ group dosed in the recommended range in only about a quarter of their new starts. This improved during the 6 months of follow-up.

Prescriptions were examined at the patient level over 6 months of follow-up. Decreases in dose (from the highest achieved during follow-up) were uncommon (two patients, both in the “favor HCTZ” group). Approximately a third of patients (45 of 138) met the criteria for nonpersistence, similar to other findings.17 In the “favor chlorthalidone” group, 37% of those started on chlorthalidone were nonpersistent, while 46% of this group who were started on HCTZ (the non-cluster drug) were nonpersistent with their starting drug. In the favor HCTZ group, 23% were nonpersistent (all HCTZ). We explored two factors potentially associated with nonpersistence that could be analyzed objectively with administrative data (ie, change in provider and adverse events). In 13 patients (9%), a different provider took over ordering the thiazide fills, yet the starting thiazide was continued in all but one patient. There were two patients (both prescribed chlorthalidone) with adverse events (both “other abnormal glucose”) associated with nonpersistence.

Findings from the first survey of site investigators regarding provider motivation and study reminders are shown in Table 3. Additional factors were elicited from the site investigators and included in a second survey to ascertain scope of agreement (see the “Methods” section). Those factors from the second survey for which there was a majority agreement included the facilitators “prospect of better blood pressure control” and “site investigator enthusiasm”, and a barrier, “provider having a habit of a different thiazide or dose”.

Table 3

Survey of site investigators

Factor	Median response*
What factors did providers report were important in their decision to participate in the trial?
To contribute to a comparative effectiveness study	Very important
The literature evidence that important questions remain for the two thiazides	Somewhat important
The opinion of colleagues	Somewhat important
For providers randomized to a thiazide other than their pre-study routine choice: What factors were helpful in reminding them of their study drug?
Study beverage mug/cup	Very helpful
Monthly reports	Somewhat helpful
Provider information letter	Very helpful
Patient information letter	Somewhat helpful
Face-to-face interactions with the site investigator	Very helpful

Note:

Response categories: very, somewhat, not at all.

Discussion

The study was designed to evaluate the feasibility of the pragmatic trial design proposed by Sabin and colleagues for use in comparing the effectiveness of drug-choice strategies.14 The clinical question chosen is one for which there has been renewed interest, given the discrepancy between the evidence base and current practice.15,16 The strategy that was assigned randomly to providers applied when (1) the provider had already made the decision to start a thiazide-type medication, and (2) when the provider thought that either chlorthalidone or HCTZ would be equally good for the patient. Providers were free to deviate as needed for initial therapy, as well as to modify therapy subsequently. All four IRBs judged this design to be eligible for waiver of patient informed consent. This is consistent with recent calls for a more measured approach to minimal-risk trials.24,25 Prior to the study, the participating providers strongly favored HCTZ over chlorthalidone, similar to other reports.16,17 Despite this, those then randomized to favor chlorthalidone did so. The most important motivation for the providers to participate was the desire to contribute to a comparative effectiveness study. The providers had no recent history of publishing in the clinical literature; thus, the study was able to extend interventional research into a primarily practice setting.

Our procedures for influencing provider prescribing behavior incorporated standard components such as an opinion leader, education, and feedback.26,27 While opinion leaders were identified at study initiation, these people were not mentioned in the surveys as having played an influential role. Interactions with the site investigators were reported to be very helpful, so it may be that the site investigators essentially substituted for the opinion leaders. The provider information letter was reported to be very helpful, but the feedback in the form of monthly reports was less helpful. The monthly feedback was hampered by the need to avoid naming providers in the reports (a requirement of the coordinating center IRB). This necessitated use of a coded identifier that providers had to use to find their own information. In post-study discussions, site investigators suggested individualized feedback via personal emails or mailed paper reports for future studies.

In assessing the generalizability of our findings, it must be noted that the VHA’s team model includes an active role for pharmacists.18 Pharmacists advise physicians on pharmacotherapy, and some have prescribing authority in their scope of practice. Three of the four site investigators were pharmacists, as were six of the 18 providers. The coordinating center was under the centralized pharmacy benefits management service for the VHA28 that facilitates evidence-based as well as cost-effective therapy. Also having an impact on generalizability is our choice of a pair of older drugs where there is evidence that the more popular drug (HCTZ) is less effective on the surrogate (ie, blood pressure16,29) and major outcome measures.15,16 The prior habit of prescribing HCTZ may have made it easy for an occasion of poor response in a patient prescribed chlorthalidone to cause that provider to lose equipoise and revert to the HCTZ habit, possibly influencing others at the site to do likewise.

In proposing a CRT in which groups are assigned to favor specific drugs, Sabin and colleagues did not expect that all members of a cluster would be treated according to that group’s assigned strategy. Thus, inferences at the individual patient level are not the primary goal; rather, the goal is to make inferences about the relative value of different treatment approaches at a population level.14 Indeed, in the Multiple Risk Factor Intervention Trial,30 there were several significant trends associated with whether HCTZ or chlorthalidone was the thiazide used “predominantly” in a study clinic (although “predominantly” was not quantitatively defined). The larger the separation in use patterns in the assigned groups, the greater the power of the trial, and the more straightforward the interpretation.7 If inferences at the patient level are of interest, the data could also be analyzed using observational techniques such as instrumental variables.5 Because much of the practice variation would have been engineered using random assignment, residual confounding by indication would be expected to be reduced compared with a sample that was not so engineered.

Our study was limited by the low number of new starts for many providers. In addition, to encourage participation by non-research clinicians, the pragmatic design necessitated putting little research burden on the providers. Thus, by design, our information about influences on provider behavior came from the site investigators. Finally, the providers were volunteers. Their behavior and patients may not be reflective of the larger population of providers.

This design shows promise as a means of engineering practice variation to facilitate comparisons of treatment approaches. The 9-month 80% adherence to the “favor chlorthalidone” strategy represents an 80-fold increase in use. This provider adherence compares well to benchmarks for patient-level adherence in clinical trials.31,32 However, future studies might consider some modifications to maintain the median 100% adherence to the assigned strategy that was seen in the first half of the trial. Randomizing clusters defined at a higher level (eg, at the clinic or facility level) could involve all participants who may influence prescriptions (eg, nurses, pharmacists). Reminders could then be more widely available (eg, posters, office supplies) within the broader cluster, with less concern about contamination of other clusters. The experience of a group of providers might serve to modulate the reaction of an individual provider with a spurious unsatisfactory response. Additionally, IRBs may grant permission to distribute feedback along with naming the cluster, since the cluster would not be an individual. The feedback would then be more useful and might stimulate conversation and interest. A disadvantage to randomizing at a higher level is that the number of clusters would likely be much reduced. This, in turn, reduces power and the ability to balance prognostic factors.7,13 A second modification could be a contingency plan, prespecified in the protocol, for responding to an apparent loss of equipoise. This would probably necessitate discontinuing a provider who has lost equipoise (as defined operationally by a low adherence). However, to preserve the integrity of the randomization, it would be necessary to have randomized in pairs, and then discontinue both members of a pair when one meets discontinuation criteria.13 Clearly, this would be costly to power and increase complexity,13 thus would need careful consideration a priori.

Conclusion

The CRT design that Sabin and colleagues propose shows promise. IRBs were willing to waive patient consent, permitting the inclusion of patients who might not otherwise be represented in the trial literature. Further broadening trial representation, non-research providers were willing and able to adhere to a study assignment to favor a particular drug. It was possible to implement the trial in a routine care setting, using only internal funds. Further work should be directed at confirming our findings in a larger number of providers, with different candidate drugs, and possibly defining clusters at a higher organizational level.