Quality improvement needed in quality improvement randomised trials: systematic review of interventions to improve care in diabetes.

OBJECTIVE: Despite the increasing numbers of published trials of quality improvement (QI) interventions in diabetes, little is known about the risk of bias in this literature.
DESIGN: Secondary analysis of a systematic review. DATA SOURCES: Medline, the Cochrane Effective Practice and Organisation of Care (EPOC) database (from inception to July 2010) and references of included studies. ELIGIBILITY CRITERIA: Randomised trials assessing 11 predefined QI strategies or financial incentives targeting health systems, healthcare professionals or patients to improve the management of adult outpatients with diabetes. ANALYSIS: Risk of bias (low, unclear or high) was assessed for the 142 trials in the review across nine domains using the EPOC version of the Cochrane Risk of Bias Tool. We used Cochran-Armitage tests for trends to evaluate the improvement over time.
RESULTS: There was no significant improvement over time in any of the risk of bias domains. Attrition bias (loss to follow-up) was the most common source of bias, with 24 trials (17%) having high risk of bias due to incomplete outcome data. Overall, 69 trials (49%) had at least one domain with high risk of bias. Inadequate reporting frequently hampered the risk of bias assessment: allocation sequence was unclear in 82 trials (58%) and allocation concealment was unclear in 78 trials (55%). There were significant reductions neither in the proportions of studies at high risk of bias over time nor in the adequacy of reporting of risk of bias domains.
CONCLUSIONS: Nearly half of the included QI trials in this review were judged to have high risk of bias. Such trials have serious limitations that put the findings in question and therefore inhibit evidence-based QI. There is a need to limit the potential for bias when conducting QI trials and improve the quality of reporting of QI trials so that stakeholders have adequate evidence for implementation.

Reliable quality improvement research is needed to make decisions about initiating or scaling up quality improvement strategies.

The number of published quality improvement trials has increased rapidly over time.

The quality of trials published in other areas of health seem to be improving over time but the risk of bias in the quality improvement literature is uncertain.

Nearly half of quality improvement trials for diabetes are at high risk of bias.

The quality of quality improvement trials does not seem to be improving over time.

Policy-makers, administrators, clinicians and research funders must carefully scrutinize the methods used in quality improvement trials to ensure evidence-based quality improvement.

This is the largest systematic review of risk of bias in the quality improvement literature and the only to assess for trends over time.

The risk of bias tool does not capture all sources of methodological bias and poor reporting interferes with the assessment of many domains.

The merits of any given trial report depend to some extent on the needs of the reader, such that some trials with high risk of bias may be of value for certain purposes.

Introduction

There is significant interest in quality improvement (QI) in healthcare, as evidenced by the rapidly increasing number of randomised clinical trials (RCTs) of QI interventions, especially in the diabetes literature.1 RCTs can provide a foundation for making statements regarding causation, but the validity of trials varies widely; trials with adequate allocation concealment and blinding generally produce smaller effect sizes.2 Since internal validity in QI trials is a necessary precursor for application to other settings,3 the ‘risk of bias’ of the findings should be assessed to ascertain the utility of the trial results. When an RCT is deemed to have high risk of bias, the study's findings become questionable.4

Evaluations to assess trends in methodological quality of RCTs have been conducted in many fields of healthcare,5 but no previous reviews have assessed risk of bias in QI RCTs or whether risk of bias in QI RCTs has changed over time. Recently, we conducted a systematic review and metaregression that included 142 RCTs evaluating QI strategies to improve care for patients with diabetes.1 In this secondary analysis of those data, we aimed to examine the risk of bias of included studies using the Cochrane Risk of Bias tool developed by the Cochrane Effective Practice and Organisation of Care (EPOC) group6 and determine whether the proportion with high risk of bias decreased over time. We also evaluated the trial and publication characteristics that might be associated with high risk of bias. Finally, we assessed whether the adequacy of reporting of risk of bias domains improved over time.

Methods

A detailed description of the methods used for searching, screening and abstracting the relevant data has been published1 and is briefly summarised here.

Search strategy

Studies were identified by searching MEDLINE and the Cochrane EPOC database (up to July 2010), and screening references of included RCTs. The search strategy has been previously published1 and is available on request.

Study selection

RCTs examining 1 of the 11 predefined QI strategies, and/or financial incentives, targeting health systems and/or healthcare professionals for the management of adult outpatients with diabetes were included. RCTs had to report at least one of the chosen process of care measures (proportion of patients taking acetylsalicylic acid, statins, antihypertensive medication, screened for retinopathy, screened for foot abnormalities and monitored for renal function) or intermediate outcomes (glycosylated haemoglobin levels, low-density lipoprotein cholesterol levels, diastolic and systolic blood pressure, proportion of patients with controlled hypertension and proportion of patients who quit smoking) for inclusion.

Data abstraction

A draft data abstraction form was developed and modified after a training exercise among reviewers. Two reviewers abstracted relevant data for each RCT independently. Discrepancies were resolved by discussion or the involvement of a third reviewer. Authors of the included RCTs were contacted to obtain further information for data items requiring clarification. Journal impact factors from journal citation reports (ISI Web of Science, 2009) were obtained. When a journal's ranking was unavailable, we used the impact ranking of the open access SMImago journal and country rank database if available.7 This ranking is calculated using a similar formula and is strongly correlated with the journal citation impact factor.8

Assessing risk of bias

As the included trials tested QI interventions, the Cochrane EPOC Risk of Bias Tool6 was used to assess the risk of bias in each study. The standard Cochrane Risk of Bias Tool includes an assessment of seven domains: sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective reporting and others. The Cochrane Handbook9 provides instructions for making judgements about the specific domains as high, unclear or low risk. When formulating summary assessments for each trial, classification of a study as ‘high risk’ indicates that bias could have affected the results, while unclear risk of bias indicates that some doubt exists about the results, and low risk of bias indicates that bias is unlikely to affect the results. It has been shown empirically that studies classified as high risk using this tool are more likely to have larger effect sizes.10

The EPOC tool was adapted to account for the unique features of QI trials. (The guidelines for applying the Cochrane EPOC tool are summarised in table 1.) For example, in many QI trials, it is not possible to blind participants. In addition, QI trials may require cluster-randomisation to avoid contamination, but in cluster-randomised trials balance at baseline is a particular concern.11 Therefore, the EPOC tool uses the same approach as the general Cochrane Risk of Bias Tool, but requires an assessment of bias in nine domains: sequence generation, allocation concealment, similarity of baseline measurements, similarity of baseline characteristics, incomplete data, blinding of outcome assessment, contamination, selective outcome reporting and others. If a given domain is deemed ‘unclear’, it was inadequately reported to determine whether it meets high risk or low risk criteria. Risk of bias assessment was conducted independently by a clinician-researcher (NMI) and a systematic review methodologist (ACT), and conflicts were resolved by discussion with an expert QI trialist (JMG).

Table 1

Cochrane Effective Practice and Organisation of Care risk of bias assessment tool*

Risk of bias domain	Low risk of bias	High risk of bias	Unclear risk of bias
Was the allocation sequence adequately generated?	A random component in the sequence generation process is described (eg, referring to a random number table)	Non-random method is used (eg, performed by date of admission)	Not specified in the paper
Was the allocation adequately concealed?	The unit of allocation was by institution, team or professional and allocation was performed on all units at the start of the study; or if the unit of allocation was by patient or episode of care and there was some form of centralised randomisation scheme, an on-site computer system or sealed opaque envelopes were used	Allocation was not adequately concealed	Not specified in the paper
Were baseline outcome measurements similar?	Performance or patient outcomes were measured prior to the intervention, and no important differences were present across study groups, or if imbalanced but appropriate adjusted analysis was performed	Important differences were present and not adjusted for in analysis	If no baseline measure of outcome
Were baseline characteristics similar?	Baseline characteristics of the study and control providers are reported and similar	No report of characteristics in text or tables or if there are differences between control and intervention providers	Not clear in the paper
Were incomplete outcome data adequately addressed?	Missing outcome measures were unlikely to bias the results	Missing outcome data was likely to bias the results	Not specified in the paper
Was knowledge of the allocated interventions adequately prevented?	The authors state explicitly that the primary outcome variables were assessed blindly, or the outcomes are objective (eg, length of hospital stay)	Outcomes were not assessed blindly and not objective	Not specified in the paper
Was the study free from selective outcome reporting?	There is no evidence that outcomes were selectively reported	Some important outcomes are omitted from the results	Not specified in the paper
Was the study adequately protected against contamination?	Allocation was by community, institution or practice and it is unlikely that the control group received the intervention.	It is likely that the control group received the intervention.	Communication between intervention and control professionals could have occurred
Was the study free from other risks of bias?	There is no evidence of other risk of biases

*Adapted for ease of presentation. See http://epoc.cochrane.org/epoc-resources-review-authors for full explanation.

Analysis

For each risk of bias domain, the proportions of RCTs meeting the criteria for high or low or unclear risk of bias were determined. To assess for trends over time in the bias classifications, year of publication was categorised into three groups demarcated by the publication of the 2001 CONSORT statement12 and the publication of the earlier version of the systematic review of diabetes QI interventions in 2006,13 as we believed that these may have spurred investigators to improve the quality of their trial. Therefore, we categorised the year of publication as before 2002; 2002–2006; and 2007–2010. We examined each of the risk of bias domains for change over time descriptively and conducted either exact or asymptotic Cochran-Armitage tests for trend for each item.

We estimated the proportion of QI RCTs at high risk of bias overall, together with 95% asymptotic CI. For this analysis, we created a dichotomous indicator for each RCT based on whether or not the study was classified as high risk of bias in at least one domain. To assess for trends in reporting over time, we dichotomised domains as ‘reported’ (low or high risk of bias) and ‘unreported’ (unclear risk of bias). We tested for trend over time in the proportion at high risk of bias overall, hypothesising that the proportion would decline over time. We used the same year of publication categories and conducted Cochran-Armitage tests for trend of the dichotomous indicator.

We also conducted a post hoc sensitivity analysis that applied an empirically based rule for assigning high risk of bias overall. Since previous meta-analyses have found that high risk of bias in four specific domains, namely allocation sequence generation, allocation concealment, blinding and selective outcome reporting are each associated with greater effect size,14–16 we repeated the analyses considering only studies with high risk of bias in these domains as high risk of bias overall.

Finally, we tested for associations between high risk of bias in at least one domain and study characteristics chosen a priori: type of diabetes (type 1, type 2, both or unclear), type of allocation (cluster randomised or patient randomised), country (USA or Canada, UK or Western Europe or others), type of intervention (single or multifaceted), journal impact factor, effective sample size and year of publication using χ2 tests (or Fisher's exact tests, as appropriate) for categorical and Wilcoxon signed-rank tests for continuous measures. We hypothesised that each of these characteristics may be associated with studies at high risk of bias overall.

All analyses were conducted in SAS V.9.2.17

Results

See figure 1 for a study flow diagram.

Figure 1

Study flow diagram.

We analysed 142 studies, with 37 (26%) published before 2002, 46 (32%) between 2002 and 2006, and 59 (42%) between 2007 and 2010. These studies evaluated the effects of QI interventions on 123 529 patients with diabetes. Trial and patient characteristics are described in table 2. The proportions of studies judged to be at low, unclear or high risk of bias for each domain are illustrated in figure 2. The domains most commonly at high risk of bias were outcome reporting bias (17%) and similarity across characteristics at baseline (16%). A lack of similarity in outcome measures at baseline (10%), and lack of adequate blinding (8%) were also relatively common domains with high risk of bias. Studies were rarely at high risk of bias due to the allocation sequence generation (4%) or allocation concealment (3%), but these domains were often unclearly reported (57% and 55% unclear, respectively). Selective outcome reporting was deemed unclear approximately 84% of the time because published protocols were rarely available and it was often plausible that many more outcomes than those reported were measured. Table 3 indicates a lack of significant trend over time in the proportion of trials at high risk of bias for any given domain. Examination of table 3 also reveals no trends over time in quality of reporting for any of the risk of bias domains.

Table 2

Study and patient characteristics

Characteristic	Result
Patient RCTs, n (%)	94 (66.2)
Cluster RCTs, n (%)	48 (33.8)
Number of clusters, median (IQR)	29 (12, 57)
Number of patients, median (IQR)	405.3 (203, 878)
Duration of intervention months, median (IQR)	12 (8.9, 15.0)
Mean age in years, median (IQR)	59.4 (54.9, 62.9)
Percentage of male, median (IQR)	49.8 (41.8, 55.9)
Type of diabetes n (%)
Type 1 diabetes	9 (6.3)
Type 2 diabetes	80 (56.3)
Type 1 and 2 diabetes	34 (23.9)
Type of diabetes unclear/NR	19 (13.4)
Number of QIs per RCT median (IQR)	2 (0, 3.5)
Administrators of patient intervention(s), n (%)
Primary care physician	30 (21.1)
Nurse	67 (47.2)
Pharmacist	19 (13.4)
Dietician	22 (15.5)
Psychiatrist	3 (2.1)
Psychologist	2 (1.4)
Ophthalmologist	2 (1.4)
Specialist/endocrinologist	21 (14.8)
Other	49 (34.5)
Location of study, n (%)
The USA	68 (47.9)
The UK	14 (9.9)
Canada	11 (7.7)
The Netherlands	8 (5.6)
South Korea	7 (4.9)
Australia	6 (4.2)
Denmark	3 (2.1)
Belgium	1 (0.7)
Israel	3 (2.1)
Spain	3 (2.1)
Norway	2 (1.4)
France	2 (1.4)
Germany	2 (1.4)
Italy	2 (1.4)
Switzerland	2 (1.4)
China	2 (1.4)
Ireland	1 (0.7)
New Zealand	1 (0.7)
Thailand	1 (0.7)
Taiwan	1 (0.7)
United Arab Emirates	1 (0.7)
Mexico	1 (0.7)

All IQRs reported as the 25th and 75th percentiles, includes investigators and community workers.

N, number; NA, not applicable; NR, not reported; QI, quality improvement; RCT, randomised clinical trial.

Figure 2

Percentage of studies judged to be at low, unclear or high risk of bias in each risk of bias domain.

Table 3

Trends over time in proportions of trials classified high, unclear or low for each risk of bias domain

Risk of bias domain	Pre-2002 N=37	2002–2006 N=46	2007–2010 N=59	p Value*	p Value†
Was the allocation sequence adequately generated?				0.41	0.43
Low	11 (30%)	19 (41%)	25 (42%)
Unclear	24 (65%)	25 (55%)	33 (56%)
High	2 (5%)	2 (4%)	1 (2%)
Was the allocation adequately concealed?				1.00	0.82
Low	15 (40%)	20 (44%)	25 (42%)
Unclear	21 (57%)	25 (54%)	32 (54%)
High	1 (3%)	1 (2%)	2 (4%)
Were baseline outcomes similar?				0.87	0.20
Low	31 (84%)	39 (85%)	44 (75%)
Unclear	2 (5%)	3 (6%)	8 (13%)
High	4 (11%)	4 (9%)	7 (12%)
Were baseline characteristics similar?				0.16	0.57
Low	30 (81%)	34 (74%)	43 (73%)
Unclear	3 (8%)	6 (13%)	3 (5%)
High	4 (11%)	6 (13%)	13 (22%)
Were incomplete outcome data adequately addressed?				0.17	0.70
Low	29 (78%)	33 (72%)	38 (64%)
Unclear	3 (8%)	8 (17%)	7 (12%)
High	5 (14%)	5 (11%)	14 (24%)
Was knowledge of the allocated interventions prevented?				0.44	0.61
Low	32 (87%)	38 (83%)	46 (78%)
Unclear	3 (8%)	5 (11%)	7 (12%)
High	2 (5%)	3 (6%)	6 (10%)
Was the study protected against contamination?				0.54	0.78
Low	25 (68%)	23 (50%)	35 (59%)
Unclear	10 (27%)	21 (46%)	19 (32%)
High	2 (5%)	2 (4%)	5 (9%)
Was the study free from selective outcome reporting?				0.84	1.00
Low	3 (8%)	4 (9%)	6 (10%)
Unclear	32 (87%)	37 (80%)	50 (85%)
High	2 (5%)	5 (11%)	3 (5%)
Was the study free from other risks of bias?				0.58	0.58
Low	27 (73%)	30 (65%)	39 (66%)
Unclear	10 (27%)	16 (35%)	20 (34%)
High	0	0	0

*Exact Cochran-Armitage test for high versus low or unclear risk of bias in each domain except the last domain which was analysed as low versus high or unclear due to absence of studies with high risk of bias.

†Exact Cochran-Armitage test for reported (high or low risk of bias) or unreported (unclear risk of bias) in each domain.

Overall, 48.6% (69/142) of the RCTs had a high risk of bias in at least one domain (95% CI 40.4 to 56.8). Figure 3 illustrates the rapid increase in number of QI RCTs published over time and the cumulative proportion of trials having at least one domain with high risk of bias up to a given year. In general, the line representing the proportion at high risk of bias runs parallel to the number of trials published, consistently accounting for almost half of the studies. Table 4 indicates a lack of significant trend over time in the proportion of trials with at least one domain with high risk of bias: these proportions were 46%, 44% and 54% before 2002, between 2002 and 2006, and after 2006, respectively. Table 4 also demonstrates a lack of significant association between any of the study characteristics considered and the presence of high risk of bias in at least one domain.

Figure 3

Cumulative number of diabetes quality improvement trial publications at high risk of bias in any domain, 1990–2010.

Table 4

Association between study characteristics and risk of bias

Characteristic	All studies (n)	Studies in high risk of bias in at least one domain number (%)	p Value*
Year of publication			0.37
Pre-2002	37	17 (46%)
2002–2006	46	20 (44%)
2007–2010	59	32 (54%)
Type of diabetes			0.11
Type 1	9	3 (33%)
Type 2	80	36 (45%)
Both	34	16 (47%)
Unclear	19	14 (74%)
Unit of allocation			0.24
Patient	94	49 (52%)
Cluster (eg, provider/clinic)	48	20 (42%)
Country/setting			0.62
The USA or Canada	79	41 (52%)
The UK or Western Europe	40	17 (43%)
Other	23	11 (48%)
Journal impact factor			0.87
Greater than 3 (median)	71	34 (47.9%)
Less than 3 (median)	71	35 (49.3%)
Effective sample size			0.87
Greater than 154 (median)	71	35 (49.3%)
Less than 154 (median)	71	34 (47.9%)
Intervention type			0.17
Multifaceted (featuring more than one QI strategy)	124	63 (51%)
Single intervention	18	6 (33%)

*Comparing proportion of studies with at least one domain at high risk of bias against studies no domains at high risk of bias. For year of publication, Cochran-Armitage test for trend was conducted. For other study characteristics, χ2 (or Fisher's exact) tests for categorical and Wilcoxon signed-rank tests for continuous variables were used.

QI, quality improvement.

The sensitivity analysis, restricting studies defined as high risk of bias overall to those with high risk of bias in one of four domains (allocation sequence generation, allocation concealment, blinding or selective outcome reporting) also revealed no trends over time—the proportions were 19%, 20% and 20% before 2002, between 2002 and 2006 and after 2006, respectively (p=0.86).

Discussion

Main findings

Using the Cochrane EPOC Risk of Bias Tool,6 we found that nearly half of RCTs focusing on diabetes had at least one domain at high risk of bias. The trials were most often at high risk of bias due to inadequate follow-up of participants, a lack of similarity at baseline across outcome measures or covariates, or inadequate blinding. We also noted that the majority of RCT reports failed to include an adequate description of the allocation process (ie, sequence generation and allocation concealment were ‘unclear’). To be interpreted appropriately, RCTs must be completely and transparently reported.18 19 Our findings indicate that greater efforts are needed to ensure both adequate reporting and methodological conduct of diabetes QI trials.

We found that poor follow-up, baseline imbalances and blinding were the most common sources of high risk of bias. Although these domains may be difficult fully control in QI trials, methodological approaches are available to mitigate and/or explore such causes of risk of bias. For example, sensitivity analyses may be used to explore the risk of bias related to loss of follow-up, and risk of baseline imbalances in QI trials may be reduced through restricted randomisation techniques, especially when trials are cluster-randomised with relatively few clusters. In addition, selective outcome reporting may be limited if more QI trial protocols were registered. Finally, although blinding may be particularly difficult to accomplish in QI trials, this should be clearly reported; if outcome assessment is not blinded, risk of bias could still be limited by using objective outcomes.

Comparison to literature

A systematic review focusing on cluster randomised trials found minimal improvement over time in either reporting or methodological conduct.20 We found no evidence for a difference in the proportion of cluster-randomised trials at high risk of bias compared with trials in which individuals were allocated. However, imbalance at baseline was a common source of potential bias in diabetes QI trials, possibly owing to the inadequate use of restricted randomisation in cluster trials.21 Another systematic review included 35 studies covering a range of health-related fields assessing trends over time in quality criteria for RCTs.5 Of these, 26 found improvement over time for at least one aspect of methodological quality. The domain most commonly noted to have improvement was allocation concealment, but the authors noted that this domain remained either poorly reported or inadequately performed in over half of the examined trials. We found a similarly low proportion of studies clearly reporting adequate allocation concealment, and no evidence of improvement over time.

Previous authors have noted that QI reports may not contain enough information to inform generalisation and allow for replication in different clinical settings.22 Standards for Quality Improvement Reporting (SQUIRE) guidelines suggest that investigators conducting trials use both SQUIRE and CONSORT to inform their manuscripts.19 Journal editors should enforce the requirements of both SQUIRE and CONSORT for QI RCTs, possibly by permitting detailed information to be posted as online appendices. Although it might seem onerous to force investigators to address all items in SQUIRE and CONSORT, the risks of poor reporting are substantial. Inadequate description of context could omit essential preconditions or important effect modifiers for a successful QI programme, while incomplete description of the programme itself might lead to failure due to partial implementation.

Strengths and limitations

To our knowledge, this is the largest analysis of risk of bias ever reported for healthcare QI RCTs and the only one to assess for trends over time. The findings are strengthened by the rigorous methods used to prepare the data for the systematic review.

QI evaluations have been criticised based on numerous criteria beyond the risk of bias domains, including short duration of intervention, lack of justification for intervention design and poor generalisability.23 24 Some important components of methodological quality do not relate to bias (eg, reporting of a sample size calculation). Thus, it is possible that studies at low risk of bias have important flaws with respect to methodology and/or reporting (and vice versa), and it is possible that using other scales to assess study quality could have led to different results.14 While the overall risk of bias assessment using the Cochrane Risk of Bias Tool has been shown to differentiate effect sizes (ie, higher risk of bias studies usually have larger effect sizes),10 studies at high risk of bias may still offer valuable knowledge for QI implementers. The merit of any given report will depend on the needs of the reader, while the current analysis provides an assessment of the progress in the literature as a whole.

Furthermore, we acknowledge that assigning trials with high risk of bias in a single domain a status of high risk of bias overall may be arguable. Nevertheless, our sensitivity analysis led to the same conclusion: there has been no improvement over time in the proportion of trials at high risk of bias in this literature and no particular study characteristics were associated with high risk of bias.

Another potential limitation stems from our analytical approach regarding change over time; collapsing publication year into three timeframes (pre-2002, 2002–2006 and 2007–2010) and testing for trends may have limited our power. These timeframes were chosen a priori based on the publication of important documents that we thought might affect the conduct and reporting of these trials. We felt the assumption of linear change over time underlying the Cochran-Armitage test for trend was appropriate and in keeping with our hypotheses (eg, high and unclear risk of bias would decrease gradually over time, while low risk of bias would increase). Risk of type 2 error is tempered by the number of tests performed; the lack of a significant p value for trend for any level of risk of bias in any domain supports our main conclusion. Finally, this review considered only RCTs from the diabetes literature. It would have been preferable to evaluate a random sample of all QI trials, but adequate QI electronic literature searches are yet to be developed.25

Implications

Published trials testing QI in diabetes are frequently at high risk of bias, producing results that may not be replicable. Clinicians must scrutinise the internal validity of the results as a first step in the process of considering the application of clinical findings for particular patients. Our findings emphasise the need for policy-makers, managers and/or clinical-administrators seeking to implement QI interventions to apply the same process.3 It is likely that QI investigators publishing RCTs desire for their work to have a broad impact. To help them in accomplishing this, research funders and journal editors can play an important role by ensuring that QI trials are reported thoroughly and transparently and are designed in a manner that limits the potential for risk of bias.