Benchmarks for acute stroke care delivery.

OBJECTIVE: Despite widespread interest in many jurisdictions in monitoring and improving the quality of stroke care delivery, benchmarks for most stroke performance indicators have not been established. The objective of this study was to develop data-derived benchmarks for acute stroke quality indicators.
DESIGN: Nine key acute stroke quality indicators were selected from the Canadian Stroke Best Practice Performance Measures Manual. PARTICIPANTS: A population-based retrospective sample of patients discharged from 142 hospitals in Ontario, Canada, between 1 April 2008 and 31 March 2009 (N = 3191) was used to calculate hospital rates of performance and benchmarks. INTERVENTION: The Achievable Benchmark of Care (ABC™) methodology was used to create benchmarks based on the performance of the upper 15% of patients in the top-performing hospitals. MAIN OUTCOME MEASURES: Benchmarks were calculated for rates of neuroimaging, carotid imaging, stroke unit admission, dysphasia screening and administration of stroke-related medications.
RESULTS: The following benchmarks were derived: neuroimaging within 24 h, 98%; admission to a stroke unit, 77%; thrombolysis among patients arriving within 2.5 h, 59%; carotid imaging, 93%; dysphagia screening, 88%; antithrombotic therapy, 98%; anticoagulation for atrial fibrillation, 94%; antihypertensive therapy, 92% and lipid-lowering therapy, 77%. ABC™ acute stroke care benchmarks achieve or exceed the consensus-based targets required by Accreditation Canada, with the exception of dysphagia screening.
CONCLUSIONS: Benchmarks for nine hospital-based acute stroke care quality indicators have been established. These can be used in the development of standards for quality improvement initiatives.

Introduction

Benchmarking has gained popularity as a health care quality performance measurement tool with benchmarks used to compare delivery of care across institutions and jurisdictions and to encourage excellent performance by ranking institutions and highlighting top performers [1]. Establishing realistic performance benchmarks that monitor implementation of evidence-based best practice has been shown to improve performance compared with audit and feedback alone [2-6].

Benchmarks based on subjective or expert panel consensus rather than empirical data may be viewed as invalid. Data-derived benchmarks based on the average or median by definition are unlikely to drive excellence. The Achievable Benchmarks of Care (ABC™) approach is a method to establish ‘real world’ performance benchmarks by examining performance across all relevant organizations or health care providers and then determining the best care achieved by at least 10% of the eligible patients across organizations to identify ‘top’ performance levels [7, 8]. This method produces benchmarks that can be seen as realistic targets, as they have been achieved by at least one provider caring for at least 10% of all eligible patients in the sample.

Currently, there is limited information on appropriate benchmarks for acute stroke care delivery [9-11]. In Ontario, a regionally based system of stroke care delivery—the Ontario Stroke System—was established in 2000 [12]. Within this system, regional stroke centers are accountable for leading the implementation of stroke care best practices across a geographic region, which includes a number of community hospitals, rehabilitation facilities (inpatient and ambulatory), community-based providers, community support agencies, health promotion practitioners, long-term care facilities and pre-hospital care providers [12]. The regional stroke centers are typically large teaching or academic hospitals with neurology and neurosurgical services, sophisticated diagnostic technologies and annual stroke/transient ischemic attack (TIA) volumes ranging from 400 to >1100 per year. The established and organized approach to stroke care within Ontario provides a unique opportunity to develop and use stroke benchmarks for quality improvement.

Methods

We used clinical data collected at all acute care institutions in the province of Ontario, Canada, and among eleven regional stroke centers, by the Registry of the Canadian Stroke Network (RCSN). The ABC ™ methodology was used to calculate benchmarks for quality indicators for acute stroke care including: (1) thrombolysis among patients with ischemic stroke arriving within 2.5 h of symptom onset, without contraindications, (2) care on an acute stroke unit, (3) neuroimaging within 24 h of hospital arrival, (4) carotid imaging among ischemic stroke patients without atrial fibrillation, (5) dysphagia screening within 72 h of hospital arrival, (6) antithrombotic therapy, (7) anticoagulation for atrial fibrillation, (8) antihypertensive therapy and (9) lipid-lowering therapy [2].

Data source

The RCSN includes: (1) a periodic random sample audit (Ontario Stroke Audit, OSA) of all acute care institutions with at least 10 stroke cases annually and (2) data on consecutive stroke/TIA patients seen at eleven regional stroke centers in the province of Ontario, Canada (www.rcsn.org). The RCSN is a ‘prescribed’ registry under provincial privacy legislation, and charts are audited without patient consent. The overall project is approved by the Research Ethics Board of Sunnybrook Health Sciences Centre as well as the Research Ethics Board of each participating stroke center. Data are collected on all aspects of acute stroke management, including demographics, comorbidities, use of the emergency medical services, emergency department and in-hospital processes of care and complications by centrally trained neurology research nurses. Chart validation by duplicate chart abstraction has shown excellent agreement (kappa scores or intra-class correlation coefficients of >0.8) for key variables in the database including age, sex, thrombolysis administration, stroke unit care and other processes of care [13].

Data sample

Patients of eighteen years of age or older with stroke or TIA seen in a hospital emergency department or admitted to hospital were identified from administrative databases the National Ambulatory Care Reporting System and the Discharge Abstract Database maintained by the Canadian Institute for Health Information using International Classification of Diseases, Tenth Revision (ICD-10-CA). Those assigned codes I60, I61, I63, I64, H34.1 and G45 (excluding G45.4) were included. To calculate the benchmarks, we used data from the 2008/2009 RCSN Ontario Stroke Audit, which captured patients discharged from 142 acute care hospitals between 1 April 2008 and 31 March 2009. Ninety-nine percent of eligible hospitals participated in this audit, and a simple random sample of 17% of eligible cases was included (n = 3931) with over-sampling at low-volume institutions to ensure each institution contributed a minimum of 10 cases and 50 cases at smaller specialized stroke centers [14]. Because of concern about a small and unequal sample size for the development of some benchmarks, secondary analyses used data from consecutive patients seen at 11 regional stroke centers between 1 April 2006 and 31 March 2008 (n = 8 109).

Acute Stroke Quality of Care Indicators

Table 1 lists the nine stroke quality indicators evaluated in this study. The indicators of stroke care performance are based on the Canadian Stroke Strategy's 2008 Performance Measurement Manual, are used for reporting within the Ontario Stroke System, reflect a subset of indicators identified by Accreditation Canada for hospitals seeking stroke care distinction status and are also recommended or reported by other organizations in other jurisdictions [2, 10, 11, 15–19].

Table 1

Acute Stroke Quality of Care Indicators

Performance indicators
1. Proportion of suspected stroke/TIA patients who receive a brain CT/MRIa within 24 h of hospital arrival to the emergency department.
2. Proportion of stroke/TIA patients treated on a stroke unit at any time during their inpatient stay.
3. Proportion of ischemic stroke patients who arrive within 2.5 h of symptom onset and receive acute thrombolytic therapy (tPA)b (excluding patients with known contraindications).
4. Proportion of ischemic stroke patients without atrial fibrillation who receive carotid imaging prior to inpatient hospital discharge.
5. Proportion of stroke (excluding TIA, unconscious patients) in patients with documentation that an initial dysphagia screening was performed within 72 h of hospital arrival.
6. Proportion of ischemic stroke/TIA patients who were prescribed antithromboticc therapy at discharge.
7. Proportion of ischemic stroke/TIA patients with atrial fibrillation prescribed anticoagulant therapyb on discharge from acute care (excluding patients with contraindications).
8. Proportion of ischemic stroke/TIA patients who were prescribed antihypertensive therapy at discharge.
9. Proportion of ischemic stroke/TIA patients who were prescribed lipid-lowering therapy at discharge.

All indicators exclude unable to determine final diagnosis except indicator 1. All indicators exclude stroke type unknown except indicator 1, 2 and 5. aCT and MRI based on records with admission date/time, scan date/time recorded. btPA, tissue plasminogen activator. At the time the data were collected, the tPA window of time was 3 h. Midway through the time of data collection (September 2008), trial results demonstrated the safe therapeutic window for tPA delivery from stroke symptom onset had increased to 4.5 h. However, we chose to base it on the longstanding therapeutic window of 3 h as practice change was not expected to change immediately. cAntithrombotic therapy includes acetylsalicylic acid (ASA), combination ASA and dipyridamole, clopidogrel and warfarin.

Statistical analysis

We calculated overall indicator performance as the proportion of eligible patients that received each stroke care indicator rounded to the nearest whole number. We then used the ABC™ methodology to calculate benchmarks for each of the nine quality indicators. The overall indicator performance was assessed using Kiefe et al.'s algorithm to determine the minimum sufficient denominator (MSD) (i.e. eligible patients) and whether a Bayesian adjustment was needed to adjust for a small number of eligible patients [7]. If a hospital's eligible patient sample did not meet the MSD (i.e. number of eligible patients) for each indicator, we applied a Bayesian adjustment to calculate the hospital's adjusted performance fraction described by Kiefe et al. [7, 8]. Starting with the highest performing hospital for the particular indicator and continuing through the next highest performing hospital, we cumulatively added each hospital's eligible patients until the total number of patients represented in the denominator included at least 15% of the total eligible patients across all hospitals. We based the benchmark on at least 15% of the total eligible patients across hospitals because the calculation of the ABC™ benchmark is based on crude data and we wanted to increase the number of eligible patients and hospitals to include in the benchmark and to reduce the influence of hospitals with small numbers of eligible patients on the benchmark. The benchmark was determined by dividing the total number of patients receiving the best practice care by the total number of patients eligible to have received best practice care in this subset. See Supplementary material, Appendix A for a sample calculation. For each indicator, hospital(s) were considered ‘top’ performers if their performance rate was at or above the ABC™ benchmark.

The ABC™ process was repeated using the validation cohort from 11 regional stroke centers, except that there was no need to apply the Bayesian adjustment as the number of eligible patients exceeded the MSD denominator at all of these hospitals. For each indicator benchmark, we report the total number of hospitals included in our benchmark calculations, as well as the range in the number of eligible patients at hospitals included in the ABC™ benchmark calculation. For each indicator, we report (1) the median, 25th and 75th percentile of eligible patients among the hospitals, (2) categories of the number of eligible patients, <10, 10–24, 25–75, 75 or more and (3) the percentage of hospitals in the audit with eligible patient samples below the MSD.

Results

Table 2 describes the characteristics of participating hospitals and patients in the 2008/09 Ontario stroke audit. Of the 142 hospitals included in the audit sample, 28 (20%) were regional or district stroke centers, 70 (49%) were high volume sites (with more than 100 patients with stroke or TIA annually), 90 (63%) had computed tomography (CT) scanners, 26 (18%) had stroke units, 70 (49%) had interdisciplinary stroke teams and 39 (28%) had a secondary stroke prevention clinic on site. The study sample included 3931 patients with a median age of 75 years. The number of patients sampled across the 142 hospitals ranged from 10 to166. Baseline characteristics of study patients are summarized in Table 2.

Table 2

Characteristics of stroke/TIA patients (N = 3931) and hospitals (N = 142) in the 2008/09 Ontario Acute Stroke Audit

Characteristic	N (%)
Hospital characteristics	(N = 142)
Designation: regional stroke centersa	9 (6)
District stroke centersb	19 (13)
Non-designated hospitalsc	114 (80)
Urban hospitals	83(58)
Annual stroke/TIA volume ≥100	70 (49)
Hospitals with stroke units	26 (18)
Hospitals with designated stroke teams	70 (49)
Hospitals with CT on site	90 (63)
Hospitals with secondary stroke prevention clinics on site	39 (27)
Patient characteristics	(N = 3 931)
Stroke	2 425(62)
Transient ischemic attack	1 167(30)
Unable to determine	338 (9)
Stroke type	(N = 2 370)
Ischemic	1896 (80)
Intracerebral hemorrhage	308 (13)
Subarachnoid hemorrhage	119 (5)
Undetermined	24 (1)
Male	1965 (50)
Median age (years, 25th and 75th percentile)	75 (60, 81)
Rural residence	590 (15)
CNS scores >8	2 948 (75)
Transported by ambulance	2 240 (57)
Independent prior to admission	3 302 (84)
Diabetes	983 (25)
Hypertension	2 555 (65)
Hyperlipidemia	1 454 (37)
Atrial fibrillationd	589 (15)
Previous TIA/stroke	1 336 (34)
Previous MI	511 (13)

aRegional stroke center: all the requirements of a district stroke center, plus neurosurgical facilities and interventional radiology. bDistrict stroke center: facilities with written stroke protocols (e.g. transport and triage, thrombolytic therapy and neuroimaging), clinicians with stroke expertise and linkages to rehabilitation and secondary prevention. cNon-designated: acute care hospital that does not fit the definition of district or regional stroke center. dAtrial fibrillation was based documented on past history of OR new onset during hospital stay.

There was a wide range in performance for all nine indicators across acute care hospitals within the OSS. The overall performance ranged from 30% (admission to a stroke unit) to 94% (prescribing of antithrombotic therapy at discharge). Only two of the nine indicators (neuroimaging within 24 h and antithrombotic therapy prescribed on discharge) had overall performance rates of >85% (see Table 3).

Table 3.

Achievable Benchmarks of Care in Ontario's acute care hospitals (2008/09)

Stroke process of care indicators	Number of eligible patients	Overall performance (%)	Benchmarks
			OSA–ABC™ (%)	Number of hospitals included in OSA benchmark calculation (median, range of eligible patients included in the benchmark calculation)	Accreditation Canada targets (10 ref) (%)	Stroke Centre ABC™ (%)
Neuroimaging <24 h	3176	86	98	11 (51, 21–138)	≥90%	99
Admitted to a stroke unit	2457	30	77	10 (46, 4–101)	75%	94
Arrived within 2.5 h of stroke onset and received tPAa	469	30	59a	5 (11, 7–33)	NA	42
Carotid imaging	1186	75	93	11 (18, 12–56)	NA	87
Dysphagia screening	1924	62	88	18 (24, 3–66)	≥90%	90
Antithrombotics on discharge	2883	94	98	11 (43, 33–100)	≥90%	97
Warfarin among patients with atrial fibrillation on discharge	456	70	94	13 (5, 3–15)	75%	88
Antihypertensives on discharge	2883	78	92	31 (16, 6–60)	NA	79
Lipid-lowering agents on discharge	2883	60	77	22 (27, 3–65)	NA	68

aonly hospitals with capacity to deliver tPA. OSA, Ontario Stroke Audit; NA, not available.

Five of Ontario's nine ABC™ benchmarks for each acute stroke quality indicator were >90%: neuroimaging within 24 h (either CT or magnetic resonance imaging (MRI) of the brain) and antithrombotic therapy prescribed at discharge, 98%; warfarin prescribed at discharge for patients with atrial fibrillation, 94%; carotid imaging prior to discharge, 93% and antihypertensive therapy prescribed at discharge, 92%. ABC™ benchmarks of <90% were dysphagia screening, 88%; admission to a stroke unit and lipid-lowering therapy prescribed at discharge 77% and thrombolysis administration, in patients presenting within 2.5 h of stroke symptom onset and without contraindications for thrombolysis, 59%.

The five Accreditation Canada indicator non-data-derived targets correspond to two of the ABC™ benchmarks (neuroimaging and antithrombotic prescribing) but are lower for stroke unit admission and warfarin prescribing and higher for dysphagia screening. Benchmarks derived from regional stroke center data were similar to those derived from the entire provincial hospital sample for neuroimaging and antithrombotic therapy, higher for stroke unit admission and dysphagia screening and lower for all other indicators (Table 3). Less than a quarter of all hospitals in the audit (22%) were included in the benchmark calculation. The median number of eligible patients included in the benchmark calculation ranged from 5 for warfarin prescribing among patients with atrial fibrillation to 51 patients for neuroimaging within 24 h. The number of hospitals that met or exceeded the benchmark for any indicator ranged from 2 hospitals for patients presenting within 2.5 h of stroke symptom onset and without contraindications for thrombolysis to 61 hospitals for prescribing antithrombotic therapy on discharge (data not shown).

Table 4 illustrates the distribution of hospitals by categories of eligible patient sample size for each acute stroke quality of care performance indicator. The number of hospitals contributing to a performance indicator ranged from 93 (for thrombolysis) to 141 (for use of medications for secondary stroke prevention). Among our nine acute stroke quality indicators, the MSD varied from 4 to 36 eligible patients based on overall indicator performance. The percentage of hospitals with eligible patients below the MSD varied from 14% for stroke unit admission to 91% for warfarin prescribing on discharge among patients with atrial fibrillation. For each indicator, at least 65% of hospitals had <25 eligible patients sampled, and for thrombolysis administration, only 1 hospital had at least 25 eligible patients and for warfarin prescription, no hospitals were sampled with at least 25 eligible patients. With the exception of prescribing antihypertensive therapy on discharge, all of the benchmarks had at least two specialized stroke hospitals (regional or district stroke centers) that achieved or exceeded the benchmark.

Table 4.

Distribution of Eligible Patient Volume in the 2008/09 Ontario Stroke Audit

Eligible patients (hospital, N)	Neuroimaging within 24 h	Stroke unit admission	tPA among patients arriving within 2.5 h of symptom onset	Carotid imaging	Swallowing assessment	Discharge medications
						Antithrombotics	Warfarin for patients with AF	Antihypertensives	Lipid-lowering
0	2	4	49	10	7	1	36	1	1
<10	62	72	76	81	76	68	94	68	68
10–24	34	26	16	43	35	25	12	25	25
25–75	37	35	1	8	22	46	0	46	46
>75	7	5	0	0	2	2	0	2	2
Overalla	140	138	93	132	135	141	106	141	141
Median eligible patients (25th and 75th percentile)b	11 (7, 32)	9 (5, 27)	3 (1, 7)	5 (2, 13)	6 (3, 21)	10 (7, 34)	3 (1, 6)	10 (7, 34)	10 (7, 34)
MSD	23	4	4	13	8	36	11	13	8
% Hospitals below MSD	33	14	58	74	53	78	91	53	33
Number of RSCc at or above the ABC™ benchmark	2/9	1/9	0/9	0/9	2/9	0/9	0/9	0/9	2/9
Number of DSCd at or above the ABC™ benchmark	0/19	3/19	2/19	3/19	1/19	5/19	3/19	0/19	4/19

aNumber of hospitals with at least 1 eligible patient. bMedian number of eligible patients among hospitals with at least 1 eligible patient. cregional stroke center: all the requirements of a district stroke center, plus neurosurgical facilities and interventional radiology. dDistrict stroke center: facilities with written stroke protocols (e.g. transport and triage, thrombolytic therapy and neuroimaging), clinicians with stroke expertise; and linkages to rehabilitation and secondary prevention. MSD, minimum sufficient denominator (i.e. eligible patients).

Discussion

This study provides benchmarks for nine important acute stroke care indicators using the ABC™ methodology with adjustment made for hospitals with small numbers of eligible patients, detailed indicator definitions to allow our work to be replicated, and the median and range of eligible patients within the hospitals included in the benchmark calculation is provided to assess the precision of the benchmark. These benchmarks are derived from a representative sample of stroke/TIA patients seen at 142 hospitals providing stroke care in a province of >13 million people within an organized system of stroke care. We also provide the distribution of eligible patients for each indicator across the hospitals to assess overall indicator performance within the Ontario Stroke System. In addition, we also provide benchmarks based on data from regional stroke centers.

Most of our benchmarks are similar to those found by Hinchey et al. [9] using the ABC™ method to calculate stroke care benchmarks for ischemic stroke patients from 17 volunteer hospitals (13 were community hospitals) across 9 US states (N = 2294). Our neuroimaging, stroke unit admission, prescribing antithrombotics and anticoagulants for patients with atrial fibrillation benchmarks are similar to the overall performance levels reported in other countries from a subset of hospitals that participate in quality improvement initiatives [11, 15–19]. Compared with the consensus-based non-data-derived targets used by Accreditation Canada to designate centers of stroke distinction, our data-derived ABC ™ benchmarks based on a sample of eligible patients across all acute hospitals in Ontario meet or exceed the Accreditation Canada targets with the exception of dysphagia screening [10].

Deciding what benchmark is appropriate depends on the purpose of that benchmark. Some might argue we only need to have consensus-based targets or performance better than the overall average especially if funding is tied to achieving a benchmark. However, under a quality improvement framework where data are essential for measuring performance, providing the number of eligible patients contributing to the benchmark provides organizations full disclosure in assessing the credibility, acceptability and achievability of the benchmark [20]. In Ontario, the OSS evaluation program has developed report cards on the quality of stroke care to be used by the OSS regions, hospitals, system planners and provincial agencies to understand the strengths and weaknesses and prioritize quality improvement initiatives in acute stroke care across Ontario hospitals [12, 21]. The benchmarks included in the report card are based on the provincial audit data (OSA). Benchmarks will need to be recalibrated over time, and the advantage of ABC™ methodology is that as all organizations improve the benchmarks rise as well while remaining achievable.

The ABC™ approach to deriving benchmarks is data-driven, represents a level of excellence and is demonstrably attainable [7]. However, chance fluctuations, small sample sizes or samples of unequal size can produce unstable performance measures [7, 22–24]. Small and unequal sample size is a limitation of our study as it is for most performance measurement studies [22-24]. We evaluated the effect of small sample size on our benchmark calculations three ways. Firstly, we excluded facilities with <25 eligible patients. This analysis resulted in no change to the neuroimaging benchmark, and minor (1 to 4%) decreases to two benchmarks and increases to four benchmarks (data not shown). In our study, four out of nine benchmarks (neuroimaging, stroke unit admission, antithrombotic and lipid-lowering agents prescribed on discharge) were not dominated by hospitals with numbers of eligible patients; median eligible patients among hospitals contributing to the benchmark were 51, 46, 43 and 27, respectively (Table 3). Five of our benchmarks were driven by small sample sizes therefore may be viewed with caution (median number of eligible patients were <25); these include warfarin prescribing on discharge among patients with atrial fibrillation, thrombolysis, carotid imaging, dysphagia screening and antihypertensive medication prescribing these benchmarks may be viewed with caution (Table 3). In particular, the warfarin prescribing benchmark did not have any hospital included in the benchmark calculation with >15 eligible patients. However, we have presented the median and range of eligible patients among the hospitals included in the benchmark calculation for the reader to assess the acceptability the benchmark based on a sample of all hospitals in Ontario.

Secondly, we explored whether the benchmarks would change if we excluded hospitals where the MSD was not met for each indicator. We found all benchmarks remained unchanged, with the exception of warfarin prescribing for atrial fibrillation where the benchmark was reduced from 94 to 76% (data not shown).

Finally, we repeated the benchmark calculations using data from 11 regional stroke centers where eligible patient sample sizes are larger and demonstrate less variation. We found that six of the nine ABC™ benchmarks were lower (1 to 17%) than those derived from the entire sample of hospitals (Table 3). However, neuroimaging, admission to a stroke unit and dysphagia screening benchmarks were higher (1, 17 and 2%, respectively). This finding may be a statistical artifact; difference in years of data may reflect real differences in processes of care at these regional stroke centers. For example, regional stroke centers are large tertiary centers and may have facility/system issues that prevent them from achieving higher levels of performance. Stroke patients compete with trauma and cancer patients for access to assessment and diagnostic imaging not faced by the smaller district stroke centers. Given the structural issues and more complex patients at regional stroke centers, the benchmarks derived from regional stroke centers may be considered to have better face validity among those centers with similar characteristics (e.g. large academic/teaching hospitals) to be used for peer benchmarking. Additionally, the time-dependent difference between the regional stroke center data (2006 to 2008), and the provincial hospital population-based data (2008/09) may also explain the lower benchmarks obtained using the regional stroke center data. However, when we recalibrated the regional stroke center-derived ABC™ benchmarks using the same fiscal year data as the OSA ABC™ benchmarks, 1 April 2008 to 31 March 2009, we did find a significant change to four of ABC™ benchmarks based on 2006–2008 data. In particular, thrombolysis increased from 42 to 48%, warfarin prescribing decreased from 88 to 76% and antihypertensive and lipid-lowering prescribing increased from 79 to 86% and 68 to 82%, respectively.

Hierarchical modeling has been cited as an appropriate statistical method to use for performance measurement and in particular for outcomes of care rather than processes of care to deal with the issue of unstable performance metrics based on small and unequal sample sizes [23-25]. Hierarchical modeling takes the performance estimate of a facility and ‘shrinks’ it closer to the mean performance among all facilities and, the degree of ‘shrinkage’ is greater for facilities with smaller sample sizes. A benchmark could be based on the 90th percentile among these ‘shrunken’ performance estimates derived from hierarchical modeling. O'Brien et al. [22] generated benchmarks based on the 90th percentile using hierarchical modeling and found this ‘shrinkage’ resulted in minimally lower benchmarks (<3%) compared with the benchmarks derived using the ABC™ methodology with the Bayesian estimator to reduce the influence of a small sample for five out of their eight ABC™ benchmarks, but more marked (7 to 32% lowering) for the remaining three benchmarks with the most dramatically lower benchmark for the indicator dominated by hospitals with <25 eligible patients (>90% of the hospitals). We chose not to use hierarchical modeling to calculate our first release of our process of care benchmarks as we wanted to provide an easily understood method to stakeholders. The ABC™ methodology uses the actual performance with a Bayesian adjustment made for facilities that are below the MSD to reduce the influence of small sample sizes and has greater transparency compared with a statistical modeling approach to adjust hospital performance rates. Furthermore, O'Brien et al.'s work demonstrated minimal difference for the majority of their process of AMI care benchmarks generated by hierarchical modeling compared with the ABC™ benchmarks, and Arling et al.'s ischemic stroke quality performance indicators demonstrated modest increases in the 90th percentile performance level when applying multilevel model empirical Bayes estimation methods compared with the unadjusted 90th percentile performance [22, 24]. Arling et al. also observed small denominators (i.e. eligible patients) for many of their quality indicators as did our study and considerable uncertainty remained with the estimated Bayes adjusted performance indicators. We, however, did not produce confidence intervals around our ABC™ benchmark because the purpose of our benchmarks is to drive quality improvement rather than identify good versus poor performance. Finally, our quality indicators and benchmarks reflect the Canadian context, and therefore our benchmarks may not be transferable to other jurisdictions. The data were collected to measure performance based on the Canadian perspective according to the Canadian Stroke Strategy Performance Measurement Handbook, which was designed to measure the implementation of the Canadian Stroke Strategy Best Practice Recommendations (2).

Conclusion

Benchmarks are an important tool for quality improvement initiatives and empirically derived benchmarks including patient sample size offers transparency to allow the credibility of the benchmarks to be assessed. This is the first Canadian study to report benchmarks from all acute care hospitals within a large province. Although the acute hospitals vary widely in size and complexity of service provision, these benchmarks for acute stroke care delivery can be used for reporting and quality improvement to strive for excellence in delivering acute stroke care. Further research is needed to examine what is the optimal benchmark for treatments that have high prevalence of contraindications such as anticoagulation in atrial fibrillation and thrombolysis.

Supplementary material

Supplementary material is available at

Funding

This work was supported by an operating grant from the Canadian Stroke Network. The Registry of the Canadian Stroke Network is funded by an operating grant from the Ontario Ministry of Health and Long-Term Care. The Institute for Clinical Evaluative Sciences is supported by an operating grant from the Ontario Ministry of Health and Long-Term Care. The results and conclusions are those of the authors and are not attributed to any of the sponsoring or funding agencies. The funding agencies had no role in the design or conduct of the study or the collection, management, analysis or interpretation of the data. The manuscript was reviewed and approved by the publications committee of the Registry of the Canadian Stroke Network. Funding to pay the Open Access publication charges for this article was provided by the Canadian Stroke Network.