Development and Implementation of ExPLORE Clinical Practice, a Web-accessible Comparative Outcomes Tool for California Hospitals and Physicians.

BACKGROUND: Hospital-based clinicians have little information about the outcomes of their care, much less how those outcomes compare with those of their peers. A variety of care quality indicators have been developed, but comparisons tend to be hospitalwide, and often irrelevant to the practice and patient group of many hospital clinicians. Moreover, information is not enough to transform clinical practice, as the human response to such comparisons is, "I'm doing the best I know how." What is needed is granular, clinically specific feedback with peer-mediated advice about how "positive deviants" achieve better results.
OBJECTIVE: This case study reports on the development and implementation of a web-accessible comparative outcomes tool, ExPLORE Clinical Practice, for hospitals and clinicians in California.
METHODS: We use iterative development and refinement of web tools to report comparative outcomes; incremental development of suites of procedure-patient outcome pairs specific to particular medical specialty groups; testing and refinement of response time metrics to reduce delays in report generation; and introduction of a comments section for each measure that assists with interpretation and ties results to strategies found to lead to better clinical outcomes.
RESULTS: To date, 76 reports, each with 115 to 251 statistically evaluated outcomes, are available electronically to compare individual hospitals in California to statewide outcomes. DISCUSSION AND
CONCLUSIONS: ExPLORE Clinical Practice is one of a number of emerging systems that attempt to lever available data to improve patient outcomes. The ExPLORE Clinical Practice system combines a clinical focus on highly specific outcome measures with attention to technical issues such as crafting an intuitive user interface and graphic presentation. This case study illustrates the important advances made in using data to support clinicians to improve care for patients. We see this information as a way to start local conversations about quality improvement, and as a means of generating peer advice for improving patient outcomes.

Introduction

There is widespread consensus that the quality of health care is far from optimal.1 Measuring and learning from health care outcomes is a central pillar of clinical outcome improvement and cost management. Suites of isolated outcome metrics (e.g., AHRQ PSI’s,2 PDI’s,3 and IQI’s4) are produced and delivered to hospitals and clinicians. However, indicators may not consistently reflect the quality of patient treatment,5,6 and seldom are directly related to the clinical tradeoffs facing physicians every day.

At present, there is no systematic method for detecting or evaluating the incidence of patient-level outcomes of care. Where rates of outcomes of care are communicated, they are usually limited to outcomes deemed preventable, although the definition of preventable often varies from one physician to the next.7,8

The Dartmouth group has demonstrated “unwarranted variation” in the provision of medical care for United States Medicare patients (e.g., Caesarean section and tonsillectomy rates, among others).9 Their work highlights variation in treatments, costs, and quality measures by geographical region, and more recently by hospital as well as other groupings (e.g., Medicare reimbursements). Moreover, evidence suggests that highlighting variation across all available patient outcomes by clinician can lead to outcome improvement.10

Single indicators may not reflect the full complexity of issues under consideration when a clinician decides to implement (or not) a treatment component. For example, an indicator for postoperative hemorrhage may simply reflect treating clinicians’ documentation preferences, which may be based on a standard volume of estimated total blood loss, the need to intervene (e.g., transfuse the patient), or on other metrics. Even the decision to transfuse a patient may be balanced with concerns regarding cardiac comorbidities, (which may or may not be weighed equally across physicians), or a protocol-defined trigger based on hemoglobin level, where the standards may vary between hospitals.

Isolated hospital- or physician-level indicators (e.g., transfusion rate) insufficiently stratify both patient acuity and the range of procedures performed to facilitate identification of the cause of variation. A suite of interconnected outcome measures applied to stratified patient groups can add both context and depth to single indicators.

The literature on best practice, clinical guidelines, and strategies to avoid patient harm is extensive and growing. Clinicians strive to stay up-to-date with this burgeoning best practice literature. Expecting clinicians to have accessed, and be able to recall, the clinical conditions under which all treatment components are to be applied is unreasonable. This idea is supported by the McGlynn et al. finding that clinicians provide recommended care for the leading causes of death and disability to as few as 55 percent of patients.11

Moreover, evidence supporting many common clinical practices has never been sought. For example, blood transfusion has saved the lives of many patients, particularly injured soldiers in danger of exsanguination. However, the net clinical benefit (benefit minus harms) for transfusion in scheduled total hip replacement has only recently been investigated12 and continues to be developed.13

Indicators most strongly associated with poor outcomes (i.e., increased length of stay, cost, and mortality) rarely point to specific outcome improvement strategies.14 There is a clear need to identify strategies that lead to better clinical outcomes, and to tie these to the data that are fed back to hospitals and clinicians. Ideally, this information would be available at the point of care, but local data can be used to stimulate conversations among clinicians to identify targets for clinical improvement.

Hospitals and clinicians have very little access to detailed, externally benchmarked clinical outcome data with the depth and granularity necessary to inform a broad range of quality improvement activities. The ExPLORE (Examining Patient Level Outcomes to Reveal Excellence) Clinical Practice program has been developed by the Palo Alto Medical Foundation Research Institute (PAMFRI) as a web-based approach to feeding detailed outcome data back to California hospitals. Reports include a full complement of readmission measures and comments intended to help interpret, and act upon, outcome data.

ExPLORE Clinical Practice aims to use de-identified, linked, routinely collected patient data to facilitate review of surgical outcome information, identify opportunities for practice and quality improvement, and to assist doctors and patients in making health care decisions.

Methods

Design

The design of ExPLORE Clinical Practice incorporates four overlapping development processes, one with clinical dimensions and three involving technical performance dimensions.

Treatment / patient outcome pairs;

Data warehousing;

Business intelligence (analysis); and

Reporting

Clinically, treatment/patient outcome pairs have been identified for a growing number of treatment groups (currently 20, stratified to create a total of 76 patient groups; see Appendix A) and patient outcomes. Each patient group has a specific set of 115 (colonoscopy) to 251 (abdominal aortic aneurysm; overall average 189) patient outcomes defined using ICD-9-CM codes. Almost half of these pairs are measured as readmissions—e.g., pulmonary embolism (PE) or deep vein thrombosis(DVT) arising within the index admission, and PE or DVT arising as a readmission. The majority of the outcomes of care are applied across all reports (e.g., PE-, DVT) although there are report-specific patient outcomes (e.g., mechanical loosening of prosthetic joint in total hip or knee replacement). Additional outcome measures will be incorporated to accommodate changes to national or statewide reporting standards. Each treatment/patient outcome pair is tested at three levels of statistical significance (80 percent Confidence Interval (CI), 95 percent CI, and 99.8 percent CI for difference of proportions, adjusted for small cell sizes and small proportions, against the remainder of California cases) and each report is provided for an average of 277 hospitals. Each full data run estimates approximately 12 million comparisons, necessitating an automated approach.

Arrangements for access, data governance, and analytics were early design priorities, with a later shift in focus to technical response times for generating reports, as the scope of the program grew (additional comparative data, additional procedure/patient outcome pairs, and additional system users).

Pilot feedback of the data to physicians elicited mixed responses. Specifically, some physicians were comfortable when presented with positive results, but expressed frustration when their results were less positive and there was no solution (pathway) for improvement. In response we developed a mechanism to identify “positive deviants” or exemplary providers at both the physician and hospital level (see Discussion section). This provides opportunities for physicians to incrementally add to their knowledge about what constitutes a best practice approach.15

Procedure-complication pairings

There is a need for procedure-complication pairings to be clinician-led and to include pairings that may fall outside the experience of the individual clinician or clinician group. To achieve this we started with the following process:

Identify a disease- or procedure group of interest and define it in ICD-9 Diagnosis and/or Procedure Codes (e.g., a suite of procedure codes defining colectomy).

Seek the outcomes of interest from a surgeon- or physician champion treating the patient group.

Identify—for the disease- or procedure group—all diagnosis codes not present on admission in these cases and all procedure codes (for all California cases).

Rank the codes in order of frequency to create a full complement of potential clinical pairings detectable within the data. Identify the outcomes of interest for the surgeon- or physician champion.

Review the disease or procedure and outcome definitions with a Health Information Manager.

Review the resulting list of outcomes with the clinician champion. The result is a suite of clinical pairings.

Continue to refine the outcomes as the program is rolled out across a wider range of clinicians.

After our first few procedure-diagnosis groups we found that step 2 could be problematic as physicians would often limit pairings to a small suite of outcomes relevant to their individual practices. For example, hyponatremia is a complication related to prostatectomy. This pairing wasn’t mentioned in step 2 when we reviewed this with the initial surgeon champion. When we reviewed the full complement of pairings (step 5) hyponatremia was identified by the surgeon champion as a known complication of care that had not had an impact on any of the champion’s cases. Interpretation of the measures (and in some cases the measures themselves) continue to be refined.

ExPLORE Clinical Practice uses bi-level stratification to manage severity of patient illness. At the first level, surgical patient treatment groups are stratified based on their presentation (scheduled or unscheduled). The majority of surgical groupings are limited to scheduled surgery (e.g., scheduled colectomy). This removes variation arising from cases where the surgical procedure under study may not be the dominant clinical issue (e.g., colectomy for penetrating abdominal trauma). The second level, surgeon initiated stratification, divides patients into their underlying pathology where appropriate. For example, colectomy is divided into four groupings: inflammatory bowel disease, benign neoplasms, primary malignant neoplasms, and metastatic or extrinsic tumors (e.g., malignant cancer of the ovary). To date these stratifications have been well accepted by surgeons.

Data Warehousing Functions

The ExPLORE Clinical Practice core data is built on a modified data warehousing design with (1) Extract, Transform and Load (ETL), (2) Analytics and Business Intelligence (BI), and (3) Reporting layers. The data warehousing functions and layers are depicted in Figure 1. Once the data dictionary metadata file has been prepared for extraction, the only manual step is to move the data from the file share to the SQL Server Reporting Services (SSRS)-based presentation server. Although this step could be automated, data security concerns led to a process that ensured that the input data and the report-ready (postanalysis) data are stored in totally disconnected systems, necessitating manual transfer.

Figure 1.

Data Warehouse Schema for the ExPLORE Clinical Practice

ETL Layer

The California Office of Statewide Health Planning and Development (OSHPD) collects data for each inpatient or ambulatory surgery admission and emergency department presentation for all nonfederal California hospitals. Each data set contains, among other variables, patient demographics, up to 25 diagnosis codes with corresponding present-on-admission flags, up to 25 procedure codes with corresponding procedure dates, admission source, and discharge destination. ExPLORE Clinical Practice uses California-wide (excluding Veterans’ Administration hospitals and physician-owned ambulatory surgery centers) data sets for benchmarking. The data also include a record linkage number (RLN) for identifying readmissions. The RLN is based on the Social Security Number (SSN) reported to OSHPD, or the medical record number (MRN) where SSN is absent or invalid. These data provide the basis for all hospital level reports.

ExPLORE Clinical Practice reports derived from historic hospital level OSHPD data include readmissions across any combination of California facilities. Readmission reports from data provided by participating hospitals are limited to readmissions to the same hospital only (Figure 1, step 1) or, in the case of a large hospital group (e.g., Sutter Health), a hospital within the group. Reports are based on the most recent data available for a two-year period. This provides a balance of timeliness of the data and the statistical power necessary for infrequent outcomes and strata with smaller patient volumes.

Physician-level data can be provided directly by participating hospitals. Each participating hospital provides data in a specified text file format that replicates the OSHPD submission with the addition of physician identifiers. The physician identifiers are a string of National Provider Identifier (NPI) numbers corresponding to procedure specific, admission, and consultation related roles. Each case is currently attributed to a single physician. The data are submitted to the ExPLORE Clinical Practice team via Secure File Transfer Protocol. Data replicating the OSHPD submission were chosen as a data set that hospitals already produce, requiring only the addition of NPI fields. Data from individual hospitals would not be necessary in the 14 U.S. states that collect physician identifiers and release them in their standard data sets.16 California has recently passed legislation permitting the collection of NPI’s,17 but is yet to commence NPI collection.

After hospitals send their data to OSHPD and before the data are provided by OSHPD for research and hospital operations (quality improvement) purposes, they undergo data cleaning routines, hundreds of edits, and transformations. A replication of the data transformations is undertaken within the ExPLORE Clinical Practice engine to create a standard internal format for the data. The Emergency Department and Ambulatory Surgery data sets provided by OSHPD are in a format that is not consistent with their Patient Discharge Datasets (e.g., Current Procedural Terminology (CPT) rather than ICD formatted procedure codes) requiring additional transformation.

A detailed data dictionary is used to define the metadata (Figure 1, step 2) including the following: patient groups for the reports; patient subgroups; diagnosis- and procedure-based outcomes; the reports (patient groups) to which each outcome is applied (clinical pairings); assignment of each clinical pairing to an outcome cluster; and determination as to whether the outcome is also to be reported as a readmission. The data dictionary, with derived metadata, enables the automation of the system for efficient expansion.

A series of processes create metadata based on data dictionary dimensions (e.g., a listing of outcomes for colectomy reports), which is loaded in the file server (Figure 1, step 3). Accuracy of the data dictionary is critical to the smooth running of the analytic routines and, although complex, it provides a single point of modification that is propagated throughout the metadata. Although automated, there are still intervening steps between the data dictionary and the metadata. These steps have been maintained, as they are a helpful adjunct for debugging the data dictionary.

Business Intelligence (BI) Layer

Driven by the metadata, the BI layer handles all data analysis including diagnosis and procedure code mapping for treatment groups, patient groups and complications, locating index cases for readmission, and analyzing data with statistical calculations.

The metadata defined by the data dictionary prescribes procedure and diagnosis code mapping for treatment groups, patient subgroups, and outcomes (Figure 1, step 4). The procedure code mapping is conducted in both ICD and CPT procedure classifications. Each diagnosis code mapping considers condition “present on admission” (POA) codes. For example, outcomes that arise within the index admission for a treatment group (e.g., peritonitis following cholecystectomy) are often reported as diagnoses accompanied by a condition “not present on admission” (NPOA) flag, while a readmission for the same outcome is likely to include diagnoses coded as POA.

The analysis module within the ExPLORE Clinical Practice engine (Figure 1, step 5) contains a suite of nests, which are depicted in Figure 2. The metadata determines the components of the nesting loops, which are executed using recursion (i.e., rather than assigning the number of loops for a nest, each loop is executed until all of the available data have been processed).

Figure 2.

Nesting Diagram for the BI Layer

A result for each hospital (or physician) is calculated for each treatment-outcome pair at three levels of significance (two-tail: 80 percent; 95 percent; 99.8 percent) to identify gradations of high and low variation compared with the remainder of California. The comparison uses difference of proportions (shrunken estimates)18 that adjusts the analyses for small case sizes (i.e., cells with < 5 cases) and infrequent outcomes (i.e., proportions <5 percent), a previously identified challenge associated with clinical outcome measures.19 Treating each outcome as a single independent variable results in analyses that are easy to explain to report users (i.e., perceived as transparent) and cannot be inadvertently confounded within the analysis by a priori assumptions (e.g., the result of transfusion-by-physician analyses cannot be modified by the physician’s postoperative-hemorrhage documentation practices).

This method results in many analyses; the analytics for the hospital reports alone involve some 12 million estimations for each new run of data (76 reports x an average of 189 outcomes/report x 3 levels of significance x an average of 277 hospitals).

Each estimate provides a value for events (number of cases) and outcomes, an outcome rate, and confidence interval for both the physician or hospital of interest and the benchmark (all other cases across California).

The report data is transformed to a single-fact table (one key value pair per data row). This data structure is efficient to query with SSRS. While we aspired to a hierarchical data structure, the compromise is a key linked star schema with a single value fact table, two dimension tables and a table defining parent-child-grandchild relationships.

Reporting Layer

A condition of access to the California Health and Human Services data is an agreement to not identify or contact any individual to whom the data pertains. In compliance with this expectation, the presentation (report) server contains only summary data presented as a one-fact-per-row table. This data structure severely limits the ability to cross-tabulate the data to discover any patient-level information. The external facing presentation server is disconnected from the secured patient level data (BI) server. Data are moved from the (analysis) data warehouse to the presentation server via external hard drive with access control—the only connection ever made between the two systems. Where report users require a case series from hospital-supplied data, an MRN listing can be acquired from the interim files stored in the data warehouse.

The ExPLORE Clinical Practice data are loaded in an SQL Server database, from which they are queried and presented through SSRS. Data architecture has focused on ensuring that page load speeds are acceptable (< 10 seconds). Many modifications have been made to increase the speed, including nesting and streamlining queries and partitioning the fact table into indexed treatment-group specific tables to improve retrieval efficiency.

ExPLORE Clinical Practice hospital reports are delivered via a website (explorecp.org) that points to the dedicated presentation (SSRS) server within the PAMFRI firewall. The system is user-name and password protected and currently has more than 100 registered users. Users (hospital representatives) have access to their own cases compared with all like cases reported to ExPLORE Clinical Practice. To ensure privacy protection, users are currently added manually by a system administrator and are not permitted to automatically change or retrieve their passwords. Dynamic tables within queries are specific to the user and session, preventing multiple user conflicts.

Reports for Sutter Health hospitals are delivered by a similar service within the Sutter Health intranet. Authentication is inherited from the existing Sutter Health Microsoft Windows NT Local Area Network Management system.

Physicians and hospitals have access only to data for their own practice and hospital. For physician data, the Chief Medical Officer has access to data on every physician treating patients at the organization. This is currently managed manually to ensure verified user credentials.

Results

Four report views are available for both hospital- and physician-level data. Figure 3 shows a sample summary report for scheduled colectomy procedures (irrespective of their indication) for the “de-identified hospital.” Outcomes for each cluster of measures are plotted (e.g., NPOA adverse drug events and cardiovascular outcomes) with column graphs illustrating the mean (star or dot) and 95 percent CI (bar) for the local measure (green) and for California as a whole (blue). The first graph reports patient outcomes arising during the index admission; the second graph reports outcomes arising as readmissions. The full complement of clusters is listed on the right-hand side with expandable information on how each outcome is defined using ICD-9-CM codes.

Figure 3.

Sample Summary Report of Groups (Bundles) of Outcome Measures

Although the cluster level outcomes themselves are of limited value, they are an important navigation aid for the user. For example, outcomes relating directly to hemorrhage (i.e., documented hemorrhage or anemia due to hemorrhage and various transfusion of blood product pairings) are clustered together. Similarly, cardiac complications are clustered together. At present, each clinical pairing is presented in only one place. Clusters can be adjusted (via the data dictionary) as more logical groupings are identified.

Figure 4 illustrates the capacity to drill down into a particular cluster to examine the component outcomes. Here, graphs are again provided for both index admissions and for readmissions. These maintain the same graphics for comparing local and California-wide rates. In addition, fields are available for comments from exemplary performers, from the Health Information Manager who contributed to the data definitions and, occasionally, comments from the ExPLORE Clinical Practice team that may assist with interpretation of the outcomes.

Figure 4.

Sample View of the Measures in Each Bundle

Figure 5 shows a screen view for the all cause 30-day readmission rate, in this case for colectomy procedures at the hospital level. The graph has space for comment from exemplary performers on practices that contribute to reducing readmissions.

Figure 5.

Sample of the All-Cause 30 Day Readmission Rate View

The graphic in Figure 6 plots the proportion of episodes with length of stay at each day up to 9 days, in this view, for total hip replacement. This example shows that the de-identified hospital has a discharge profile for total hip replacement procedures that is a little shorter than the profiles for other hospitals in this comparison (Figure 6—faint grey lines). However, two hospitals have length of stay profiles that are quite different to the comparator hospitals (Figure 6—red circles): more than half of the patients are discharged on day one (left-hand red circle), while almost half of the patients at another hospital are discharged on day 4 (right-hand red circle). This variation is further emphasized in the average length of stay (ALOS) graph where ALOS varies between 1.5 days for Hospital 15 and 5 days for Hospital 392. A second graph illustrates that the ALOS for hospitals 15 and 392 is not highly influenced by the proportion of patients staying more than twice the statewide ALOS for this procedure (a long-stay outlier problem). However, this may be an issue for Hospital 395.

Figure 6.

Sample View of Length of Stay (LOS)

Discussion

We believe that we are well advanced in making ExPLORE Clinical Practice an easy to use, intuitive, quick loading, and low maintenance feedback system for California surgeons and hospitals. The design accounts for busy clinicians who need timely and highly specific feedback about their patient outcome metrics. Loading times have been a particular focus for minimizing clinician frustration in using the system. We see this information as the start of local conversations about improvement of the quality of patient outcomes, and as a means of generating peer advice around how to achieve such improvements.

The ExPLORE Clinical Practice program is an innovative approach to improving outcomes for patients. While other programs include some of its features, this is the only program that we are aware of that incorporates the following:

A deductive approach to identifying a broad suite of outcome variables. This results in a less prescriptive suite of outcomes variables that can be used for exploring outcome improvement opportunities, reducing resource utilization, and improving documentation;

Independent non-Gaussian statistical tests for a broad range of outcomes;

An interactive interface;

Outcome data that is used to identify a pathway to improving the reported outcomes through peer comments;

Readmission outcomes that include all readmissions irrespective of the California hospital to which the patient is readmitted;

The identification of practice variation between hospitals

A platform for creating data-driven jurisdictionwide best practice; and

An automated process designed for frequent (e.g., weekly to monthly) data updates.

During the piloting process, physicians with positive results received the data enthusiastically. Physicians with below-average results, the ones most likely to benefit from feedback, were quick to discard the results.20 When we investigated this further, we found that a critical missing ingredient is a pathway to improved outcomes. Specifically, the feedback was implicitly making physicians feel accountable for outcomes without providing access to tools with which they could improve their outcomes.21 This need for a coupling of results with pathways to clinical solutions resulted in the program adopting a focus on positive deviants.22 The goal is to identify the best performers, learn how they achieved those results, and then make those lessons available to the ones who could do better.

The first of the pathway projects, Hospital Inpatient Transfusion Reduction, is uncovering some distinguishing approaches from hospitals with very low transfusion rates. This project is helping us to develop our methods for efficiently harvesting information from positive deviants. Preliminary findings have identified hospital-initiated structures and processes that may contribute to transfusion reduction. These include creative approaches to correcting anemia preadmission, preventing iatrogenic anemia, and educating physicians and nurses on the appropriate use of blood products.

The ability to discriminate positive deviants from apparent good results due to random variation is a strength of the non-Gaussian statistical approach applied to ExPLORE Clinical Practice reporting. In addition, the statistics (CI estimates) are valid for both frequent and infrequent (<5 percent) outcomes and small numbers of cases (even single case cells).

Historically, one of the limitations of quality improvement is the inability to identify practices that result in high quality care and then transfer this learning between consulting rooms, organizations, states, and countries. Working in two countries, the ExPLORE Clinical Practice team has noted that the same clinical questions arise. For example: Should an intraoperative cholangiogram (a radiological procedure used to prevent common bile duct injury and detect gall stones) be conducted as a routine component of a cholecystectomy (gall bladder removal operation)? This is one of the first surgical outcome questions that was asked both in Victoria, Australia in 2005 and California in 2011. Despite this potentially significant change to practice, it is only recently that this clinical issue has been the focus of a randomized, controlled trial.23

Being able to distinguish positive deviants from random variation is important, as many seemingly logical “good ideas” have not resulted in better outcomes for patients (e.g., extracranial and intracranial bypass surgery for stroke prevention).24,25 The concept of the continuous learning health care system envisages just such outcome evaluation as that we have designed into the ExPLORE Clinical Practice system.26 By using positive deviants to generate small modifications to practice that might not otherwise be tested or generalized in clinical practice, we believe that the speed of quality improvement innovation can be enhanced, with built-in mechanisms to evaluate outcomes.

Data driven, consensus based, best practice protocols have been a staple of the Palo Alto Medical Foundation (PAMF)’s Variation Reduction Program.10 The program seeks to eliminate tests and procedures that do not contribute to the quality of patient care through consensus-based protocols established by small groups of physicians. The program estimates cost savings to PAMF patients of $31 million. The ExPLORE Clinical Practice program takes a similar approach, albeit with a hospital (rather than consulting room) focus, a larger group of participants, and more sophisticated statistics.

The true impact of treatments can only be done efficiently with large—preferably jurisdictionwide—routinely collected patient data sets. As information technology capacity and electronic data acquisition improves, jurisdictionwide health information becomes more robust and captures more of the information required to answer the questions arising from this complex domain. Although this approach has been advanced by very sophisticated mathematical methods,27 there is an opportunity to use jurisdictionwide data to detect rates of detailed, interconnected complications of care.

ICD-10, which will be introduced in the United States in October 2015, will bring implementation challenges, but it will also bring opportunities. For example, Januel et al. found that comorbidity detection improved over a five-year period (1999–2003) following the introduction of ICD-10 in Canada.28 There will also be an opportunity to implement the ICD-10 based Classification of Hospital Acquired Diagnoses (CHADx).29 CHADx uses ICD-10’s richer detail for describing complications, and takes account of multiple code sequences that better define untoward patient outcomes. This will allow ExPLORE Clinical Practice to expand the current suite of outcomes.

The abstraction of routinely collected data from medical records is prescribed through legislation and collected in a similar form across jurisdictions in most, if not all, developed countries including all Australian states; 48of the 50 states in the United States; all Canadian provinces; New Zealand; England; Scotland; Ireland; and most other European health care systems. This provides an opportunity to identify positive deviants across a much larger pool of clinicians and hospitals. Collaboration between Australia and the United States has already commenced through ExPLORE Clinical Practice, and that could be readily adapted for data from other jurisdictions.

Reports are currently shared with hospitals participating in the Hospital Inpatient Transfusion Reduction Study through http://www.explore.org. A new tranche of reports that include a pathway to improvement (comments and suggestions) section are being shared with select Sutter Health Hospitals and physicians in an attempt to implement the preliminary findings of the Hospital Inpatient Transfusion Reduction Study.

Although the administrative data used in ExPLORE Clinical Practice has been reported to be highly reliable for many purposes,30 capable of improvement.31 and more accurate than comparative registry data,32 some skepticism remains regarding its use. The limitations in the sensitivity (capture rate) of the data are well documented.19,33 For example, in the ExPLORE Clinical Practice pilot a surgeon could not find a known case involving paralytic ileus. Upon review, the medical record documented treatment for the ileus but no specific documentation of ileus diagnosis. We have revised this complication pairing to use naso-gastric tube insertion to assist in identifying these cases. Our finding is not unique. Casez et al. have found that data collected by technicians or physicians did not record DVT in the medical record of more than 40 percent of diagnosed and treated cases; documentation of a more serious condition, pulmonary embolus, was more frequently recorded.34 There appears to be a high level of specificity; we are yet to find a clinical pairing in the ExPLORE Clinical Practice data that has been questioned and not found in the clinical record.

There have also been instances where the ExPLORE Clinical Practice team has misinterpreted the data. For example, admission for rehabilitation can be represented with two code sequences, which, for a short period, led to overrepresentation of readmissions statewide. The ExPLORE Clinical Practice team’s online comments draw attention to such limitations, and are designed specifically to assist with data interpretation by adding details regarding the data source and definitions.

Future Developments

Physician data, where they exist within ExPLORE Clinical Practice reports, assign a single surgeon or physician to each case. We intend to allow for multiple attributions (i.e., the ability to attribute each case to all treating physicians in accordance with their role in the admission or readmission) as a better approach to feeding back data. It resolves “not my patient” issues and ensures that surgeons and physicians accepting payment for treating the patient are also accepting responsibility for the outcomes of their treatment. This will enhance teamwork and will prevent cases that have involved a lack of coordination from being placed in the “too hard basket” or being dismissed as “nobody’s problem.” This analytic approach will entail significant modification to the ETL layer.

Security access to the aggregated data reported through the external website (http://www.explorecp.org) is currently set manually. A feature permitting automated password assignment, retrieval, and reset will permit streamlining the enrollment of ExPLORE Clinical Practice report users.

Conclusion

ExPLORE Clinical Practice is one of many initiatives attempting to lever available data to improve patient outcomes. The program has evolved from the initial intent of automated, statistically robust, clinically relevant detailed data feedback alone to a more sophisticated user friendly, web-based reporting service that identifies and propagates exemplary practice. We feel we have made important advances in using data to support clinicians’ desire to improve care for patients, and we look forward to undertaking evaluative studies on uptake and comparative outcomes. The continual and pragmatic approach to improving the accessibility and usefulness of ExPLORE Clinical Practice bodes well for ongoing development to improve outcomes of clinical care for patients.

Patient group	Patient sub-group
Colectomy Procedures	All Procedures
Colectomy Procedures	Crohn’s Disease and Ulcerative Colitis
Colectomy Procedures	Diverticula and Benign Neoplasm
Colectomy Procedures	Laparoscopic Procedures
Colectomy Procedures	Malignant Neoplasm Colon
Colectomy Procedures	Open Procedures
Colectomy Procedures	Other Diagnoses
Colorectal Procedures	Crohn’s Disease and Ulcerative Colitis
Colorectal Procedures	Diverticula and Benign Neoplasm
Colorectal Procedures	Laparoscopic Procedures
Colorectal Procedures	Malignant Neoplasm of Rectum
Colorectal Procedures	Open Procedures
Colorectal Procedures	Other Diagnoses
Colorectal Procedures	All Procedures
Lobectomy Procedures	All Procedures
Lobectomy Procedures	Malignant Neoplasm of Lung
Lobectomy Procedures	Other Diagnoses
Bariatric Procedures	All Procedures
Lumpectomy Procedures	Benign Neoplasm
Lumpectomy Procedures	All Procedures
Lumpectomy Procedures	Malignant Neoplasm
Lumpectomy Procedures	Malignant Neoplasm with Lymph Node Excision
Lumpectomy Procedures	Other Diagnoses
Oophorectomy Procedures	Laparoscopic Procedures
Oophorectomy Procedures	Malignant Neoplasm Ovaries
Oophorectomy Procedures	Benign Neoplasm Ovaries
Oophorectomy Procedures	All Procedures
Oophorectomy Procedures	Open Procedures
Oophorectomy Procedures	Other Diagnoses
Prostatectomy Procedures	BPH
Prostatectomy Procedures	Ca Prostate
Prostatectomy Procedures	All Procedures
Cholecystectomy Procedures	Acute Cholecystitis
Cholecystectomy Procedures	Biliary Cholic
Cholecystectomy Procedures	All Procedures
Cholecystectomy Procedures	Bile Duct Stones
Cholecystectomy Procedures	Laparoscopic Procedures
Cholecystectomy Procedures	Open Procedures
Cholecystectomy Procedures	Other Diagnoses
Hysterectomy Procedures	Benign Neoplasm
Hysterectomy Procedures	All Procedures
Hysterectomy Procedures	Laparoscopic Procedures
Hysterectomy Procedures	Malignant Neoplasm
Hysterectomy Procedures	Malignant Neoplasm with Lymph Node Excision
Hysterectomy Procedures	Open Procedures
Hysterectomy Procedures	Other Diagnoses
Hysterectomy Procedures	Vaginal Hysterectomy
Total Hip Replacement	All Procedures
Total Knee Replacement	All Procedures
Diagnostic Colonoscopy Procedures	Benign Neoplasm
Diagnostic Colonoscopy Procedures	All Procedures
Diagnostic Colonoscopy Procedures	Inflammatory Bowel Desease
Diagnostic Colonoscopy Procedures	Intestinal Infection
Diagnostic Colonoscopy Procedures	Malignant Neoplasm
Diagnostic Colonoscopy Procedures	Screening Colonoscopy for Malignant Neoplasm
Mastectomy Procedures	All Procedures
Mastectomy Procedures	Malignant Neoplasm
Mastectomy Procedures	Other Diagnoses
Heart Failure, Procedures	All Cases
Heart Failure, Procedures	Angioplasty
Heart Failure, Procedures	Pacemarker for Unspecified HF
Heart Failure, Procedures	Resynchronization
Heart Failure, No Procedures	All Cases
Heart Failure, No Procedures	Diastolic Heart Failure
Heart Failure, No Procedures	Unspecified Heart Failure
Heart Failure, No Procedures	Systolic Heart Failure
Pneumonia (CMS Definition)	All Cases
Myocardial Infarction (CMS Definition)	All Cases
AAA Repair, Unscheduled	Not Ruptured
AAA Repair, Unscheduled	All Procedures
AAA Repair, Unscheduled	Ruptured
AAA Repair, Unscheduled	Other Diagnoses
AAA Repair, Scheduled	Not Ruptured
AAA Repair, Scheduled	All Procedures
AAA Repair, Scheduled	Other Diagnoses
Cataract Procedures	All Procedures