Monitoring intensive care unit performance-impact of a novel individualised performance scorecard in critical care medicine: a mixed-methods study protocol.

INTRODUCTION: Patients admitted to a critical care medicine (CCM) environment, including an intensive care unit (ICU), are susceptible to harm and significant resource utilisation. Therefore, a strategy to optimise provider performance is required. Performance scorecards are used by institutions for the purposes of driving quality improvement. There is no widely accepted or standardised scorecard that has been used for overall CCM performance. We aim to improve quality of care, patient safety and patient/family experience in CCM practice through the utilisation of a standardised, repeatable and multidimensional performance scorecard, designed to provide a continuous review of ICU physician and nurse practice, as well as departmental metrics. METHODS AND ANALYSIS: This will be a mixed-methods, controlled before and after study to assess the impact of a CCM-specific quality scorecard. Scorecard metrics were developed through expert consensus and existing literature. The study will include 19 attending CCM physicians and approximately 300 CCM nurses. Patient data for scorecard compilation are collected daily from bedside flow sheets. Preintervention baseline data will be collected for 6 months for each participant. After this, each participant will receive their scorecard measures. Following a 3-month washout period, postintervention data will be collected for 6 months. The primary outcome will be change in performance metrics following the provision of scorecard feedback to subjects. A cost analysis will also be performed, with the purpose of comparing total ICU costs prior to implementation of the scorecard with total ICU costs following implementation of the scorecard. The qualitative portion will include interviews with participants following the intervention phase. Interviews will be analysed in order to identify recurrent themes and subthemes, for the purposes of driving scorecard improvement. ETHICS AND DISSEMINATION: This protocol has been approved by the local research ethics board. Publication of results is anticipated in 2019. If this intervention is found to improve patient- and unit-directed outcomes, with evidence of cost-effectiveness, it would support the utilisation of such a scorecard as a quality standard in CCM. © Article author(s) (or their employer(s) unless otherwise stated in the text of the article) 2018. All rights reserved. No commercial use is permitted unless otherwise expressly granted.

Scorecard design derived from evidence-based, validated tools that are used in critical care medicine worldwide.

Scorecard designed with ‘balanced composites’, which aims to reduce performance that is targeted specifically toward scorecard metrics rather than patient-centred care.

Multidisciplinary focus on both physician and nursing care.

Qualitative interviews used to gauge impact and attitudes towards scorecard utilisation, for the purposes of driving improvement of this tool.

Study will only use 19 attending critical care physicians.

Introduction

In this era of improving patient safety and healthcare provider performance, the popularity of scorecards and use of metrics-based analysis has expanded.1–3 Add to this the increased pressures of financial constraints and austerity policies, an increased emphasis and almost cult-like dependency on scorecards and metrics has arisen in healthcare.4 5 This practice in medicine follows the path taken by other fields including education and business administration. Dependency on the use of scorecards and metrics in other fields has potentially contributed to nebulous actions including teaching to the exam, and directing performance towards the scorecard measures instead of the overarching goal of the profession.5 A further challenge in medicine is that patient care is often provided by teams, and the most responsible provider changes over time. Additional challenges include that, for many measures, a value can be two-tailed (ie, too high and too low) and some measures are in direct opposition to others. Thus, tools must be able to account and assess performance variables that providers influence and control, while also reflecting important patient outcomes.

By their very nature, patients admitted to critical care environments, including an intensive care unit (ICU), are most susceptible to adverse and catastrophic outcomes if healthcare provider performance is suboptimal.6 The ICU is also an environment where invasive procedures and treatments are frequently performed, which involves increased risk to patient safety. Critical care environments typically have a multidisciplinary team approach, and the most responsible physician (MRP) usually changes on a frequent basis (with occasionally some units routinely having at least two physicians provide care daily).4 Thus, if a patient stays in an ICU for even just 2 weeks, easily five or six physicians and over 20 nurses could have provided care to that one patient and thus influenced their outcomes. While the ICU promotes a team-based approach to care, the opportunity for individual variations in bedside care to impact patient outcomes is very high. Examining individual performance is aimed at improving care by promoting adoption of agreed-upon best practice, and identifying both positive and negative deviance from peer groups delivering care. Add to this the financial impact of critical care units, which often account for over 10% of most hospital budgets with a clear expectation of marked growth in the coming decade,7 a strategy to optimise healthcare provider performance in terms of patient care and efficient use of resources is required. Hospitals also seek methods of monitoring performance, in order to further drive changes in overall quality of healthcare delivery.

To address this need, hospitals and organisations are quickly adopting various performance measure tools.8 Such tools have been applied in a wide array of healthcare applications, including hospital-wide venous thromboembolism prophylaxis, vaccination delivery and medical education.9–11 These performance measure systems are thought to be beneficial, as they highlight shortcomings in care delivery, and identify areas for further improvement.8 Specifically, performance scorecards have been used by healthcare institutions and departments for the purposes of monitoring clinical performance and driving quality improvement.12 In addition, evidence-based evaluation and recommendations have been made regarding quality indicators that must be regularly assessed, and are believed to be associated with improved patient outcomes.13 Unfortunately, most of these performance scorecards are using overall outcomes that are hard (if not impossible) to directly attribute to an individual. For example, ventilator-associated pneumonia (VAP) is a common measure. As VAP takes days to develop, it is difficult to attribute this complication to a single individual. Further, many of the measures on existing scorecards are assessed in a one-tailed manner. For example, mortality rates are assessed with a view that lower values are ideal. However, as most high-acuity ICUs in Canada have a mortality rate of approximately 20%, a very low value would raise questions regarding an individual physician’s practice. To achieve a very low value, a physician would have to either admit patients who are less sick than other critical care physicians and/or not conduct end-of-life patient discussions—both of which are undesirable characteristics for effective practice.14 An ideal analysis of ICU performance should include patient safety measures but also measures of throughput, resource utilisation, cost, wellness and patient/family satisfaction.15 Such a tool should be able to assess the potential impact of each individual healthcare provider and ideally be transferable from one ICU to another with applicability in a variety of settings and centres, and account for changing healthcare providers. The tool needs to adjust for measures that are two-tailed in nature and also include strategies to prevent ‘gaming’ (ie, targeting performance to optimise scorecard performance, and not the best interests of the patient). While attempts have been made to identify quality indicators that may comprise such a scorecard,13 16 17 at present there is no widely accepted or standardised scorecard that has been used specifically in critical care.

Aim

The primary purpose of this study is to perform a mixed-methods, controlled before and after study to assess the impact of an individualised, multidisciplinary clinical quality scorecard (specifically designed for utilisation in the ICU) on various performance metrics for ICU attending physicians and nurses, along with barriers and facilitators to its utilisation. Performance metrics have been identified as factors associated with improved patient outcomes, and are related to daily bedside care (see table 1). These include ventilation weaning strategies, sedation, delirium, analgesia and glycaemic control. Secondary aims include impact of critical care nursing operation.

Table 1

Validated tools for scorecard inclusion

Variable	Tool to be used
Pain (for patients that can self-report)	Numeric Rating Scale, Visual Analogue Scale19 24
Pain (for patients that are unable to self-report)	Critical Care Pain Observation Tool25
Delirium	Confusion Assessment Measurement for the Intensive Care Unit (ICU)26
Sedation	Richmond Agitation and Sedation Scale27
Mobility	ICU Mobility Scale28
Mechanical ventilation weaning	Spontaneous breathing trials36
Fluid balance in acute lung injury	Conservative fluid strategy29
Blood glucose management	Avoidance of hyperglycaemia30 31
Physician feedback	Ottawa assessment tool
Family and patient satisfaction	Ottawa assessment tool

Variables for inclusion in the scorecard will be measured using evidence-based, validated clinical tools from the critical care literature.

Methods and analysis

This will be a mixed-methods, sequential explanatory study design.18 The quantitative component will be a single-centre, controlled before and after analysis of a novel, individualised, ICU-specific performance scorecard. The qualitative component will be based on interviews with ICU attending physicians and registered nurses about their experiences with the scorecard. Integrating these findings will allow for a better understanding of how the intervention may or may not have influenced staff behaviour, whether there were any unintended consequences and what factors were influential. The study will be conducted in an academic health centre with two urban mixed medical/surgical ICUs in Ontario, Canada, with a combined annual census of over 3000 admissions per year.

Participants

This study will include 19 attending critical care physicians and approximately 300 critical care registered nurses. All attending physicians who were hired in the last two decades have completed fellowship training in adult critical care medicine (CCM), and all have faculty positions in the University of Ottawa. Study flow for attending physician participants is depicted in figure 1.

Figure 1

Study flow. Flow of physician subjects through the study. Data will be collected 6 months prior to the intervention (the initial provision of scorecard data to subjects). Following the intervention, there will be a 3-month washout phase, where no data will be collected. After this washout phase, postintervention data will be gathered for 6 months. Subjects will continue to receive scorecard data on a quarterly basis.

Sample size

This intervention has not been tested in any previously published study or clinical trial protocol. Therefore, sample size will be pragmatically determined based on the existing attending critical care physicians and critical care registered nurses currently employed at The Ottawa Hospital, without exclusion.

Procedure

Development of the scorecard

The scorecard has been developed in collaboration with staff at both hospitals. The scorecard criteria were selected based on expert opinion and existing literature. A group of senior critical care clinicians and administrators (with responsibilities for governing quality and safety within The Ottawa Hospital ICUs) proposed developing and piloting a scorecard specifically for monitoring comprehensive indicators of critical care performance, given that existing scorecards focus particularly on overall patient safety outcomes, but little on other performance measures.6 Canadian efforts have been directed at developing national quality indicators at a unit level.16 Most of these measures are thus difficult (if not impossible) to attribute to quality of care delivered by individual physicians or nurses. Our scorecard was specifically developed to examine quality of care at the bedside for individual physicians and nurses, and therefore the majority of measures consist of once or twice daily recorded measures. Both physicians and nurses will use the same scorecard, though some components (eg, admission frequency, discharge frequency) will only be measured for physicians. The measures chosen for nurses were selected after review with nurse managers and educators, in identifying clinically impactful measures that nurse bedside actions could influence. As these measures have an expected incidence, scores are similarly designed in relation to their peer group, as opposed to an arbitrary target.

Attempts have been made to address ‘gaming’ actions by incorporating multiple measures and creating so-called ‘balanced composites’. The balanced composites combine three or four individual measures that are counter or opposed to one another. For example, the respiratory balanced composite includes extubation numbers (ideally high), extubation 48 hours failure rate (ideally low) and percentage of patients ventilated (ideally not too high or too low). Thus, if an intensivist were to direct performance towards the extubation failure rate by extubating very few patients, then their extubation number would be low and they would become an extreme for ventilation percentage.

As stated, incorporation of a variety of measures attempts to make such ‘gaming’ less probable, as it becomes too difficult to direct performance at all variables. Further attempts to avoid directing performance to a specific measure have been to incorporate the use of several ‘two-tailed’ variables. Such a variable is one that is ideally neither too high or too low. Examples of this include mortality percentage (as previously mentioned), admission frequency and number of discharged patients.

There are currently several areas of bedside clinical care that have been demonstrated to improve patient outcomes in ICUs. For example, optimising analgesia and sedation management, and reducing/preventing delirium are important variables that have been shown to decrease the duration of mechanical ventilation and ICU length of stay, and also improve the patient experience.19–23 Performance with regard to these variables will be conducted using existing standardised, validated and guideline-recommended scales and tools (see table 1).19 24–31

Multidisciplinary feedback on physician performance including communication and professionalism will also be incorporated and analysed through the use of The Ottawa Hospital assessment tool—a tool in which anonymous feedback is collected from up to 10 physician colleagues and 10 non-physician allied health professionals.

Scorecard data collection and feedback to staff

Patient data will be collected daily from patient bedside charts and flow sheets by Critical Care Database clerks, entered into a database and then compiled into individual score cards. The charts and flow sheets contain all of the patient information used in the scorecard (see table 2). Audits of quality review have been conducted from January 2016 to January 2017, to ensure data entry by Critical Care Database clerks correlated with actual patient data. For each patient day, an MRP will be assigned, according to the attending physician responsible that day. For each patient day shift and night shift, a corresponding ICU nurse will be assigned. Patient data will be compiled on a web-based server platform at The Ottawa Hospital that uses the Excel platform for data reporting. Scorecard data for each attending physician and nurse will be compiled from cumulative patient data using a preprogrammed algorithm. The population’s raw data median score, target score and the individual attending physician’s score will be displayed.

Table 2

Performance scorecard

Measure	Individual score	Population median score	Target score
Mortality (%)
Efficiency
No. of admitted patients/day
No. of patients discharged/day
Sensorium
% of CAM+ patients
% of CAM+ patients receiving sedation
% of patients with RASS score −4 or −5
Analgesia
CPOT daily score
% of patients with CPOT >2
Mobilisation score
Ventilation and weaning
% of patients maintained on ventilator
Extubation %
Reintubation rate <48 hours after planned extubation
% SBT of ventilated patients
Fluid balance
Median fluid balance
Median fluid balance/24 hours
Prevention
% patient days with blood glucose >12
% patient days with central line
% patient days receiving nutrition
Physician feedback
Ottawa assessment tool score
Family/patient satisfaction
Ottawa assessment tool score

An example of the proposed scorecard, including the subject’s individual score, the population median score and the ideal target score.

CAM, Confusion Assessment Measurement; CPOT, Critical Care Pain Observation Tool; RASS, Richmond Agitation and Sedation Scale; SBT, Spontaneous Breathing Trial.

For all variables (other than fluid balance and family meetings) in addition to the raw data above, an individual score will be assigned. A target score range will be determined from the variable in question (eg, delirium rate target rate is lowest possible). For attending physicians who are in the target range, a score of 0.75 will be given. Scores that fall greater than 2 SDs above the target score will then be assigned a score of 1.0. This scorecard data will be calculated using preprogrammed algorithms, and are planned to be distributed to members on a quarterly basis.

Study flow is depicted in figure 1. Scorecard data will be provided to participants either quarterly (in the case of attending physicians) or annually (in the case of nurses). Each attending physician will meet with the department head, following the 6-month period of baseline data collection. At this meeting, the participant will be informed of the purpose and function of the scorecard, and will be provided with their individual scorecard report, based on the baseline data collection. There is otherwise no defined structure to this meeting. Following this meeting, attending physicians will continue to directly receive scorecard reports every 3 months, but only meet with the department head every 6 months. Participants will be provided scorecard performance during the study period, but will not receive feedback from the department head. The ICU nursing manager at each hospital will meet with individual nurses on an annual basis, and will have access to both the semiannual and annual scorecard data for each nurse. No study data will be gathered in the immediate 3 months following the release of the first scorecard report. After this 3-month period, postintervention data will be collected for 6 months.

Primary outcome

The primary outcome will be change in performance metrics following the provision of scorecard feedback to subjects. We will perform descriptive statistics in order to analyse data between-groups (ie, cumulative ICU departmental performance preintervention and postintervention) and within-subjects (ie, change in individual participant performance metrics preintervention and postintervention). Statistical analyses will be performed using commercially available software. Summary results will be presented as mean values (with SD) or median (with IQR). To test differences before and after scorecard implementation, the Student’s t-test will be used.

Qualitative data

Qualitative data collection

At each campus, interviews will be conducted once the intervention phase has been completed. Interviews are expected to last between 30 and 60 min via telephone or in locations agreed on with interviewees. Subjects will be asked questions related to the performance metrics they received. They will also be asked questions related to their experiences of receiving scorecard feedback and any potential impact on their practice.

Qualitative data analysis

All interviews will be audio recorded and transcribed. Interviews will be analysed using data-driven thematic analysis,32 beginning with listening to audio recordings while reading transcripts, followed by multiple close readings of each transcript. Two investigators will independently analyse a preliminary number of transcripts to inductively develop preliminary codes. After this, the investigators will meet to discuss and negotiate the framework in order to arrive at a consensus. One investigator will then proceed with coding of the remaining transcripts using constant comparison. The data will then be charted, mapped and interpreted in order to describe the themes and subthemes.

Ethical perspective

Participants will receive written and verbal information regarding the study, followed by written informed consent forms. Information will be provided on the purpose, the procedures, the voluntary nature of their participation and the option to withdraw at any time. Participants will also receive information regarding confidentiality, protection of their data and secure data storage.

Discussion

ICU care is focused on a very complex patient population, and as a result often requires significant resource utilisation for optimal functioning.33 For this reason, ongoing monitoring of performance metrics is necessary for the purposes of maximising efficiency and patient care. To our knowledge, while previous scorecards have been developed to monitor patient safety specifically,6 at present there is no single scorecard that has been developed to comprehensively monitor all aspects of ICU performance, particularly at the individual level. While patient safety is of particular importance, there are also other variables that are integral to optimal ICU functioning, including throughput, output, management of delirium, pain management and patient/patient family satisfaction.16 17 Our scorecard also incorporates many quality indicators that have been identified through consensus recommendations as important for improving outcomes, but are also attributable to individuals. An analysis on the effect of scorecard implementation (pre and post) on standard quality measures13 will also be considered, given the potential impact improving scorecard measures on individual performance may have on other system-level quality measures. We feel that this will provide a more complete perspective of ICU performance. Understanding of ICU performance is integral to identifying gaps in the quality of healthcare delivery. Utilising this scorecard, therefore, may allow for recognition of areas where both positive and negative deviance occur, and offer opportunities for promotion of positive deviance, quality improvement among individuals and promotion of overall patient safety and quality improvement for departments and hospitals.

What is also unique to this study is the mixed-methods design. While quantitative data are important for depicting change in performance, performing qualitative analysis allows for the identification of themes or factors that may facilitate or hinder the utilisation of the scorecard. Further to that, the concept of monitoring performance is still fraught with controversy, and how to do this task optimally is still a matter of debate.34 35 Identifying themes related to the function of this scorecard will allow for appropriate alterations to be made. To our knowledge, a mixed-methods design has not been previously employed in the development or evaluation of any critical care-specific scorecards.

While this proposed study presents a novel method for analysing performance in the ICU, there are some inherent shortcomings that should be discussed. First, even though this study will be carried out in two different ICUs, they occur within the same system. Given the known geographical variation in ICU admission criteria and daily practice that exists worldwide, our findings may not be equally applicable in all centres. However, while the practices of ICU admission and management may vary geographically, the before and after nature of the analysis should provide a valid representation of how the intervention (ie, the scorecard) ultimately influences performance. Furthermore, our sample of physicians is relatively small (19 participants), and therefore may result in more skewed results if there are outliers present. While this remains a possibility, it again should not impact the validity of the scorecard effect, as both ‘before’ and ‘after’ groups will use the same participants.

The overall goal of this study is to identify a systematic means of gathering comprehensive data on individual clinical performance, and feeding it back to individual clinicians in order to provide them with information on current performance, and also identify areas for improvement. As mentioned, no such scorecard currently exists in CCM. The development and utilisation of such a scorecard will provide a standardised method for monitoring ICU performance, and will hopefully lead to improvement in patient care and optimal patient outcomes.

Ethics and dissemination

Findings from this study will be submitted for peer review in an appropriate journal. Findings will be presented to researchers and clinicians at suitable conferences.