Voluntary peer review as innovative tool for quality improvement in the intensive care unit--a retrospective descriptive cohort study in German intensive care units.

INTRODUCTION: Quality improvement and safety in intensive care are rapidly evolving topics. However, there is no gold standard for assessing quality improvement in intensive care medicine yet. In 2007 a pilot project in German intensive care units (ICUs) started using voluntary peer reviews as an innovative tool for quality assessment and improvement. We describe the method of voluntary peer review and assessed its feasibility by evaluating anonymized peer review reports and analysed the thematic clusters highlighted in these reports.
METHODS: Retrospective data analysis from 22 anonymous reports of peer reviews. All ICUs - representing over 300 patient beds - had undergone voluntary peer review. Data were retrieved from reports of peers of the review teams and representatives of visited ICUs. Data were analysed with regard to number of topics addressed and results of assessment questionnaires. Reports of strengths, weaknesses, opportunities and threats (SWOT reports) of these ICUs are presented.
RESULTS: External assessment of structure, process and outcome indicators revealed high percentages of adherence to predefined quality goals. In the SWOT reports 11 main thematic clusters were identified representative for common ICUs. 58.1% of mentioned topics covered personnel issues, team and communication issues as well as organisation and treatment standards. The most mentioned weaknesses were observed in the issues documentation/reporting, hygiene and ethics. We identified several unique patterns regarding quality in the ICU of which long-term personnel problems und lack of good reporting methods were most interesting
Conclusion: Voluntary peer review could be established as a feasible and valuable tool for quality improvement. Peer reports addressed common areas of interest in intensive care medicine in more detail compared to other methods like measurement of quality indicators.

Introduction

The daily routine in intensive care medicine is very rich in complex processes, has numerous different acting groups and the likelihood of adverse events is high [1], [2]. Such an adaptive system is difficult to monitor with regard to quality and a “gold standard” of quality assessment in intensive care units (ICUs) remains to be defined [3], [4]. Two important questions arise in this setting: “Does a single patient receive care based on proven evidence based principles?” and secondly: “Does the patient really get what we think he gets?” [5], [6]. Quality improvement and especially methods that enable clinicians to improve quality have been proposed [7], but implementation remains difficult especially when there is no formal obligation. Moreover, controlling implementation as well as introduction of quality improvement measures is difficult. In this article we propose voluntary peer review in intensive care medicine as a new method to improve quality. This method addresses major obstacles for quality improvement in single intensive care units by focusing on process oriented quality indicators. It is intended to involve the participating unit in a quality improvement process employing a respectful dialogue among peers. So the visitation looses a potential negative connotation as external control visit.

The method of peer-to-peer assessment as a quality improving tool is not unknown e.g. in high risk areas like the nuclear industry [8]. Perhaps the earliest description of medical peer review with regard to quality assessment is found in the Ethics of the Physician written by Ishap bin Ali al-Rahawi (854–931) of al-Raha. It mentions the duty of Arabic physicians to duplicate their patients notes for evaluation by a local council of physicians who proved whether the physician had performed according to the common standards [9]. In 1994 Richard Grol published the method of peer review for quality improvement in medical care [10]. In his observations peer review is an effective method to induce change of a process. It offers also the possibility of life long learning in medical care providers. The herein proposed form of voluntary peer review as a newly developed strategy uses self- and external assessment of quality indicators and reports strength and weakness to the reviewed in a sense of mutual respect excluding blaming and negative sanctions. By using this strategy of “no blaming” the acknowledgement of potential “bad practice” in a visited unit leads to a higher motivation to solve such problems. This leads to quality improvement in the reviewed unit by engaging the acting groups via motivation to change. Vice versa this method offers the visiting peers ideas of particularly good practice in an ICU. So both sides are learning from each other.

The method of voluntary peer review in intensive care medicine has been developed in Germany as a pilot project in 2007 supported by the German Society of Anesthesiology and Intensive Care (DGAI) and the German Interdisciplinary Association of Intensive Care (DIVI) and is conducted in cooperation with the State Chambers of Physicians (Landesärztekammern).

The objective of this study was to analyse the results of the pilot phase of the intensive care peer review procedure in order to assess the feasibility of process and to obtain first insights regarding patterns of ICU assessments. Another intention was to analyse the peer review procedure itself and to potentially improve it if necessary.

Methods

Population

At study begin 35 ICUs had completed a voluntary peer review. A formal request to the participating state chambers of physicians was sent to get permission for anonymous analysis of the peer review reports. 22 ICUs that were representing over 300 patient beds consented to participate. Eighteen peer review reports were complete, including results from questionnaire and SWOT analysis. In 4 reports only a SWOT analysis was available.

The method of voluntary peer review

Peer review is a voluntary tool for single intensive care units to evaluate structural, process and outcome quality. The procedure of a peer review is described in Figure 1 (Fig. 1) and Table 1 (Tab. 1). In short: Upon request a group of peers consisting of experienced intensive care physicians and nurses who were certified following a special peer training (Curriculum published by the German Medical Association (Bundesärztekammer)) performs the scheduled visit (“peer review”) to the ICU. The curriculum can be accessed at http://www.bundesaerztekammer.de/downloads/CurrAerztlPeerReview2013.pdf. A member of the State Chamber of Physicians is accompanying the peers in each review of this project. A questionnaire has been sent beforehand to the institution consisting of questions regarding certain quality aspects (structured questionnaire approved by the DGAI/DIVI). This questionnaire includes quality indicators derived from national and European intensive care societies [11], [12] and is filled in by the institution representatives as means of self-assessment. The same structured questionnaire is then used for external assessment during the peer visit. Comparison of self and external assessment is an essential part of the review process. The visit of the ICU itself consists of a bedside analysis of quality indicators and a peer-to-peer dialogue giving direct feedback including a description of quality goals which concludes the peer review visit. Following this visit a written – strictly confidential – SWOT report is sent only to the head of the ICU. This report is divided into two parts:

Figure 1

Schematic overview over the peer review process

The upper part of the figure shows the steps necessary for preparation of the visitation (encircled with dashed line) consisting of contact between the representatives, the state chamber of physicians and the group of peers taking part in the review. Further steps consisting of the peer review visitation and the preparation of the report are shown.

Table 1

Planned timetable of ICU peer reviews

The fulfilment rate of the questionnaire of the peer visit, which includes aspects of structure, process and outcome. The fulfilment rate that can be reached varies depending on the questionnaire used. Grade of fulfilment is divided in four levels:

(1) Not established

(2) Planned

(3) Partly established

(4) Fully established

The domains address the issues: organization, personnel, patients and relatives, quality indicators and controlling/reporting. The latest version of the questionnaire is available at http://www.divi.de/images/PeerReview/IMPR_52-Fragen-Bogen_V2.5.pdf

The SWOT analysis. It covers all aspects relevant to the peers to describe an overall impression of the unit as well as detailed mentioning of certain aspects included in the quality indicators used for assessment. The application of SWOT analysis is relatively uncommon in a setting like this, but has been used successfully in the context of medical quality before [13].

Data collection

Data were retrospectively recorded from 22 anonymous reports of peer reviews. Data were analysed with regard to number of topics addressed and results of assessment questionnaires. The reports were analyzed in more detail regarding strengths, weaknesses, opportunities and threats (SWOT report). All reports were anonymous. External assessment points were derived from the consented checklist included in the questionnaire mentioned above.

Anonymized peer review protocols were studied and all mentioned items were then listed in a table. Then we categorized these items into thematic clusters that were representing almost all mentioned items of the reports in a meaningful fashion. The SWOT report is separated into four categories in which strengths and opportunities are considered as representative of positive aspects whereas weakness and threats were considered negative. Strength and weakness were regarded as rather short-term items that origin from the internal situation of the respective institution. In contrast, opportunities were regarded as long-term items with a focus on external influences on an institution. Strength and opportunities were assigned an arbitrary value of “+1” and consequently weaknesses and threats were assigned “–1”. These items were also counted and then analysed by a plus-minus analysis resulting in an overall positive or negative sum. The authors had only access to reports without identification of the addressed ICU. We limited the analysis to the complete sample without further differentiation based on ICU characteristics. This was due to data protection reasons.

Statistical analysis

In this analysis descriptive statistics are used. Numerical variables are presented as means with standard deviation where appropriate. No comparative statistical analyses are performed to avoid measures of rank between reviewed institutions.

Results

Characteristics of ICUs

The visited ICUs represented 288 patient beds. From two ICUs the exact number of beds was not available. Since the mean number of beds was 14.4 (Range 7–36) we estimated the total number of beds covered in this study to be over 300. The units were representative for ICUs in Germany regarding level of care, number of beds and responsible discipline organizing care for the patients (anaesthesiology 12 and medical 5 and others/interdisciplinary 4). Basic characteristics of the visited ICUs are shown in Table 2 (Tab. 2). Partly missing values were caused by anonymization of the confidential reports.

Table 2

Characteristics of the ICUs that underwent peer review

Analysis of numerical values representing quality indicators

For all peer-reviewed units the assessment of the peers revealed values of the quality dimensions that were estimated “good”. All units reached a mean value of 85.3% (standard deviation (SD): 6.5, Range: 70.7–93.3%) of maximally achievable points. The results for the different quality dimensions are presented in Figure 2 (Fig. 2). Notably structure indicators had the best values with a mean fulfilment rate of 88.6% (standard deviation (SD): 8.2, Range: 67.9–100.0), whereas process indicators revealed slightly lower values (mean 86.4%, SD: 9.1, Range: 59.1–100.0). The outcome related process indicators reached 82.7% (SD: 10.4, Range: 59.7–97.2) fulfilment rate. Fulfilment rate of reporting/controlling was lowest with 75.9% (SD: 12.2, Range: 59.7–88.6). These results represent the external assessment by the peers. As was stated in most of the reports these observations were in almost complete accord to the self-assessment of the units before the visitations.

Figure 2

Spider chart representing the results of the external assessment questionnaires

Dots represent mean percent of maximally achievable points for one dimension. All units with questionnaires completed (n=18) were evaluated. The questionnaire consisted of 52 questions each of which could reach 1 to 4 points. The percentage of maximally achievable points for each dimension was computed.

Analysis of the SWOT reports

The overall count of items extracted from the records was 632. The median number of items covered per report was 26 (Range 14–45). About 60% of the items are represented by four thematic clusters consisting of personnel (115), organisation (95), standards and procedures (87) and team/communication (69). The remaining 40% of the counts were in a range from 57 to 12 where architecture/equipment had the highest count and ethical, legal had the lowest. Topics concerning personnel, organisation, SOPs, and communication were mentioned more often and in almost any of the SWOT reports. Issues like ethics, management of relatives, hygiene, reporting/documentation, risk management and architecture were mentioned less frequently. Personnel items were found in each report more than five times, whereas for example mentions of hygiene were found less frequently (2.1 per unit). An overview over these results is shown in Figure 3 (Fig. 3).

Figure 3

Spider chart representing the plus/minus analysis of the SWOT reports

Black triangles represent the sum of positive and negative counts per thematic cluster (range from –40 to +40). Stronger line represents zero. Black letters represent high values (above +20). Grey letters represent medium level values (+10 to +20). Grey letters in boxes represent low (0 to +10) and black letters in boxes negative values. The plus/minus analysis was extracted from the 22 SWOT reports (strength, weakness, opportunity, threat) evaluated. Every item for each thematic cluster was appointed either “+1” when mentioned in the strength or opportunity part of the SWOT reports or “–1” when mentioned in the weakness or threat part. Numerical values are also presented in Table 3.

We then compared the overall numbers regarding their positive (strengths and chances) or negative (weaknesses and threats) values. Overall positive counts represented 373/632 (59.0%) of the items compared to 259 negative counts (41.0%) a relation of 1.44/1. In a further analysis we were interested whether strength and weakness in each item were related in a similar way. The overall comparison revealed 242 strength items to 181 (ratio 1.34:1) weaknesses. The relation between opportunities and threats was slightly higher with 131 positive values and 78 negative (ratio 1.68:1).

We selectively analysed these numbers for strength and weaknesses as well as opportunities and threats over all the items. In the analysis of strength and weakness personnel and team/communication had the most positive relation. Negative relations were seen for documentation, reporting and ethics/legal issues. When comparing opportunities and threats personnel and documentation had relevant negative values (+19/–24 and +6/–11). A presentation of further plus/minus analyses is provided in Table 3 (Tab. 3).

Table 3

Items extracted from SWOT analysis of peer review reports. Descriptive numbers extracted from SWOT reports.

Discussion

Peer review for quality assessment is considered an important part of quality control in intensive care medicine. Valentin et al. published on behalf of the working group on quality improvement of the ESICM the recommendation on basic requirement for ICUs. They conclude that a “benchmark process involving peers using quality indicators” is needed in intensive care medicine for quality assessment and quality improvement programs [14]. Applied indicators should reflect quality and safety of care and should also cover structure, processes and outcome in intensive care medicine. Currently there are different lists of indicators proposed in the literature for use in intensive care units [7], [15], [16] covering a broad number of topics regarded as important for quality and safety in the ICU. These indicators focus mainly on structure and outcome as they are easier to assess. However, there is a debate over the different quality dimensions that should be assessed [3], [11], [17]. In the German peer review project the evaluation is based on consented and recently published indicators that are included in the assessment questionnaire [18]. The results of our analysis show fulfilment rates of those – mainly process based – quality indicators in the questionnaires with values mostly above 75%. We considered this an overall good value. In comparison to fulfilment rates of other studies that tended to be lower with median values of approximately 60%. However, the tested quality dimension differed and the values of these results were achieved by another method, potentially influencing the results [4].

The use of checklists based on proven quality indicators has been valuable and the use of treatment bundles as based on evidence based therapy processes included in quality indicators affect outcome positively. However, the use of a specific set of quality indicators has only shown mixed results with regard to outcome effects [19], [20].

Additionally to the indicators we analysed the peers’ SWOT reports to evaluate whether we could confirm the results obtained by the questionnaires. We were able to show that these SWOT reports – that are not directly related to specific quality indicators – are at least as relevant as the questionnaires. We extracted 11 clusters of topics with relevance to patient safety and treatment quality. Some of these clusters address in fact commonly known problems in ICUs. However, the specific peer comments together with the results of the questionnaires showed a more complete picture of single ICUs. In the following paragraphs we describe examples of our findings.

The sufficient amount of qualified personnel is a key for providing adequate care to patients. Comments on staffing of an ICU as well as management of staff were components of the cluster mentioned most often in the SWOT reports. The importance of adequate nurse to patient ratios and staffing of ICUs with specialized intensivists is well described in the literature [21], [22], [23], [24]. One of the main findings of our analysis was that personnel issues were not only mentioned most frequently in the SWOT reports, but were in a high rate associated with weaknesses and threats. Our analysis revealed that checking the quality indicators apparently showed fairly adequate structural and process quality regarding personnel. However, the more specific and detailed SWOT reports showed that formal adherence to those quality indicators in the area of staffing does not automatically mean that an adequate amount of well-trained personnel in single ICUs is present. Additionally, in this cluster further issues like professional development of staff through education and mandatory regulations regarding knowledge of technical equipment and its proper use, were addressed in more detail in the SWOT analysis than in the questionnaires’ quality indicators. The reports additionally show that those (related) issues could be more specifically described as aspects of staff development. Even the “general mood” in an ICU recognized by the peers may show a way in which understaffed units were compensating this problem through good “team spirit”. We saw indications of that in our plus/minus analysis (see Table 3 (Tab. 3)) and consider this result as very important because apparently this is not a robust way to maintain adequate treatment quality.

As a second important result of this analysis we could demonstrate that there is a lack of good documentation tools and reporting systems in most ICUs. This was shown in both evaluations the questionnaires and the SWOT reports. Only a minority of ICUs used a patient data management system, and only a minority of units received data reports of key outcome measures like mortality. The general use of specific quality indicators was also lacking. Improved outcome documentation and reporting seems necessary in most evaluated units. Without reporting of outcome effects of quality improvement programs cannot be measured.

A third important cluster included ethical issues and end-of-life-decisions (EOLD) which are of increasing importance in intensive care medicine [25], [26]. Surprisingly, this was the at least represented thematic clusters in the SWOT reports with only half of all the reports mentioning this issue. We concluded, that this topic has to be covered in the SWOT reports regularly to make sure that this topic is not underrepresented in German ICUs. The recently developed German quality indicators therefore include a weekly structured communication with patients’ relatives where the definition of therapy goals is mandatory. This may include therapy limitations and palliative medicine.

All other thematic clusters mentioned in the SWOT reports showed a stable ratio of positive to negative mentions (strength and opportunities vs. weakness and threats) that was 1.44:1. The above mentioned clusters had aberrations from this values that were considered significant. Apart from single ICUs this value shows that most visited ICUs had been evaluated positively also in the SWOT reports.

External visitations have long been a part of quality control in medicine as stated above. In contrast to the proposed method most of these visitations use a summative evaluation method since they are primarily intended to maintain minimal quality standards. They usually end with the “Check” of the PDCA-cycle. Voluntary peer review uses a formative evaluation process meaning to induce change and hence develop a system further [27]. It is therefore intended to close the gap usually seen between “Check” and “Act”.

Using a voluntary approach we intended to take advantage of a potentially existing or developing climate of change and motivation. However this voluntary approach could be problematic because in our experience potential “candidate” ICUs tend to assume that they are not ready yet for peer review.

Another very important distinction to other external assessment methods is the inclusion of “good practice” exchange. This is an integral part of the voluntary peer review process. It has recently been shown that networking between ICUs – which is a relevant part of this method – is driving a friendly competitiveness between ICUs [28]. In the future the described method is intended to do exactly this.

Establishing a new quality culture, should involve all the professions participating in the therapy process. It has been shown that a collaborative approach involving a group of participants will better aid in the cultural changes necessary for a sustained quality improvement [29]. Although the method mentioned there has an approach that focuses in single goals for quality improvement the networking aspect is comparable. The method of voluntary peer review is basically such a collaborative effort using the advantages of a network. However there are multiple goals addressed and therefore the positive effects of collaborative approaches are not proven yet in this context [30].

Finally we are aware of the fact this analysis has certain limitations. It is a retrospective study of the peers’ impressions. The use of a standardized protocol including a mandatory questionnaire for the peer reviews diminishes subjectivity and makes this method more reliable. Furthermore the fact that these results are probably not representative of all German ICU limits in part the results we found. Also this study is limited by missing outcome results. This is mainly due to the study design, but also to the structure of the German medical institutions, which limits the possibility of trans-sectoral data exchange between hospitals and others (i.e. data from health insurances and general practitioners outside of formal scientific studies). Future prospective trials are necessary to prove this concept as effective but it is crucial to mention that the study design is difficult especially with regard to outcome measures apart from morbidity and mortality with a stronger focus on quality of life. We are currently evaluating ways to prospectively study the effects of voluntary peer review in intensive care medicine as compared to other quality improvement measures with regard to outcome of patients.

To further elucidate the success of this method we intend to develop a data registry. It will be used to collect the anonymized peer review reports – depending on the consent of the visited institutions – to evaluate methodological aspects as well as measures of quality improvement.

Conclusion

We were able to show that voluntary peer review in intensive care units is a feasible method with unique features with regard to quality improvement. It is methodologically sound and follows key rules for quality evaluation methods. We were able to identify thematic clusters like personnel, reporting that – if not addressed in the future – pose risks for treatment quality in ICUs. This would most likely not have been detected by the mandatory quality measures in German ICUs. The success of the voluntary peer review method clearly depends on its effect on motivation to change and the will for mutual learning. Especially the included SWOT analysis enables the visited units to begin quality improvement.

Notes

Competing interests

The authors declare that they have no competing interests.

Acknowledgements

We thank all the participating intensive care units – especially their head of departments – who made the peer review protocols accessible. We are especially thankful to the members of the DGF for their support on nursing specific aspects of the peer review method.