Towards an International Classification for Patient Safety: a Delphi survey.

OBJECTIVE: Interpretation and comparison of patient safety information have been compromised by the lack of a common understanding of the concepts involved. The World Alliance set out to develop an International Classification for Patient Safety (ICPS) to address this, and to test the relevance and acceptability of the draft ICPS and progressively refine it prior to field testing.
DESIGN: Two-stage Delphi survey. Quantitative and qualitative analyses informed the review of the ICPS.
SETTING: International web-based survey of expert opinion. PARTICIPANTS: Experts in the fields of patient safety, health policy, reporting systems, safety and quality control, classification theory and development, health informatics, consumer advocacy, law and medicine; 253 responded to the first round survey, 30% of whom responded to the second round.
RESULTS: In the first round, 14% felt that the conceptual framework was missing at least one class, although it was apparent that most respondents were actually referring to concepts they felt should be included within the classes rather than the classes themselves. There was a need for clarification of several components of the classification, particularly its purpose, structure and depth. After revision and feedback, round 2 results were more positive, but further significant changes were made to the conceptual framework and to the major classes in response to concerns about terminology and relationships between classes.
CONCLUSIONS: The Delphi approach proved invaluable, as both a consensus-building exercise and consultation process, in engaging stakeholders to support completion of the final draft version of the ICPS. Further refinement will occur.

Introduction

Although the results of the first large-scale study of adverse events were published over 30 years ago [1], the field of patient safety has only gained widespread attention in the last decade [2-4]. In this time, there has been a rapid increase in the number of publications and reports in this area, but interpretation and comparisons have been compromised by a lack of common understanding and language. A need was thus identified to develop an International Classification for Patient Safety (ICPS) that would pave the way for researchers to understand each others' work and facilitate the systematic collection, aggregation and analysis of relevant information on patient safety [5].

An opportunity to address this need was presented by the launch of the World Alliance for Patient Safety of the World Health Organization (WHO) [6]. A Drafting Group of international experts was formed under the auspices of the World Alliance. After reviewing the relevant literature and examining existing classifications [7-10], an initial version of a framework was developed, with definitions of a few basic concepts. Between August and November 2006, this framework and its accompanying concepts were subjected to a two-stage web-based modified Delphi survey to test the relevance and acceptability of the ICPS, including the conceptual framework and the underlying terms and concepts. The Delphi methodology is a research technique designed to obtain opinions from experts in a particular field through the use of subsequent rounds of questionnaires combined with the provision of feedback to build consensus [11]. The aim of the two-stage survey was to test the relevance and acceptability of the ICPS and to support the Drafting Group in producing an initial draft version of the conceptual framework for the ICPS for field testing. An iterative process was undertaken through which the initial draft conceptual framework (Fig. 1), concepts and definitions were progressively refined and improved. This paper concentrates on the overall Delphi process to illustrate its impact on the conceptual framework and high-level classes; definitions for key concepts and preferred terms were agreed as part of this process and are discussed in a companion paper [12].

Figure 1

Initial draft conceptual framework.

Methods

Participants

Over 300 experts in the field of patient safety, health policy, reporting systems, safety and quality control, classification theory and development, health informatics, consumer advocacy, law and medicine were directly invited to participate in the Delphi survey. Input was also sought from a wider stakeholder constituency. Open invitations to take part in the Delphi process were placed in an article published in the International Journal for Quality in Health Care [13] and on the websites of the WHO's World Alliance for Patient Safety, the Australian Patient Safety Foundation, The Joint Commission in the USA and the UK National Patient Safety Agency.

Experts who accepted the invitation to participate, and those who responded to the wider invitation, were asked to complete two confidential questionnaires.

Questionnaire development

The web-based questionnaire was developed and designed using Survey Monkey [14] as the platform. Survey Monkey is a web-based, flexible, scalable, secure survey development tool. It was pre-tested with 18 participants with clinical knowledge and/or experience with coding/classifying patient safety incidents from the Australian Patient Safety Foundation, the National Patient Safety Agency and The Joint Commission. This was performed on two occasions to support face and content validity and to understand the practical elements of the design. Considerable refinements were made to the survey tool, particularly to the navigational structures.

Round 1 survey

The first round of the web-based Delphi survey took place between 18 August and 22 September 2006. The questionnaire included sections on: (i) participant demographics including age, gender, country, native language, whether the respondent was a healthcare professional and expertise and practical experience in collecting, classifying and/or analysing patient safety data; (ii) the conceptual framework, including questions on the (a) adequacy, coverage and meaningfulness of the conceptual framework itself, (b) the clarity, precision and adequacy of the terms and definitions for each of the 10 classes within the conceptual framework and (c) views on the meaningfulness and usefulness of each of the 10 classes and (iii) overall comments. In any case where respondents thought there could be improvement, open comments were sought. The questionnaire was designed to direct respondents to a comments page to be completed before proceeding.

Round 2 survey

The second survey, sent only to those who responded to the first round, took place between 1 and 27 November 2006 and included the same sections and questions as the first survey, with two additions: an enhanced introductory explanation of the classification, and quantitative and qualitative feedback from the first survey for each question. The feedback included an explanation for the modifications made, or not made, to the classification based on the first round results.

Analysis and redrafting

The results for both surveys were tabulated using quantitative and qualitative analyses. Quantitative analysis of the structured responses was undertaken to describe the overall responses, whereas all open responses were reviewed in detail to identify common themes and key issues.

At the end of round 1, using face to face, teleconference and email discussion, the Drafting Group made appropriate modifications to the ICPS and to the supporting information and the detailed method for round 2.

Following the second round, the Drafting Group reviewed the results in a 2-day face-to-face meeting. After this meeting, it was agreed to undertake further detailed review of the open Delphi comments. Each individual comment was reviewed by one of the three members of the Drafting Group, and a response to each comment was provided, including a description of any change (if appropriate) made to the ICPS as a result. This was shared with the Drafting Group, and the conceptual framework was further iteratively revised by the Drafting Group between December 2006 and July 2007. The revised version ‘The Conceptual Framework for the International Classification for Patient Safety Version 1.0 for use in field testing 2007–2008’ was produced at the end of July 2007 [15].

Results

Round 1 survey results

Two hundred and fifty-three people speaking 29 different languages from 43 countries responded to the first round; 161 responded to the personalized invitation letter sent by the Drafting Group and 92 responded to the open invitation. The respondents included healthcare professionals, health policy experts, developers/managers of patient reporting systems, patient/public representatives, academics, representatives from professional associations for a variety of healthcare specialties, litigation experts, classification/taxonomy experts, risk managers and representatives from organizations responsible for assessing and monitoring patient safety performance. Eighty-one percent (205/253) had practical experience in collecting, classifying and/or analysing patient safety data.

All comments were reviewed. The number of comments ranged from 11 to 137 per question; 146 general comments were made. In all, there were 1011 comments.

The responses to the main questions are shown in Table 1 and Figs 2 and 3, comparing the results of both rounds of the survey. The main deficit in initial response was the fact that 14% (35/253) of the respondents felt that the conceptual framework was missing at least one class; this was 19% (14/75) of those who responded to both surveys. However, upon analysis of respondents' comments, it was apparent that most of the respondents were referring to ‘concepts’ they felt should be included within the classes instead of ‘classes’ that should be included within the conceptual framework. The distinction between a concept and a class is dealt with in the companion paper on definitions of key concepts [12].

Table 1

Questions on the conceptual framework as a whole

	First round—all respondents	First round–second round respondents	Second round
Q1. Is the conceptual framework an adequate model for use in describing a patient safety event?	85.7	86.7	92
Q2. Do you believe any classes are missing from the conceptual framework?	86.2	81.3	88
Q3. Is the conceptual framework a meaningful and useful tool for translating disparate information into a common format conducive to learning and improving patient safety?	85	89.3	78.7

Q1, % answering yes or yes with modification; Q2, % answering no; Q3, % agree or strongly agree.

Figure 2

Are the definitions for each class clear, precise and accurate? (1st round survey results for actions taken not available). Percent agree or strongly agree.

Figure 3

Is the class meaningful and useful within the ICPS's conceptual framework? Percent agree or strongly agree.

The main themes elicited from the review of open comments and the changes made in response to them are summarized in Table 2. A key finding was the need for clarification of several components of the classification, particularly its purpose, structure and depth.

Table 2

Results from the thematic analysis

Results from thematic analysis	Modifications made
Round 1 survey	Round 1 survey
A need for clarification, particularly with respect to the: purpose of the classification structure and depth of the conceptual framework intention of the classification to include both adverse events and near misses ability of the conceptual framework to serve as a model to classify a patient safety event concepts contained within each of the classes	An overview of the classification was developed that: provided background on the development of the classification differentiated and discussed the relationship between a classification and a reporting system described the classification, including a detailed explanation of its structure and composition explicitly illustrated how the concept of a patient safety incident (both adverse events and near misses) was captured by the classification demonstrated how to classify an incident using the classification's conceptual framework as a model, including two examples. delineated each classification tree and the concepts contained therein
	The definitions of the terms for several classes were clarified—event type (adverse events and near misses), patient impact/outcomes; contributing factors; actions taken and recovery factors
	The relationships among contributing factors, preventive factors, recovery factors and mitigating factors were explained
Round 2 survey	Round 2 survey
The main themes to emerge were: the need for field testing and an instruction manual the conceptual framework was too complex the classes were incomplete and, in some instances, inappropriately organized confusion about the relationship between contributing factors, preventive factors, recovery factors, mitigating factors and actions taken confusion about the role of the preventive factors class a view that ‘patient procedures’ belonged under the event characteristics class instead of patient characteristics class concern over the term for and purpose of recovery factors concern over the ‘behaviour’ concept concern over use of the term ‘event’	The conceptual framework was revised in iterative stages to clarify the purpose of each class and to explicitly show the relationship between them The classes were refined to ensure that the concepts within a class were organized hierarchically and fell into categories which are brief and easily and commonly understood The concepts contained within the third level concept ‘behaviour’ (‘contributing factors’, ‘human and performance factors’ and ‘behaviour’) were modified and the definitions of the several classes were refined The term ‘event’ was replaced by ‘incident’

Round 2 survey results

Thirty percent (75/253) of round 1 respondents replied to the second survey, of whom 68 completed the entire survey. Again, the respondents included the full range of backgrounds within the first survey.

Comments were less critical than those to round 1 (Figs 2 and 3). Only 12.0% (9/75) thought that the conceptual framework was missing at least one class, a reduction from 19% (14/75) of those who responded to both surveys compared with round 1. Once again, upon further analysis of comments, it became apparent that most of the respondents were referring to concepts they felt should be included within the classes instead of classes that should be included within the conceptual framework.

Views on whether the conceptual framework was an adequate model were more positive, although there was a fall in the proportion believing that ‘the conceptual framework is a meaningful and useful tool for translating disparate information into a common format conducive to learning and improving patient safety’. More respondents felt that the conceptual definition for each class was clear (Fig. 2) and that each class was meaningful and useful (Fig. 3).

All comments were reviewed. The number of comments ranged from 5 to 14 per question; 26 general comments were made. In total, there were 227 comments.

The main themes elicited from the review of open comments and the changes made in response to them are summarized in Table 1. In particular, confusion remained about the relationships among the classes of ‘contributing factors’, ‘preventive factors’, ‘recovery factors’, ‘mitigating factors’ and ‘actions taken’.

Major modifications to the ICPS

The initial class for describing what happened was an ‘event’. The Delphi responses indicated that the term ‘event’ was problematic because it is not generally perceived to include underlying hazards and potentially dangerous circumstances. The term ‘incident’ is widely used in healthcare patient safety literature and is understood to include events, hazards and circumstances. ‘Event’ was therefore changed to ‘incident’ [8].

The Delphi analysis also revealed confusion about the meanings of, purposes for and relationships between the classes, specifically ‘contributing factors’, ‘preventive factors’, ‘recovery factors’, ‘mitigating factors’ and ‘actions taken’. In addition, it was felt that some classes were incomplete and, in some instances, inappropriately organized. The classes were therefore refined to ensure that the concepts within a class were organized hierarchically and fell into categories, which are brief and easily and commonly understood.

The classes—contributing factors, preventive factors, recovery factors, mitigating factors and actions taken—were introduced or revised to address this confusion and provide building blocks for the important meta-concepts of system resilience and recovery. The concept of recovery is particularly important if learning from patient safety incidents is to occur. However, the term ‘recovery factors’, widely used in industrial safety settings to describe recovery from an incident, produced confusion among some clinical respondents who equated the term ‘recovery’ with clinical recuperation. Recovery factors were defined as the ‘actions or circumstances that follow detection of an incident and prevent or moderate its progression so as not to result in harm’. Recovery thus has both a detection phase and a mitigation phase [16]. Therefore, to alleviate the confusion, the Drafting Group decided to create ‘detection’ and ‘mitigating factors’ as classes to signify the detection and action phases and to delete the class ‘recovery factors’.

‘Detection’ is defined as ‘an action or circumstance that results in the discovery of an incident’. An incident could be detected by noticing an error, via a monitor or alarm, by a change in the patient's status, or via an audit, review or risk assessment. Detection mechanisms may be built into the system as formal barriers or informally developed. Detection is thus a necessary component of the system to allow subsequent mitigation or amelioration. ‘A mitigating factor’ is defined as ‘an action or circumstance which prevents or moderates the progression of an incident towards harming a patient’. ‘Mitigating factors’ are generally actions, but they can be due to good luck or chance. Therefore, they were labelled ‘factors’ rather than ‘actions’. Thus detection by a dispensing pharmacist of prescription of an interacting drug by a doctor could lead to a change in the prescription, thus mitigating the potential adverse effects.

‘Ameliorating actions’ was created as a new class. It was borne from the ambiguities in the meanings of mitigation/recovery, action taken and preventive factors. To develop this class, the Drafting Group revisited the literature and reviewed real incident reports. The concepts contained in this class are not circumstances, but are actions undertaken by the healthcare professional or organization to ‘right a wrong’. The concepts signify that specific action was taken, i.e. ‘doing something’, so the descriptor ‘action’ was substituted for ‘factor’. Ameliorating actions take place after the incident has already caused harm to the patient. An example would be the resuscitation of a patient who had suffered a cardiac arrest as a result of inadvertent injection of high-concentration potassium chloride or treatment of a post-operative wound infection with antibiotics.

The original concepts for the organizational outcomes class were adapted from the International Classification of Functioning, Disability, and Health (ICF) [17]. It became clear from the Delphi responses that these concepts, even as modified, were not fit for purpose. Revised concepts populating this class were derived from incident reports and are concepts widely used in this context in health care.

A significant outcome of the Delphi analysis was recognition of the need to clarify the semantic relationships between the ‘preventive factors’ class and the other classes. The class was relabelled as ‘actions taken to reduce risk’. Its component concepts are both informed by and inform the classes ‘contributing factors/hazards’, ‘detection’, ‘mitigating factors’ and ‘ameliorating actions’ and focus on learning and system resilience. In the ICPS, resilience is defined as the degree to which a system continuously prevents, detects, mitigates or ameliorates hazards or incidents.

The final conceptual framework (Fig. 4) continued to organize the high-level classes into a framework reflecting the relationships between the classes, modified so as to incorporate as many ideas of the Delphi respondents as possible.

Figure 4

Final draft conceptual framework.

Summary of changes

The Delphi analysis revealed that the classes were incomplete and, in some instances, inappropriately organized. The responses of the Drafting Group were as follows:

to relabel ‘event type’ as ‘incident type’;

to relabel ‘patient impact/outcomes’ as ‘patient outcomes’;

to relabel ‘event characteristics’ as ‘incident characteristics’;

to relabel ‘contributing factors’ as ‘contributing factors/hazards’ to indicate that circumstances, agents and actions in addition to factors can and do play a part in the origin or development of an incident and/or increase the risk of an incident;

to add a new class, ‘detection’. To recover, one needs to detect and then mitigate to prevent or moderate the progression of an incident. Recovery comprised detection plus mitigation. The revised conceptual framework is designed to reduce the substantial confusion with respect to the intent and meaning of these classes;

to revise and relabel ‘preventive factors’ as ‘actions taken to reduce risk’. This class encompasses concepts previously contained in the now deleted ‘action taken class’;

to delete ‘recovery factors’;

to add ‘mitigating factors’ as a class. To recover, one needs to detect and then mitigate;

to add ‘ameliorating actions’ as a class. These are actions taken or circumstances altered to make better any harm after an incident.

Discussion

Recognition of the need for an internationally agreed framework and key concepts for patient safety led to the development of a process for achieving the development of the conceptual framework for the ICPS. An early draft was based on existing classifications. A consultation process using a modified Delphi survey resulted in changes being made to the presentation and structure of the initial framework and to the definitions of its major classes. The conceptual framework and classes are described in greater detail, as are the agreed definitions of other key concepts and their relationships to the framework, in companion papers. The final conceptual framework is shown in Fig. 4.

The Delphi approach engaged disparate stakeholders in the refining of the original drafts of ‘The Conceptual Framework for the International Classification for Patient Safety Version 1.0 for use in field testing in 2007–2008’. This approach enabled the engagement of an international array of experts and safety managers to critically comment on the early drafts of the ICPS. The Delphi survey allowed the original framework, built around existing classifications and theoretical foundations, to be tested against expert opinion. When the views from respondents demonstrated misunderstanding of the framework (e.g. over classes/concepts) or diverged from the initial framework for other reasons, the Drafting Group discussed the appropriate response that both responded to the feedback, but retained the underlying theoretical foundations.

Limitations include the fact that considerable time and effort were required from participants to complete the survey, although their commitment to doing so is applauded. There was no immediate suggestion of response bias, and although the second round had a much lower response rate, this is not uncommon in a Delphi process.

Changes were made as a result of the Delphi survey, but this takes us only to the end of the first phase of an ongoing work programme. The framework has been developed on the basis of a range of available classifications, a wide mix of expertise and opinion, enhanced by the Delphi surveys and by iterative discussion, drafting and redrafting. There is likely to be a balance to strike between the comprehensiveness and complexity of the conceptual framework and its ease of use to derive a classification. However, the ultimate test of this comes in its application in practical settings, including the viability of the framework as a practical foundation for classification and coding. This task will be initiated in the field testing phase.

Several lessons were learnt in undertaking the Delphi survey. The use of a web-based survey platform, with structure and questionnaires designed with Survey Monkey, simplified the process and allowed ready tracking and analysis. The survey instrument was improved by two rounds of piloting. The open-ended questions provided invaluable feedback and enabled in-depth understanding of respondents' views. The second round was essential to test the changes made and did lead to further changes. We would use the Delphi process again, but had underestimated the time required to analyse and respond to the findings.

In conclusion, this article has described the development of the conceptual framework for the ICPS to the point of field testing, concentrating on the method and impact of a modified Delphi process that supported an expert group to refine the initial drafts. The Delphi methodology serves as both a consensus-building exercise and a consultation process. The consultation gained through the Delphi methodology proved to be valuable and had an influence on recasting the framework and refining the definitions and labels of key concepts. Further refinement will occur following field testing.

Full Drafting Group Membership

The full membership of the Drafting Group in alphabetical order was: Gerry Castro, Martin Fletcher, Martin Hatlie, Peter Hibbert, Robert Jakob, Richard Koss, Pierre Lewalle, Jerod Loeb, Thomas V. Perneger, William Runciman, Heather Sherman, Richard Thomson, Tjerk van der Schaaf and Martti Virtanen.

Funding

The Project to Develop the ICPS was funded by the World Health Organization World Alliance for Patient Safety, Geneva, Switzerland. The conceptual framework for the ICPS in the intellectual property of the World Health Organization.