Development of a taxonomy for characterising medical oncology-related patient safety and quality incidents: a novel approach.

Introduction

The publication of To Err is Human by the Institute of Medicine (IOM, now the National Academy of Medicine) in 2000 proved to be a watershed moment in healthcare.1 For the first time, national attention focused on flaws in the healthcare system that regularly led to patient harm. The report drew widespread attention from the press and began an important national discussion that continues today. A follow-up report from the IOM in 2004, Patient Safety Achieving a New Standard, recommended a series of sweeping changes.2 Principal among them, the IOM argued that ‘all healthcare organisations should establish comprehensive patient safety programmes operated by trained personnel…’ The main aim of the proposed programme was to create systems for collecting and analysing patient safety data and using the knowledge to reduce risk and prevent future incidents.

Systematically identifying and tracking patient safety incidents—defined as events occurring during an episode of patient care that have the potential to, or actually cause injury or harm to a patient—is now regarded as the cornerstone of safety reporting systems.3 The major tool for collecting patient safety reports from frontline clinicians and other patient-facing staff is a voluntary incident reporting system; this tool allows data to be collated and analysed to identify areas of safety risk, and to ameliorate them.3 4 Incident reporting systems are now mandated by The Centers for Medicare and Medicaid Services, the National Health Service and other agencies around the globe. Many hospitals now rely on web-based proprietary reporting systems such as RLDatix, which supports >5000 customers worldwide.

The starting point for any incident reporting system is a common language.5 General classification systems for patient safety incidents have been published by the Joint Commission and by WHO.6 7 Although the development methodologies used were different (structured literature review for the former, Delphi consensus panel for the latter), both stressed the critical importance of consistency of language. Both are intended to have broad applicability across locations of care, healthcare facility type, medical specialty and clinician category. In addition to broadly applicable ones, safety incident classification systems have been developed for use in various specialties including internal medicine, surgery, paediatrics, obstetrics and gynaecology and psychiatry.5 8–13

We have been able to identify only three reports describing the use of an incident reporting system in medical oncology. Two of these describe the implementation and early evaluation of a paediatric oncology incident reporting system implemented at Vanderbilt University Medical Center in 2003.14 15 Ninety-seven incidents were described; ordering errors were the most common. Details of the classification system, including how it was designed, were not reported. Lennes et al described the implementation of an incident reporting system at the Massachusetts General Hospital Cancer Center in 2016.16 The classification system focused exclusively on chemotherapy medication errors, was developed by a group process, but not described further. Our review, therefore, has identified a potentially important gap in the literature—the lack of a high-quality, validated incident taxonomy focused on the practice of medical oncology.

In this report, we describe the development of a comprehensive, validated taxonomy for medical oncology-related incidents. We combined qualitative analysis with multiple, rapid cycles of testing at three different healthcare facilities to validate the classification system. Our goal was to create a taxonomy that will be broadly applicable to the practice of medical oncology across different sites of care.

Methods

Neither patients nor the public were involved in the design, conduct, reporting or dissemination of our research.

Setting and sample

The project was performed across three different healthcare facilities based in the USA. Site 1 is a free-standing National Cancer Institute-designated cancer centre that provides adult and paediatric oncology care. Site 2 is based in an academic medical centre (AMC) that provides adult cancer services. Site 3 is an AMC that provides both paediatric and adult cancer services, but only adult services are included in this project.

The data source included all ambulatory incident reports from site 1, all hematology-related and medical oncology-related incidents from site 2 and all oncology-related incidents from site 3. The measurement period was 1 January 2019 through 31 December 2020. All sites record incident reports using the same version of a proprietary incident reporting system (RLDatix). The details for each site, including incident report inclusion and exclusion criteria, are provided in table 1. Data fields were downloaded into a spreadsheet (Microsoft 365 Excel), de-identified for all patient and staff information and randomly ordered at each site.

Table 1

Site Characteristics

	Site 1	Site 2	Site 3
Type	Free-standing NCI-designated cancer centre	Academic medical centre-based cancer centre	NCCN-designated comprehensive cancer centre based in academic medical centre
Community sites included	Yes	Yes	No
Total available reports for review	1919	677	858
Ambulatory	Yes	Yes	Yes
Inpatient	No	Yes	Yes
Clinical trials	Yes	Yes	Yes
Incident Learning System software application	RL Solutions Version 6 (RLDatix)	RL Solutions Version 6 (RLDatix)	RL Solutions Version 6 (RLDatix)
Excluded incidents	Falls, skin/tissue issues, primary radiation oncology incidents	Falls, skin/tissue issues, primary radiology incidents, adverse reactions	Primary radiation oncology incidents
Enrichment for incident-type categories	Diagnostic and therapeutic issues; care and service coordination	No	No
Fields used for review	Incident description; suggestions for avoiding future incidents; outcome	Incident distribution	Incident description; suggestions for avoiding future incidents
Recharacterisation of incident reports by quality and patient safety analysts	Yes	No	Yes

NCCN, National Comprehensive Cancer Network; NCI, National Cancer Institute.

Selection and modification of coding framework

Development of the taxonomy used an iterative process involving a literature review, content analysis, rapid cycle testing of a draft taxonomy and expert panel review (online supplemental figure 1).

For the purposes of this project, we define a taxonomy as a scientifically driven framework that hierarchically arranges patient incidents into logically organised domains and categories. It is distinguished from a classification system which arranges incidents by simple grouping only.6

To develop a taxonomy, we systematically reviewed the published literature for safety incident classification systems including those that are intended to be broadly applicable as well as those that focused exclusively on medical oncology. In addition, we reviewed related classification systems such as those designed to catalogue patient and family complaints. We sought a clinically intuitive system that would lend itself to easy applicability by patient safety analysts who are responsible for maintaining incident reporting systems.17 We concluded that the patient and family complaints taxonomy developed by Reader et al was easily applicable to incident reporting, and that it was amenable to modification for medical oncology-related incidents.18 The taxonomy includes three broad domains, each with two to three categories: clinical (categories: patient safety and quality); relationships (categories: communication; humaneness/caring and patient rights) and management (categories: institutional issues and timing/access). An expert advisory panel approved moving forward with the Reader framework.

We employed content analysis on a crosswalk of incident types currently used at the three sites to identify key components for the incident taxonomy and to refine the taxonomy.19 Thirty distinct high-level incident types were identified. We compared these with the categories included in the report by Lennes et al.16 After removing duplicates, an additional five categories were added bringing the total to 35 incident types. These were then fitted into the Reader framework resulting in 26 categories and 22 subcategories divided between the three domains. Next, two coders (JOJ, TM) independently reviewed 50 incident reports from site 1 and coded them using the draft taxonomy. The coders then met to clarify areas of agreement and disagreement and ways in which the coding framework could be improved. Thereafter, one of the coders (JOJ) met with the qualitative researcher (ACR) to review the results and modify the draft taxonomy as needed. Two coders (JOJ and an oncology physician assistant) then separately analysed an additional 62 site 1 incidents during three separate sessions, repeating the process described above, modifying the coding framework between each review cycle. This single site incident coding exercise allowed for draft coding instructions to be created and enabled us to proceed to rapid cycle coding across all sites.

Rapid cycle testing of the taxonomy

The next phase of the project entailed a series of rapid cycle tests based on the principles of the plan-do-study-act methodology employed in quality improvement.20 The process is highlighted in figure 1. The goal of the rapid cycle process was to accelerate the taxonomy development process. Volunteer coders were instructed to read each incident carefully, searching for key words or phrases that best explained why the incident reporter placed the entry, and to search the incident and contributing factor code sheets to identify the best fit for each. Each incident was coded for primary and secondary incident types and up to two contributing factors. Coders were instructed to ignore whether the incident resulted in harm or not, or whether the incident was completed or interrupted (ie, a near-miss). In addition, coders were instructed to concentrate only on the incident as described and not to draw inferences or bring specialised knowledge to bear (see online supplemental coding instructions).

Figure 1

Rapid cycle testing diagram

Coders catalogued the incidents in a spreadsheet employing dropdown menus to select incident types and contributing factors. Coders were required to select at least one incident type, and to select a second one only if the incident report contained two distinguishable occurrences (eg, the primary incident described a pharmacy delay, but the patient also reported being treated rudely by staff members). Coders were encouraged to select at least one contributing factor if one could be identified. Finally, coders were also asked to provide a one-sentence or two-sentence justification for their choice of an incident type.

Each cycle was designed for a turnaround time of <1 week. One of the coders (JOJ) and eight different secondary coders met sequentially for an hour each to review the coding instructions and the incident code sheet and contributing factors code sheet; this included coding five cases together. The coders then separately analysed approximately 20 incidents and met again within 48 hours to compare results side-by-side and to provide feedback on the coding tools. One of the coders (JOJ) then met with the qualitative researcher (often with SOS, an epidemiologist) to review the results of the rapid cycle with focus on disagreements. Using a structured process in which comments from the coders were catalogued and reasons for disagreements were clarified, the coding instructions and code sheets were updated, and another rapid cycle was performed.

An interdisciplinary expert advisory panel supervised the progress of the project and provided feedback at key inflection points. The panel reviewed and approved the final incident taxonomy. The panel included adult oncologists, a paediatric oncologist/health service researcher, a senior director of patient safety, a systems safety expert and members of the safety teams from all three sites who regularly code medical oncology incidents.

Descriptive statistics and other analyses were calculated using Excel.

Results

Three hundred seventy-three incidents were reviewed including the content analysis and rapid cycle tests. A total of 10 rapid cycle tests were conducted over 12 weeks during which 221 incidents were coded, including 81 from site 1 (JOJ plus four different coders), 89 from site 2 (JOJ, JAZ and two other coders) and 51 from site 3 (JOJ, TM and one other coder). The coding instructions were updated following each rapid cycle to maximise clarity. The incident code sheet was revised a total of 13 times. The final version of the incident coding sheet is displayed in table 2. The final versions of the instruction guide and contributing factors code sheet are included in the online supplemental materials.

Table 2

Incident coding guide

Incident	Description
A. Clinical	Events related to the delivery of care.
Safety	An event placing the patient at immediate or delayed risk of harm.
1. Drug prescription (generally applies to providers)	Selection of wrong drug or treatment regimen.
a. Change in order or new order not communicated	Change in order or new order not communicated.
b. Wrong regimen, drug, drug dose ordered or wrong patient	Wrong treatment is ordered including for wrong patient.
c. Ordered regimen or drug contraindicated	Ordered regimen or drug contraindicated because of risk, etc.
d. Dose adjustment not ordered	Dose adjustment not ordered.
e. Delays	Includes delays related to signing orders, etc.
f. Other	Other prescription-related issues.
2. Pharmacy preparation (generally applies to pharmacy staff)	Pharmacy error in preparation, labelling (incorrect or damaged) or dispensing; incompatible medication; contraindicated medication.
a. Drug preparation, labelling or release	Errors related to drug preparation including right drug, dose, concentration, patient, etc.
b. Prepared drug is incompatible or contraindicated	Incompatible or contraindicated medication is prepared.
c. Delays	Includes delays directly related to pharmacy.
d. Other	Other pharmacy preparation-related issues.
3. Medication administration and management (generally applies to infusion staff)	Errors in administering medication: dosage, strength, day, date, frequency. Includes recording or using wrong weight or height.
a. Activation before assessment	Release of chemotherapy orders to pharmacy before patient assessment.
b. Administration error	Administration error—wrong patient, drug, dose, route, schedule. Includes omitted doses or extra doses. Includes near-misses.
c. Height and/or weight	Misentered height or weight, use of wrong date, etc.
d. Laboratories	Failure to adjust treatment based on laboratory data.
e. Intravenous catheter management	Improper use of peripheral or central intravenous catheter. Includes catheter malfunction or malposition and extravasation.
f. Delays	Delays related to medication administration.
g. Other	Other medication administration/management issues.
4. Adverse drug reaction	Noxious and unintended response to a drug.
a. Potentially avoidable reaction	History not available or not considered or interaction not considered.
b. Other	Includes non-preventable adverse drug reactions.
5. Equipment or product issue	Improper equipment design or function or lack of availability. Includes issues related to blood bank.
6. Procedure	Unnecessary procedure or complication of a procedure.
7. Anaesthesia, sedation complication	Complication of anaesthesia or sedation.
8. Lab error or consequential delay	Result issue (wrong patient, incorrect result, delayed report, lost specimen, etc.).
9. Imaging error or consequential delay	Reporting error (wrong patient, incorrect result, delayed report, etc.).
10. Infection, exposure, contamination	Preventable patient or staff exposures.
Quality/knowledge management/skills	A clinical incident related to patient management.
11. Evaluation and assessment (generally applies to providers)	Incomplete assessment prior to treatment decision.
a. History, physical examination	Incomplete history, physical examination.
b. Review of relevant data	Incomplete review of relevant data including laboratory, imaging, pathology, outside information.
c. Evaluation	Incomplete, faulty or untimely evaluation.
12. Therapeutic decision making (generally applies to providers)	Failure to provide the most suitable treatment for an individual patient in a timely manner.
a. Decision making	Non-optimal or non-evidence-based decision.
b. Follow-up	Failure to arrange appropriate, timely, short-term or long-term follow-up care.
13. Evaluation or therapeutic decision making, other (generally applies to providers)	Includes other events related to evaluating and/or treating patients, including delays.
14. Staff knowledge, skills or action (can apply to any staff member)	Issues related to staff knowledge, skills or action including those that result in delays. Includes administrative errors (eg, wrong patient identifier).
B. Relational	Events related to interpersonal issues.
15. Documentation error	Incomplete, inaccurate or untimely record keeping.
16. Communication	Communication breakdown: inadequate, delayed or absent communication—generally between two parties (limited to isolated communication issues, and therefore distinct from coordination of care, see below).
17. Coordination of care	Failure to coordinate complex care across clinical staff, services, sites. Larger than communication failure. Uncoordinated or untimely service including failure to establish follow-up appointments, tests or treatments. Includes failure to complete expected service. Includes delays.
18. Other relational issues	For example, contacting the wrong patient or staff to schedule an appointment.
19. Unprofessional behaviour	Inadequate attention to patient needs (cultural, linguistic, etc), staff attitude (rudeness disrespect, insensitivity, improper behaviour); lack of respect for patient needs and preferences including those related to race, ethnicity, gender, language, etc. Includes unprofessional behaviour between staff members.
20. Patient factors	Angry or aggressive patient or caregiver; unrealistic expectations or demands, especially time-related.
C. Institutional/Management	Events related to
21. Waste or inefficiency (applies to all phases of care and staff)	Any process or event that leads to wastage of resources including staff time, equipment and medications.
22. Patient rights, equity, discrimination	Consent/Coercion; confidentiality; discrimination, abuse; failure to provide privacy. Breech of protected health information.
23. Policies, procedures (applies to all phases of care and staff)	Problem with policies or procedures. For this event, procedure refers to hospital policies and procedures rather than a clinical procedure.
a. Design issue	Policy or procedure incorrect, confusing, contradictory, non-existent or not readily available.
b. Failure to adhere	Failure to adhere to known and available policy or standard procedure.
24. Protocols and guidelines	Failure to follow existing standard of care or research protocols or clinical guidelines.
c. Design issue	Protocol or guideline incorrect, confusing, contradictory, not readily available or non-existent.
d. Failure to adhere	Failure to adhere to known and available protocol or guideline.
25. Facilities and environment	Unsafe conditions including inadequate staffing or resources to support care and unsafe environment.
D. Event category not otherwise specified	Other event category (please specify why no other category was appropriate including insufficient information).

*Contraindication. The prescription, preparation and/or administration of a medication that should not be given to an individual patient because of existing knowledge that predicts for an untoward reaction or because of concern for a negative interaction between it and another medication.

†Extravasation. Leakage of a parenteral medication into surrounding soft tissue due to malposition or malfunction of an intravenous catheter that can result in injury.

The final incident coding taxonomy conforms closely to the Reader framework. The ‘clinical’ domain includes 10 safety-related and 4 quality/knowledge management skills incident categories (along with 19 and 5 subcategories, respectively). There are six ‘relational’ domain categories and five ‘institutional/management’ domain categories (including zero and four subcategories, respectively). Most of the modifications to the incident coding guide were minor changes to the descriptive language (table 2, column 2). A few significant deviations from the Reader framework were necessary. For example, ‘delay in care’ is listed under ‘institution/management’ in the Reader framework. But, because it is a broad, high-level issue, and not easily actionable, the category was eliminated. In its place, delays were reassociated with discreet incident categories (eg, pharmacy-related, provider-related, etc). Other changes to the taxonomy were necessary to encompass safety incident reports related to employee risk rather than patient risk. A category of ‘patient factors’ was added under the relational domain. ‘Waste and inefficiency’ was added under the institutional/management domain. ‘Policies/Procedures’ and ‘protocols and guidelines’ required modification to distinguish between design issues and adherence.

The results of the rapid testing cycles are summarised in table 3.

Table 3

Rapid cycle test results

Rapid cycle	Location coded	Incidents evaluated	Domain agreement (%)	Category agreement (%)	Unclassifiable incident (single coder)	Unclassifiable incident (both coders)
1	1	21	67	67	1	0
2	1	21	86	67	0	0
3	1	21	81	81	0	0
4	1	18	72	67	0	0
5	2	20	75	60	1	0
6	3	22	95	86	0	0
7	2	22	67	50	2	0
8	2	23	78	56	0	0
9	2	24	75	62.5	0	0
10	3	29	72	59	1	0
Median			75%	65%

Only five incident types (0.03%) could not be assigned by one of the coders; there were no incidents in which both coders were unable to select an incident type. There was domain-level agreement between the two coders in 75% (median, range 67%–95%) of incident reports. The median level of agreement at the category level was 65% (range 50%–85%). During the debriefing sessions, a variety of reasons emerged to explain disagreements between coders. Early in the process, these disagreements were often due to ambiguous language in the event coding guide and in the coding instructions. Over time, as the coding language became better clarified, disagreements were most often attributable to limitations in the reporting narrative. Clarity of the reports varied widely as did report length, with some reports as brief as a single sentence and others many paragraphs in length. A common area of ambiguity related to delays in the receipt of chemotherapy. It was sometimes impossible to distinguish, for example, whether a delay was due to slowness in signing an electronic order or a backup in the pharmacy.

Discussion

Incident reporting systems have been an important tool for evaluating and managing safety in several high-risk industries, most notably commercial air travel, for nearly 50 years.21 Today, they are widely employed in healthcare organisations throughout the world.5 Their usefulness, especially for clarifying risky practices, has been documented in high-complexity, high-risk areas of medicine such as surgery, anaesthesia and critical care.9 22–25 Another high-risk area of medicine is cancer care, which often requires exquisite coordination between different specialties including medical oncology, radiation oncology, surgery, pathology and radiology.26 In addition, there have been major therapeutic advances in cancer care in recent years; new treatments are often complex and carry the potential for significant harm if administered improperly or if poorly coordinated. In medical oncology, whole new therapeutic categories have been introduced including engineered cell therapy, immune checkpoint inhibition and highly targeted agents (often orally administered). In contrast to radiation oncology, where incident reporting has become common place and where a large international registry is active, medical oncology-related incidents have not been well studied.27–33

The aim of the current project was to develop a medical oncology incident taxonomy that will have widespread applicability and that is intuitive, easy to implement and easy to use. Using a qualitative approach, we modified an existing framework for classifying patient and family complaints to create a medical oncology incident taxonomy. Then, employing multiple rapid testing cycles including feedback from coders, we finalised the taxonomy by testing it across three different sites of care. The final taxonomy is hierarchically arranged with 3 domains, 44 categories and 28 subcategories.

To the extent possible, we sought to confirm the validity of the taxonomy, recognising that there is controversy about whether qualitative research is amenable to validity testing.34–36 We chose to determine validity based on the following criteria: assessment of bias, transferability, usability, confirmability and transparency.35 We managed bias in the following way: structured review of the literature at project inception designed to maximise inclusivity; random selection of incidents; double coding of incidents and use of highly structured training of coders using concise language. Bias was further managed during the analysis phase of each testing cycle by including team members with different areas of expertise (qualitative research, epidemiology and a medical oncology). Transferability was addressed by testing at three different facilities (including various combinations of ambulatory and inpatient events, adult and paediatric patients, patients receiving standard chemotherapy and those on research trials and academic and community settings). Usability was determined by the degree to which incidents could be successfully coded (99.7% of incidents). We assessed dependability based on the rate of intercoder agreement and on input from a highly qualified interdisciplinary expert advisory panel. Finally, we addressed transparency by sharing our entire methodology and all the materials that we developed (tables, figure and online supplemental materials).

We believe that the high level of intercoder agreement that we achieved was at least partly due to the emphasis on training. One of the investigators (JOJ) met with each coder for a full hour before each testing cycle to provide detailed instruction (described in the ‘Methods’ section); this was followed by another hour for debriefing, focusing on opportunities to improve the incident coding sheet and training guide. Early in the process, most disagreements between the two coders were due to limitations in the coding tools. Later in the process, most disagreements were due to ambiguity in the incident narratives themselves. We cannot overemphasise the importance of training for anyone who might wish to employ the methodology.

It is important to point out the limitations of our project. Key among them is the narrative reports themselves. By their very nature, the reports are subjective.18 They are often emotive, submitted at a time of anger, distress or frustration. Length and quality of the reports ranged widely. Another limitation is that we focused the project on developing an incident coding taxonomy. Although we collected information on contributing factors, the data are exploratory only. Finally, we were unable to collect information on incident severity.

We recognise that incident taxonomies must be viewed as ‘living documents’. That is, based on accumulating knowledge garnered through usage in different environments, periodic updates will be necessary. At the same time, we understand that there will need to be an ongoing dynamic balance between sensitivity and specificity; adding additional incident categories will increase the ability to pinpoint specific safety areas, but at the risk of making the taxonomy unwieldy. An example in which the current taxonomy lacks specificity is subcategory A3b, ‘administration error’. The current version of the taxonomy does not distinguish between high-risk medications (chemotherapeutic agents) and lower risk ones (eg, antinausea drugs). In addition, the current taxonomy lumps wrong patient incidents with other categories of errors. ‘Wrong patient’ errors are particularly high-risk incidents; the taxonomy will benefit from calling them out specifically.37 These issues will be addressed in a future version.

In this publication, we share how we developed the first validated incident reporting taxonomy for medical oncology. Our approach was novel, using a qualitative research framework in combination with multiple rapid testing cycles based on the plan-do-study-act principles of quality improvement. The combined approach allowed for accelerated completion of the project in less than a year. Our next step is to test the taxonomy at our institutions to determine its usefulness for tracking and analysing incidents, for discerning areas of patient risk and for identifying opportunities for improvement interventions. The taxonomy also has the potential to be deployed as part of a national or international medical oncology incident reporting registry akin to the Radiation Oncology Incident Reporting System which has been in place since 2014.28 Finally, the methodology and framework that we have described may be applicable to other specialties that wish to develop an incident taxonomy.