Comparing the Evolution of Risk Culture in Radiation Oncology, Aviation, and Nuclear Power.

OBJECTIVES: All organizations seek to minimize the risks that their operations pose to public safety. This task is especially significant if they deal with complex or hazardous technologies. Five decades of research in quantitative risk analysis have generated a set of risk management frameworks and practices that extend across a range of such domains. Here, we investigate the risk culture in three commercial enterprises that require exceedingly high standards of execution: radiation oncology, aviation, and nuclear power.
METHODS: One of the characteristics of high reliability organizations is their willingness to learn from other such organizations. We investigate the extent to which this is true by compiling a database of the major publications on risk within each of the three fields. We conduct a bibliographic coupling analysis on the combined database to identify connections among publications. This analysis reveals the strength of engagement across disciplinary boundaries and the extent of cross-adoption of best practices.
RESULTS: Our results show that radiation oncology is more insulated than the other two fields in its adoption and propagation of state-of-the-art risk management tools and frameworks that have transformed aviation and nuclear power into high reliability enterprises with actuarially low risk.
CONCLUSIONS: Aviation and nuclear power have established risk cultures that cross-pollinate. In both nature and extent, we found a distinct difference in radiation oncology's engagement with the risk community, and it lags behind the other two fields in implementing best practices that might mitigate or eliminate risks to patient safety.

Medical error constitutes one of the leading causes of avoidable death in the United States.[1] Although there remains much debate about the exact number, most systematic analyses suggest that it likely ranges in the hundreds of thousands.[1-4] One recent study lamented the industry's profoundly unfortunate record, exclaiming that in no other industry “would such a record be tolerated, let alone defended… We can and must do better.”[4]

All organizations should seek to minimize the risks that their operations pose to public safety. Although all medical errors are regrettable, errors within the field of radiation oncology generate special and considerable attention among the general public.[5] This may be in part due to the dread associated with ionizing radiation[6] and the dread associated with cancer.[7] This dread has also made radiation oncology a unique specialization within medicine for regulatory reasons. Despite being a civilian field, it entails the use of equipment, materials, technology, and expertise that are deemed sensitive and highly complex. These are all subject to a qualitatively greater level of regulation because of their sensitivity and the potential consequences of accidents involving their use.

As a result, the task of minimizing risks to the public gains special prominence in radiation oncology and the field requires exceedingly high standards of execution. It is not unique in requiring such high standards; many complex enterprises require high reliability organizations to manage risk, including commercial aviation, nuclear power, space travel, aircraft carriers, the military, and wildland fire management. Ideally, the radiation oncology department within any medical center ought to be run like other such high reliability organizations. Research on such organizations stretches back decades, and we do not review it in this article; the interested reader is referred to the following references for an introduction to some of the concepts in this literature.[8-12] One of the characteristics of a high reliability organization—the one we are most interested in for the purposes of our study—is its development and cultivation of a positive risk culture.[13,14]

Risk culture (alternatively, reliability culture) describes the collective risk assessment and management ethos that is established and cultivated by high reliability organizations whose operations could potentially pose a threat to life or fundamentally compromise their mission. For the past four decades, there has been considerable elaboration of what this culture consists of, how to measure it, and how to maintain or enhance it.[13,15] Broadly speaking, it encompasses the risk assessment methods, knowledge, values, beliefs, and attitudes systematically instilled and reinforced in practitioners by the institutions that comprise a profession.[16]

Among the characteristics of a healthy risk culture are a preoccupation with failure, which leads to frequent self-reflection and a willingness to continuously refine procedures to learn from mistakes.[17] These characteristics form part of an organization's resilience.[15] When it comes to risk assessment methods, this involves developing (or adapting) appropriate tools and frameworks, including from other fields if they complement or enhance the reliability mission. The field of risk analysis has generated tools and frameworks that are applicable across a range of high reliability organizations. Although diagnoses and prescriptions for mitigating risk tend to rhyme regardless of domain, differing treatments of risk generate different results.

Here, we investigate the evolution of risk culture in the following three fields that demand exceedingly high standards of reliability: radiation oncology, commercial aviation, and nuclear power. We choose these three for several reasons. First, unlike space travel or wildland fire management, a large cross-section of the U.S. public is statistically exposed to their risks, making safety a paramount concern. Second, they have existed long enough for an evidence-based risk culture to develop. Third, the evolution of this culture is eminently traceable in the publications within each field. Fourth, all three are civilian fields, although they exploit dual-use equipment, materials, technology, and expertise. By commercial aviation, we refer to Part 121 air carriers, so-called because they have been certified by the Federal Aviation Administration under Part 121 of Title 14 of the Code of Federal Regulations.[18] This excludes both general aviation and Part 135 carriers, which are mostly commuter or on-demand carriers.[19] Similarly, in analyzing the nuclear power industry, we focus on commercial, civilian nuclear power reactors as opposed to either test reactors or military assets.

Because commercial aviation and nuclear power might not be as familiar to readers as radiation oncology, we provide a brief discussion of the evolution of regulation and actuarial risk in these two fields. Table 1 provides a nonexhaustive summary of when regulations that govern commercial aviation and nuclear power in the United States came into effect, by decade.

TABLE 1

A Brief Outline of When Major Regulations That Govern Commercial Aviation and Nuclear Power Came Into Effect

Aviation	Decade	Nuclear Power
Air Commerce Act; mandatory inspections; pilot licenses	1920s
Federal Air Traffic Control; Civil Aeronautics Act; Civilian Pilot Training Act	1930s
Civil Aeronautics Authority split; standardization centers; federal tower control; federal aircraft certification requirements; Chicago Convention	1940s	Atomic Energy Act of 1946; Atomic Energy Commission established to control nuclear science and technology
Federal Aviation Agency established; mandatory data recorders; inception of NASA	1950s	Atomic Energy Act of 1954 encourages the commercial development of nuclear power
Forbidding intoxicated passengers, mandating tall towers, cockpit doors, cockpit voice recorders, and life preservers	1960s
Antihijacking mandates; radar improvements; crew resource management; standardized communication; Airline Deregulation Act	1970s	Nuclear Regulatory Commission established; WASH-1400 report establishes probabilistic risk assessment
Satellite guidance and tracking systems	1980s	Post-TMI safety measures, including revised control room design; inception of Institute of Nuclear Power Operations
Travel collision avoidance systems start to be incorporated	1990s
Post-9/11 security measures	2000s	Post-9/11 security measures

There are no similar oversight regulations for the medical physics field.

TMI, Three Mile Island.

Two points bear emphasizing: first, increased regulation of these complex technologies was mainly spurred by accidents, which tend to have high visibility and generate loud demands for enhanced safety from both organized critics and the public at large. Second, new regulations tend to be instituted in tranches. Accidents serve as policy windows[20] during which government, industry, and civil society labor to alter the existing safety paradigm in pursuit of risk mitigation. In commercial aviation, these policy windows included the 1977 Tenerife air disaster, which encouraged the adoption of standard phraseology in radio communication and crew resource management to enhance intercrew communication[21]; the 1996 Charkhi Dadri midair collision, which accelerated the adoption of traffic collision avoidance systems[22]; and the September 2001 attacks in the United States, which instigated a radical shift in security protocols across the airline industry.[23]

Risk management in the nuclear industry, meanwhile, changed after the 1979 Three Mile Island incident, when control rooms were redesigned and the Institute of Nuclear Power Operations was established[24]; after the September 2001 attacks, when the design-basis threat for new plants was revised to include potential aircraft impact[25]; and after the 2011 Fukushima disaster in Japan, when the industry was forced to explore the implications of beyond design-basis accidents and enhance emergency response protocols,[16] among others.

In Figure 1, we present the evolution in the number of U.S. accidents and fatalities within commercial aviation and nuclear power as a result of these enhanced risk mitigation practices. The figure shows a substantial decline in the number of U.S. accidents and fatalities in commercial aviation; commercial nuclear power, meanwhile, has been responsible for a small number of accidents and fatalities in the United States over its history. By even the most conservative standards, the decline in accidents and fatalities in commercial aviation is remarkable, whereas the risk of dying from nuclear power plant accidents in the United States is virtually nil. The statistics on aviation fatalities are the least controversial, whereas those on fatalities due to medical error are the subject of much debate.[1-4] Meanwhile, the numbers we use for nuclear power fatalities come from the only systematic (though haphazard) assessment of nuclear power's risk of which we know.[27] Although there remains much to be said about the history and evolution of risk within these two fields, we refrain from doing so in this article.

FIGURE 1

Accidents (A) and (B) fatalities in commercial aviation[26] (blue), nuclear power[27] (orange), and the entire field of medicine[1–4] (red). Industries with evidence-based risk cultures—commercial aviation and nuclear power—have experienced lower and decreasing accident and fatality rates.

METHODS

We apply the principles of network theory[28] to perform a bibliographic coupling analysis[29] on a database of the major risk publications in each field of study—radiation oncology, commercial aviation, and nuclear power. To build this database, we consult with experts in each of the three fields to elicit what are—in their judgment—its seminal risk publications. Upon compiling this database, we perform a content analysis on the documents within it to obtain the top key words each field uses to describe risk. We do this because the language used to describe, assess, and manage risk can vary within each field despite the common goal of ensuring high reliability and safety.

We then use these key words to cast a wider net in our search of the literature. This second phase of the search was conducted through Web of Science.[30] Combining the field name with the top key words used to describe risk in the field, as derived from the previously mentioned first step, we conduct a search of Web of Science and add the resulting articles to our database. Finally, we conduct a bibliographic coupling analysis of the entire database to establish the degree to which fields reference each other, borrowing methods and best practices in the process. We present the methodological design in Figure 2, before explaining each step in greater detail below.

FIGURE 2

An overview of key steps and methods in our research.

Extraction of Key Words Within Seminal Risk Publications

Different fields use different terminology to describe their approach to risk assessment, risk management, and risk communication. Moreover, regulation within each field is undertaken differently, which occasionally necessitates jargon that is context specific. Conducting a blanket search of the literature using standard risk terminology is therefore inappropriate. Instead, we need to identify the core risk terminology within each field and search the literature for that terminology. We consult with experts in aviation, nuclear power, and radiation oncology to assemble a corpus of the seminal risk documents within each field. These came to include key publications that gave birth to risk analysis frameworks and methods, regulatory manuals and postaccident investigations, and reviews of the state of risk analysis conducted by professional organizations within each field.

We conduct a content analysis of these documents' abstracts using the Text Mining and SnowballC packages in R.[31,32] This text analysis produces a list of key word stems that describe risk culture within each field. Figure 3 lists the 10 key word stems that most frequently appear in our database and displays the frequency of their presence in radiation oncology (in red), commercial aviation (in blue), and nuclear power (in yellow). It demonstrates the extent of similarity in the terminology used to describe risk across the three fields.

FIGURE 3

Top key words describing risk culture, ranked by count of appearance in the abstracts of seminal risk documents in each of the three fields.

The main words used to describe risk culture contain the following stems: safeti, risk, system, use, manag, requir, accid, license, event, and cultur. The frequency with which these word stems appear in each domain is summarized in Table 2. All of these key words appear in each of the three disciplines with the sole exception of license, which is referred to almost exclusively in the nuclear power literature—licensing being a key requirement for every phase of nuclear power development, including design, construction, and operation.

TABLE 2

Frequency With Which the 10 Most Repeated Key Word Stems Appear in the Risk Literature Across Domains

Word Stem	Frequency in Commercial Aviation	Frequency in Nuclear Power	Frequency in Radiation Oncology	Total Frequency
safeti	2700	1820	3167	7687
risk	1394	3573	791	5758
system	1660	2194	1187	5041
use	761	2562	1191	4514
manag	991	1457	880	3328
requir	378	2391	348	3117
accid	869	1835	257	2,961
license	0	2636	3	2639
event	431	1481	601	2513
cultur	612	44	1813	2469

Systematic Literature Review to Generate a Corpus of Risk Publications

The key word stems in each field form the basis for a systematic literature review. We conduct a key word search through Clarivate Analytics' Web of Science,[30] the largest global database of accredited scholarly work. By using the top 10 key words across fields, we reduce bias in the search results. For the full list of articles in the corpus, please see Appendix A, http://links.lww.com/JPS/A201.

We further analyze the database of articles to determine whether our three searches captured out of context articles and, if so, their number and subject. We find virtually no out-of-context matches in our database of radiation oncology articles and limited out of context matches in our database of nuclear power articles (9%). Commercial aviation has a substantial number of out-of-context matches (20%), which we expected because aviation is often held up as a role model for effective risk assessment and management. Therefore, we found articles dedicated to adapting the lessons of aviation risk culture to kayaking, scuba diving, epidemiology, pharmacology, dentistry, diabetes, disaster response, wildfire management, wildlife management, surgery, patient safety, and various industries. We manually remove articles that are not relevant from each database before proceeding with a bibliographic coupling analysis.

In Table 3, we present summary statistics that describe the risk articles in our database, which is composed of more than 2000 unique items written by more than 8000 unique authors. The eight journals that appear most frequently happen to include some of the premier risk analysis and management journals in the world, which serves as a valuable sanity check. The earliest documents in the database are from 1984.

TABLE 3

Summary Statistics Describing our Database

		Journals
Corpus Statistics		Name	Field	Article Count
Articles	2045	International Journal of Radiation Oncology Biology Physics	Radiation oncology	63
Sources	944	Journal of Clinical Oncology	Radiation oncology	61
Period	1984–2017	Safety Science	Aviation, nuclear power	40
Average citations per article	15.9	Reliability Engineering & System Safety	Aviation, nuclear power	35
Authors	8015	Radiotherapy & Oncology	Radiation oncology	34
Author appearances	9679	Gynecologic Oncology	Radiation oncology	31
Single authorships	263	Cancer	Radiation oncology	25
Multiple authorships	7752	Risk Analysis	Aviation, nuclear power	25

Bibliographic Coupling Analysis

Finally, we perform a bibliographic coupling analysis on the entire database using the bibliometrix software package in R.[33] This specific type of network analysis[29] captures the connectedness of the research within each field and across fields. Articles are said to be bibliographically coupled if they have at least one cited reference in common, capturing how often one field references the work of another. The mathematical formulation of the bibliographic coupling analysis is presented in Equation 1, with B representing the coupling matrix and A representing a bipartite matrix of articles by references.

As risk analysts, it is especially interesting to compare reference lists in different fields. This helps us characterize the extent to which state-of-the-art risk management tools and frameworks are applied in each field, and whether each engages with (and thus learns from) other high reliability enterprises—specifically, with each other.

RESULTS

Figure 4 illustrates the bibliographic coupling of the top 900 most coupled articles within our corpus, with red representing literature in the field of radiation oncology, blue representing articles in the field of commercial aviation, and yellow representing articles in the field of nuclear power. Broadly, there exist two clusters: the risk culture literature within radiation oncology constitutes one such cluster, shown on the left, and another on the right that betrays the strong intermingling of risk analysis in the fields of commercial aviation and nuclear power. Of the 977 articles in the field of radiation oncology within our database, only 11.4% reference the enormous and more advanced risk literature outside the field. This low interreference rate is in stark contrast to the risk literature in the fields of commercial aviation and nuclear power, where 34% and 25% of the articles refer to the risk literature in the other fields, respectively.

FIGURE 4

Bibliographic coupling of articles within the database of risk documents that we constructed. This network graph highlights the difference between the field of radiation oncology and the other two: while commercial aviation and nuclear power are tightly coupled—borrowing liberally from advances in risk study in each other—radiation oncology constitutes a cluster unto itself.

Of the radiation oncology literature, only 9.2% references the risk work in commercial aviation, whereas only 2.2% references the risk work in nuclear power. Figure 5 illustrates the evolution of this referencing to the commercial aviation and nuclear power risk literatures over time. The stacked columns are composed of bibliographic links between radiation oncology and commercial aviation (depicted in blue) and links between radiation oncology and nuclear power (depicted in yellow). Across all the interreferenced articles in radiation oncology, the commercial aviation literature is cited more than twice as often as the nuclear power literature.

FIGURE 5

Evolution of reference coupling from radiation oncology to commercial aviation (blue) and nuclear power (yellow).

This relative dearth of bibliographic coupling hints at a lack of engagement with the fundamental risk analysis frameworks and methods developed in the fields of commercial aviation and nuclear power. However, a few articles in the radiation oncology literature stand out as early adopters of the fundamental precepts of risk analysis in the pursuit of patient safety. These articles almost exclusively reference safety practices in the field of commercial aviation, despite the fact that the technology used in radiation oncology is arguably closer to that of nuclear power. This may well be the result of an availability bias[34]: aviation safety is arguably a more familiar topic to the general public and to practitioners of risk analysis than nuclear power operations, whereas medical errors occur as a single fatality and escape public attention. It may also reflect a bias against nuclear power, which is a controversial technology that engenders substantial dread and with which parallels might not be desired.

DISCUSSION

Our analysis highlights the relative isolation of the radiation oncology risk literature from that in arguably two standard-setting fields: commercial aviation and nuclear power. The field of radiation oncology objectively lags behind other high reliability, mission critical enterprises: there is no overarching regulatory authority; there are no mandatory training protocols; and there are no mandatory staffing protocols. These three happen to be basic prerequisites in other high reliability enterprises, including commercial aviation and nuclear power. Commercial aviation and nuclear power make extensive use of simulation training. In addition, the training in these two fields amount to weeks per year.

The goal of this study is not to provide a comprehensive explanation of the shortcomings in radiation oncology. Our work does illustrate, however, that the state of the art in its assessment and management of risk does not borrow from other fields. As the literature on high reliability organizations makes clear, failure to engage with and borrow from other fields that require similarly high standards is a clear demonstration of radiation oncology's poor risk culture—specifically, its ignorance of the wealth of risk management frameworks and methods used in commercial aviation and nuclear power. Although we are not the first to point out this failure,[4,35] we confirm it statistically and quantify its extent in this article. We also note that the mere fact that it has taken so long for medical physics and radiation oncology to react to this problem[35] and to consider applying risk management practices that other fields adopted decades ago and continuously refine betrays a reticence in the ingrained culture that we consider intolerable. Consequently, it is not surprising that errors in radiation oncology do occur, nor that most are entirely preventable.

Our method can be generalized beyond radiation oncology to other disciplines within health care that affect patient safety. Comparing different disciplines within health care would also be valuable. However, we encourage researchers to engage subject matter experts within each discipline before proceeding. In all likelihood, these experts would be able to recount the central risk publications within each discipline, its standard-setting bodies, its regulations, and any ongoing (i.e., unpublished) work to enhance risk culture. Knowing as much as possible about the discipline provides valuable “sanity checks” as the quantitative analysis proceeds.

CONCLUSIONS

In this work, we identified seminal risk publications in the fields of radiation oncology, commercial aviation, and nuclear power. We then used key risk terminology derived from these seminal publications to build a database of the risk literature in each field. A bibliographic coupling analysis was conducted on this database to rigorously compare one aspect of the risk culture in these three high reliability enterprises—namely, their willingness to explore or borrow state-of-the-art practices from each other. We found that commercial aviation and nuclear power have established risk cultures that reference and borrow best practices from advances in each other. However, there seems to be a distinct difference in radiation oncology's engagement with the risk community. Specifically, the risk literature within radiation oncology seems to be more isolated, which raises significant concern of a reticence to engage with other high reliability enterprises. Based on our observations, we recommend that leaders in the field enhance their engagement with the broader risk community to expeditiously integrate best risk practices and reduce errors, nurturing healthier attitudes to risk throughout the enterprise in the process.