The effect of a safety crisis on safety culture and safety climate: The resilience of a flight training organization during COVID-19.

Flight training programs incur great risks as they instruct aspiring pilots to complete flight training and licensing procedures. It is instrumental for these programs to create strong safety cultures and climates to promote the highest level of safety possible. The COVID-19 pandemic raised many safety concerns across a variety of industries, especially the aviation industry. This study investigates the impact that the COVID-19 pandemic had on the safety culture and safety climate of the flight training organization at Embry-Riddle Aeronautical University (ERAU). For this purpose, we collected longitudinal data for the years 2018 through 2021 including the variables of safety culture and safety climate. The results of this research suggest that various safety culture and safety climate variables were impacted during the COVID-19 pandemic. Based on these results, leadership of the flight training program was able to mitigate and adjust safety policies and procedures to improve the safety culture and climate and ensure continuous accident-free performance. Finally, the viability of these and other safety procedures for the safe management of future crises are discussed.

Introduction

Continuous assessments of safety culture and safety climate are particularly important for minimizing accidents, incidents, and hazards in high-risk industries such as nuclear power, commercial space travel, health care and aviation. These continuous assessments assist companies in understanding the impact that critical safety events (accidents, incidents, or hazards), major organizational changes (changes in policies, procedures, or leadership), and other system changes (changes in resources, training, or support systems) have on the safety culture and climate within an organization (e.g., Casey et al., 2017; O'Connor et al., 2011; Zohar, 2010). A flight training organization is a high-risk organization operating in the High-Risk Industry of aviation. Additionally, the COVID-19 pandemic can be characterized as a critical safety event for flight training organizations since it introduced a significant hazard to flight training operations. Specifically, the requirement of a student and instructor to be in close proximity of each other while operating within a small, confined aircraft introduced the hazard of spreading COVID-19 thus increasing the likelihood of a large-scale COVID-19 outbreak within a flight training organization.

There is a high level of inherent risk associated with flight training. It is critical for flight training organizations to operate using a safety management system that ensures the highest level of safety possible. As part of this safety system, it is important for leadership to develop and embed a positive safety culture and safety climate within the flight training organization. Research suggests that organizations with strong safety cultures and safety climates are less prone to experiencing safety related incidents and accidents (e.g., Christian et al., 2009; Clarke, 2006; Guldenmund, 2000; Zohar, 2010). Hence, it is important for flight training organizations to assess both safety culture and safety climate on a regular basis. These assessments assist leadership in monitoring the safety culture and safety climate as well as providing feedback related to their effectiveness and possible needs for improvement.

The purpose of this study is to investigate the impact that the COVID-19 pandemic had on the safety culture and safety climate of the flight training organization at Embry-Riddle Aeronautical University (ERAU). To complete this study, safety culture and climate were assessed using a modified version of an assessment tool originally developed by Ayyalasomayajula (2007) and refined by Grabowski et al. (2010). Although the assessment tool was originally developed for the marine transportation industry, Byrnes (2015) adapted the instrument for use within a flight training organization. This adaptation is appropriate since maritime and aviation activities are both considered high-risk industries. Additionally, human and organizational error are prevalent in both industries with limited in-person oversite; this makes risk mitigation in these industries similarly complex. The aforementioned data collection instrument uses several variables to assess safety culture and safety climate. The variables for safety culture include employee perceptions of: (1) effective formal learning system, (2) promotion of safety, (3) prioritization of safety, (4) effective communication, (5) problem identification, (6) quality of new hires, (7) safety orientation, (8) reward safety, and (9) multicultural operations. The variables for safety climate include employee perceptions of: (1) perceived risk, (2) employee empowerment, (3) personal safety responsibility, (4) anonymous reports, (5) effective feedback, (6) respect, (7) integrity, and (8) willingness to change. ERAU has collected safety culture and safety climate data using this assessment tool since 2011. Analyzing these data between 2018 and 2021 suggest that various safety culture and safety climate variables were impacted during the COVID-19 pandemic.

We will first review the extant literature on safety climate and safety culture in general and then focus on the literature as it relates to the safety climate and safety culture. Then we will explain the hypothesized impact of COVID-19 on the safety culture and safety climate at ERAU and analyze the data collected. We will discuss the results of this research and provide suggestions for addressing concerns related to the safety culture and safety climate that emerged during COVID-19. In particular, we will discuss the changes leadership implemented to address these problems and how safety policies and procedures were adjusted to ensure continuous accident-free performance. Finally, the viability of these and other safety procedures for the safe management of future crises is discussed.

Literature review

Safety climate and safety culture

Organizational climate is generally defined as shared perceptions nd meanings attached to organizational policies, procedures, practices employees experience and behaviors that are observed as being rewarded, supported, and expected (Schneider et al., 2013). Schneider (1975) suggested that organizational climates should be studied at the focused level where their strategic impact is perceived and experienced. Hence, in order to observe and monitor safety related issues, it is important to define and assess the safety climate of the organization. Zohar (1980) defined safety climate as employee perceptions related to the values and role of safety in the organization. In particular, he defined safety climate as the unified cognitions regarding safety, safety policies, procedures and practices (Zohar, 1980). A consistent stream of research has emerged to study the effect of safety climate on various organizational outcomes, employee behaviors and attitudes (Schneider et al., 2013). Meta-analytic results demonstrate a consistent relationship between safety climate and accidents (Christian et al., 2009; Clarke, 2006, 2010; Nahrgang et al., 2011). The authors identified that increased levels of accidents inversely affect the shared perceptions of the climate for safety such that safety climates are perceived to be poor. The safety climate of an organization is directly related to accidents, reporting of these accidents and that poor safety climates are related to underreporting of accidents (Probst et al., 2008). Weak safety climates are often the result of inconsistencies related to policies and procedures (Zohar and Luria, 2005). Casey et al. (2017) explain that positive safety climate is encouraging safety behaviors due to the norms of reciprocity that develop when individuals and teams perceive that management is focusing on workplace safety. Safety climate has also been found to be a leading and lagging indicator of safety; correlations between safety climate and prior accidents was marginally stronger than correlations between safety climate and future accidents (Beus et al., 2010).

Safety is important in a high-risk industry such as the aviation industry. Gao et al. (2015) found that safety climate of the commercial airline under investigation was perceived to be strong across all four professions (pilots, cabin crew, engineers and ground operations). Respondents were willing to report incidences, perceived a commitment of senior management toward safety, and felt they had access to current safety-related information. In a study involving pilots from Australia, Evans et al. (2007) found no difference regarding safety climate perceptions when studying pilots from regular public transport, charter and aerial work pilots. Kao et al. (2009) studied safety climate among Taiwanese cabin crew members and found that high management commitment to safety was positively related to crewmember participation in safety procedures. Also, the safe cabin environment was significantly related to crewmember's individual safety behavior. A study including 281 pilots demonstrated that organizational affiliation and proactive management were the main correlates of safety outcomes and main drivers of safety outcomes (Block et al., 2007). In a study of US Naval squadrons, Desai et al. (2006) identified a positive relationship between minor and intermediately severe accidents and safety climate scores but no effect for major accidents. Adamshick (2007) found that US Navy Strike-Fighter pilots recognized leadership to be positively related to safety climate. Hernandez (2001) did not identify a difference in safety climate between U.S. Naval squadrons that had experienced maintenance incidents and those that had not. Overall, the literature suggest that positive safety climate is inversely related to accidents and investigations of fatal accidents. Research suggests the importance of safety climate as an attributing factor in accidents (McInerney, 2005). However, the safety climate literature from the aviation industry provides mixed evidence and no dominant safety climate measure has emerged (Evans et al., 2007).

Organizational culture is generally defined as the shared assumptions, values, and beliefs of a work environment, that are taught to newcomers as “how things are done around here,” ow to think, feel and act, and that are communicated by the myths and stories people tell about how the organization came to be (Schneider et al., 2013). Schein (2010) proposed three levels of organizational culture including artifacts, espoused beliefs and values, and underlying assumptions. The first studies that identified the construct of safety culture and identified its impact on accidents and safety related incidences originated with the Chernobyl accident in 1986 (NSAG-1, 1986, as updated in INSAG-7, International Nuclear Safety Advisory Group, 1992). Major accident investigations including the Oil Platform Piper Alpha, Space Shuttle Challenger and Space Shuttle Columbia disasters subsequently identified safety culture as a leading factor in accidents (e.g., Cullen, 1990; NASA, 2003).

Taylor (2010) defined safety culture as an organization's values and beliefs toward safety. Poor risk practices such as ignoring organizational protocols and disregarding risk influences the behaviors of individuals in these organizations and were common practices regarding “how things are done around here” (Pate-Cornell, 1993; Pidgeon, 1991). JeffcottPidgeon et al. (2006) found that safety culture is directly related to trust in the organization. A study by Håvold (2010) assessing the safety related attitudes and culture of fishermen from 63 tankers found that the safety attitude of management strongly influenced a company's safety policy such that fishermen that were not involved in accidents demonstrated more positive attitudes towards the organization's safety culture. A study involving expert and management staff from 17 countries found evidence that while safety cultures can exist consistently across national borders they differ across industries (Reader and O’Connor, 2014). Using semi-structured interviews using front-line operational staff members from the railway industry identified that senior management had a direct impact on how safety culture was perceived and that employees do not necessarily consider safety their responsibility and tend to look for an easy and comfortable way of achieving task even if this involves increased risk.

A body of literature has emerged studying safety culture in the aviation industry. For example, Ek et al. (2003) studied the safety culture in air traffic control centres and found that safety culture varied across centres and personnel levels. Employees perceived a lack in safety training regarding emergency communication procedures. Managers seemed to be overly positive in reporting a positive safety culture. Ek and Akselsson (2007) studied ground handlers in Sweden and found a strong and positive safety culture among employees. Similarly, Kelly and Patankar (2004) identified a positive safety culture among maintenance personnel. They attributed variances in responses to differences in participants’ age and experience. A study including employees from the New Zealand aviation industry revealed that employees did not place much importance of safety culture or safety management and that pilots revealed that luck significantly contributed to safety (Gill and Shergill, 2004). Adjekum (2014) studied flight students and certified flight instructors in a U.S. 4-year university flight program and found a positive correlation between the perceived positive safety culture and longevity in the program. In addition, international students perceived the safety culture to be less favorable than domestic students.

It is evident that organizational safety climate and safety culture are related to safety performance including accidents and safety related incidences. Safety culture and safety climates are perceived differently across ages, industries and positions in the organization. Also, management and effective leadership are important in developing a positive and strong safety culture and safety climate (O'Connor et al., 2011). However, a review of the literature also reveals that definitions and measures of safety culture and safety climate vary greatly from study to study (Casey et al., 2017). Many studies define and use safety culture and climate as interchangeable constructs (e.g., Nævestad et al., 2018) which muddies the results and our understanding of these constructs and their impact on safety related performance and incidences. In addition, high-risk industries generally adhere to strict safety policies and procedures and safety incidences are rarely observed. Hence it is difficult to determine with certainty the impact of safety culture and safety climate on the prevention of accidents and safety related incidences.

In the following section, we will discuss the development and implementation of a safety culture and safety climate at the flight training organization at Embry-Riddle Aeronautical University.

Safety culture and safety climate in a flight training program

As we have discussed, aviation is a safety-critical industry that requires a specific culture and related processes focused on behavior that not only avoids accidents but also instills a safety-oriented mindset across all levels of the organization. One of the most fundamental ways to instill the tenets of a safety culture in an industry such as aviation is to do so during the training process. While the following discussion will focus on flight training programs, parallels can be drawn to other areas such as aircraft maintenance. In the United States, the Federal Aviation Administration (FAA) regulates flight training and certification, which consists of two different pathways. The first is typically referred to as a “Part 61” program, referring to FAA regulation 14 CFR (Code of Federal Regulations) Part 61: Certification of Pilots, and Flight Instructors, and Ground Instructors. While Part 61 establishes the eligibility, knowledge requirements, and flight time needed for certification of all pilots, it also refers to a program that simply follows the overall requirements of the Part. Part 61 schools tend to be more flexible and do not have a firm training structure other than what is required within the regulation. The other type of program is regulated by 14 CFR Part 141: Pilot Schools. Part 141 oversees the certification and operation of flight schools to include a specific FAA-approved curriculum and a fairly regimented certification process for students within such programs. While pilot schools may operate under either Part 61 or Part 141, with the expected same quality of training, the structure, style, and other differences can be quite significant. The remainder of this discussion will focus on the safety culture at Part 141 programs since they have very structured processes and are typically found at universities such as ERAU, especially for students preparing for a career in aviation.

A particular focus of the literature on aviation safety culture, and more specific to flight training, is the implementation of a Safety Management System or SMS. The use of an SMS focuses on the proactive management of risks and hazards through data analysis and is often an organization's official implementation of predetermined safety goals. This is in sharp contrast to the more customary reactionary approach to safety which reacts to an accident or incident, and then takes measures to avoid reoccurrences (Gilliam, 2019). The FAA defines an SMS as, the formal, top-down, organization-wide approach to managing safety risk and assuring the effectiveness of safety risk controls. It includes systematic procedures, practices and policies for the management of safety risk (FAA, 2019). Some universities offer undergraduate degrees in Safety Management which not only brings the flexibility and management of an SMS into the industry but also equips future leaders with the tools needed to face the many challenges to safety as technology and the industry advances (Velequez and Bier, 2015). An important feature of an SMS is that it provides for the assessment of safety conditions and offers an iterative process for improvement. A positive safety culture can be viewed as a willingness to work as a team to promote safety and is often included as assessments within an SMS to measure progress towards safety goals (Wheeler et al., 2019).

While the FAA does not currently require a Part 141 school to have an SMS program within its training operation, many collegiate programs that are accredited by the Aviation Accreditation Board International (AABI) have implemented programs that meet the requirements for specialized safety accreditation. This process involves a formal and verifiable safety program involving students, flight instructors, faculty, and other key stakeholders. Programs earning this designation represent only about 10% of all aviation programs (Canders, 2016). Studying a safety culture and specifically the SMS that helps facilitate it necessitates the need to benchmark safety performance in an effort to achieve continuous improvement. Such measurements can be tedious due to the fact a large amount of forensic data is needed to aid the benchmarking process. Examples of such data includes accident rates, accident costs, insurance claim filings, productivity days lost, etc. It is also recommended that multiple measures be used to evaluate a safety system such as accident records, results of safety audits, and perception survey results (Goodheart, 2017).

While the establishment of a strong safety culture within a flight training program will likely have the side-effect of instilling safe practices within the student pilots, which they will hopefully take with them to future employers, we will look more closely at the actual flight training programs and their specific safety culture as it relates to flight instructors, policies and procedures, and management. Wheeler et al. (2019) performed a study during the fall of 2018 to measure the safety perceptions of flight students, instructors, and maintenance employees. The goal of the study centered on different perceptions by instructors, their students, and those who maintain the aircraft. The overall results of the study showed an overall positive safety culture across all groups with the highest average scores given to the instructors and maintenance personnel. Takeaways from this study included the need for continued assessment as well as measures to improve processes as needed. A positive safety culture is vital to managing the risks of flight training and can be encouraged through the cultivation of suitable safety performance and practices (Adjekum et al., 2015).

Adjekum et al. (2015) used a variation of the Collegiate Aviation Perception of Safety Culture Assessment Survey (CAPSCAS) to perform a cross-sectional assessment of safety culture perceptions and reporting behaviors in collegiate flight training programs. Findings of this study showed that age, safety perceptions, and the reporting system used were reliable predictors of safety reporting behavior. Also noteworthy was that there was no significant difference in safety reporting behavior between students and flight instructors. An additional study by Adjekum et al. (2016) with aviation students in non-flight programs showed disparate results on safety culture perceptions and reporting as compared to students in flight programs. This is possibly due to lack of exposure to safety and operational professionals that routinely provide feedback to student pilots. Goodheart and Smith (2014) looked at whether a positive safety culture measured by an established instrument actually reflected positive safety performance. The research involved a meta-analysis of five studies and while the results showed a positive correlation of culture to performance, the sample size was relatively small which makes it challenging to correlate the results given difference with the studies.

As we have seen, there is oversight along with specific processes and tools that are used to manage and monitor the safety culture within an aviation flight training program. Even with the most cutting-edge resources, the human element is crucial to any successful safety initiative and includes group and individual attitudes, competencies, values, etc. (Adjekum, 2017). Implementing a safety system may actually be the easy part whereby engagement of key stakeholders requires buy-in at all levels as well as integration into the core mission of the organization. Effective systems are not simply installed and left to run on their own, but need constant monitoring, assessment and revision throughout its’ useful life.

ERAU's safety culture and safety climate assessment

Methods

This mixed methods study was conducted using a survey instrument to measure organizational safety culture and organizational safety climate. Additionally, a focus group comprised of flight instructors was assembled to provide qualitative feedback on the survey results. The research described in this paper is based on an ongoing safety program at Embry-Riddle Aeronautical University. One component of this effort is an annual survey of all flight instructors in the flight training organization. Survey methodology is an efficient and timely way to measure unobservable constructs such as culture and climate (Molenaar et al., 2009). This specific survey has been administered within the flight department since 2010. As a standard practice, the flight department leadership reviews the survey results with a focus group of instructor pilots each year to gain deeper insight into any safety culture or climate variations. The survey results used for this research spanned across a four-year period from 2018 through 2021. These survey results provided safety culture and climate data from before the COVID-19 pandemic (June 2018 and 2019), during the height of the pandemic (June 2020), and a period that the pandemic had appeared to subside (June 2021). The survey results from these four years were analyzed longitudinally using descriptive statistics to determine general trends and to gather feedback from the focus group. Further quantitative analysis was conducted using a Random Forest regression model to investigate any potential relationships between survey questions and pre, during, and post-pandemic conditions.

Survey

Safety culture and safety climate were measured using an instrument developed by Ayyalasomayajula (2007) to identify leading indicators (LIs) of safety for the shipping industry. Grabowski et al. (2010) continued to refine this survey which consisted of 102 items on employee attitudes and perceptions of safety culture and climate. Byrnes (2015) adapted this survey for use within flight training organizations.

All items in the survey used for this research used a 7-point Likert scale with 1 = Disagree Strongly and 7 = Agree Strongly. Following a second order factorial analysis, the following composite measures were created for the LIs of safety culture: (1) effective formal learning system (9 items), (2) promotion of safety (9 items), (3) prioritization of safety (3 items), (4) effective communication (7 items), (5) problem identification (4 items), (6) quality of new hires (6 items), (7) safety orientation (5 items), (8) reward safety (8 items), and (9) multicultural operations (4 items). The composite measures for safety climate include: (1) perceived risk (4 items), (2) employee empowerment (5 items), (3) personal safety responsibility (7 items), (4) anonymous reports (4 items), (5) effective feedback (4 items), (6) respect (8 items), (7) integrity (7 items), and (8) willingness to change (8 items) (See Appendix I for the individual items). All organizational safety culture and organizational safety climate observed variables were deemed reliable and internally consistent with alpha coefficients greater than 0.75 (Byrnes, 2015).

Sample

The sample consisted of all flight instructors actively employed in the flight operations department at Embry-Riddle Aeronautical University. The sponsoring institution has one of the largest accredited AABI flight training programs. The total sample size was 585 for the period 2018 to 2021. The yearly sample size ranged from 124 in 2020 to 150 in 2021. Table 1 identifies the age frequency and percentage of sample. As might be expected at a university-based flight program, most respondents were in the 19–24 age range.

Table 1

Age range of respondents.

Age	Frequency	Percent
19–24	335	57.3
25–30	190	32.5
31–40	34	5.8
41–50	6	1
Over 50	13	2.2

Note. Table 2 shows the flight hours for the respondents. Almost 66% of the sample fell in the range of 600–1999.

Table 2

Total flight hours.

Total Flight Hours	Frequency	Percentage
250–299	13	2.2
300–599	120	20.5
600–999	195	33.3
1000–1999	189	32.3
2000–2999	32	5.5
3000–3999	9	1.5
4000–5999	5	.9
Over 6000	15	2.6

Note. Table 3 shows the total hours for the responding CFI.

Table 3

Total CFI hours.

Total Hours	Frequency	Percentage
1–99	45	7.7
100–299	94	16.1
300–499	96	16.4
500–799	113	19.3
800–999	61	10.4
Over 1000	170	29.1

Administration of survey

The questionnaires were administered in a computer laboratory during a normally scheduled 60-min meeting. Participants completed the survey for safety culture, safety climate, and a related professionalism scale electronically using the internet based SurveyMonkey software.

Analysis

To investigate the impact of the COVID-19 pandemic on the safety culture and safety climate at ERAU, Radom Forest (RF) models were built on the data from surveys administered from 2018 and 2019 (pre-pandemic), 2020 (during-pandemic), and 2021 (post-pandemic) to investigate how results from the survey questions (i.e., indices) predict the pandemic-related categories. Initially, the results of each question are tallied and averaged for each leading indicator. Averages of each leading indicator are calculated, and the data is displayed visually to allow annual trends to be identified. To provide a more robust statistical analysis, an RF regression model was then built. This model can show how the survey questions (i.e. indices) predict the pandemic-related categories. RF uses decision trees through bootstrapping different samples of a data set. A node in the decision tree is divided between subsets of predictors at random. The model reports which input variables (in this case the survey questions) are of importance as they predict the output. RF method was selected in this study because it is efficient, interpretable, and non-parametric for various types of datasets, and strong enough to overcome the problem of overfitting (Ali et al., 2012). These features fit our dataset because it is not normally distributed and has a complex set of independent variables (e.g., survey questions) which might cause overfitting problem using traditional classification models.

Prior to statistical analyses, we first pruned the data for missing values and multicollinearity. Of the 586 participants, 117 participants had incomplete data and were removed leaving 469 observations (n = 469). A check on the 124 variables was performed to control for multicollinearity and 23 pairs of questions which highly correlated (absolute r > 0.699) were identified. From these 23 pairs, a representative variable from each pair was selected based on its importance in identifying that the safety management system is working appropriately to mitigate risk. The remaining 101 variables were then used to build an RF model. RF uses decision trees through bootstrapping different samples of a data set. A node in the decision tree is divided between subsets of predictors at random. RF is also useful with data which is not normally distributed as is the case in this study. The model reports which input variables (in this case the survey questions) are of importance as they predict the output. In general, the two parameters for RF are 1) the number of trees and 2) the number of variables at each split.

A set of RF regression models were built using the CARET package (Kuhn et al., 2016) using R (R Core Team, 2017). To check for accuracy of the models (Brieman, 2001), the data was randomized and split into a group of 281 observations (60%) to be used as training data and the other 188 (40%) to be used as the test set. The RF models were first trained and evaluated on the training data. The trained RF model was then fit to the test set, and a confusion matrix was created using confusionMatrix function from CARET package to display the number of correct and incorrect classifications. In addition, an F-score was calculated based on the prediction results from the confusion matrix. Finally, the final RF model was selected based on their performance as shown by the accuracy scores and F-scores.

To unravel the detailed links between the survey questions and the pandemic-related categories in the final RF model, we also built a multinomial logistic regression model using R (R core Team, 2015) with the nnet (Croissant, 2012) package. The survey questions as selected in the final RF model were entered as independent variables and the pandemic related categories (during-pandemic was chosen as the baseline category) were entered as dependent variable. To find the best fit multinomial logistic regression model, a backward elimination approach was used. The independent variables were checked for evidence of separation through visual analysis of crosstabs of the independent and dependent variable. None of the independent variables was shown to perfectly predict the outcome.

Results

General results

Leadership within the flight department uses the annual results from the safety survey as a tool to determine the trend and overall perceptions of employees. Specifically, averages of each safety culture and safety climate leading indicator are reviewed to get a general sense of employee perceptions. The results are then reviewed with an instructor pilot focus group to gather additional feedback and insight. Department leadership then develops an annual plan based on these results to improve safety culture and climate within the department. Reviewing the overall average responses for safety culture and safety climate (Fig. 1 ) highlights the decrease in the flight instructors’ perception of safety culture and climate during the pandemic and the subsequent increase in safety culture as the pandemic subsided in June of 2021. The measurement scale in the chart is associated with the 7-point Likert agreement scale employed in the survey instrument. The results show that the average responses fall between “somewhat agree” (5) and “agree” (6) suggesting that the overall safety culture and climate positive level.

Fig. 1

Safety culture and safety climate responses.

Fig. 2 provides a breakdown of each specific LI specific to safety culture and clearly shows a decline in most of the LIs associated with safety culture starting in 2019 and continuing into 2020. An improvement in the perception of the LIs begins in 2021. The largest overall decline occurred for Prioritization of Safety which dropped to 4.86 in 2020. The decline in 2020 was inspected further through the use of a focus group (unpublished). The focus group attributed this decline to the reopening of the flight training department in May of 2020 (a month into the COVID-19 pandemic). There was a sense among the instructor group that if department leadership truly cared about safety then we would have remained closed until the pandemic subsided. As a result, employee perception of safety culture declined overall according to the focus group. One exception to the general pattern was observed for Multicultural Operations, which experienced its largest decline in 2019 and began to rise in 2020. The focus group attributed the improvement of multicultural operations to the increase in experience among the flight instructor group as a result of reduced flight instructor turnover during the first year of COVID-19. Additionally, the focus group suggested that the introduction of a new program in August of 2019 that was designed to improve aviation English among individuals who use English as a second language had a positive impact on multicultural operations.

Fig. 2

Safety culture leading indicator averages.

A similar pattern was present for the safety climate LIs. The results in Fig. 3 show a decline starting in 2019 and sharp decline in 2020 before rising again in 2021. Again, the focus group attributed the 2020 decline to operating during the pandemic. The data suggests that employees’ perceptions of the safety feedback that they received from leadership was most impacted. The focus group attributed this to individuals who were concerned about operating during a pandemic that were not satisfied with the mitigations that were implemented to minimize the risk of COVID-19. As such, it also is logical that perceived risk, employee empowerment, and willingness to change were reduced for the same reason.

Fig. 3

Safety climate leading indicator averages.

Advanced results

The results discussed above are a critical tool for management to visualize, trend, and communicate the state of the safety culture and climate within their organization. However, the results do not provide any statistical evidence of which leading indicators have been impacted by the COVID-19 pandemic. To provide this higher level of analysis, a Random Forest regression model was built. The results from the confusion matrix (see Table 4 ) shows the training set most reliable in predicting pre-pandemic questions as compared with pandemic and post-pandemic questions.

Table 4

Confusion matrix for pandemic prediction.

		Actual
		Pre-pandemic	Pandemic	Post Pandemic
Predicted	Pre-Pandemic	82	15	30
	Pandemic	5	12	9
	Post-Pandemic	9	15	11

*Accuracy = 0.5585.

A best fit RF model was found which contains 26 questions (see appendix for the list of questions and their importance in the RF model) as the predictors. The model performs well in predicting the outcomes, as can be seen from the accuracy number, and F1 scores (see Table 5 ). Multinomial logistic regression models were then built with the 26 survey questions as the independent variables and pandemic-related categories as the dependent variable. The final model after backwards elimination contains 11 variables. The results of the final multinomial logistic regression model are presented in Table 6 . Overall, when pre-pandemic questions were compared with pandemic, participants responded to 5 questions at a greater level of agreement prior to the pandemic and at a 5 greater level of disagreement. Looking at post-pandemic as compared to pandemic results, there were far fewer statistically significant differences between the baseline of the pandemic compared to the post-pandemic condition. Two questions showed stronger agreement after the pandemic while three questions showed greater disagreement.

Table 5

Random Forest results for pandemic prediction.

	Precision	Recall	F1
Pre-Pandemic	0.646	0.854	0.735
Pandemic	0.461	0.286	0.353
Post-Pandemic	0.314	0.220	0.259

Table 6

Results of multinomial logistical regression analysis.

	Estimate	B(SE)	Z	Pr (>|z|)
Pre-pandemic vs. Pandemic
Intercept	0.579	1.497	0.387	0.698
Managers in this department would stop us from working due to safety concerns even if it meant losing money.	0.424**	0.143	2.955	0.003
My company will stop work due to safety concerns even if they are going to lose money.	0.677***	0.167	4.049	5.131e-05
I am sure management will never compromise safety for profitability.	0.435**	0.157	2.771	0.005
I have a high level of job satisfaction.	−0.644***	0.143	−4.497	6.866e-06
The hiring process in my organization is effective in identifying the right people for jobs.	−0.356**	0.129	−2.744	0.006
It is important to avoid negative comments about the procedures and techniques of other team members	−0.185*	0.090	−2.056	0.039
The workload does not increase when I work with multicultural employees.	−0.350**	0.108	−3.238	0.001
It is inappropriate to share negative thoughts or opinions about my organization with my students.	0.228*	0.096	2.367	0.017
I have fair opportunity to influence the decisions being made by my superiors.	−0.185	0.118	−1.563	0.117
I am fully prepared for every flight training activity.	−0.514*	0.235	−2.183	0.028
My colleagues are enthusiastic in improving safety around the work place.	0.467**	0.166	2.808	0.004
Post-pandemic vs. Pandemic
Intercept	−1.278	1.481	−0.863	0.387
Managers in this department would stop us from working due to safety concerns even if it meant losing money.	0.123	0.130	0.946	0.343
My company will stop work due to safety concerns even if they are going to lose money.	0.327*	0.146	2.235	0.025
I am sure management will never compromise safety for profitability.	0.098	0.145	0.677	0.498
I have a high level of job satisfaction.	−0.376**	0.136	−2.763	0.005
The hiring process in my organization is effective in identifying the right people for jobs.	0.010	0.130	0.081	0.935
It is important to avoid negative comments about the procedures and techniques of other team members	−0.201*	0.087	−2.287	0.022
The workload does not increase when I work with multicultural employees.	−0.073	0.102	−0.712	0.476
It is inappropriate to share negative thoughts or opinions about my organization with my students.	0.100	0.088	1.138	0.254
I have fair opportunity to influence the decisions being made by my superiors.	−0.276*	0.112	−2.463	0.013
I am fully prepared for every flight training activity.	0.158	0.228	0.695	0.486
My colleagues are enthusiastic in improving safety around the work place.	0.323*	0.155	2.077	0.037

*p < .050, **p < .010, ***p < .001.

Three questions were identified as showing the most significantly positive or negative impact as compared to the pandemic category. The question, “I have a high level of job satisfaction”, had the largest change showing participants’ disagreement before (p < .001) and after the pandemic (p < .010) which indicates a higher level of perceived job satisfaction during the pandemic. Similarly, the question, “My colleagues are enthusiastic in improving safety around the work place” showed significantly greater agreement both in pre (p <. 010) and post-pandemic (p <. 050) conditions. These questions are associated with the safety culture leading indicator quality of new hires. Additionally, the question, “My company will stop work due to safety concerns even if they are going to lose money” to which participants indicated a higher level of agreement both before (p < .001) and after (p <. 050) the pandemic. This question is associated with the safety climate leading indicator willingness to change.

Discussion

The purpose of this study was to investigate the impact that the COVID-19 pandemic had on the safety culture and safety climate of the flight training organization at Embry-Riddle Aeronautical University (ERAU). Reviewing the results using descriptive statistics allowed trends to be identified and assumptions to be made about each leading indicator associated with safety culture and safety climate. With the exception of multi-cultural operations, the perception of all leading indicators among flight instructors declined during the height of the pandemic (2020). While the decline is dramatic in comparison to previous years, the overall levels are still indicative of a positive safety culture and climate.

Use and visualization of these descriptive statistics are good for management purposes; however, they fail to provide any statistical evidence. To analyze the results further, this research employed a Random Forest regression model. The results identified that questions associated with the safety culture LI quality of new hires and the safety climate LI willingness to change were statistically significant. Specifically, the question “I have a high level of job satisfaction” had the greatest change and the strongest level of significance pre and post COVID-19. This result makes sense, as flight instruction is a difficult job that many seek employment as temporarily to build required experience to further their aviation career. COVID-19 had a tremendous impact on the aviation industry, and many pilots lost their jobs or were furloughed from employment. During the pandemic, Embry-Riddle did not furlough any pilots. Even while the department was closed for a few weeks, the organization kept the instructor pilots employed by assigning ancillary tasks that used their expertise. The focus group confirmed that the majority of flight instructors were thankful that they had employment during a period where employment opportunities in the aviation industry were severely suppressed.

The second statistically significant question, “My colleagues are enthusiastic in improving safety around the work place” showed a decrease during COVID-19. This decrease may be attributed to the upsurge in mandatory COVID-19 safety protocols. To mitigate the risk of transmitting COVID-19 within the flight department, many new processes, procedures, and rules were instituted which added a significant burden to the training process. Mitigations included wearing of masks, key cleaning, use of gloves, aircraft and simulation disinfecting procedures, to name a few. The focus group supported this by reporting that they were fatigued by all of the additional protocols that were in place due to the pandemic.

Lastly, the question “My company will stop work due to safety concerns even if they are going to lose money” showed lower agreement from the flight instructors during COVID-19. The focus group suggested that this reduction was created by the belief among some flight instructors that the department was taking an unnecessary, increased risk by remaining open during the pandemic. While the department did close for the month of April 2020 when the pandemic started, it was reopened in May. The perception of the focus group was that the reduction in this question was a result in the belief among some individuals that the department had opened too soon, reopening primarily for financial reasons when it should have remained closed to ensure safety.

Conclusion

Measuring safety culture and climate on a regular basis is important as it allows leadership to monitor the leading indicators of safety culture and climate. An analysis of the longitudinal safety culture and climate survey responses collected at Embry-Riddle Aeronautical University has highlighted a change in the overall safety culture and climate levels during the COVID-19 pandemic. Descriptive statistics clearly show a reduction of instructor pilot perception in all safety culture and climate LIs, with the exception of multicultural operations. Further analysis using inferential statistics suggested that during the pandemic employees had a higher level of job satisfaction, were less enthusiastic about improving safety around the work place, and believed that management would prioritize generating revenue over stopping work due to safety concerns. The use of a focus group confirmed these discoveries. These findings are important as they can assist leadership in high-risk industries insight in managing their organizations through future critical safety events such as a pandemic. Specifically, this research suggests that leaders should focus on finding methods to keep staff motivated on improving safety in the workplace. Additionally, organizational leaders should communicate early and often with employees when reopening operations after a critical safety event. Doing so would assist in creating an understanding that the rationale for reopening is not solely financial.

Safety Culture and Safety Climate Survey Variables	Overall
Managers in this department would stop us from working due to safety concerns even if it meant losing money.	100
My company will stop work due to safety concerns even if they are going to lose money.	78.042
I am sure management will never compromise safety for profitability.	65.392
I have a high level of job satisfaction.	41.811
The hiring process in my organization is effective in identifying the right people for jobs.	36.099
It is important to avoid negative comments about the procedures and techniques of other team members.	34.740
The workload does not increase when I work with multicultural employees.	34.093
Safety is the top priority when work is scheduled in this department.	32.127
It is inappropriate to share negative thoughts or opinions about my organization with my students.	30.323
I am always given feedback on accidents, incidents, and near misses that occur in the department.	27.552
Junior employees should not question their senior employee's decisions.	26.312
The individuals that are hired in this company are the best qualified individuals for the job.	26.124
Accident investigations are mainly used to identify faults in the system rather than to identify who is to blame.	25.748
Language differences in multi-cultural employees are not a threat to safety.	24.418
I have fair opportunity to influence the decisions being made by my superiors.	21.945
I am fully prepared for every flight training activity.	21.334
There is good communication in this department about safety issues.	20.892
Employees are not reluctant to report a co-worker's failure.	20.458
My colleagues are enthusiastic in improving safety around the work place.	19.543
People in this department refuse to do work if they feel the task is not safe.	19.332
I can influence health and safety performance in my department.	17.932
There is a lot I can do to further improve health and safety here.	17.678
Good proposals on how to improve safety are not stopped even if they cost too much.	17.671
A consistent message that work pressures must not compromise safety is communicated by the department management to the workforce.	17.583
On the whole, good work is rewarded.	17.234
I am always on time for work.	17.133

Table 7

Radom Forest variable importance.

Construct	Questions/Item
Effective formal learning system	Accident investigations are mainly used to identify faults in the system, rather than to identify who is to blame.
	Appropriate personnel are always given feedback on accidents and near losses that occur in the organization.
	I am encouraged by senior personnel to report any unsafe conditions I may observe.
	This organization is very concerned about learning from accidents, incidents, and near misses.
	My colleagues are willing to report accidents.
	My colleagues are willing to report incidents.
	My colleagues are willing to report perceived hazards.
	Information on recurring causes of accidents, incidents, hazards, and near misses is effectively disseminated to all appropriate personnel.
	This organization is continually improving its mechanisms for learning about safety.
Promotion of safety	Sufficient resources are available for health and safety in my organization.
	People can always get the equipment that is needed to work to my organization's health and safety procedures, instructions or rules.
	Top management is personally involved in safety activities on a routine basis.
	Personnel are actively encouraged to participate in initiatives that can improve safety.
	The company really cares about the health and safety of the people who work here.
	Management places a high priority on health and safety training.
	Corrective action is always taken when management is told about unsafe practices.
	I am satisfied with the recognition, praise and rewards given in this organization for working safely.
	Safety matters are given high priority in company meetings.
Prioritization of safety	Managers in this department would stop us from working due to safety concerns, even if it meant losing money.
	A consistent message that work pressures must not compromise safety is communicated by the department management to the workforce.
	Safety is the top priority when work is scheduled in this department.
Effective communication	I am satisfied with the way I am kept informed about what takes place in my department.
	There is good communication in this department about safety issues.
	I am always informed about the outcome of meetings that address health and safety.
	I am kept informed about the company's safety policies, safety goals and safety performance assessments.
	I am always given feedback on accidents, incidents, and near misses that occur in the department.
	There is good communication between dispatchers, mechanics and flight supervisors.
	Our department is very effective in communicating safety information.
Problem identification	I am encouraged to conduct job safety analyses and report unsafe conditions.
	In my department all jobs (Instructor, Flight Supervisor, Mechanic, Etc.) have safety procedures, instructions or rules.
	In my department all jobs (Instructor, Flight Supervisor, Mechanic, Etc.) have safety procedures, instructions or rules.
	Department rules make it easy for me to identify procedures that are not safe.
Quality of new hires	My colleagues consider safety issues seriously while performing their job duties.
	The hiring process in my organization is effective in identifying the right people for jobs.
	I have a high level of job satisfaction.
	Safety is a primary focus in hiring new personnel in this company.
	The individuals that are hired in this company are the best qualified individuals for the job.
	My colleagues are enthusiastic in improving safety around the work place.
Safety orientation	I fully understand the health and safety procedures and instructions associated with my job.
	The training I had covered all the health and safety risks associated with the work for which I am responsible.
	Initial safety training is provided in this company at the earliest possible opportunity.
	I am clear about what my responsibilities are for health and safety.
	Training has given me a clear understanding of all those aspects of my job that are critical to safety.
Reward safety	Error reporting is rewarded in this organization.
	Personnel are not punished for errors reported through incident reports.
	I get praised for working safely.
	On the whole, good work is rewarded.
	Incentives do not encourage me to break rules.
	I do not receive financial rewards for breaking the rules.
	I receive praise for working safely.
	Employees who work here are recognized for working safely.
Multicultural operations	Language differences in multi-cultural employees are not a threat to safety.
	I enjoy working with multi-cultural employees.
	The workload does not increase when I work with multicultural employees.
	There are no differences in the performance of employees from different cultures.
Perceived risk	I trust most of the people with whom I work with to work safely.
	The standard of safety is very high in my work place.
	Safety is taken seriously in my department; it is not just a cosmetic exercise.
	People in this department refuse to do work if they feel the task is not safe.
Employee empowerment	I can influence health and safety performance in my department.
	I have a good control over the safety outcomes of my job.
	I am very satisfied with my involvement in safety in my department.
	I feel involved when health and safety procedures, instructions or rules are developed or reviewed.
	I have fair opportunity to influence the decisions being made by my superiors.
Personal safety responsibility	Employees perceive safety as a top priority while performing their duties.
	Employees are involved in informing management of important safety issues.
	Employees are accountable for reporting safety violations (actual or potential) in my department.
	I perceive safety as a top priority while performing my duties.
	Whenever I see safety regulations being broken in my department, I point it out.
	I am accountable for reporting safety violations -actual or potential- in my department.
	I am involved in informing management of important safety issues.
Anonymous reporting	An effective anonymous reporting system exists in our organization.
	My organization has a ‘no-blame culture’.
	Mistakes are corrected without punishment and treated as a learning opportunity in my department.
	Employees are not reluctant to report a co-worker's failure.
Effective feedback	Employees are always given feedback on accidents, incidents, near misses or injuries that occur in my department.
	I am very satisfied with regard to follow-up and measures taken after accidents, incidents, injuries and near misses have taken place in this department.
	I am always given feedback on accidents, incidents, near misses, and injuries that occur in my department.
	I am very satisfied with regard to follow-up and measures taken after accidents, incidents, near losses and injuries have taken place.
Respect	Junior employees should not question their senior employee's decisions.
	It is important for me to work safely if I am to keep the respect of others in my team.
	Employees trust senior management in this department.
	It is important to avoid negative comments about the procedures and techniques of other team members.
	Even if I am in a hurry, I try to listen and not interrupt or “talk over” others.
	When appropriate, I take the initiative and time to share my knowledge and experience with others, even if this means that a task takes more time.
	The management in this department listens to us and cares about our concerns.
	Department managers are respected role models in our company.
Integrity	The organization's rules should not be broken - even when I think it is in the company's best interest.
	I am sure management will never compromise safety for profitability.
	I do not ignore safety regulations to get the job done.
	I do not carry out activities which are forbidden.
	I do not take chances to get the job done.
	I do not bend the rules to achieve a target.
	I do not think I get the job done better by ignoring some rules.
Willingness to change	Sometimes it is not a good idea to depart from requirements for production's sake.
	Good proposals on how to improve safety are not stopped even if they cost too much.
	Suggestions to improve health and safety are always acted upon.
	My company will stop work due to safety concerns even if they are going to lose money.
	If I am interrupted while carrying out a procedure, I will always back up a few steps to start again to ensure that I have not made a mistake.
	Supervisors should not reject suggestions for change.
	There is a lot I can do to further improve health and safety here.
	My immediate boss is receptive to ideas on how to improve health and safety.