Assessing Reliability and Validity of an Instrument for Measuring Resilience Safety Culture in Sociotechnical Systems.

BACKGROUND: Safety culture, acting as the oil necessary in an efficient safety management system, has its own weaknesses in the current conceptualization and utilization in practice. As a new approach, resilience safety culture (RSC) has been proposed to reduce these weaknesses and improve safety culture; however, it requires a valid and reliable instrument to be measured. This study aimed at evaluating the reliability and validity of such an instrument in measuring the RSC in sociotechnical systems.
METHODS: The researchers designed an instrument based on resilience engineering principles and safety culture as the first instrument to measure the RSC. The RSC instrument was distributed among 354 staff members from 12 units of an anonymous petrochemical plant through hand delivery. Content validity, confirmatory, and exploratory factor analysis were used to examine the construct validity, and Cronbach alpha and test-retest were employed to examine the reliability of the instrument.
RESULTS: The results of the content validity index and content validity ratio were calculated as 0.97 and 0.83, respectively. The explanatory factor analysis showed 14 factors with 68.29% total variance and 0.88 Kaiser-Meyer-Olkin index. The results were also confirmed with confirmatory factor analysis (relative Chi-square = 2453.49, Root Mean Square Error of Approximation = 0.04). The reliability of the RSC instrument, as measured by internal consistency, was found to be satisfactory (Cronbach α = 0.94). The results of test-retest reliability was r = 0.85, p < 0.001.
CONCLUSION: The results of the study suggest that the measure shows acceptable validity and reliability.

Introduction

In an efficient safety management system, safety culture may be seen as necessary as the oil [1]. The concept of safety culture was born in the aftermath of the devastating Chernobyl disaster (1986) and has today become an established concept in all major safety-critical domains [2]. Therefore, one of the earliest definitions of the safety culture was also presented by the International Atomic Energy Agency [3] in 1991 as “Safety culture denotes the assembly of characteristics and attitudes in organizations and individuals which establishes that, as an overriding priority, nuclear plant safety issues receive the attention warranted by their significance” [3]. Since then, a plethora of definitions for safety culture has been provided (see more in [4], [5], [6], [7]).

The literature review shows that to date many studies have been conducted in this area (for example, [2], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14], [15], [16]), but seldom do they explicitly tell us about the weaknesses of employing safety culture in practice. Some of these weaknesses include the following.

Focusing only on one aspect/dimension of safety culture, such as the Just culture. As Guldenmund [10] has noted, the safety culture is a multidimensional construct. Accordingly, the exact dimensions consisting of safety culture have still been debated and no agreement has been reached in this area [10]. Therefore, the focus of most safety culture and safety climate studies has been on identifying the basic dimensions of the concept, but the identified dimensions are not always commensurate [2]. As Díaz-Cabrera et al. [17] pointed out, most conceptualizations of the dimensions of safety culture include tacit or abstract dimensions (such as commitment, learning) as well as dimensions that relate more concretely to working practices (such as rules, staffing). In other words, considering the various arenas of safety, safety culture is in fact becoming so context-dependent that the common features among these different forms of safety introduce an emerging problem. Therefore, competitions among the various forms of safety and attempts to justify the demands in one arena can lead to sacrifices in another [2]. Hence, sometimes an area of safety culture can be more or less forgotten or even be considered. For example, “just culture” is very important for some industries, such as aviation and nuclear power plant. In this context, justness is very important for good reporting and learning from incidents. In this light, the weakest link in just culture, reporting and learning the culture chain is often just culture. Therefore, the risk exists that system focuses so much on getting just culture and forgets all about the reporting and learning culture. Accordingly, some safety areas can be more or less forgotten [1].

Another example to be presented here is when workplace hazards are thrown into the shade of hazards for major accidents or vice versa. The ESSO gas plant explosion, Longford, Australia (1998) is a good example for such an issue. In this accident, a systematic lack of attention to major hazards was found as a contributing cause [1]. As Hopkins and Andrew expressed [18], over-reliance on simple accident statistics in the management of high hazard installations had led to such accidents. In a gas plant, there may be the risk that attention to workplace hazards is insufficient, which itself may be an indirect risk for major accidents [1].

Ignoring the dynamic interactions among people, technology, and administration. Sociotechnical systems comprise many interactions between people, between people and procedures, and between people and hardware/software technical systems [19]. However, safety is a system property that emerges from a conglomerate of components, subsystems, software, organizations, human behavior, and their interactions. By contrast, safety is something that a system does, instead of something that a system has [20]. Therefore, safety cannot be understood or managed by understanding or managing its constituent parts in isolation. Consequently, if safety culture is regarded as a component of a sociotechnical system, the overall system will have emergent properties that cannot be deduced from the study of safety culture alone. Hence, safety culture studies “often seem to lack a proper conceptualization of the relationship between culture, technology, and structure in high-risk systems” [21].

Trying to link the concept of safety culture to various negative consequences such as injuries and adverse events. Good safety culture is often viewed as a capability or as an absence of injuries or accidents. However, as Weick [22] pointed out, safety is a “dynamic nonevent.” Therefore, the absence of accidents in a system in the past cannot guarantee the safety of the system in the future. In other words, the past successes cannot be an indicator of good safety culture in the future [22]. In this line, review of the literature shows that many of the major reported accidents have evolved from a poor safety culture. For example, there can be cited accidents, such as the Chernobyl disaster [23], Columbia accident [24], Clapham Junction railway disaster [25], and Herald of Free Enterprise disaster [26]. In this context, research on safety culture has faced a challenge in dealing with such accidents. The reason is the studies should combine the human related concepts (e.g., assumptions, values) with more concrete objects including the structure and the quality of the artifacts such as the technology. Therefore, it is sometimes preferred to appeal to an “excuse” for not dealing with some more basic safety problems involving technological designs. This problem is not new and has long been discussed in traditional account of safety and in old accident models where individuals should have sometimes been adapted to technology rather than the technology to the individual [2].

Not considering the resilience aspects. Two major features of sociotechnical systems are complexity and uncertainty. These systems can conceivably be assumed to exhibit emergent properties. Thus, it is very difficult to identify and manage emergent risks by proactively using traditional methods used in the safety culture. This is due to emerging risks as non-linear combinations of performance variability [27]. Therefore, conventional risk assessment methods which are based on linear philosophy fail to focus on the dynamic interplay among factors [2]. Furthermore, the traditional risk analysis approaches applied to the sociotechnical systems cannot cover the structures of complex systems, the interactions between systems and human behaviors, the interrelated factors among sub-systems, and the safety culture of specific societies [28].

Another point is that many accident models in traditional conceptualizations of safety are also based on this approach. According to this philosophy, larger the cause, larger the effect, i.e., serious the effects (e.g., accidents) will be. The accidents are believed to be caused by serious or big causes (e.g., major negligence or ineptitude). On the contrary, in the non-linear philosophy that is based on resilience engineering (RE), there is not necessarily a proportion between inputs and outputs. In other words, small causes can produce arbitrarily large effects and vice versa [29].

The failure to make full use of systemic theories in the sense of recognizing some of the fundamental principles common to system theories and so on [2].

The philosophy of systems theory was developed in the 1930s. The reason for the emergence of this theory was the need for a response to the limitations of the classic analysis techniques and their possibilities to cope with more and more complex systems [30]. This theory focuses on the system as a whole and its interactions, not on some components of the system in isolation. In this line, Leveson [31] said: “It assumes that some properties of a system can only be treated adequately in their entirety, taken into account all facets relating the social to the technical aspects. These system properties derive from the relationships between the parts of systems: how the parts interact and fit together.”

Therefore, as Skyttner [32] pointed out, one important concept in these systems is the emergence which “results from the interaction of independent parts when they stop being independent and start to influence each other.” This means that it is the relationship between the elements of a system and not the nature of the elements themselves that determines its properties and behavior. However, as Hollnagel (2004) has noted, the traditional safety approach fails to focus on the dynamic interplay among factors [29]. Accordingly, safety culture research and practice has so far missed the opportunity to be integrated with systemic perspectives (for more see [2]).

Therefore, the need for a new approach to identify loop holes in the safety culture and mitigate them is evident. For this purpose, and for using principles of RE, a new concept termed as “resilience safety culture” (RSC) has been proposed in order to cover the weaknesses of working with the current safety culture. RSC is not different from safety culture in theory; the difference is all about how it is used in practice. However, the definition of RSC is “an organizational culture that fosters safe practices for improved safety in an ultra-safe organization striving for cost-effective safety management by stressing the RE, organizational learning, and continuous improvements” [1]. This new concept is developing in the world, and there is lack of literature in this area. Therefore, the need for measuring the RSC with a convenient and efficient instrument is obvious (for more, see [33]). Accordingly, the purpose of this study was to design a valid and reliable instrument based on the RE principles and safety culture.

Materials and methods

Instrument design

The research team carried out a substantial literature review to find an instrument for measuring the RSC in different countries, but they could not find any questionnaire in this area. Therefore, the team decided to develop a self-designed questionnaire. Stages of developing the instrument included the following. (1) Determining the goal. The objective of the instrument design was to analyze the understanding of operators and managers about resilience safety culture and identify strong and weak points of the system included in the study. There are many instruments to measure safety culture, but this is not true for RSC. On this basis, and considering the existing gap, the need for designing such an instrument was evident. (2) Doing a preview interview. In order to start, the team conducted a comprehensive literature review of the material on the topic under research. The main scales of the RSC were extracted from four research articles and one book [1], [34], [35], [36], [37]. Furthermore, the researchers carried out a structured preview interview with some of the members of the population under study in order to modify the gathered information and adapt it to the plant under study. (3) Development. As the initial instrument, a questionnaire was constructed according to Table 1 and was based on a 5-point Likert-type scale (from “disagree strongly” to “agree strongly”). The reasons for selecting these scales included: (1) their pivotal role in building the safety culture and RE; (2) their focus on dynamic interactions among people, technology, and administration; and (3) their efforts to avoid linking the concept of safety culture only to negative consequences.

Table 1

The scales of resilience safety culture and the number of items

Scales	No. of items	Scales	No. of items
Just culture	8	Management commitment	7
Management of change	6	Awareness	4
Learning culture	8	Safety management system	3
Risk assessment/management	5	Accident investigation	3
Preparedness	4	Involvement of staff	3
Flexibility	6	Competency	4
Reporting culture	5

At first glance, it might perhaps be thought that these scales are similar to those commonly used in measuring the safety culture. As was aforementioned, however, the difference between safety culture and RSC is all about how it is used in practice. As an example, the traditional fields of practice such as risk analysis and probabilistic safety assessment, which have been used in the safety culture, are firmly rooted in oversimplified models of accidents as cause-effect chains. In the RSC, however, the risk analysis is based on the combinations of normal variabilities. Moreover, the safety culture tries to understand why things get out of hand and why control is lost; but the RSC has a different view in this regard. It tries to understand how a system can actively ensure that things do not get out of hand and how control is not lost [20]. Therefore, the research team tried to design the questions in all scales in such a way that they could measure people's attitudes about RE. By contrast, the questions were aimed at revealing why normal performances succeed (the main trait of the RE) rather than looking for the underlaying failures and malfunctions. Table 2 illustrates the scales (factors) applied in this study. Attempts were made to use the RE approach in the definition of the factors.

Table 2

Description of the variables of the research

Variable	Description
Just culture	An atmosphere of trust that workers are encouraged to report essential safety concerns and issues [38]
Management of change	A best practice used to ensure that safety risks are controlled when a plant makes changes in their facilities, documentation, personnel, or operations [33]
Learning culture	How much does the plant respond to problems with denial versus modification? [34]
Risk assessment/management	A systematic process of evaluating the potential risks that may be involved in a process or activity [33]
Preparedness	Actively anticipates various threats and prepares for them [36]
Flexibility	Ability to restructure in response to various changes and variabilities [33]
Reporting culture	Cultivating an atmosphere where employees have confidence to report safety-related issues without fear of blame [33]
Management commitment	Recognizing the human performance concerns and tiring to address them, devoting to safety above or to the same extent as the other goals in the plant [27], [34]
Awareness	Aware of risks and systems' boundaries and know how close it is to their edge [34], [39] as well as aware of the safeguards and procedures efficiency.
Safety management system	Systematic approach to proactively managing safety, including the necessary organizational structures, accountabilities, policies and procedures [33]
Accident investigation	Process of detailed and systematically collecting and analyzing information relating to an accident [33]
Involvement of staff	How much employees are contributed in decision making and planning for safety [33]
Competency	What an employee is capable of doing [40]

After determining the main scales, a questionnaire comprising 66 items was designed. Its reliability and validity were evaluated as outlined in the following subsections.

Determination of validity

The validity of the instrument was evaluated using the content and construct validity methods. The content validity was used to determine the relevance of the items in the instrument. To determine the content validity, the following steps outlined by Lynn [41] were undertaken. In the first step, the experts had to confirm that the items were valid. Then, a different group of experts had to assert the validity of the entire instrument. In each step, each item's content validity index (I-CVI) was calculated using a 10-point ordinal rating scale, where 1 indicated an irrelevant item and 10 indicated an extremely relevant item. The I-CVI shows the proportion of agreement among the experts as to each item. The criteria for selecting experts included knowledge and experience related to the area as well as relevant training. By contrast, experts with more than 5 years of experience in the unit and those who were familiar with the concepts of safety and RE were selected for this study. In addition, there are also standards for selecting these experts which emphasize the necessity of their relevant training, experience, and qualifications [42]. The scale's content validity index (S-CVI) was also calculated for the entire instrument. It was the proportion of all the judged content validity in all items. The minimum acceptable score for the I-CVI in this study was 0.70, but an I-CVI with a score of ≥ 0.80 was generally considered to be an excellent content validity.

This study also calculated the content validity ratio (CVR) which was outlined by Lawshe [43] for each item.

The construct validity of the instrument was also determined because it is a major component in testing all the outcomes of the instruments. The construct validity means that the scales in the questionnaire behave as expected.

Moreover, the exploratory factor analysis (EFA) was utilized to extract the factors through a principal component analysis and varimax rotation with Kaiser Normalization. In this process, the total variance was explained and scree plots were used to extract the factors. The factors that had eigenvalues > 1 were retained in the study. Furthermore, to extract items in each factor, the rotated component matrix was used. Items were put into each factor according to their values and the nature of each factor. That is, items which had values > 0.40 in each factor [44] were loaded at the same factor. Then, with regard to the nature of items in every factor, the factor was named by one of the variables in Table 2. Furthermore, in order to specify the number of factors required in the data and to decide which measured variable (item) is related to which latent variable (factor), confirmatory factor analysis (CFA) was employed. The purpose of using CFA was to confirm or reject the measurement theory. In other words, it was used to test whether measures of a construct are consistent with a researcher's understanding of the nature of that construct (or factor).

Before extracting factors, sampling adequacy and sphericity were tested by Kaiser-Meyer-Olkin and Bartlett's test, respectively [45]. Kline [46] believed that in EFA, the sample size is 10–20 participants per variable, but the minimum sample size of 200 is reasonable. Given that the sample size is determined based on factors and not variables in CFA, the minimum sample size is 20 participants per factors [47], [48].

Determination of reliability

Kline [49] stated that a minimum sample size of 300 was sufficient for estimating alpha, as did Nunnally and Bernstein [50]. Segall [51] believes that a sample size of 300 is “small”. Charter [52] suggested a minimum sample size of 400 for a sufficiently precise estimate of the population coefficient alpha. However, determination of reliability was undertaken using two methods: (1) Cronbach α, the internal consistency of the responses for each scale and the entire instrument were determined by the Cronbach α coefficient; (2) test–retest, the reliability was also tested using a test–retest design with a 2-week interval between the measurements [33].

Study population and data collection

The study population consisted of 1,274 employees working in a petrochemical plant in the southwest of Iran. Due to the specific nature of the population, the sample size was estimated as 295 (with 95% confidence coefficient) in accordance with Cochran formula. To prevent any statistical loss, the total sample size with a 20% increase was determined to be 354. The data were processed by SPSS IBM 23 (SPSS Inc., Armonk, Il, USA) and LISREL 8.8 (LISREL version 8.80. Lincolnwood, IL: Scientific Software International, Inc. USA).

Results

The context of the study was a petrochemical plant in southwest Iran. The plant was established in 1998 with more than 1,200 employees. Therefore, the 59-item questionnaire was distributed among 354 randomly selected employees of 12 units in the mentioned plant. In total, 312 valid questionnaires were gathered. Therefore, the sample size was adequate for the EFA and CFA criteria. The employees worked in three 8-hour shifts. Table 3 lists the demographic data of the respondents.

Table 3

Demographic data for 312 employees who completed the resilience safety culture instrument

Demographic characteristics	Value
Mean age (y)	37.70 ± 12.44
Sex
Male	307 (98.40)
Female	5 (1.60)
Education status (y)
>12	255 (81.30)
<12	57 (18.70)
Occupational status
Manager	61 (19.60)
Operator	251 (80.41)
Work experience (y)
>10	105 (33.62)
<10	207 (66.40)
Employment status (y)
Employee	185 (59.31)
Contractor	127 (40.70)

Data are presented as n (%) or mean ± SD.

The validity results

Content validity. The results of CVI and CVR are presented in Table 4. As reported in many other studies [53], [54], [55], the CVI values > 0.78 were considered appropriate. In addition, according to Lawshe [43], the minimum acceptable CVR is 0.78. However, if a question has a value < 0.78 and the mean of judgments is > 1.50, it is accepted.

Table 4

The content validity index and content validity ratio in the primary instrument

No.	Items	I-CVI	CVR	Judgments mean	Accept/reject
1	If a change associated with the job occurs in the procedure, I am timely and well aware of them.	1	0.78	1.89	A
2	To do any change in instructions and to modify the current procedures in my workplace, there are certain processes of which all are aware.	1	1	2	A
3	All employees follow the procedures related to changing the instructions and work processes in my workplace.	1	0.78	1.89	A
4	Any change in the processes and procedures is well documented (both electronically and on paper) in my workplace.	1	1	2	A
5	Temporary changes, like permanent changes, are well publicized.	0.89	0.56	1.78	A
6	A risk assessment is performed after any change in the processes.	0.78	0.78	1.89	A
7	Guidelines and procedures associated with work processes will be reviewed in appropriate intervals.	0.67	0.33	1.44	R
8	I believe the management of change was performed in my workplace as well.	0.44	0.11	1.33	R
9	Discussing the risks in the organization or workplace where I work is very important.	0.89	0.56	1.78	A
10	Feedbacks obtained from accident investigations were used to implement corrective measures in the organization where I work.	1	1	2	A
11	Risk assessment was performed through systemic methods in my workplace.	1	1	2	A
12	All staff are aware of the risks they are facing.	1	0.78	1.89	A
13	Accidents that happen in my workplace are analyzed.	1	1	2	A
14	Detailed analysis of defects and failures and putting them at the disposal of other staff brings about learning from them and their prevention.	1	0.78	1.89	A
15	Risk assessments related to my work in the specified period will be revised.	0.89	0.78	1.89	A
16	Control and corrective measures are performed in relation to the risks and hazards of my work.	1	1	2	A
17	If I have concerns about safety and my work, I can talk with my supervisor.	1	0.78	1.89	A
18	My superior director tolerates to hear any news, especially bad news.	0.78	0.56	1.78	A
19	Staff can affect the decisions of their superiors in my workplace.	0.67	-0.11	1.44	R
20	If the staff has concerns regarding the safety of their work, they can make it stop.	0.89	0.78	1.89	A
21	In my workplace, staff from different departments and levels takes part in safety/resilience meetings.	0.89	0.56	1.78	A
22	The spirit of teamwork “is completely tangible” in my workplace.	1	0.78	1.89	A
23	Top management pays attention to safety at all times, and not just after the accident.	1	1	2	A
24	The employees always do their jobs safely, even when they are not monitored.	1	1	2	A
25	Suggestions made by the employees on safety/resilience are examined and they are welcome.	1	1	2	A
26	If necessary, the staff can easily meet senior executives.	0.67	0.11	1.44	R
27	Decisions about the work and safety issues in which I am working are participatory.	0.89	0.56	1.78	A
28	My supervisor talks about safety and related matters with me.	1	1	2	A
29	I can report near misses without concern and fear.	1	1	2	A
30	Incidents that occur in the company have always been reported.	1	1	2	A
31	The staff who report a problem of safety/resilience or offer a mechanism to improve safety are encouraged.	1	1	2	A
32	In my workplace, the staff share their experiences with their colleagues.	1	0.78	1.89	A
33	My superior manager appreciates my work.	1	0.78	1.89	A
34	Safety is always the first priority and invaluable for top management in my workplace.	1	1	2	A
35	When the safety/resilience is in danger, the operator can stop production and the staff should be encouraged to do so.	1	1	2	A
36	My supervisor provides enough resources and facilities to maintain and upgrade safety/resilience.	1	1	2	A
37	My supervisor is always aware of organizational, human and technological risks of my workplace.	1	0.56	1.78	A
38	I receive constructive feedbacks regarding work safety/resilience from my superior manager.	1	0.78	1.89	A
39	The appreciation that the superior manager shows for my work is completely visible.	0.67	–0.11	1.44	R
40	Managers and supervisors are committed to what other people are advised to adhere on safety issues.	1	0.78	1.89	A
41	I have to do my job even with little information about it.	0.89	0.56	1.67	A
42	The organization where I work has the facilities and procedures for responding to unpredictable and unexpected changes and disruptions.	1	1	2	A
43	In the emergency condition and rapid response, it is easy to follow the procedures.	1	0.78	1.89	A
44	In case I face a problem so that I have to sacrifice safety/resilience or production, I prefer to select safety/resilience for keeping the system.	1	1	2	A
45	In a major emergency condition for important decision making (such as stopping production), permission from my supervisor is necessary.	1	0.78	1.89	A
46	The organization where I work has the ability to adapt to stressful situations caused by internal and external pressures.	1	0.56	1.78	A
47	If the system collapses, it has the ability to recover and return quickly to a stable state.	0.89	0.78	1.89	A
48	There are good safe ways to do my job that I'm aware of.	1	1	2	A
49	RSC and rules governing my organization are appropriate and they can be used in the future.	1	0.78	1.89	A
50	In the organization where I work, the issues related to safety and resilience are regularly discussed at top management level, not the case, and only after a catastrophic accident.	1	0.78	1.89	A
51	In the organization where I work, holding group meetings in the areas of safety/resilience is the perfect solution to expect potential problems in the future.	1	1	2	A
52	After accidents, the first priority of management is to find and correct faulty barriers, not to search for scapegoats to blame.	0.89	0.56	1.78	A
53	Accountabilities and responsibilities of individuals for safety/resilience are clearly defined and understood.	1	0.78	1.89	A
54	In the organization where I work, safety/resilience performance is a part of the staff evaluation system.	1	0.78	1.89	A
55	Administrative paperwork influences my workplace's safety/resilience.	0.56	–0.56	1.22	R
56	Information about the flaws and shortcomings of the system must be reported to the competent people of the organization.	1	0.56	1.78	A
57	In the system in which I work, training courses are proper for promoting safe behaviors.	1	0.78	1.89	A
58	In the organization where I work, retraining courses are conducted regularly and at the appointed times.	1	1	2	A
59	In the system in which I work, I have received the necessary training to do the job properly and safely.	1	1	2	A
60	In the organization where I work, the results of investigating incidents and accidents are used to develop training programs.	1	1	2	A
61	Experienced staff also need health and safety training.	1	0.78	1.89	A
62	Procedures and manuals have been updated and are suitable for performing tasks safely.	1	0.78	1.89	A
63	My colleagues and I have skills needed to perform our functions at work.	1	0.56	1.78	A
64	When faced with challenges at work, I have full confidence in dealing with them.	1	0.78	1.89	A
65	My colleagues and I know procedures and safety issues related to our work.	0.56	0.11	1.44	R
66	I can devise a way to learn a lesson from the current defects in my workplace.	1	0.78	1.89	A
	Mean	0.97	0.83

CVR, content validity ratio; I-CVI, item's content validity index.

Construct validity. To extract underlying factors, a principal component analysis with varimax rotation was performed. Prior to this, the sampling adequacy and sphericity were tested by Kaiser-Meyer-Olkin and Bartlett's test, respectively. The results showed a strong significance for Bartlett's test (χ2 = 9951 and p < 0.001). Additionally, Kaiser-Meyer-Olkin was measured to be 0.88.

The EFA with varimax rotation extracted 14 factors with eigenvalues > 1. Therefore, these 14 factors should be retained in the study (Fig. 1 and Table 5). These factors accounted for 68.29% of the total variance. It seems necessary to mention that only Question 33 in the instrument is loaded in a separate factor (Factor 14). However, considering the relatively high load of Factor 6 and the concept of the mentioned question, which is close to Factor 6, it was moved to this factor. The extracted factors, therefore, decreased in number from 14 to 13. As mentioned earlier, the extraction of items associated with each factors was based on their values; for e.g., Question 1 to Question 6 (in Table 6, as a part of the rotated component matrix) were categorized into Factor 2 due to the values well above 0.40. Other items were all categorized in the same manner.

Fig. 1

Scree plot of eigenvalue against the number of factors is used for the extraction of factors.

Table 5

Total variance explained∗,†

Component	Initial eigenvalues			Extraction sums of squared loadings			Rotation sums of squared loadings
	Total	% of variance	Cumulative %	Total	% of variance	Cumulative %	Total	% of variance	Cumulative %
1	14.14	23.97	23.97	14.14	23.95	23.95	3.85	6.58	6.58
2	3.24	5.49	29.46	3.24	5.44	29.49	3.82	6.48	13.06
3	2.95	5.01	34.47	2.95	5.01	34.49	3.71	6.28	19.35
4	2.78	4.72	39.20	2.78	4.75	39.24	3.67	6.22	25.57
5	2.47	4.19	43.39	2.47	4.11	43.35	3.32	5.63	31.20
6	2.20	3.74	47.13	2.20	3.73	47.17	3.29	5.59	36.79
7	2.05	3.48	50.62	2.05	3.43	50.60	3.11	5.27	42.07
8	1.94	3.30	53.92	1.94	3.30	53.90	2.85	4.84	46.92
9	1.68	2.85	56.77	1.68	2.81	56.71	2.61	4.43	51.35
10	1.54	2.62	59.39	1.54	2.60	59.31	2.33	3.96	55.32
11	1.53	2.60	61.99	1.53	2.61	61.92	2.31	3.92	59.24
12	1.46	2.48	64.47	1.46	2.44	64.46	2.16	3.66	62.91
13	1.14	1.94	66.41	1.14	1.93	66.49	1.88	3.19	66.11
14	1.10	1.87	68.29	1.10	1.84	68.23	1.28	2.18	68.29
15	0.95	1.62	69.91
16	0.90	1.54	71.45

Extraction method: principal component analysis.

Fourteen factors were extracted.

Table 6

Some of the rotated component matrix∗,†,‡

	Component
	1	2	3	4	5	6	7	8	9	10	11	12	13	14
Q1	0.07	0.75	0.04	0.18	–0.01	0.06	0.07	0.04	–0.01	0.07	–0.01	0.08	0.09	0.10
Q2	0.14	0.73	0.08	0.05	0.07	0.03	0.03	0.08	0.09	0.02	0.07	–0.04	–0.05	–0.03
Q3	0.11	0.65	0.15	0.04	0.04	0.10	0.10	0.06	0.03	0.02	–0.06	0.08	–0.05	–0.11
Q4	0.04	0.78	0.06	0.12	0.07	0.08	0.04	0.04	0.06	0.05	0.04	0.16	0.09	0.02
Q5	0.14	0.65	0.01	0.05	0.07	0.06	0.02	0.18	0.12	0.07	0.13	0.03	0.04	–0.03
Q6	0.04	0.80	0.01	0.21	–0.01	0.04	0.01	0.01	0.01	0.09	0.10	0.06	0.03	0.13
Q9	0.12	0.23	0.08	0.80	0.14	–0.04	0.12	0.13	0.02	0.07	0.14	0.07	0.01	0.07
Q10	0.24	0.10	0.14	0.28	0.14	0.11	0.15	0.17	0.03	0.09	0.73	0.10	0.05	0.15
Q11	0.12	0.14	0.11	0.79	0.14	0.06	0.03	0.11	0.04	0.09	0.15	0.05	0.02	0.02
Q12	0.06	0.07	0.11	0.71	0.17	0.13	0.21	0.06	0.05	0.01	0.04	0.15	0.04	0.01
Q13	0.14	0.06	0.05	0.30	0.08	0.17	0.05	0.08	0.08	0.11	0.78	0.01	0.05	0.01
Q14	0.12	0.13	0.07	0.23	0.08	0.08	0.07	0.06	0.06	0.04	0.81	–0.01	0.11	–0.03
Q15	0.11	0.22	0.01	0.77	0.09	0.09	0.03	0.10	0.08	0.12	0.19	0.03	0.03	–0.07
Q16	0.13	0.10	0.15	0.67	0.07	0.09	0.18	0.12	0.07	0.09	0.21	0.03	0.09	0.07
Q17	0.62	0.02	0.01	0.14	0.06	0.10	0.29	0.20	0.07	–0.01	0.09	0.11	0.04	–0.17
Q18	0.68	0.10	0.21	0.02	0.08	0.01	0.22	0.15	0.10	–0.01	0.05	0.04	0.02	0.12
Q20	0.68	0.06	0.01	0.09	0.11	0.01	0.11	0.06	0.11	0.01	0.02	–0.01	–0.01	0.22
Q21	0.70	0.09	0.14	0.07	0.07	0.13	0.07	0.08	0.07	–0.03	0.17	0.07	0.08	–0.21
Q22	0.60	0.22	0.02	0.05	0.08	0.02	–0.06	–0.02	–0.10	0.03	–0.01	0.11	0.17	0.22
Q23	0.69	0.08	–0.01	0.20	–0.05	0.12	0.16	0.07	0.03	0.16	0.02	0.12	–0.01	–0.12
Q24	0.71	0.12	0.08	0.05	0.07	0.08	0.02	0.11	–0.01	0.04	0.13	0.12	0.01	0.04
Q25	0.15	0.10	0.16	0.18	0.18	0.01	0.78	0.12	0.03	0.04	0.09	0.01	0.12	0.02
Q27	0.19	0.19	0.14	0.14	0.12	0.07	0.05	0.16	0.06	0.09	0.04	0.80	0.02	–0.02
Q28	0.27	0.07	0.13	0.12	0.11	0.02	0.08	0.11	0.14	0.11	0.01	0.69	0.08	0.27
Q29	0.11	0.06	0.16	0.10	0.10	0.12	0.78	0.05	0.04	–0.01	–0.02	0.01	0.03	0.09
Q30	0.16	0.05	0.01	0.13	0.14	0.06	0.76	0.14	0.10	0.06	0.17	0.03	0.09	–0.05

The bold values show the extracted components.

Extraction method: principal component analysis.

Rotation method: Varimax with Kaiser Normalization*.

Rotation converged in eight iterations.

The researchers also performed a CFA using factors found in EFA because item overloading in EFA could not represent a problem in CFA. The results of CFA (Table 10) thoroughly confirmed the results of EFA. As seen in the table, the factor loading for almost all items was well above 0.60 [44], [56]. In other words, the relationship between the latent variables (factors) and the observational variables (questions) had a full agreement. The model (Fig. 2) shows good fit χ2 (2453.49, n = 354), p < 0.001, i.e., the model is consistent with the observed data. The root mean square error of approximation (RMSEA) equaled 0.04. Considering the RMSEA values which are categorized into four groups, close fit (0.00–0.05), fair fit (0.05–0.08), mediocre fit (0.08–0.10), and poor fit (over 0.10) [57], the model shows a close fit. In other words, the proposed model shows more significant relationships between the variables. The results in Table 10 show that the correlation (t-value > 1.96) between items (observational variables) and factors (latent variables) is highly significant.

Table 10

Results of confirmatory factor analysis

Factors	Items	Factor loading	T values
Just culture	Q17	0.70	13.32
	Q18	0.71	13.63
	Q20	0.61	11.14
	Q21	0.72	13.70
	Q22	0.51	9.01
	Q23	0.67	12.60
	Q24	0.67	12.66
Management of change	Q1	0.76	14.93
	Q2	0.66	12.38
	Q3	0.58	10.65
	Q4	0.77	15.32
	Q5	0.65	12.24
	Q6	0.80	16.04
Learning culture	Q57	0.74	14.60
	Q58	0.75	14.80
	Q59	0.60	11.10
	Q60	0.77	15.27
	Q61	0.68	12.91
	Q62	0.73	14.37
Risk assessment/management	Q9	0.67	19.17
	Q11	0.67	17.33
	Q12	0.51	14.13
	Q15	0.72	16.59
	Q16	0.61	14.20
Preparedness	Q49	0.60	19.47
	Q50	0.77	18.89
	Q51	0.68	16.21
	Q53	0.73	17.43
Flexibility	Q33	0.62	10.97
	Q42	0.73	12.70
	Q44	0.76	11.76
	Q45	0.75	11.71
	Q46	0.99	17.33
	Q47	0.64	11.84
Reporting culture	Q25	0.78	18.89
	Q29	0.71	14.75
	Q30	0.77	16.58
	Q31	0.72	15.59
	Q34	0.75	12.90
	Q35	0.63	9.69
Management commitment	Q36	0.53	9.20
	Q37	0.64	10.44
	Q38	0.66	12.81
	Q40	0.49	8.89
Awareness	Q41	0.85	12.70
	Q43	0.88	11.75
	Q48	0.79	16.21
Safety management system	Q52	0.73	15.37
	Q54	0.80	15.87
	Q56	0.80	15.78
Accident investigation	Q10	0.85	20.07
	Q13	0.75	17.30
	Q14	0.73	15.81
Involvement of staff	Q27	0.86	17.42
	Q28	0.76	14.01
	Q32	0.71	15.24
Competency	Q63	0.82	12.95
	Q64	0.50	8.15
	Q66	0.62	10.14

Q: question.

Fig. 2

Correlation between observational and latent variables in the resilience safety culture. RMSEA, root mean square error of approximation.

The researchers hypothesized that the functional scales in the RSC instrument (just culture, management of change, learning culture, risk assessment/management, preparedness, flexibility, reporting culture, management commitment, awareness, safety management system, accident investigation, involvement of staff, and competency) would be significantly correlated with each other. For example, a safety management system with poor preparedness would have poor work performance against accidents. Table 7 lists the inter-correlations (Pearson's r) among the summated scales for the 13 scales derived from EFA.

Table 7

Inter-correlations (Pearson's r) among the summated scales

	1	2	3	4	5	6	7	8	9	10	11	12
Management of change
Risk assessment/management	0.40∗
Just culture	0.33∗	0.37∗
Reporting culture	0.22∗	0.39∗	0.38∗
Involvement of staff	0.35∗	0.35∗	0.42∗	0.28∗
Accident investigation	0.27∗	0.54∗	0.37∗	0.31∗	0.25∗
Flexibility	0.24∗	0.28∗	0.29∗	0.30∗	0.31∗	0.34∗
Awareness	0.25∗	0.26∗	0.22∗	0.20∗	0.30∗	0.25∗	0.30∗
Preparedness	0.26∗	0.43∗	0.30∗	0.37∗	0.35∗	0.34∗	0.27∗	0.32∗
Learning culture	0.27∗	0.35∗	0.30∗	0.34∗	0.38∗	0.30∗	0.36∗	0.28∗	0.43∗
Safety management system	0.30∗	0.37∗	0.23∗	0.30∗	0.32∗	0.32∗	0.23∗	0.36∗	0.59∗	0.43∗
Competency	0.17∗	0.23∗	0.23∗	0.29∗	0.27∗	0.27∗	0.20∗	0.28∗	0.24∗	0.31∗	0.25∗
Management commitment	0.32∗	0.42∗	0.40∗	0.37∗	0.37∗	0.37∗	0.40∗	0.22∗	0.35∗	0.37∗	0.30∗	0.21∗

Correlation is significant at the 0.01 level.

The results of reliability

Before estimating reliability, it is essential to determine the sample size for reliability analysis. According to Kline [49] and Nunnally and Bernstein [50] who suggested a minimum sample size of 300, the sample size of 354 is sufficient for the reliability analysis in the current study. However, the reliability of the instrument was evaluated by calculating Cronbach α and test–retest [33]. The results of calculating (α) are presented in Table 8. In addition, the instrument was tested with a 2-week interval between measurements. For this purpose, 56 participants randomly filled out the questionnaires a second time 2 weeks later. The conditions of retest and test were the same. In this context, scores were calculated, and the correlation between scores for both measurement times was determined using Spearman correlation coefficient. The result of the Spearman correlation was 0.85 at p < 0.001.

Table 8

Coefficient alpha for 13 factors of the resilience safety culture

Scales	Items	α	Mean	SD	Scales	Items	α	Mean	SD
Just culture	7	0.84	3.35	1.04	Management commitment	6	0.77	2.897	1.07
Management of change	6	0.85	2.96	1.03	Awareness	3	0.89	3.046	0.99
Learning culture	6	0.86	3.05	1.03	Safety management system	3	0.84	2.856	1.24
Risk assessment/management	5	0.89	2.79	1.03	Accident investigation	3	0.88	2.893	0.86
Preparedness	4	0.91	2.93	1.09	Involvement of staff	3	0.83	3.285	0.97
Flexibility	6	0.83	3.20	1.29	Competency	3	0.67	3.409	0.72
Reporting culture	4	0.88	3.08	0.88			Total alpha = 0.943

Discussion

The philosophy of safety culture is one of the cornerstones in complex systems in order to prevent catastrophic accidents/incidents. Safety culture can be seen as the framework and the prerequisite for the implementation of other aspects of system safety in such organizations [15]. The effect of poor safety culture can be tracked in most severe accidents. However, there are weaknesses in the current conceptualization of the safety culture and its utilization as a concept in practice. To overcome these weaknesses, a new concept termed as the RSC has recently been proposed. However, this concept is in its infancy and needs a reliable and valid instrument for measuring factors that affect it. As mentioned earlier, there is no theoretical difference between the safety culture and RSC. The difference is on how it is used in practice. For instance, the RSC believes that risk assessment and accident are two sides of the same coin; therefore, both are constrained in equal measures by the underlying models and theories. As a result, developments in risk assessment have matched developments in accident analysis [20]. On the contrary, according to the safety culture view, the nature of risk assessment and accident are two different things. Moreover, according to the RSC view, hazards or risks emerge from combinations of normal variability in the sociotechnical systems. Hence, looking for efficiency–thoroughness trade-off [29] and sacrificing decisions are necessary to deal with them [58]. By contrast, the safety culture believes that hazard risks are due to component failure or degradation of components (organizational, human, technical); therefore, it is necessary to look for failure probability or drift, degradation, and weakness. Another issue that can be raised here is the learning culture. Whereas the RSC emphasizes not only on learning from incidents but also from normal works [59], the safety culture focuses on learning from incidents, accidents, and abnormities. The RSC strives to develop resilience and use feed forward control in order to keep processes within safe boundaries, whereas the safety culture is focused on feedback for improvement on system safety. The safety culture often focuses on one aspect and other aspects are forgotten, as was already mentioned in the Introduction.

However, to the best of our knowledge, this is the first instrument developed to allow stockholders to systematically collect information from employees about the RSC of their system. In testing the validity of the RSC instrument, two types of validity were examined: content validity and construct validity. Content validity determined whether there was content representativeness or relevance of the items of the instrument. For this purpose, the validity of the instrument was tested by CVI and the CVR. The CVR and the CVI values resulted in removing seven items because these items had values < 0.78 (Table 4). To determine whether the scales in the instrument behaved as expected, the construct validity was determined using inter-correlations (Pearson's r) among the summated scales (Table 7).

Several pilot studies were conducted to derive the main factors considered central to the measurement of outcomes for the RSC. Thus, the researchers performed an EFA with varimax rotation in order to reduce the factors. However, the EFA could extract 14 factors (see Table 5 and the Introduction). Accordingly, the first factor that comes out from the data (considering all the items) is the just culture, and last factor is the competency. Table 9 shows the other emerging factors.

Table 9

The extracted factors of the exploratory factor analysis and the related items

Factor	Items	Factor	Items
1# Just culture	17,18,20,21,22,23,24	8# Management commitment	34,35,36,37,38,40
2# Management of change	1,2,3,4,5,6	9# Awareness	41,43,48
3# Learning culture	57,58,59,60,61,62	10# Safety management system	52,54,56
4# Risk assessment/management	9,11,12,15,16	11# Accident investigation	10,13,14
5# Preparedness	49,50,51,53	12# Involvement of staff	27,28,32
6# Flexibility	33,42,44,45,46,47	13# Competency	63,64,66
7# Reporting culture	25,29,30,31

The obtained results from CFA, including factor loading (> 0.60) and t-value (> 1.96, p < 0.05) completely confirmed that putting items into related factors was perfectly done and the correlation between them was highly significant. The results of evaluating the observational and latent variables (Fig. 2) also show that the model is close fit (RMSEA < 0.05).

The results presented in Table 7 indicated that there was a high correlation among the 13 factors which measured the functional outcomes. Therefore, as expected, the factors correlated significantly with each other. As an example, the results of Table 7 show that there was a high correlation in the instrument between the safety management system and the preparedness factor as well as between the accident/incident investigation and the risk assessment/management factor. This finding is entirely consistent with the results of field observations. Consequently, this correlation is an evidence of good construct validity for the instrument.

In reliability testing, the RSC instrument was found to have high test–retest reliability and internal consistency. For test–retest reliability, Spearman's correlation exceeded 0.8, indicating that items were highly reliable in repeated testing and were stable over time. The value of alpha indicated that all factors had a value > 0.70 (Table 6). This value of alpha verified that all the items in the factor were highly correlated with each other, providing assurance that random errors were minimized in the factor.

This study, however, has several limitations. First, the instrument does not contain all the proposed dimensions of resilience and safety culture because integration of all dimensions could result in developing an instrument with many items, making it very difficult to be used. Second, the validity and reliability of the instrument were tested in only one particular plant [33]. Therefore, it strongly advises that the generalization of the findings should be done with caution. More studies are required to refine items and generalize the findings to other industries or organizations. In addition, another limitation of this study is that no bias analysis was performed between the participants.

Conclusion

Safety culture can be considered as the necessary oil for any efficient safety management system; however, there are weaknesses in the current conceptualization of the safety culture and its utilization as a concept in practice. Accordingly, the authors combined the principles of the RE and safety culture, and an efficient instrument was proposed for measuring and evaluating the RSC. Therefore, the new instrument can be used as a useful tool to assess the RSC in sociotechnical systems, such as petrochemical industries, chemical industries, and oil refineries. The results of the study suggest that the measure shows acceptable validity and reliability although further development is necessary to refine items and ensure that each construct is adequately represented.

Conflicts of interest

All authors have no conflicts of interest to declare.