The Development and Deployment of a Maintenance Operations Safety Survey.

OBJECTIVE: Based on the line operations safety audit (LOSA), two studies were conducted to develop and deploy an equivalent tool for aircraft maintenance: the maintenance operations safety survey (MOSS).
BACKGROUND: Safety in aircraft maintenance is currently measured reactively, based on the number of audit findings, reportable events, incidents, or accidents. Proactive safety tools designed for monitoring routine operations, such as flight data monitoring and LOSA, have been developed predominantly for flight operations.
METHOD: In Study 1, development of MOSS, 12 test peer-to-peer observations were collected to investigate the practicalities of this approach. In Study 2, deployment of MOSS, seven expert observers collected 56 peer-to-peer observations of line maintenance checks at four stations. Narrative data were coded and analyzed according to the threat and error management (TEM) framework.
RESULTS: In Study 1, a line check was identified as a suitable unit of observation. Communication and third-party data management were the key factors in gaining maintainer trust. Study 2 identified that on average, maintainers experienced 7.8 threats (operational complexities) and committed 2.5 errors per observation. The majority of threats and errors were inconsequential. Links between specific threats and errors leading to 36 undesired states were established.
CONCLUSION: This research demonstrates that observations of routine maintenance operations are feasible. TEM-based results highlight successful management strategies that maintainers employ on a day-to-day basis. APPLICATION: MOSS is a novel approach for safety data collection and analysis. It helps practitioners understand the nature of maintenance errors, promote an informed culture, and support safety management systems in the maintenance domain.

Introduction

Collecting information from daily operations has substantial learning potential, allowing the causes of failures, their organizational roots, and successful recoveries to be identified (O’Leary, 2002). Achieving an in-depth understanding of the error chain helps to develop specific management strategies (Hobbs & Williamson, 2003) and to manage human error like any other business risk (Reason & Hobbs, 2003). Predictive tools, such as flight data monitoring and the line operations safety audit (LOSA), provide information on routine operational performance. However, these tools are predominantly focused on flight operations. Other areas of the complex aviation system, like aircraft maintenance, remain reliant on reactive tools, such as investigations of reported events.

Despite research efforts categorizing the types of human error (see Dhillon & Liu, 2006), the maintenance industry still lacks an understanding of maintenance errors in the organizational context (Hobbs & Kanki, 2008). This lack of understanding is mainly because maintenance errors tend to remain hidden (latent), which makes them more challenging to detect (Hobbs, 2004; Reason, 1997). Although the flight crew usually get feedback on their actions and see the operational effect almost immediately, the consequences of maintainers’ actions may not become apparent unless an incident or an accident occurs (International Civil Aviation Organization [ICAO], 2002a; McKenna, 2002). Hobbs (2004) highlights that until such unsafe actions are uncovered, maintainers may continue using the same unsafe practices.

Aircraft maintenance is considered a highly error-prone activity due to the extensive human involvement and system complexities (Hobbs, 2008; Reason & Hobbs, 2003). Maintenance errors reduce safety margins and cause financial losses in terms of schedule disruptions; thus effective error management can lead to considerable cost savings (Hobbs & Kanki, 2008).

The maintenance field suffers from a restricted supply of information about maintainer and system performance. Several researchers highlight that maintenance still shows signs of blame culture (Hobbs, 2008; McDonald, 2006; Patankar & Taylor, 2004), which can discourage open reporting of maintenance errors (Hobbs & Kanki, 2008). The volume of maintenance incident reports is 10 times lower than reports associated with flight operations (Nisula & Ward, cited in Ward, McDonald, Morrison, Gaynor, & Nugent, 2010; Patankar & Driscoll, 2004).

Maintenance audits usually focus on quality control, regulatory compliance, and adherence to procedures but fail to effectively audit performance in progress (McDonald, 2003; Reason & Hobbs, 2003). Given the evaluative conditions, maintainers are likely to alter their behavior. Dekker (2003) highlights that many organizations fail to understand and monitor the practical drift between procedures and practice, even though there is an apparent conflict between the managers’ perception of how work should be carried out (i.e., follow procedures to the letter) and actual performance, where rigorous adherence to procedures causes delays (McDonald, Corrigan, Daly, & Cromie, 2000).

LOSA, defined by 10 operating characteristics (see Table 1), is a tool designed to collect data during routine flight operations (see Klinect, Murray, Merritt, & Helmreich, 2003). Its origin dates to 1994, when the University of Texas was approached to develop an observational tool able to measure how well crew resource management training translates into actual flight operations. It uses peer-to-peer direct observations to capture how flight crew manage, or indeed, mismanage, everyday threats and errors. The observers (experienced and trained flight crew) write a “story” of a flight, which is then coded according to a threat and error management (TEM) framework. TEM takes into account the operational complexities (threats) that are always present in the system as well as active failures (flight crew errors) and responses to threats and errors. It provides structure for the data collection and enables the quantification of results (Klinect, 2005).

Table 1:

LOSA Operating Characteristics

1. Jump seat observations of routine flights	6. Trusted and trained observers
2. Anonymous and nonpunitive data collection	7. Trusted data collection repository
3. Voluntary flight crew participation	8. Data verification roundtables
4. Joint management/pilot union sponsorship	9. Data-derived targets for enhancements
5. Systematic/safety targeted observation instrument	10. Results feedback to line pilots

Note. LOSA = line operations safety audit.

The developers claim that by establishing flight crew trust, LOSA has the ability to collect information about flight crew performance much closer to operational reality than any other safety tool (Klinect et al., 2003). It complements existing safety data sources and has the capability to assess safety margins, rationalize allocation of resources, and provide insights into flight crew shortcuts and workarounds and serves as a baseline for measuring organizational change (Federal Aviation Administration [FAA], 2006; Ma et al., 2011; Thomas, 2004).

LOSA was also successfully applied in air traffic control (Henry et al., 2010; ICAO, 2008), the medical industry (Helmreich, 2000, 2003; Thomas, Sexton, & Helmreich, 2004), and recently, aircraft maintenance. The FAA in collaboration with the Air Transport Association in the United States developed a maintenance line operations safety assessment (M-LOSA), which collects observational data using checklists (Ma et al., 2011) instead of written narratives. Checklists are simple to use, so observers require little training. They are also associated with low cost. However, the context of the observation cannot be fully captured (Stanton et al., 2013), and the predefined codes limit what observers record (Bakeman, 2000).

In this research we aimed to adapt the original LOSA concept (i.e., collecting narrative data) for maintenance operations. Even though analysis and coding of free text are very time-consuming and rather laborious (Stanton et al., 2013), thoroughly written narratives can capture the complexities between the operational context and maintainer performance. They can be retrospectively coded for quality purposes and uncover previously unidentified codes, and a timeline can be produced to establish the links between threats, errors, and undesired states. Importantly, the narrative text helps to identify specific actions that maintainers employ in response to threats and errors.

Method

This section describes the conceptual background and development (Study 1) of the maintenance operations safety survey (MOSS) and the empirical application of the developed tool (Study 2). The research project was conducted in accordance with the code of conduct and ethical guidelines of Cranfield University.

Study 1: MOSS Development

We aimed to test whether the 10 LOSA operating characteristics (see Table 1) can be applied in aviation maintenance. To achieve the aim, we studied the practicalities of conducting peer-to-peer observations and writing meaningful narratives in the maintenance context. We also tested whether the collected information could be quantified using the TEM framework.

MOSS Design Considerations

Maintenance environment versus flight operations

Maintainers work in a hazardous (Hobbs, 2008; Kinnison, 2004; Lind, 2008; Reason & Hobbs, 2003) and variable environment (Reiman, 2011) frequently associated with poor lighting, temperature variations, humidity, noise, and other adverse conditions (Strauch, 2004). In contrast, the flight deck offers the comfort of a uniform and a highly ordered and air-conditioned environment (ICAO, 2002a).

Although flight operations appear to benefit from a culture wherein direct observations are embraced, maintainers are not regularly observed for training and assessment purposes (Hobbs & Kanki, 2008). Therefore, the observer becomes more visible to the observed maintainer and may feel more intrusive than in flight operations. Maintainers move around the aircraft, collect spares from stores, and frequently return to the office to access documentation as part of a check/task. So rather than observing a maintainer discretely, the observer must follow the maintainer, potentially affecting the latter’s behavior. Although it is possible to keep a distance during certain tasks, such as a walk-around, the observer has to get much closer when accessing the cabin and the flight deck.

Unit of observation

The first step in adapting the LOSA methodology was to identify a common unit or event that could be observed as a complete phenomenon from beginning to end (Wilkinson, 2000). In flight operations, this common unit is a flight, with clear starting and finishing points, always following the same flight phases (e.g., taxi out, take-off, climb, cruise). This common breakdown means that the collected information can be compared across flights independently from other attributes, such as the operator, departure or arrival points, or the length of the flight.

In line maintenance, scheduled tasks are usually organized into a check (e.g., transit, daily, weekly). Unscheduled tasks, such as troubleshooting and defect rectification, may be performed outside of a scheduled check. Irrespective of the type, any maintenance task follows three stages as per the manufacturers’ publications: setup, procedure, and close-up (Liston, 2005). This deconstruction is considered equivalent to the flight phases. Therefore, a line check including a set of tasks (both scheduled and unscheduled) and involving all three stages represents the unit of observation.

The setup stage includes any tasks conducted in preparation for the check and usually takes place in the line office before attending to the aircraft. This stage may involve reviewing and printing documentation, preparing tools and spares, and communicating with other members of the team. The procedure stage starts when the maintainer arrives at the allocated aircraft. During this stage, the maintainer carries out the tasks required by the specific checklist and defect rectification according to the technical log or any other tasks communicated from maintenance control. This stage finishes with the maintainer having completed all required tasks and returning back to the line office. During the close-up stage, the maintainer carries out necessary actions to complete the line check, (e.g., complete appropriate company documentation), which usually takes place in the line office.

TEM in the maintenance context

A systematic observation requires a coding scheme so measurement can occur. Categories or codes are an important means of retrieving and organizing segments of text relating to particular themes (Miles & Huberman, 1994), in this case, to MOSS TEM definitions. This section therefore outlines the key TEM definitions and the initial subcategories of the TEM model applicable in the maintenance context.

Before conducting test observations, the TEM code lists for flight operations published by the International Air Transport Association (IATA; 2010), ICAO (2002b), and Klinect (2005) were examined. The researcher and an experienced maintainer jointly reviewed all of these sources. The aim was to select relevant TEM subcategories and propose other appropriate threat and error subcategories that reflect the maintenance operational complexities. Utilizing existing categories as a provisional “start list” prior to fieldwork is recommended in research literature (Miles & Huberman, 1994).

The top-level categories (threats, errors, and undesired states) come from the TEM framework and represent mutually exclusive categories, as defined by the developers (Klinect, 2005).

Threats are events or errors that occur outside the influence of the maintainer, increase the operational complexity of a task, and require the maintainer’s attention and management if safety margins are to be maintained. Following the review process, we identified two threat types for MOSS: environmental and organizational. In addition to weather threats, the environmental category also includes factors related to the air traffic control (ATC), airport authority, and external operational pressure. These four subcategories have similar characteristics. The organization has no influence over their occurrence and frequency. For example, the allocation of aircraft stands for arrivals, departures, and overnight parking is controlled solely by the local airport authority. Therefore, if a maintainer is in the middle of a line check and the airport authority requests the aircraft to be towed, it will have an implication on the task in progress and the overall check. Organizational threats originate within the organization. The subcategories are defined according to the source of the threat, such as flight or cabin crew, dispatch, maintenance control center, aircraft, and so on. For example, inaccurate information about a defect in the technical log written by the flight crew represents a threat for the maintainer addressing the specific defect and falls under the category “organizational threats–flight/cabin crew.”

Errors are defined as maintainer actions or inactions that lead to a deviation from maintainer intentions or organizational expectations and reduce or have the potential to reduce safety margins. Following the review process, we identified three types of errors for MOSS: aircraft handling, communication, and procedural. Aircraft-handling errors include incorrect interactions with aircraft systems and task action errors, such as installation error, removal of wrong parts, or servicing errors. Procedural errors are defined as maintainer deviations from regulations or company standard operating procedures. For example, a maintainer performing a transit check without the relevant documentation commits a procedural error coded under the category “checklists/worksheets.” Written communication errors, such as technical log entries, are also considered procedural and fall under “documentation errors.” Communication errors include inadequate or absent verbal communication between maintainers or between maintainers and external personnel, such as flight crew, cabin crew, or ground agents.

Undesired states are maintainer error–induced states or situations that reduce or have the potential to reduce safety margins. An undesired state is always preceded by an error. According to the TEM model, all threats, errors, and undesired states have to be recoverable/manageable (Klinect, 2005), which means that normal operations can be regained following the threat/error management. This is an important characteristic because undesired states in particular can be misinterpreted for negative outcomes, such as damage or injury.

Participants

The participating organization under study was a large U.K. airline with an aircraft engineering subsidiary. The subsidiary provides base and line maintenance for Boeing and Airbus fleets, engine services, and material logistical support both for the company aircraft and for third-party organizations. Considering these characteristics, it is likely that findings from this research will transfer to other maintenance organizations.

A company line maintainer with European Aviation Safety Agency (EASA) Part 66 license, category B1, and over 20 years of experience acted as an observer and provided expert advice throughout the development phase. The observed 12 participants were company line maintainers with current EASA Part 66 license (category type was not collected at this stage).

Procedure overview

The MOSS development process (Figure 1) involved an iterative process of testing and editing of the observation instrument, protocol, and TEM coding (discussed in the next sections). The continuous collection of observational data and subsequent revisions were discussed between the researcher, the observer, company line maintenance manager, and quality manager until consensus was achieved.

Figure 1.

Maintenance operations safety survey (MOSS) development process. LOSA = line operations safety audit; TEM = threat and error management.

Overall, 12 test observations of line maintenance checks were conducted. We ensured that most of the LOSA operating characteristics (see Table 1) were in place before test data collection. Targets for enhancements were the only exception, as we did not expect the operator to act on the TEM results from the development phase. We had support from the union and management to conduct peer-to-peer observations. The test observations were selected purposely (Yin, 2011) during day and night shifts and collected at two different line maintenance stations. Two observers, the researcher and an experienced maintainer, trained in TEM jointly conducted the test observations. At the end of each observation, the maintainer was asked if he or she felt uncomfortable or threatened during the observation and if this feeling led to a change of behavior. All observations were anonymous, nonpunitive, and conducted with prior consent from observed maintainers. To minimize information loss as a result of memory decay, both observers jointly reconstructed the observed events and wrote the narrative immediately after the completion of each observation (Robson, 2002; Weick, 1968; Wilkinson, 2000).

Observation instrument

MOSS is based on structured observations requiring an observation instrument (Robson, 2002). We consulted the LOSA observation forms published by ICAO (2002b) and by Klinect (2005) and made adaptations to suit this research. Specifically, we changed the demographic information and narrative breakdown by task stage. The initial draft MOSS form included demographic information about the observer, aircraft type, shift type, station, date, start and finish time of a check, and maintenance type (line, hangar, store, engine shop). Also, basic information about the observed maintainer was collected: license type, years of experience, and years in this position. The narrative was broken down into setup, procedure, and close-up as discussed previously.

As the collection of test observations progressed, we decided to remove the date from the demographic section to further protect the observed maintainer from identification. We inserted additional information regarding the task(s) observed, which was split into routine and nonroutine (unscheduled), as line checks often include both types. Also, the narrative procedure stage was separated accordingly. Following the completion of the 12 test observations, the MOSS observation form was finalized and ready for the deployment stage (Study 2). A final version of the deidentified MOSS observation form is presented in the appendix.

Appendix

Maintenance Operations Safety Survey Observation Form

Derivation of TEM codes

The code derivation process used principles from content analysis (Krippendorff, 2004). With the TEM definitions in mind, the narratives were repeatedly reviewed to identify threats, errors, and undesired states. Separate categories were created for responses and outcomes to establish the TEM event chain. For example, in the narrative Example 1 (Table 2), the duty manager approached the maintainer, asking to board. This event is clearly a threat, as the interruption has an impact on the maintainer’s performance and the situation must be managed to ensure safety. According to its origin, it is coded as “organizational threat–dispatch/flight operations–request to commence boarding.” The maintainer responded to the threat appropriately, so it did not lead to any errors, nor did it have any negative impact on the check. Therefore the threat was coded as inconsequential. The uncoded text provides an important context, which, on many occasions, reveals the links between TEM events.

Table 2:

Derivation of TEM Codes From Narratives

Narrative Sample (coded text is in italics)	Codes in Hierarchical Order:TEM Top LevelThreat/Error TypeThreat/Error CategorySpecific Subcategory
Example 1: “The maintainer went to the front of the aircraft and started an anticlockwise walk-around. After completing the walk-around, he then went back to the flight deck and, at the boarding door, was asked by the duty manager if boarding without flight crew could commence [threat] and answered that it could. He then went to the flight deck and configured the aircraft for that procedure [response to threat]. This had no impact on the flow of the check [threat outcome = inconsequential]. The maintainer then left the flight deck and went to the fuel bowser, and the flight crew arrived at the aircraft and saw that the refuel team had dialed up the departure fuel.”	ThreatOrganizationalDispatch/flight operationsRequest to commence boarding
Example 2: “The maintainer took parts, AMM, and tools into the forward cargo bay to complete the job (replace forward cargo trim valve), entering the freight bay via the access hatch in the flight deck. Although he had the AMM, including diagrams, the maintainer proceeded to remove the wrong panel (aircraft right-hand side) [error outcome = linked to additional error], probably because the panel had a decal on it that said ‘forward cabin trim valve’, he didn’t realize it was not the correct panel [lack of response to error]. The maintainer then looked at the AMM diagram again and confirmed his mistake (spoke to himself) that he misread the diagram [error]. The actual panel was on the aircraft left-hand side, which also had a decal but it was in poor condition (this read ‘forward cargo trim valve’).”	ErrorProceduralAMMMisinterpreted or missed information
Example 3: “A mechanic was allocated to aircraft with other two maintainers. No transit checklist printed but good communications between all three maintainers prior to aircraft arrival [error]. The mechanic was specifically asked to do engine oils and cabin. No checklist (standard practice) [lack of response to error] resulted in freight bay inspections being missed so the check was incomplete [error outcome = linked to undesired state].”	ErrorProceduralChecklists/worksheetsCheck performed from memory

Note. TEM = threat and error management; AMM = aircraft maintenance manual. Text in italics is specific to individual codes. Roman text provides the context.

The same process was used in identification of errors, undesired states, and their management. Example 2 (Table 2) presents a scenario of an error leading to an additional error. Both errors were counted in the results. The maintainer misread the aircraft maintenance manual (AMM) diagram, which was coded as “procedural error–AMM–misinterpreted or missed information.” The maintainer did not initially notice this error; therefore it was not managed. From a TEM perspective, the outcome was consequential and coded as linked to additional error because the maintainer proceeded with a removal of a wrong panel (additional error). The additional error was coded as “task action error–removal of wrong parts, panels.” The maintainer realized his mistake when looking again at the AMM diagram (after the additional error was already committed); therefore, he was able to manage the additional error by reinstalling the wrong panel and a removal of the correct panel on the left-hand side. This action means that the additional error was successfully managed and coded as inconsequential. The maintainer then continued with the task.

In Example 3 (Table 2), the allocated maintainer did not use a checklist for the transit check, which was coded as “procedural error–check performed from memory.” The error was not managed, and as a result, some tasks were not completed. From a TEM point of view, the error outcome was consequential, meaning it was linked to an undesired state. It was considered to be an undesired state because of the potential that the aircraft would depart in unairworthy condition (likely reduced safety margins), which was coded as “failure to complete all checklist items before certification.”

The MOSS TEM coding scheme was systematically tested, discussed with the operator, and refined with the continual clarification of individual codes (Bakeman, 2000; Miles & Huberman, 1994).

Key Findings

Following the collection and analysis of results from the 12 test observations, we concluded that all 10 operating characteristics (as seen in Table 1) could be achieved in the maintenance field. Peer-to-peer observations were feasible in the busy line environment, and maintainers felt comfortable during observations. When questioned, all 12 observed maintainers confirmed that they trusted their peer conducting observations and did not change their behavior as a result. Importantly, we did not experience any refusals. Test observations were coded according to TEM and findings discussed with the operator. The results were deemed to reflect the nature of line maintenance operations.

Maintainer trust

MOSS success is dependent on maintainer trust. The key to success was the thorough face-to-face introduction of the project to all participants and union representatives, and frequent communications with the maintainers throughout the development phase.

However, the participants’ main concern was the use of collected information. It highlighted an existing blame culture and concerns over disciplinary action. Therefore, it was emphasized that we did not collect any information that could lead to identification of the observed maintainers. All collected information was managed and securely stored by the researcher. The independence and credibility of the researcher was an important success factor. As a result, it is recommended that future MOSS applications be carried out in cooperation with an independent and trustworthy organization.

Observation procedure

Because the line check is not necessarily a continuous process, remaining unobtrusive is challenging and observers must be adequately trained. For example, while the passengers are boarding or disembarking the aircraft, the maintainer is often unable to access the flight deck and is forced to wait, sometimes for extended periods. Experience has shown that the maintainer is likely to instigate a casual conversation with the observer under these circumstances, even asking for a professional opinion or help. Such instances need to be carefully managed so the observer remains friendly but does not become part of the line check, thus contaminating the observation.

An observer follows one person at a time, usually the maintainer responsible for the certification of the check/task. If multiple maintainers were involved, they would communicate with the certifying maintainer, which would be captured in the written narrative.

Study 2: MOSS Deployment

After completing the development of the MOSS, we tested the tool to determine if it could be used reliably to gather meaningful data about maintenance tasks.

Seven maintainers from the organization described previously volunteered to be trained as observers. All observers held current EASA Part 66 license: category A (2), category B1 (2), and category B2 (3). Five observers were line maintainers, one was a shift leader, and one was a training officer.

The observed participants (N = 56) were company-approved line maintainers with current EASA Part 66 license: category A (12), category B1 (15), and category B2 (29).

Procedure

The observers completed 2 days of classroom-based training acquiring skills to recognize and record TEM events and carry out MOSS observations. They were instructed to approach maintainers an hour before each observation to ask for consent and to carry out observations as discretely as possible. Although MOSS aims to record maintainer errors and undesired states, the observers would stop an observation and intervene if they observed any event that may lead to injury, damage, or a breach in health and safety regulations. In cases where maintainer errors (particularly, omissions of tasks) may affect aircraft airworthiness, the observers would inform the responsible maintainer after the observation so corrective actions could take place. The observers were trained to record only key information during the actual observation, thus limiting the observer impact on maintainer behavior, and to write the full narrative as soon as possible (Klinect, 2005; Robson, 2002; Wilkinson, 2000). This approach allows the observer to fully concentrate on the situation and the environment.

Following the comprehensive training, all observers completed a coding exercise recognizing and recording TEM events in a sample narrative, which aimed to ensure consistent data collection. Cronbach’s alpha coefficient, measuring observer consistency, was calculated in SPSS Version 21. The coefficient was 0.6, which was satisfactory, considering the exploratory nature of the research (Nunnally, 1967, cited in Peterson, 1994).

Prior to data collection, all maintainers were introduced to MOSS and its anonymous, nonpunitive nature. Overall, seven expert observers conducted 56 peer-to-peer observations across four line maintenance stations. Narrative data were based on unobtrusive direct observations of routine line maintenance checks (Kerlinger & Lee, 2000; Stanton et al., 2013). All observations were conducted with prior consent from the maintainers allocated to carry out the selected line checks. Four different locations were chosen purposely to account for different biases: team size, number of daily flights, workload, complexity, and culture (Miles & Huberman, 1994). Stratified purposeful sampling was used to determine the number of observations (see Table 3). The basis for selecting samples was the number of departures in the operator’s flying program for a calendar year. Observations were selected by the researcher, taking into account observer availability and operational requirements. The proportions of day/night departures, aircraft types, and peak/off-peak periods were also considered. The average observation time was 2.5 hr. The breakdown of observations by line check type is presented in Table 4. All checks were observed in their entirety. At the end of each observation, the maintainer was asked if he or she felt uncomfortable or threatened during the observation and if this feeling led to a change of behavior.

Table 3:

Proportions of Observations by Line Station and Shift

	Shift
Station	Day	Night	Total
A	19	4	23 (41%)
B	14	7	21 (38%)
C	7	2	9 (16%)
D	2	1	3 (5%)
Total	42 (75%)	14 (25%)	56 (100%)

Note. Station A based at a U.K. international airport with 50+ maintainers. Station B based at a U.K. international airport with 50+ maintainers. Station C based at a U.K. regional airport with 12 maintainers. Station D based at an overseas international airport with five maintainers.

Table 4:

Observed Line Checks by Type (N = 56)

Line Check Type	Number of Observations
Line A check	1
Supplemental check	1
Weekly	6
Terminal/daily	21
Predeparture	29
Transit	8
ETOPS	8

Note. ETOPS = extended-range twin-engine operations. Categories are not mutually exclusive. For example, an ETOPS check may be performed together with a daily check and count as one observation.

Following each observation, the observer wrote a detailed narrative (“story”) of the check and coded TEM events from the text. The completed observation form was then sent directly to the researcher. The large volume of textual data posed a challenge for the researcher. It became clear that it was not practical to analyze the data manually. To speed up the analytical process, the collected observations were uploaded into qualitative data analysis software, NVivo Version 9. The software is designed to deal with multimedia information and text-rich data sets. It supports effective data management and enables coding and quantification of narrative data, queries, analysis, data visualization, and output of analysis. The researcher then reviewed the narrative and coding. If there were any discrepancies between the text and the codes assigned by the observers, the researcher corrected the coding. As a second round of the quality process, all codes were reviewed again, this time jointly with an experienced maintainer. Before commencing analysis, the coding was finalized at a verification roundtable meeting with a review group comprising six company experts. The code derivation process followed the same principles as previously illustrated in Table 2.

One indication of MOSS success is the degree of acceptance from the maintainers, which is measured by the number of refusals for observations. A high denial rate could indicate a lack of organizational trust in the methodology and compromise the quality of results. MOSS noted only two refusals, which suggests that it achieved a high level of confidence with observers and participants. When questioned, 55 out of the 56 participants declared that they trusted the observer and did not feel uncomfortable during the observation. The observation did not lead to any change in their behavior. One participant reported that he felt uncomfortable but did not feel that it resulted in a change of his behavior.

Results and Discussion

The findings show that observations of routine maintenance operations are feasible and provide further knowledge about the nature and management of maintenance errors.

Threats

Threats were present in 100% of collected observations at an average of 7.8 threats per observation. Although the threat prevalence is similar to flight operations, the average frequency is higher in the maintenance field, indicating the complexity that maintainers routinely deal with. The vast majority of threats, 95%, were organizational and mainly related to ground/ramp and aircraft (see Figure 2).

Figure 2.

Proportion of threats by category (N = 439). Organizational threats are depicted in white; environmental threats are shown in gray.

These findings support previous reports claiming that the flight deck appears to be separated from the organizational context, whereas aircraft maintenance is closely linked with and influenced by the wider system (McDonald, 2006; Pettersen & Aase, 2008). According to the LOSA Archive (4,532 TEM observations collected between 2002 and 2006 from 25 airlines), organizational issues usually influence flight crew before the airborne flight phases; on average, 73% of airline (organizational) threats were recorded during predeparture/taxi out (Merritt & Klinect, 2006). From the take-off phase until arrival to gate, the flight crew operate in a seemingly remote environment, away from the organization.

Maintainers are influenced by various organizational factors and carry out tasks in a busy ramp environment at the same time as ground operations, such as loading and unloading aircraft, refueling, and servicing (see Figure 2), supporting previous findings that the origin of human failure is largely associated with organizational and management factors (Antonovsky, Pollock, & Straker, 2014; Hobbs, 2008; McDonald et al., 2000; Reason & Hobbs, 2003; Saleh, Marais, Bakolas, & Cowlagi, 2010; Tsagkas, Nathanael, & Marmaras, 2014).

Errors

Errors were noted in 86% of observations, at an average of 2.5 errors per observation, but 79% were procedural and mainly associated with noncompliance (see Figure 3). The level of noncompliance corresponds with analysis of engineering reports submitted to the U.K. Confidential Human Factors Incident Reporting Programme, which noted 88% noncompliance with procedures (Skinner, 2010). LOSA experience suggests that increased noncompliance leads to declining safety margins, as the operator usually experiences higher rates in mismanaged threats, errors, and undesired aircraft states (Merritt, 2005).

Figure 3.

Proportion of errors by category (N = 138). Procedural errors are depicted in white; aircraft handling errors are highlighted in gray; communication errors are shown in black. AMM = aircraft maintenance manual; SOP = standard operating procedure.

Some consider work-around practices as a source of system resilience to compensate for inadequate resources (e.g., time, tools, documentation) available to complete certain tasks (Dekker, 2003; McDonald, 2003; Pettersen, McDonald, & Engen, 2010; Reiman, 2011; Tsagkas et al., 2014). Although such practices are externally viewed as routine nonadherence, the ability to adapt, compromise, and improvise is considered internally as a mark of expertise, pride, and professionalism (Dekker, 2003; McDonald, 2003).

When interpreting MOSS results, it is important to distinguish if an error was spontaneous (no observable reason for the error) or linked to a threat or a previous error. For example, if a maintainer failed to use a checklist because the printer did not work, it would be classified as an error linked to a threat, which can be eliminated by implementing a suitable strategy to manage the threat. If there was no observable reason for the failure to use a checklist, it would be a spontaneous error, which requires a different corrective action. Although the errors are essentially the same (noncompliance), understanding the differences in the error chain helps to develop appropriate remedial strategies.

A spontaneous error may be individual or cultural. An individual error is associated with personal physiological factors, such as stress or lack of sleep. A cultural error is the same category of error (e.g., failure to use checklists/worksheets) committed by different individuals and repeatedly observed. Spontaneous individual errors are essentially much harder to predict and manage because the underlying factors are not usually observable. In MOSS, 45% of all errors were spontaneous, frequently associated with the failure to use checklists/worksheets. This finding may point toward cultural error, given that this specific error was observed in 50% of the 56 collected observations. The reason may be that frequent performance of a task, such as the line check, becomes an automatic process for the individual (Dismukes & Berman, 2010). The checklist is viewed as guidance for less experienced maintainers (Pearl & Drury, 1995), or there is an issue with the actual checklist as suggested by Drury and Dempsey (2012). They concluded that checklists can be powerful tools but only when the design is appropriate and validated by the actual users to ensure that the format, sequence of tasks, and content result in reliable performance.

MOSS findings showed that maintainers generally used checklists for weekly and A checks but did not use them for daily, transit, and preflight checks. According to the maintainers, the records department required only weekly/A checklists to be archived. More importantly, the checklists were too long, contained tasks that were not possible to complete during the allocated time, and were not designed according to the usual sequence of tasks that maintainers follow. As a result, maintainers questioned the use of such checklists and perceived them unworkable. As a response to the findings, the operator set up a working group to review the checklist design. This example shows that MOSS can present the operators with information about the scale of a specific issue and an opportunity to develop appropriate management strategies.

Undesired States

Ineffective error management can lead to additional errors or undesired states. Even though the majority of observed errors were inconsequential, 26% of errors resulted in undesired states, which were noted in 34% of collected observations. There were 36 observed undesired states, mainly associated with aircraft areas not being checked for damage, the auxiliary power unit left running unattended, or failure to complete all checklist items before certification. The LOSA Archive suggests that in flight operations, an average of 30% of undesired states originated in threats (Merritt & Klinect, 2006). In contrast, our MOSS study identified 86% of undesired states linked to threats. For example, on 20 occasions, the event chain started with a ground/ramp threat disrupting the walk-around. The maintainer committed an error because the threat was not effectively managed. Then, due to ineffective error management, it resulted in an undesired state.

Inadequate walk-arounds, frequently linked to threats (disruptions and interruptions), and thus a maintainer’s failure to notice missing or incorrectly closed panels, cowling, or door latches, have previously led to serious incidents, damage, and considerable costs (e.g., Air Accident Investigation Branch, 2013). Therefore, MOSS presents an opportunity for significant safety improvements if the appropriate threats are addressed. The data also show that maintenance error is usually a result of organizational issues; hence, addressing such factors, rather than punishing the individual, can lead to progress in safety (Dekker, 2006; Leveson, 2004; Reason & Hobbs, 2003). Managers can target specific threats, help maintainers better recognize and manage those threats, and consequently eliminate associated errors and undesired states.

TEM

Although threat, error, and undesired-state frequencies provide valuable information, the examination of TEM strategies is more diagnostic. Currently, safety in aircraft maintenance is measured negatively, based on the number of audit findings, reportable events, incidents, or accidents. MOSS highlighted not only the weaknesses but also the strengths of the system in terms of successful threat and error recoveries. As depicted in Tables 5 and 6, the majority of threats and errors were inconsequential, 84% and 70%, respectively. Helmreich (2001) argues that such findings show the merit of observing routine operations. Given the absence of negative outcomes, these events would not be picked up by any other safety tool currently available in the maintenance field.

Table 5:

Proportions of Inconsequential and Mismanaged Threats by Category (N = 439)

Threat Category	Threat Count
	Inconsequential	Mismanaged
Ground/ramp	101	36
Aircraft	101	19
Flight/cabin crew	41	4
Engineering personnel	39	3
Organizational operational pressure	36	3
Environmental	18	2
Dispatch/flight operations	16	0
Documentation	11	1
Tools parts/equipment/servicing fluids	4	1
Maintenance control	2	1
Total threats	369 (84%)	70 (16%)

Note. A mismanaged threat is linked to a maintainer error. It means that the threat was either not managed appropriately or not managed at all.

Table 6:

Proportions of Inconsequential and Mismanaged Errors by Category (N = 138)

Error Category	Error Count
	Inconsequential	Mismanaged
Procedural errors
Checklists/worksheets	47	28
Documentation	17	3
AMM	6	1
Nonadherence to company SOPs	2	5
Aircraft handling errors
Task action	10	2
Aircraft systems	2	0
Other	3	1
Communication errors	9	2
Total errors	96 (70%)	42 (30%)

Note. AMM = aircraft maintenance manual; SOP = standard operating procedure; documentation = cabin and technical logs. A mismanaged error is linked to an additional error or linked to an undesired state. It means that the error was either not managed appropriately or not managed at all.

For effective TEM, the threats and errors must first be detected. According to Klinect (2010), flight crew typically notice the undesired state first rather than the error or errors contributing to that state. The detection is linked to the immediacy of feedback following flight crew actions because the error is usually less noticeable than the undesired aircraft state. Because undesired aircraft states are associated with aircraft deviations or incorrect configurations, the flight crew are likely to receive cues in a form of warnings, instrument readings, or aircraft behavior.

The level of errors that were undetected or unattended was significantly higher in maintenance than in flight operations. In our study, 93% of errors were undetected or unattended, compared with an average of 45% identified in the LOSA Archive (Merritt & Klinect, 2006). This difference can be attributed to the lack of feedback to maintainer actions, a distinct feature of maintenance work. In contrast to the typical multicrew environment, maintainers frequently work alone and must remain vigilant to catch and correct their own errors. Maintainers do not benefit from the immediate cues received by the flight crew except when maintenance tasks require subsequent functional checks or duplicate inspections. Even in the latter case, functional checks or inspections may be conducted with significant time delay (e.g., by a different shift). Therefore, it may not be immediately apparent if safety margins are reduced as a result of maintainer actions.

Limitations and Future Research

The practical experience and observer feedback identified improvements for future MOSS applications, particularly in terms of observer training. It is recommended that MOSS observer training be extended from 2 days to 3 days. The training should also include a team debrief after each observer has conducted his or her first test observation. Furthermore, the experience highlighted that future MOSS applications would benefit from testing observer consistency following completion of his or her first test observation in the operational environment. This test observation provides an opportunity to practice all of the skills accumulated during the observer training and to clear any possible misconceptions. As a result, observer knowledge and application of TEM is likely to improve; hence the internal consistency score would be expected to increase. It was not possible to extend observer training during the first MOSS application due to time constraints and the observers’ limited availability. Also, the internal consistency among observers measured by Cronbach’s alpha coefficient was found at the minimum acceptable level. Although this level was adequate, considering the exploratory nature of this research, subsequent MOSS applications should increase the number of items measuring observer consistency. However, it is possible that the TEM knowledge retention decays over time; hence a test-retest reliability measure would be perhaps more appropriate, especially in cases when a prolonged period is required for data collection.

Moreover, the strict anonymity associated with the data collection and the naturalistic characteristic of observations made it impossible to determine if an individual maintainer was observed more than once. Although it is believed that this limitation did not have negative impact on the collected information, it should be considered when interpreting the results.

This study involved a single organization, so efforts are under way to apply MOSS in other maintenance organizations to further refine the tool and enable benchmarking. Importantly, the application of MOSS at other organizations and the development of an archive of MOSS data for benchmarking is likely to facilitate future research and enhance the knowledge of maintenance errors and strategies for effective TEM in the maintenance field.

Another direction for future research would be to link MOSS results with findings from other safety data sources. This linking of results would be particularly desirable given that MOSS is limited to observable events only. For example, the relationship between spontaneous errors and nonobservable causal factors, such as stress or fatigue, could be explored. Incident reports collected by the IATA global aviation safety data-sharing program, STEADES, could be a suitable data set especially because IATA uses the TEM framework for analysis. Research could also establish whether MOSS findings correlate with specific incident outcomes.

Implications for Practitioners

MOSS is a new diagnostic tool that provides management with information about routine operations. Unlike other safety tools, it has the potential to identify organizational strengths by capturing the successful TEM strategies that maintainers routinely employ. The findings support an effective safety management system, training, and systemic change. Conducting MOSS also leads to improvements in communication between line staff and managers. The organization under study noted an increased reporting rate and improved trust in management as the maintainers received timely feedback and were subsequently involved in working groups to address key findings.

The MOSS implementation cycle is shown in Figure 4. A key challenge of this tool is the vast volume of textual data, which requires proficient knowledge of the TEM framework and resources for data analysis. Considering the prevalence of blame culture within maintenance organizations, it is advisable to follow the LOSA Collaborative model. For a successful implementation of MOSS, a credible third party—ideally, an independent research organization—should work closely with the maintenance operators to act as a trusted data repository and to deliver the MOSS data analyses. Therefore, confidentiality and trust can be maintained while industrywide learning through benchmarking is facilitated.

Figure 4.

Maintenance operations safety survey (MOSS) implementation cycle.

SMS = safety management system.

We attempted to develop a practical safety tool that would encourage organizational learning and benchmarking across the aircraft maintenance industry and that would ultimately lead to safety improvements. Although it is possible for an organization to conduct its own MOSS, the current research program has been designed around the implementation of MOSS by a third-party provider, in a similar fashion to the approach taken by the LOSA Collaborative. Therefore, continuous collaboration between the research community and the industry is needed to further enhance the MOSS tool in order to produce similar safety contributions to those realized by LOSA in flight operations.

Conclusion

This study provides evidence that LOSA principles, in their original form (i.e., based on narrative data collection), are applicable to the maintenance environment. MOSS provides a novel approach for safety data collection and analysis in aircraft maintenance. Unlike incident and accident data, MOSS collects comprehensive and standardized information that can be communicated effectively to managers to support organizational change. We have developed a practical predictive safety tool that will assist practitioners in promoting an informed culture and support an effective safety management system in the maintenance domain.

MOSS records maintainers’ responses and the outcomes for each threat, error, and undesired state; therefore the full chain of events can be examined. Understanding the link between specific types of threats and errors helps managers to develop targeted strategies to reduce or eliminate specific types of maintenance error.

The theoretical gain from this research lies in the application of the TEM concept in the maintenance context. This study generated new knowledge to enhance the understanding of maintenance errors in the detection of and recovery from threats, errors, and undesired states. TEM-driven data collection and analysis highlights the proportion of detected and successfully managed threats and errors, and allows the identification of management strategies that maintainers employ on a day-to-day basis.

Key Points

Line operations safety audit principles and the threat and error management framework are found to be applicable to the line maintenance environment.

The maintenance operations safety survey (MOSS) is a novel safety data tool that collects standardized information from routine maintenance operations and promotes an informed culture and supports effective safety management systems in the maintenance domain.

Besides systemic weaknesses, MOSS results also illustrate the strengths of the maintenance organization. Examination of inconsequential threats and errors helps managers to understand the successful strategies that maintainers employ on a day-to-day basis.