How Safety Performance Indicators Help Airlines Improve Their Safety Management System

In November 2019, revision 1 to ICAO Annex 19 became extant. Consequently, your regulator is likely to start taking a deeper interest in your Safety Management System. Specifically, the need to demonstrate continuous improvement in safety and to show how you are achieving your stated safety objectives will come under increased scrutiny. This is because ICAO will start to audit NAAs to determine their compliance with the new Annex 19 which, in turn, will result in them taking a keener interest in your SMS. How you go about demonstrating that you are meeting these emerging requirements starts with your organisation’s safety policy and its objectives.

Progress made in the achievement of stated objectives should be monitored and reported on a regular basis. This is accomplished through the identification of safety performance indicators (SPIs), which are used to monitor and measure safety performance. Through the identification of SPIs, information obtained will allow senior management to be aware of the current situation and support decision- making, including determining whether actions are required to further mitigate safety risks to ensure the organisation achieves its safety goals.

Safety Performance Management Process

The generic safety performance management process is shown below in figure 1 (source: ICAO Document 9859)

Safety data is entered into the process via the Safety Data Capture and Processing System (SDCPS). By using appropriate analysis techniques, the captured safety data can be turned into useful information that can then be used to monitor the organisation’s safety performance. This is achieved by the establishment of appropriate safety objectives and their associated safety performance indicators (SPI). Additionally, captured and analysed safety information will allow management to identify any actions required to maintain a safe operation.

Unless the safety performance management process is communicated to the entire workforce, it will fail to meet its objectives. The safety promotion aspect cannot be overstated, it vital that all personnel across all departments are informed and engaged in this important process.

Safety performance management helps the organisation to ask and to answer the four most important questions regarding safety of their operation:

• What are the organisation’s top safety risks?
  
  
• What does the organisation want to achieve in terms of safety and what are the top safety risks that need to be addressed? The organisation’s safety objectives.
  
  
• How will the organisation know if it is making progress toward its safety objectives? Through SPIs, SPTs and, if practicable, safety triggers.
  
  
• What safety data and safety information are needed to make informed safety decisions? Including the allocation of the organisation’s limited resources to manage and mitigate identified risks. SDCPS and safety analysis.

Safety Objectives

Safety objectives are the starting point for safety performance management systems. They are brief, high-level statements of the desired safety outcomes to be accomplished. Safety objectives provide direction to the organisation’s activities and should therefore be consistent with the safety policy that sets out the organisation’s high-level safety commitment. They are also useful to communicate safety priorities to the workforce. Establishing safety objectives provides strategic direction for the safety performance management process and provides a sound basis for safety related decision-making.

Safety objectives may be:

• process-oriented: stated in terms of safe behaviours expected from operational personnel or the performance of actions implemented by the organisation to manage safety risk; (e.g. a year on year increase in voluntary reporting) or:
  
  
• outcome-oriented: encompass actions and trends regarding containment of accidents or operational losses; (e.g. a reduction in the number of unstable approaches per 1000 sectors from 30 to 10 over the next 2 years).

As a general guide, safety objectives should be based on SMART criteria: Specific, Measurable, Achievable, Relevant and Time-Bound; e.g. Increase the number of voluntary, safety reports by 100% over the next 4 years. Additionally, a mix of process and outcome objectives should be determined. When senior management define an organisation’s safety objectives, they need to consider the size and complexity of their operation and also the past performance in terms of safety. For example, management may already be aware of a number of areas of concern such as an increased number of runway incursions. The safety objective could be either process orientated such as the establishment of local runway safety teams at each airport in the next year or it could be outcome orientated e.g. a reduction in runway incursions from 10 every thousand movements to 2, over the next 12 months.

Safety Performance Indicators

In order to monitor progress towards achieving the stated safety objectives, an organisation must develop a means to measure performance towards these goals. This is usually in the form of Safety Performance Indicators (SPI). SPIs provide management with an overview of how well the organisation is progressing in achieving its stated safety objectives. There are a number of different types of SPI that could be considered (all are equally valid):

• Quantitative and Qualitative SPIs Quantitative SPIs record a specific measure of an event type or issue. For example, it could be the number of unstable approaches each month. Usually, an organisation will wish to normalise these figures to maintain a “level playing field” month on month. For example, it might be that only half the number of flights are conducted in January compared to July and so the January total of unstable approaches is much less than July. To normalise this data, a more realistic measure might be the number of unstable approaches every 1000 flights. Use of quantitative SPIs is preferred from a safety management standpoint because they return measurable, numeric data that can be presented graphically at each monthly safety meeting.
  
  Qualitative SPIs are equally valid if chosen correctly but they are more difficult to measure and report on. An example of a qualitative SPI could be the improvements in safety culture; this might be monitored by the use of staff surveys.

• Leading and Lagging SPIs Lagging SPIs record things that have already happened. For example, the number of Altitude Busts per month. These SPIs record the more negative outcomes that the safety management team are trying to avoid. Lagging SPIs are very good indicators of long-term trends and they can be tailored to specific areas of concern. For example, the number of runway incursions at airfield XXX.
  
  Lagging SPIs are also very useful for measuring the effectiveness of safety risk mitigations / controls e.g. number of runway incursions at airfield XXX following the installation of illuminated stop bars.
  
  Leading SPIs tend to focus on processes or initiatives that have been implemented to improve safety in the organisation. An example of a leading SPI could be “the number of pilots who have completed Altitude Bust training in the simulator”.

It is important that, when setting SPIs, they are aligned with the organisation’s safety objectives. Furthermore, a mix of leading and lagging SPIs should be used. Primarily, quantitative SPIs will be used because they are easier to monitor and ultimately present during management meetings. Of most importance is that SPIs provide decision makers with relevant safety information such that any decisions made are based on reliable and defensible data. Furthermore, well thought out SPIs will allow for the development of meaningful Safety Performance Targets that will show progress towards the achievement of stated safety objectives.

When defining their SPIs, organisations should try to introduce a mix of leading and lagging indicators. However, the two types of SPI should not (if possible) be defined in isolation. There should be some form of link between the leading SPI and the lagging SPI. This is best explained by reference to the diagram at figure 2 (source: ICAO Document 9859).

Figure 2 shows the relationship between leading and lagging indicators. Of note is the fact that the main SPI (Number of runway excursions/1000 landings) is supported by two other SPIs. There is a leading SPI that accounts for the number of pilots who have received formal training in the issue. Additionally, the organisation has identified that a primary cause of runway excursions is unstable approaches. With this in mind, they have set this as an SPI which will serve as an indicator towards the primary event. These “indicator” issues are often referred to as “pre-cursor events”.

It is important that, when setting SPIs, they are aligned with the organisation’s safety objectives. Furthermore, a mix of leading and lagging SPIs should be used. Primarily, quantitative SPIs will be used because they are easier to monitor and ultimately present during management meetings. Of most importance is that SPIs provide decision makers with relevant safety information such that any decisions made are based on reliable and defensible data. Furthermore, well thought out SPIs will allow for the development of meaningful Safety Performance Targets that will show progress towards the achievement of stated safety objectives.

The contents of each SPI should include:

• A description of what the SPI measures;
• The purpose of the SPI (what it is intended to manage and who it is intended to inform);
• The units of measurement and any requirements for its calculation;
• Who is responsible for collecting, validating, monitoring, reporting and acting on the SPI (these may be staff from different parts of the organisation);
• Where or how the data should be collected; and
• The frequency of reporting, collecting, monitoring and analysis of the SPI data.

To support your SPI dashboard it is essential that the right safety management system software is in place. A well-designed and supported SMS software solution will help safety managers obtain information and data that can support their respective safety management priorities, and thereby provide the wider organisation with a set of SPIs that can be measured and monitored regularly. It will result in a safer airline organisation.

Learn how the right safety management software can help support your airline’s safety investigation, risk and audit management and compliance solutions. Download our free guide to find out how to select one that benefits your airline.

Note: Information in this article is taken from ICAO Document 9859.