53RD ANNUAL CONFERENCE, Gran Canaria, Spain, 5-9 May 2014WP No. 162ATCO PerformancePresented by PLC |
Summary
Seek to lead or make a meaningful contribution to the debate on what are the elements of Air Traffic Controller (ATCO) performance.
Introduction
1.1. Performance of the Air Traffic Service (ATS) system is seen as a key component in the productivity and sustainability of the whole aviation system. Controller performance is an important component of the ATS system, especially in regard to system capacity.
1.2. Debates on Just Culture and Safety Management Systems are hampered by the lack of understanding or ambiguity about the role of the individual and personal responsibility for performance.
1.3. IFATCA already has policy regarding performance indicators.
1.4. IFATCA Technical & Professional Manual 2013 Page 4219:
| WC 8.1.8 PERFORMANCE INDICATORS
IFATCA Policy is: Performance Indicators as published and used by Air Navigation Service Providers (ANSP) must not be linked in any way to the pay and/or working conditions of individual ATCO’s. IFATCA urges Member Associations (MA) to be involved in the creation of and application of an Air Traffic Management (ATM) Performance Measurement System. |
1.5. IFATCA also has relevant policy on ATM SAFETY MONITORING TOOL (ASMT) concerning performance.
1.6. IFATCA Technical & Professional Manual 2013 Page 42123:
| WC 8.2.12 ATM SAFETY MONITORING TOOL
IFATCA Policy is: ASMT should not be used as a performance monitor for individual controllers. |
1.7. Performance definition:
1.7.1. The Oxford Dictionary offers these performance definitions, other than artistic as:
“the action or process of performing a task or function.”
However, for this paper, performance should be defined as:
“a task or operation seen in terms of how successfully it is performed.”
Discussion
2.1. Overall Performance Metrics
2.1.1. Decades of modernization of government controlled activities, namely commercialisation, led to new expectations on former public services. Objectives of this new trend have been defined to offer the same service, cheaper, faster and more efficiently thus introducing a quest for performance metrics and targets.
2.1.2. Performance measurement is not an end in itself. Why should performance be measured?
2.1.3. Because managers may find such measures helpful in achieving eight specific managerial purposes. As part of their overall management strategy, managers can use performance measures to evaluate, control, budget, motivate, promote, celebrate, learn, and improve.
2.1.4. Unfortunately, no single performance measure is appropriate for all eight purposes. Consequently, managers should not seek the one magic performance measure. Instead, they need to think seriously about the managerial purposes to which performance measurement might contribute and how they might deploy these measures.
2.1.5. Only then can they select measures with the characteristics necessary to help achieve each purpose. Without at least a tentative theory about how performance measures can be employed to foster improvement (which is the core purpose behind the other seven), managers will be unable to decide what should be measured.
2.1.6. Between meeting expected performance targets of a unit and production pressure, ATCOs often make trade-offs on a local level. One example could be accepting additional traffic increasing the workload above the sector’s established capacities. This can lead to frustration, disillusionment or incomprehension of the expectations at the operational level.
2.1.7. Experiences within IFATCA members has shown the potential for dangerous practices, particularly where performance objectives have been misused by ANSPs such as bonus payment for those “super-controllers” willing to work above the target sector flows, supervisors having to work controller shifts when there are staff shortages and avoid being sanctioned, withholding safety incident reports, all to meet the safety and operational targets etc.
2.1.8. Performance measurement is still a new trend in ATM. The impact of it on the overall performance of a unit or a region can be very difficult to quantify. At the ATCO level it is perceived either as a nuisance or a hindrance.
2.1.9. Can ATCO performance be measured and if so, how and for what purposes?
2.2. ICAO Performance Metrics
2.2.1. ICAO has chosen a good approach by outlining what has to be provided by way of performance in the future, so that sustainable growth of aviation can be achieved. This forward-looking way of having ATM performance measurement established could be a way to positively improve the overall, global ATM service.
2.2.2. High-level expectations are listed in ICAO Doc 9854. Out of these expectations are 11 Key Performance Indicators (KPI) that are to be used in performance management;
- Access and Equity
- Capacity
- Cost Effectiveness
- Efficiency
- Environment
- Flexibility
- Global Interoperability
- Participation by the ATM community
- Predictability
- Safety
- Security
2.2.3. The ATM community expects that each of these KPI will be met by pursuing more specific performance objectives to produce relevant and timely enhancements (operational improvements) to a given region’s ATM system.
2.2.4. Care has to be taken to ensure that the agreed performance objectives are “SMART” (specific, measurable, achievable, relevant and timely).
2.2.5. Experiences at local, national or regional levels have shown to IFATCA that there is still a long way to go before ATM performance measurement will not be perceived as a hindrance to improving the overall performance of the ATM system.
2.3. Eurocontrol Performance Metrics
2.3.1. One example of regional performance measures is Eurocontrol. It created in 1998 the Performance Review Commission (PRC). They analyse the performance of the ATM system in the following Key Performance Indicators:
- Safety
- Capacity
- Environment
- Cost-Efficiency
2.3.2. The latest Performance Review Report (PRR) was published in 2012, and a link is provided as information in appendices a) and b).
2.4. U.S.A./EU Operational Performance Comparison
2.4.1. In 2012, the FAA and Eurocontrol published a 106-page report entitled: Comparison of Air Traffic Management-Related Operational Performance: U.S./Europe (see appendix c). Also provided as information, this document is a joint publication of the Air Traffic Organization System Operations Services of the FAA and the Performance Review Commission of EUROCONTROL in the interest of the exchange of information.
2.4.2. The objective was to make a factual high-level comparison of Air Traffic Management performance between the US and Europe. The initial focus was to develop a set of comparable performance measures in order to create a sound basis for factual high-level comparisons between countries and world regions.
2.4.3. Locally, nationally and internationally there are performance measures taken and reports published by different instances. However, even though they are very comprehensive in many aspects, they still are very high level to the average ATCO.
2.5. ATM users expectations
2.5.1. ICAO Doc 9854 shows what performance expectations the ATM users can foresee. Here are the broad lines that concern ATM performance.
2.5.2. Access and Equity
A global ATM system should provide an operating environment that ensures that all airspace users have right of access to the ATM resources needed to meet their specific operational requirements and that the shared use of airspace by different users can be achieved safely. Generally, the first aircraft ready to use the ATM resources will receive priority, except where significant overall safety or system operational efficiency would accrue or national defence considerations or interests dictate that priority be determined on a different basis.
2.5.3. Capacity
The global ATM system should exploit the inherent capacity to meet airspace user demands at peak times and locations while minimizing restrictions on traffic flow. The ATM system must be resilient to service disruption and the resulting temporary loss of capacity.
2.5.4. Cost-effectiveness
The ATM system should be cost-effective, while balancing the varied interests of the ATM community. The cost of service to airspace users should always be considered when evaluating any proposal to improve ATM service quality or performance.
2.5.5. Efficiency
In all phases of flight, airspace users want to depart and arrive at the times they select and fly the trajectory they determine to be optimum.
2.5.6. Environment
The ATM system should contribute to the protection of the environment by considering noise, gaseous emissions and other environmental issues in the implementation and operation of the global ATM system.
2.5.7. Flexibility
Flexibility addresses the ability of all airspace users to modify flight trajectories dynamically and adjust departure and arrival times, thereby permitting them to exploit operational opportunities as they occur.
2.5.8. Global interoperability
The ATM system should be based on global standards and uniform principles to ensure the technical and operational interoperability of ATM systems and facilitate homogeneous and non-discriminatory global and regional traffic flows.
2.5.9. Predictability
Predictability refers to the ability of airspace users and ATM service providers to provide consistent and dependable levels of performance. Predictability is essential to airspace users as they develop and operate their schedules.
2.5.10. Safety
Safety is the highest priority in aviation, and ATM plays an important part in ensuring overall aviation safety. Uniform safety standards and risk and safety management practices should be applied systematically to the ATM system. In implementing elements of the global aviation system, safety needs to be assessed against appropriate criteria and in accordance with appropriate and globally standardized safety management processes and practices.
2.6. Balance between acceptable performance and safety tolerable risk levels
2.6.1. Air traffic control, as can be said of many other safety related activities, is mainly managing risk to an acceptable risk tolerable level. It is generally accepted that zero risk in ATC is unattainable without immediate grounding and immobilisation of all aircraft. This would however mean a zero performance on an individual, ANSP or overall ATM level.
2.6.2. To find the best possible balance between utmost performances, which could be viewed as to offer ATC services in a given space to an unlimited amount of traffic in all possible conditions, and utmost safety where zero accidents or incidents occur, we need to have guidelines for what is acceptable or tolerable risk. So to what level of performance and how safe do we need to be? As an example, Safety Integrity Levels (SIL) has grown substantially in the oil/gas, petrochemical and other process industries over the last 10 years. SIL is still a somewhat ambiguous concept that often is misinterpreted and incorrectly implemented. To fully understand SIL and its implications, it is important to grasp the overarching concept known as Functional Safety.
2.6.3. Functional Safety, as defined by IEC standard 61508, is the safety that control systems provide to an overall process or plant. The concept of Functional Safety was developed in response to the growing need for improved confidence in safety systems. Major accidents around the world, as well as the increasing use of electrical, electronic or programmable electronic systems to carry out safety functions, have raised awareness and the desire to design safety systems in such a way as to prevent dangerous failures or to control them when they arise. In certain ATM systems, there are examples of control systems or tools at the disposal of ATCOs such as, early conflict detection, collision/minima separation prediction or short-term conflict alert, etc. These tools, mostly software integrated in radar displays or synthetic traffic displays, purposely enhances the ATCO’s traffic analysis, thus increasing both performance and safety at the same time.
2.6.4. Previous safety standards were generally prescriptive in nature, not performance based. Functional Safety is achieved when every safety function is successfully carried out and the process risk is reduced to the desired level.
2.6.5. Risk analysis brings many benefits. It ensures that continued operation is done within an acceptable risk or safety level. It should also allow for objective and consistent assessment of unsafe conditions.
2.6.6. So, what is an acceptable risk or safety level?
SIL is a measure of safety system performance in terms of Probability of Failure on Demand (PFD). This convention was chosen based on the numbers as it is easier to express the probability of failure rather than that of proper performance (e.g., 1 in 100,000 vs. 99,999 in 100,000). Total system risk is controlled by individual system or component failure risk. This means that the ATCO is an important part of the total risk, however its metrics remains a highly variable and difficult measure to value.
2.6.7. There are four discrete integrity levels associated with SIL: 1 to 4. The higher the SIL level, the higher the associated safety level. Although these numbers are not related to the aviation industry, it shows how some metrics can be applied to systems components.
2.6.8. In a recent FAA study, each system component involved in commercial airline transport should ideally achieve a PFD of 10-9, (1 in 1,000,000,000) for every flight hour. The study adds that this number should be achievable only when the cumulative risk of the entire system is controlled through individual component risk assessments and that no single component failure results in a catastrophic accident, (fail-safe philosophy).
However, the study states that testing a system that ensures a 10-9 PFD isn’t practically feasible, and that any incorrect assessment of a lone component can result in a catastrophe.
2.6.9. Public opinion
One factor that is arguably not insignificant in nature is public opinion. Any State, industry, regulator or enterprise balancing performance and safety can be challenged in some way by public opinion. The level of performance and safety in which the public believes it is entitled mandates most activities.
2.6.9.1 Following the Fukushima catastrophe, Japan has since stopped all electrical production from nuclear reaction. The controversy in Japan swirling around the topic of restarting idled nuclear reactors is showing no signs of simmering down; instead uncertainty and frustration appear to be increasing. In July 2013, the Nuclear Regulation Authority began processing restart applications for ten reactors from eight utilities under tougher post- Fukushima guidelines. At that time the process was only expected to take six months, but has continued for the last eight months with no clear end in sight.
2.6.9.2 The Swiss government announced in 2011 their decision to withdraw from the use of nuclear energy on a step-by-step basis. The intention is to decommission Switzerland’s five nuclear power plants when they reach the end of their service life and not to replace them with new ones. Is this the end of nuclear power worldwide? Certainly not for now as many other States, where public opinion might not have strongly influenced the authorities, are still pursuing this type of energy.
2.6.9.3 In most countries, today’s public mobility by automobile is a routine, daily non-event despite causing an average of 1.3 million deaths per year. That’s more than 3500 per day even if 2013 is considered to be one of the safest years in recent history. This accounts for 0.02% of the population dying yearly on the roads. While family and close relatives consider a single death a tragedy, public opinion generally considers this statistic satisfactory and would not demand an increase in safety if it means a reduction in mobility (less performance).
2.6.9.4 2013 was also a record low fatality in air travel. 224 deaths were recorded in 2013 for 3 billion commercial passengers. 453 fatalities are accounted for when considering all aircraft able to carry at least 6 passengers, excluding helicopters, balloons, hot air balloons, airships, gliders, and fighters but including military aircraft intended for troop transport, reconnaissance, surveillance and logistical support, provided they are able to carry at least six people. This amounts to 0.000008 and 0.000015% fatality rate.
2.6.9.5 Even if both statistics are difficult to compare, public opinion still considers any aviation related fatality as highly dramatic. What would it be if on an average day, 23 A320 crashed, killing 3500 passengers? Questionably, commercial aviation would not survive. Proof is that most, if not all commercial aviation accidents (17 in 2013), are regularly seen in daily electronic or paper news. This can be considered positively as public opinion arguably pushed the whole aviation industry to strive for and achieve the actual high safety standard we enjoy.
2.7. Factors that enhance or deter performance
2.7.1. Performance, on a whole, can be varied tremendously by many different factors. Compared to only a few decades in past, many ATM systems today incorporate electronic tools with varying functions intending to help the ATCO in the routine work performed on a daily basis. Understandably, these tools play a role in enhancing safety but also, to a certain limit, may also improve the ATCO performance as, in some instances, can lessen some of the workload. It is recognised that any downgraded component of an ATC system, radios, radar consoles, other technical mishap or external phenomena like weather storms, will normally reduce the overall ATM performance.
2.7.2. The ATCO, being essential to the overall performance of the ATM system, may be considered to be the most variable aspect because of its human nature. Some well- known factors that can enhance or deter the personal performance of an individual may be; general health condition, state of rest or fatigue, diet choices, personal life or family concerns, anxiety levels, pressure from peers, age, recent professional achievements, experience level, self-esteem, professional satisfaction, shift work cycles, etc.
2.7.3. IFATCA has produced, in the past and still does today, numerous papers and Policies concerning performance, such as; Fatigue in ATC, Sleep Apnoea, Ageing, Stress Management and CISM to name a few, that can be guidance or a source of information concerning factors that can enhance or deter personal performance.
2.7.4. Fatigue in ATC; T&P Manual, Page 4 2 2 9, MED 9.2.5, Policy is:
| Management has the prime role for providing fatigue management and prevention of fatigue-related catastrophes. Any situation where increased fatigue, decreased sleep, or performance loss can be demonstrated, is a situation where the margin for error is reduced, albeit by some unknown amount, and should be avoided in ATC. |
2.7.5. Ageing Controller; T&P Manual, Page 4 2 1 30, WC 8.3.7, WP 161 – Punta – Cana 2010.
2.7.6. Stress Management; T&P Manual, Page 4 2 2 6, MED 9.2.3.
2.7.7. CISM; T&P Manual, Page 4 2 2 8, MED 9.2.4, Policy is:
| Professional critical incident stress support services should be made available to air traffic controllers involved in ATC incidents / accidents and any other occurrences that have potential to create critical stress reactions influencing the ATCO’s performance. It is the controllers’ choice whether or not to take advantage of these support services. |
2.8. Measuring performance for ATCO training evaluation and certifying
2.8.1. Generally speaking, measuring the performance of ATCOs is mostly done in 2 ways. By performance, it can be understood to refer to the theoretical knowledge necessary for doing correctly the required job, and the level of skill demonstrated in either a live or simulated environment.
2.8.2. ATCO initial training programmes vary greatly from one ANSP to another. Some have an in-house school, others depend on specialised schools and, in certain cases ATCOs are contracted out to foreign ANSPs for training. In all cases, acquiring the basic ATC skills should be done using proven and constructive learning techniques that include the periodical testing of knowledge, up to the required standard of the school and allowing the candidate to move on.
2.8.3. Further schooling will normally be more specific to the future work environment of the student. Formal classroom teaching and testing mixed with simulator work and evaluations are to be expected. When all this is achieved, live, on the job training follows where learning continues with more or less continual evaluations. Positive progression normally leads to a final knowledge test and evaluation.
2.8.4. Throughout this training process, it is expected to be continuously evaluated by varying methods. Theoretical knowledge can be testing by written or oral exams while jobs skills can be evaluated in a simulated or live environment.
2.8.5. Certified ATCOs should periodically be submitted to testing and evaluation. Regulators, sometimes ANSPs, usually define the content, frequency and nature of these tests. Again, theoretical knowledge and practical skills should be assessed. When regular, comprehensive and constructive evaluation is done, it will normally maintain, otherwise enhance the ATCO knowledge and skill ensuring continued best performance.
2.8.6. To obtain the best possible results from these tests and evaluations, they should be as close as feasible to the expected work profile. Many ANSPs today revert to what some call Competence Assessments (CA) to help maintain their workforce skill and knowledge to the highest degree. For these CA to be constructive they should not be punitive where lack of knowledge or skill is discovered, but should be there to identify weaknesses either on an individual or global level and propose solutions to deal with them.
2.9. IFATCA’s perspective
2.9.1. The production and use of Performance Indicators (PI) by ANSPs cannot be controlled by IFATCA. However, they can be examined and questioned as for their relevance to the ATC system. Presently, Performance Indicators are from a “user” perspective. Their definitions bring no particular problem and they are a fair indication for the user of the system as a whole. IFATCA does understand the necessity for government to act as a control mechanism of commercialized ANSPs.
2.9.2. Arguably, from an ATCO perspective, can the actual Performance Indicators be considered valid metrics for an ATM system? Probably not as there are too many factors beyond the control of any individual ATCO and ATC Centres that can influence the outcome of these measurements.
2.9.3. Examples of these influences can be: equipment capability and serviceability, capacity, military activity, traffic priorities, resourcing, training, airspace design, over-demand etc. Also, these results can be markedly different whether they are measured at a system level, national level, district level or unit / facility level.
2.9.4. Controllers and Members Associations must be extremely careful where these results are broken down further to sector, console or individual ATCO level.
2.9.5. Safety PIs, as one example out of the many Indicators, tend to measure rates of occurrences per distance flown or per the number of flights per volume. Typically, they measure reported events that occurred after the fact. This can be more indicative of the robustness of the reporting system than the actual number of events or the safety level of the unit. What they don’t measure are preventative actions, professionalism, caution nor separation assurance executed by the ATCO. To sum it up, they definitely don’t measure ATCO performance. From an ATCO’s perspective, perhaps such PIs could best be summarized as quantitative rather than qualitative or a measure of performance.
2.9.6. PIs usually assessed and published by ANSPs should not be read as being totally indicative of ATC performance from an individual, sector or unit perspective but rather published for general, user or industry use to measure the overall system efficiency.
2.9.7. PIs as currently used have little direct relevance to the operational ATCO’s performance.
2.9.8. IFATCA isn’t aware of PIs used to measure more accurately the operational performance of an ANSP or ATCO. Suggestions to develop such PIs have been made. These could include: the successful number of ATCOs trained to operational working level, the number of occurrence reports submitted vs. actual occurrences, the number of recommendations from investigations made/accepted/implemented, the degree of success due to change management processes, etc.
Conclusions
3.1. Measuring performance in ATM is relatively new and may be difficult to quantify correctly.
3.1.1. Experience has shown that potential risky practices may exist within ANSPs when performance driven objectives are in place.
3.2. With 11 KPIs, ICAO draws a good outline of what future performance metrics should be, paving the way to what could be a globally improved ATM system.
3.3. Eurocontrol started to analyse some KPIs and is reporting its findings.
3.3.1. The FAA and the E.U. published a joint performance comparison, highlighting the similarities and differences between their respective ATM systems.
3.4. ATM users have defined their expectations concerning the system’s performance. These include; equity, accessibility, comprehensive capacity, cost efficiency, flight plan flexibility and efficiency, environmentally friendly, interoperability to ensure smooth global traffic flows, predictability and safety.
3.5. Today in the aviation industry, there are too many uncontrollable variables and undefined metrics preventing any practical ways of finding true balance to provide the best performance vs. the best safety.
3.5.1. Despite this, it is generally observed to be the case globally as the aviation industry is enjoying a record safety level while demand is steady, otherwise on the rise.
3.6. ATM systems are subject to many variables that will influence performance. Weather conditions, technical equipment qualities, and the human beings will determine what performance level to expect.
3.6.1. On the human vs. performance aspect, policies concerning this matter already exist.
3.7. Student and certified ATCOs expect to be regularly tested for knowledge and continuously evaluated for skills. These tests and evaluations should not be punitive in nature but should strive to maintain or improve the knowledge and skills.
3.8. IFATCA believes that published performance indicators should not reflect ATC performance and should only be used by and for the industry to measure its overall efficiency.
3.8.1. MAs and their members must be extremely careful when these results are directly related to them as they have little or no relevance to ATC performance.
Recommendations
4.1 That this paper be accepted as information.
References
Alexis Brathwaite; ATC Professionalism and ATC Performance, PLC Brief.
Marc Baumgartner; Performance Measurement in ATM? A necessity or a hindrance to efficient ATCO work? September 2006.
Robert D. Behn, Harvard University; Why Measure Performance? Different Purposes Require Different Measures.
IFATCA; Think Performance Regulation in ATM, Case Study Europe.
ICAO; Doc 9854 Global Air Traffic Management Operational Concept.
Eurocontrol; Performance Review Report 2012 (https://www.eurocontrol.int/publications/performance-review-report-prr-2012),
Eurocontrol; Comparison of Air Traffic Management-Related Operational Performance: U.S./Europe (https://www.eurocontrol.int/sites/default/files/publication/files/2012-US-EUR-comparison-of-ATM-related-OPS-performance.pdf)
General Monitors Inc. SIL 101, How safe do I need to be?
FAA; Aviation Safety Tolerable Risk Principles, March 19 2008.
Wikipedia; IEC 61508.
IFATCA Technical & Professional Manual 2013.
Reuters, Fukushima Update, February 20 2014.
Swiss Federal Office of Energy, SFOE.
Association For Safe Road Travel.
Airbus Industries, A320 specifications.
AFP; Air travel fatalities hit lowest ever in 2013, January 09 2014.
Bureau of Aircraft Accidents Archives.
Appendices
Performance Review Report (2012):
A) 10-page Executive Summary
B) The complete 137-page report
C) Comparison of Air Traffic Management-Related Operational Performance: U.S./Europe
Last Update: September 30, 2020