Summary

Performance Measurement in ATC continues to be an issue in several parts of the world. Existing IFATCA policy is strongly against Performance Measurement in ATC. The question remains the same: Does Performance Measurement improve safety?

This paper once again brings attention to the policy of IFATCA concerning Performance Measurement while highlighting some of the challenges related to the subject.

Introduction

1.1. IFATCA does not support using Performance Measurement (PM) for assessing ATCOs’ competence.

1.2. Performance Management systems are guided by good intentions for making things better. However their design and implementation in ATM does not seem to make things better.

1.3. Performance Measurement is meant to improve performance in ATC. When not used as it should be as a Human Resource Management function only, it can have negative effect on safety in ATC.

1.4. Performance Measurement in an ANSP should be used for:

• Improving productivity and reducing costs;
• Identifying best practices in the organization;
• Making better and faster budget decisions;
• Increasing accountability and incentives based on real data and not subjective judgments;
• Motivating staff to perform better and contribute more to the organizational goals;
• Deciding how to handle poor performance;
• Helping staff to continually learn and develop;
• It can help in planning and promotion decisions.

1.5. Performance Measurement in an ANSP shall NOT be used for:

• Denying promotion or demoting ATCOS;
• Withdrawing licenses;
• Transferring ATCOs to different positions – if they were viewed as not having performed in the current one;
• Denying allowances.

1.6. Competence Assessment should be in place to ensure a safe working environment (Competence Assessment WP), instead of Performance Measurement.

Discussion

2.1. Definition:

Performance Measurement is the process of collecting, analyzing and /or reporting information about the performance of an individual, group, or organization system or component to see whether output is in line with what was intended or should have been achieved. (Bernard Marr, Leadership and management, 2014)

2.2. The current IFATCA Policy on Performance Measurement is:

2.3. IFATCA policy states very clear that PM must not be linked in any way to the pay and/or working conditions of individual ATCO’s.

2.4. Some MAs cited the following challenges they have with the performance management systems:

• Set targets/goals were not SMART (Specific Measurable Achievable Realistic and Time bound);
• They were being used as a basis for promotion;
• As a basis of annual salary increments;
• Some countries did not have any Performance Measurement tools;
• Others were being used to know individual skills;
• For some it was a union issue;
• Withholding safety incident reports in order to meet safety and operational targets.

2.5. IFATCA definition for Just Culture is:

“A culture in which front line operators or others are not punished for actions, omissions or decisions taken by them that are commensurate with their experience and training…”

using Performance Measurement for assessing ATCOs safety opposes Just Culture principles.

2.6. Competence Assessment is the tool to ensure ATCOs are competent and working professionally. Performance Measurement is a tool that might be useful to management for ANSP/ATM performance overall, but NOT on an individual ATCO or team level.

2.7. Criticism of PM

2.7.1. When carrying out an evaluation or appraisal, it is difficult to say what an individual ATCO’s performance is, what they really produce or what is their contribution to the organisation.

2.7.2. Appraisals can end up driving wrong behaviors e.g. people already know that they are being judged by their individual output. They will therefore try to do a lot, or work alone so that they distinguish their contribution to that of others without necessarily focusing on outcomes and teamwork. In an ATM environment this means poor coordination and overload of traffic which can lead to serious incidents. In this case performance gets worse instead of getting better.

2.7.3. Some ANSPs don’t seem to agree on appropriate performance standards in ATM – Is it safety, punctuality, discipline etc… and how do you measure who is safer than another? Is it by how many incidents they have or by how severe the incidents are? Different ideas exist as to what is possible and achievable and what can be assessed in the performance appraisal process.

2.7.4. It can be difficult to hold an individual ATCO responsible for the results they achieve because they are not in complete control of those results. E.g. if an ATCO is judged on how much revenue they contributed to the organization, it may be unfair since an ATCO does not determine how much traffic passes through their airspace.

2.7.5. When performance reviews are tied to remuneration, it can be unethical and sometimes difficult to determine who should or shouldn’t get a bonus.

2.7.6. Individuals who stick to performance plans are rewarded whereas those who deviate from a performance plan, while they may produce innovative solutions to organisational problems, they may not be rewarded. E.g. when measuring performance of an ATCO against set procedures, those who stuck to procedures are rewarded whereas those who diverted from procedures (even though by their action it leads to greater safety) may not be rewarded.

2.7.7. Those who follow a safe road of predictable results are recognised and rewarded whereas those who make mistakes which may contribute to learning and knowledge are punished. In ATC, safer systems have been developed through learning from mistakes such as incidents.

2.7.8. In some of the ANSPs reward is given to individuals who achieve targets despite their inaction whereas those who have focused on monitoring, analysing, and managed situations to eliminate a bad situation/trend fail to be rewarded if they don’t achieve targets e.g. some ATCOs may be rewarded for controlling a lot of traffic safely while overlooking the fact that they come to work drunk, late or are inconsistent and unreliable whereas another ATCO may be punished for not being able to control high volume of traffic even though they are reliable and don’t show up late or drunk to work.

2.7.9. Most performance appraisals are tied to Management performance targets. This puts pressure on management to sometimes set unrealistic goals for employees to achieve. E.g. if one of the organisation KPI is increased revenue, the performance target for individual ATCO becomes measuring how much revenue they bring to the table which becomes unrealistic.

2.8. The concept of productivity in ATC

2.8.1. During the IFATCA Conference in 1998 (WP151 TOULOUSE 1998) it was reported that some countries were already using these performance indicators to define the complexity of work in certain ATS divisions. ATCOs within the same country sometimes received and continue to receive different salaries depending on the specific ATS division they work in. Often, no research had been made to obtain a form of scientific background to justify these indicators. As both items had, a great influence on the work of an ATCO, it was important to define IFATCA’s policy on the interpretation of Productivity and the correct use of performance indicators.

2.8.2. In trying to define productivity in ATC, in the WP151, TOULOUSE 1998 it was found to be a difficult if not an impossible task. Was it more appropriate to talk about ATC productivity as the main task of an ATCO or was it more appropriate to talk about “promoting safety, efficiency and regularity in international air navigation as this was the main objective of IFATCA as it is laid down in the IFATCA mission statement.

2.8.3. An increase in ATC productivity measured as “Safety productivity” (a new kind of productivity), would mean an increase in safety? This would be measured by maintaining a ratio of ATC related occurrences against the number of aircraft moving through the unit’s airspace.

2.8.4. As ATC does not produce goods, productivity in this respect could probably be defined as working productivity regardless of whether the ATS organisation has been privatised or is a governmental institution. It is accepted that ATS organisations are willing to embrace an increase in productivity and it is important to employer and employee that the correct understanding of productivity is determined.

2.8.5. Productivity as a relation between the annual turnover and the number of invested working hours would normally consider each employee of the company and thus is not only an ATC problem. An increase in ATC productivity could as well mean to be capable of handling more traffic for instance in a certain sector. As this increase might have an effect on air traffic safety, it is very important to find reliable criteria by means of which it is possible to state what the present level of productivity is.

2.8.6. An increase in productivity in this context must never lead to overload or unsafe situations. The capability of controllers to handle a certain amount of traffic differs from day to day, and within a day, for various reasons (e.g.: traffic situation, complexity of traffic, partial or complete system failure, private problems, fatigue and a lot more).

2.8.7. Another problem: An ATCO may have been trained to handle high density of traffic, but operationally it may be that there is not much traffic to handle. In such case someone looking from outside might conclude that productivity of the ATCO is low without regard for prevailing traffic situation. Counting only the number of aircraft per hour or miles flown within a certain sector or facility.

2.8.8. The issue of performance was discussed again as a continuing agenda item for SC4 (Sub Committee 4) in Santiago 1999 – WP 171, “Performance Indicators in Context”. This paper discussed Performance in the context of Human Resource Management.

2.8.9. HRM main purpose is to ensure that employees of a company are used in such a way that the employer obtains the greatest possible benefit. Traditionally air traffic service providers were located in the civil service or in the wider public sector. Commercialisation and privatisation had become major issues for air traffic management professionals and their representative bodies.

2.8.10. The pressure on service providers to become more aware of the needs of their customers, the airlines, and the competitive environment in which these airlines exist has resulted in “change management” becoming the oft heard buzz words in the ATM industry. The “necessary” change is often demanded by the customers, the target areas identified by the corporate accountants and internal auditor and the mechanisms whereby change can be brought about identified and decided upon by the Human Resources Director or Manager. This change can also lead to what Drucker calls “conflict between professional and bureaucratic management”. In our context this means between managers with an ATM / ATC background and those with an accountancy or HRM background.

2.8.11. The use of corporate mission statements, corporate communications policies and the need to cut operational costs and increase profits were some of the approaches being used by Human Resource managers to set performance targets for ATC and to “change the culture of an organisation”.

2.8.12. This has many implications for all employees but particularly for ” professional workgroups” such as air traffic controllers because inevitably there will come a conflict between what is perceived by management as being in the interests of the company (remember we are talking about a commercial environment) and what controllers perceive as being required in order to maintain their professionalism.

2.9. Performance Indicators

2.9.1. It is surely no coincidence that the first policy document which CANSO (Civil Air Navigation Service Providers) set about drafting concerned P.I.s. Nor it is a coincidence that the revised Eurocontrol Convention sets up a PRC and PRU.

2.9.2. Performance has become a battle ground for payment, status and image within the profession and with outside groups.

2.9.3. Ideologically some general managers’ views are in conflict with professionals and in this context Air Traffic Controllers. Often air traffic controllers are managed by people who know nothing about ATC. Concerns have often been raised by controllers’ representative bodies at the emphasis in various documents produced by Eurocontrol, IATA, CANSO, etc. on cost control rather than safety.

2.9.4. The WP 165 Performance Indicators, Hong Kong 2004 dealt with the broader topic of which performance indicators were used in our industry, how they are used and also discussed the IFATCA position towards them. It was generally accepted that the use of Performance Indicators is a widespread standard practice in many industries, and this is the case among many Air Navigation Service providers. Eurocontrol already having set up the performance review commission to define, monitor and report about a set of key performance indicators in defined key performance areas.

2.9.5. CANSO undertook a major benchmarking study, which includes a detailed analysis of Performance Indicators.

2.9.6. Whilst the production and use of these PIs by ANSPs cannot be controlled by IFATCA, we can examine the type of PIs typically used and question their relevance to the ATC system. And perhaps suggest more relevant PIs. Typically, the PIs used by ANSPs take the perspective of the “user” or customer of the ATC system. To use Eurocontrol’s PRC 1999 document “The PRC’s European ATM Performance Measurement System” as an example, the PRC defined its KPAs as:

• Safety – the conformance of air transport to specified safety targets;
• Delay – The time taken in excess of the optimum time that it takes a user to complete an operation;
• Cost effectiveness – The value for money that users receive from the supply of air traffic services;
• Predictability – The ability of a user to predict variation and to build and maintain optimum flight schedules;
• Access – The accessibility of airspace, ATM services and airport facilities under controllable conditions;
• Flexibility – The ability of ATM to accommodate changing user needs;
• Flight Efficiency – The ability of the ATM system to allow a user to adopt the preferred flight profile in terms of flight level and route;
• Availability – the availability of critical ATM resources and of the ATM services provided to users;
• Environment – The conformance of air transport to environmental regulations;
• Equality – Equality of treatment of flights by all aircraft operators within and between specific classes of airspace.

2.9.7. As it can be seen, these are from a “user” perspective. Indeed there is no particular problem with these definitions from that point of view and they are a fair indication for the user of the system as a whole. But are they a true measurement of the ATM system from the ATCO’s perspective? Arguably not.

2.9.8. There can be many factors beyond the control of any individual ATCO, or ATC Centre, that will influence the outcome of these measurements. Some of these influences are:

• equipment capability;
• capacity and serviceability;
• military activity;
• traffic priorities;
• resourcing;
• training;
• airspace design;
• over demand etc.

2.9.9. These concerns can be particularly valid for “cost effectiveness” PIs where so much can be influenced by the structure and use of the base data. Also, these results can be markedly different dependent upon whether they are measured at a system level, national level, district level or unit or facility level.

2.9.10. Controllers and Members Associations must be extremely careful where these results are attempted to be broken down further to sector, console or individual ATCO level.

2.9.11. From the ATCO’s perspective, perhaps such PIs could best be summarised as “quantitative” rather the “qualitative”.

2.9.12. “Safety” PIs tend to measure rates of occurrences per distance flown or numbers of flights per volume etc. They measure “reported events” after the event. They can be more indicative of the robustness of the reporting system than the actual number of events. But do they measure “preventative” actions by the ATCO’s? Can they measure professionalism, caution and separation assurance? Whilst the PIs may “infer” the success of these factors by low occurrence rates, are these successes then reflected “negatively” in the “efficiency”, “accessibility” and “cost effectiveness” PIs?

2.9.13. Similarly, “delay” PIs can only be assessed qualitatively from an ATCO perspective if all possible contributory factors are taken in account. A delay expectation from the user perspective cannot be seen as a true measure of ATC performance.

2.9.14. The conclusion from SC 4 was that PIs as usually assessed and published by ANSPs should not be read as being totally indicative of ATC performance from an individual, sector or unit perspective. These PIs are published for general or at least industry use. They are an indication of overall “system” efficiency, but not directly ATC performance.

2.9.15. It was noted that such PIs were being published within ATC units on a daily basis, for all on duty ATCO’s to see. Concerns were raised about the “passive influence” on operational ATCO’s of having this sort of data displayed in operational areas. Will the ATCO change his controlling techniques after reading these PIs on his way to the console? What could they do differently? Be ultra-safe that shift at the expense of efficiency?

2.9.16. Any such suggestions would be to denigrate the professionalism and training of ATCO’s. Indeed most of the PIs as currently used have little direct relevance to the operational ATCO.

2.9.17. There was the belief that Performance Indicators must never be linked to the pay and conditions of operational ATCO’s. With an increase in ANSPs being privatised or corporatised, more and more mangers of ATC facilities, down to lower and lower levels, are moving to contract employment.

2.9.18. In some circumstances managers receive bonuses if various PI targets are met. Such PIs can include average cost per flight, and average delay per flight.

2.9.19. There were concerns about PIs being linked to pay received by operational managers. As stated above, most of these factors are beyond the control of the individual ATCO’s on a day to day basis. Any link between payment or bonuses should be restricted to higher level managers, and only those not having direct operational supervision.

2.9.20. There were no known PIs being used that perhaps more accurately measure the operational performance of an ANSP. Such PIs have been suggested to be developed. These could include:

• number of ATCO’s trained to operational working level;
• training successes;
• number of occurrence reports received vs. investigated;
• number of recommendations from investigations made/accepted/implemented;
• success of change management process.

2.10. Performance Incentives

2.10.1. Performance incentives can take a number of forms; for example, there can be salary payments or working conditions based on performance or there may be benefits for airspace users (or others) for achieving a particular performance level early in a process (that is “an equip early incentive”).

2.10.2. The issue of incentives for particular performance levels can apply to airspace users, service providers, aerodrome operators, etc. This is performance in relation to a capability level and is usually for reaching a capability level sooner rather than later. However, in future ATM it is expected that aircraft will be able to use services as they become capable (rather than waiting for a particular implementation date for all users) and so there will be benefits even without special incentives.

2.11. ICAO position

2.11.1. In relation to required human performance, ICAO Doc 9854 Global ATM Operational Concept Appendix F paragraph 2.5.23.m states “human performance: humans within the ATM system must demonstrate appropriate competence at all levels. Such competence should be continuously monitored and frequently tested, and competence regimes should be established sufficient to provide an assurance of human performance. Appropriate mechanisms may include licensing, certificates of competence, technical qualifications, training certificates or similar”.

Conclusions

3.1. ATC is not a production line or a commercial company! IFATCA strongly recommends not using Performance Measurement for individual ATCO’s assessment. Instead of Performance Measurement, Competence Assessment should be used, and not be linked to pay or change in ATCOs’ working conditions.

3.2. Whilst there is increasing linkage between PIs and pay or bonuses for certain management positions, such linkage should be limited to only those managers not having direct operational supervision.

3.3. Performance Measurement of an individual ATCO is contradictory to Just Culture. PM should not be used as a tool for punishment since it might lead to unsafe situations e.g. if an ATCO is demoted or sacked because of ‘poor performance’.

3.4. Performance Measurement should not be used as a way of promoting safety among ATC e.g. by measuring number of air misses or unsafe situations directly involving an ATCO as a way of giving feedback. It should help them to manage safety better.

3.5. Performance Indicators are not necessarily a bad or inappropriate tool for assessing overall air navigation system performance. However PIs as commonly published and used today do not give a totally accurate picture of ATC performance at unit, sector, or individual ATCO level.

3.6. PM should be used to identify areas of improvement and take action to improve performance of the unit or the whole ANSP.

Recommendations

4.1. This paper shall be accepted as information.