Introduction

At the 42nd Annual Conference in Buenos Aires SC4 was tasked to re-visit the subject of Performance Indicators (PIs).

There have been numerous previous working papers produced by SC 4 on the subject, two which are still relevant being Toulouse 1998 – WP 151 – Productivity and Performance Indicators, & Santiago 1999 – WP 171 – Performance Indicators in Context. The topic of Monitoring Eurocontrol Performance Review Commission (PRC), which produces an annual report concerning the Performance Indicators within Eurocontrol, remains an on going work item for SC 4, and is the subject of a separate working paper to Committee C at this conference (see agenda item C.5.7.).

This paper will deal with the broader topic of which such Performance Indicators are used in our industry, how they are used, and will discuss the IFATCA position towards them.

Discussion

It must be accepted that the use of Performance Indicators (PIs) is widespread standard practice in many industries, and this is the case amongst many Air Navigation Service Providers. Eurocontrol has set up the Performance Review Commission to define, monitor and report about a set of Key Performance Indicators (KPIs) in defined Key Performance Areas (KPAs). One of the major tasks undertaken by the Civil Air Navigation Services Organisation (CANSO) has been to conduct a major benchmarking study, which includes a detailed analysis of Performance Indicators.

Whilst the production and use of these PIs by ANSPs cannot be controlled by IFATCA, we can examine the type of PIs typically used and question their relevance to the ATC system. And perhaps suggest more relevant PIs.

Typically, the PIs used by ANSPs take the perspective of the “user” or customer of the ATC system. To use Eurocontrol’s PRC 1999 document “The PRC’s European ATM Performance Measurement System” as an example, the PRC defined its KPAs as:

Safety – the conformance of air transport to specified safety targets;

Delay – The time taken in excess of the optimum time that it takes a user to complete an operation;

Cost effectiveness – The value for money that users receive from the supply of air traffic services;

Predictability – The ability of a user to predict variation and to build and maintain optimum flight schedules;

Access – The accessibility of airspace, ATM services and airport facilities under controllable conditions;

Flexibility – The ability of ATM to accommodate changing user needs ;

Flight Efficiency – The ability of the ATM system to allow a user to adopt the preferred flight profile in terms of flight level and route;

Availability – the availability of critical ATM resources and of the ATM services provided to users;

Environment – The conformance of air transport to environmental regulations;

Equality – Equality of treatment of flights by all aircraft operators within and between specific classes of airspace.

As can be seen, these are from a “user” perspective. Indeed there is no particular problem with these definitions from that point of view and they are a fair indication for the user of the system as a whole.

But are they a true measurement of the ATM system from the ATCO’s perspective? Arguably not. There can be many factors beyond the control of any individual ATCO, or ATC Centre, that will influence the outcome of these measurements. Some of these influences are equipment capability, capacity and serviceability, military activity, traffic priorities, resourcing, training, airspace design, and over demand etc. These concerns can be particularly valid for “cost effectiveness” PIs where so much can be influenced by the structure and use of the base data.

Also, these results can be markedly different dependant upon whether they are measured at a system level, national level, district level or unit or facility level. Controllers and Members Associations must be extremely careful where these results are attempted to be broken down further to sector, console or individual ATCO level.

From the ATCO’s perspective, perhaps such PIs could best be summarized as “quantitative” rather the “qualitative”. For example, “Safety” PIs tend to measure rates of occurrences per distance flown or numbers of flights per volume etc. They measure “reported events” after the event. They can be more indicative of the robustness of the reporting system than the actual number of events. But do they measure “preventative” actions by the ATCO’s? Can they measure professionalism, caution and separation assurance? Whilst the PIs may “infer” the success of these factors by low occurrence rates, are these successes then reflected “negatively” in the “efficiency”, “accessibility” and “cost effectiveness” PIs?

Similarly, “delay” PIs can only be assessed qualitatively from an ATCO perspective if all possible contributory factors are taken in account. A delay expectation from the user perspective cannot be seen as a true measure of ATC performance.

SC 4 concludes that PIs as usually assessed and published by ANSPs should not be read as being totally indicative of ATC performance from an individual, sector or unit perspective.

These PIs are published for general or at least industry use. They are an indication of overall “system” efficiency, but not directly ATC performance. SC 4 is also aware of such PIs being published within ATC units on a daily basis, for all on duty ATCO’s to see. SC 4 has some concerns about the “passive influence” on operational ATCO’s of having this sort of data displayed in operational areas. Will the ATCO change his controlling techniques after reading these PIs on his way to the console? What could they do differently? Be ultra safe that shift at the expense of efficiency? Any such suggestions would be to denigrate the professionalism and training of ATCO’s. Indeed most of the PIs as currently used have little direct relevance to the operational ATCO.

SC 4 is firmly of the belief that Performance Indicators must never be linked to the pay and conditions of operational ATCO’s

With an increase in ANSPs being privatized or corporatized, more and more mangers of ATC facilities, down to lower and lower levels, are moving to contract employment. SC 4 is also aware of some circumstances where managers receive bonuses if various PI targets are met. Such PIs can include average cost per flight, and average delay per flight. SC 4 has concerns about PIs being linked to pay received by operational managers. As stated above, most of these factors are beyond the control of the individual ATCO’s on a day to day basis. Any such linkage between Managers pay or bonuses should be restricted to higher level managers, and only those not having direct operational supervision.

So far, SC 4 is not aware of PIs being used that perhaps more accurately measure the operational performance of an ANSP. Such PIs have been suggested to be developed. These could include:

• number of ATCO’s trained to operational working level, training successes;
• number of occurrence reports received vs. investigated;
• number of recommendations from investigations made / accepted / implemented;
• success of change management process.

Conclusions

SC 4 believes that Performance Indicators are not necessarily a bad or inappropriate tool for assessing overall air navigation system performance. However SC 4 believes that PIs as commonly published and used today do not give a totally accurate picture of ATC performance at unit, sector, or individual ATCO level.

There are other PIs that SC 4 would like to see developed that more realistically assess the operational performance of ANSPs from the ATCO’s perspective.

Whilst SC 4 acknowledges the increasing linkage between PIs and pay or bonuses for certain managerial positions, such linkage should be limited to only those managers not having direct operational supervision.

SC 4 is firmly of the belief that PIs should never be linked to pay and/or working conditions for operational ATCO’s.

Recommendations

Insert in the IFATCA Manual, on new page 4113 , new section 1.7:

1.7 Performance Indicators

1.7.1 Performance Indicators as published and used by Air Navigation Service Providers must not be linked in any way to the pay and/or working conditions of individual ATCO’s.

References

IFATCA Manual.

IFATCA Working Paper, Toulouse 1998 – WP 151, Productivity and Performance Indicators.

IFATCA Working Paper, Santiago 1999 – WP191, Performance Indicators in Context.

Report on Eurocontrol Performance Review Commission Meeting, Brussels 19/5/03 – EVP Europe.

“Safety Today was 16” – The use of Performance Indicators in Air Traffic Control, article in The Controller 2/99 by Bert Ruitenberg, IFATCA Human Factors Specialist.

Airservices Australia.