Summary

Performance Schemes for the Air Traffic Management industry have been developed and introduced in regions around the globe for several years. The corresponding targets have an influence on how the ATM business is conducted nowadays. This paper looks at the shift towards a performance based system. All the major players’ respective positions are viewed in-depth. Consequences, actual and conceivable, are discussed and examples from sectors highlighted.

Introduction

1.1. The ICAO Air Navigation Conference, held in September 2003, urged ICAO to develop a performance framework for Air Navigation Systems (The target performance of the new system is described in terms of Key Performance Areas, performance  targets and metrics gathered into a Performance Framework. (Eurocontrol, Performance Framework and Influence Model in ATM, 2009). The ICAO Assembly of September 2004 adopted a resolution, in which it tasked ICAO to ensure that the future global ATM system is to be performance based and that performance objectives and targets are to be developed. The idea behind this was to initiate a shift from prescribing solutions to specifying performance.

1.2. At global level only 3 regions of the world are known to have adopted a form of performance approach based on certain features of ICAO. The European Union and associated countries with the Performance Scheme as part of the liberalization package called Single European Sky. The US with the performance assessment carried out via the FAA Performance assessment office and Australia. ICAO EUR/NAT (EUR Performance Framework Document (EUR Doc 030)) has published 5 KPAs as part of their approach to performance.

1.3. Further Eurocontrol’s Performance Review Commission and the FAA performance assessment organization have published several EU-US studies on performance. The latest has been published in November 2013.

1.4. The situation in Europe then was dominated by airlines’ growing economic problems after the immense downturn in passenger figures in the aftermath of 9/11. Pressure was high to reduce all costs not directly connected to a flight. In parallel and as a response to the crisis on delays, excarberated by the “Kosovo” crisis, the European Commission launched the liberalisation initiative for a Single European Sky. It formulated an ambition for this undertaking, which aimed to reduce costs by half, increase safety by a factor of ten, reduce the emission impact by 10% and have a significant reduction in delays. In order to achieve the liberalisation of the ATM Market the Commission put forward subsequent liberalisation packages (SES I in 2004 and SES II in 2009). Further legislative acts and commission decisions have introduced the performance-based approach. EC implementation regulation 691/2010 laid down that the overarching element of SES would be performance and it was decided to have two reference periods (RP1 2012-2014, RP2 2015-2019).

1.5. The SES is being applied in the European Union and so-called “European third countries”. The application of performance schemes outside Europe, the U.S. and Australia will follow in the years to come. Therefore the interest will reach a far more global status, especially in view of the fact that ICAO did not restrict the performance framework to Europe etc.

1.6. While the targets are created to improve ANSPs’ performance, in consequence they will reach “down” to the individual air traffic controller along with all other employees.

1.7. This paper will look at the impact, which the setting of performance targets may potentially create for ANSPs and ATCOs. It will focus on Europe, as this seems the most complex field. Due to the number of countries, ANSPs and varied cultures involved, compared to the single authority and provider nature of the U.S. and Australia, the European region is a more difficult task to regulate.

Discussion

2.1. While it might be argued that performance schemes and their associated Key Performance Indicators may deliver benefits, disadvantages and risks may also be introduced.

The following questions require detailed discussion:

• What are the aims of setting performance targets?
• Which advantages and risks are created by setting such targets?
• What are the possible effects on the ANSP and/or the individual ATCO?
• Are there any interdependencies between the targets?
• Does the Emmissions Trading Scheme have any influence?
• How will possible penalties for under-achievement affect an ANSPs ability to act?
• Will service provision change from the way we see it today?

2.1.1. The Performance Scheme will be looked at and explained in detail, covering the relevant information by the following:

• ICAO documents
• EC regulations
• IFATCA view and policy
• CANSO view and statements
• other publications by key people

2.1.2. IFATCA has discussed the issue of Performance Indicators at several previous conferences and has issued respective policy (see 2.2.5). Based on this policy, the aforementioned will be viewed against European Union Regulations, the work of the Performance Review Body and statements and opinions of the concerned providers, either by themselves or through CANSO, their international representative body.

2.1.3. Given the multilateral aspect of an ATM performance scheme, one can not expect to come to a one-sided conclusion. We need to understand that different performance targets have a correlation to each other. Turning one wheel in one direction will affect others, although not necessarily in a positive way.

2.2. Current status and organisations’ positions

2.2.1. ICAO

2.2.1.1. The 11th Air Navigation Conference in 2003 endorsed the global ATM operational concept. It urged ICAO to develop a performance framework for Air Navigation Systems. In 2004 the ICAO Assembly called upon States, PIRGs (Planning and Implementation Groups) and the aviation industry to use the ICAO Global ATM Operational Concept as the common framework. ICAO was also urged to ensure that the future global ATM system was to be performance based and that the performance objectives and targets for the future system would be developed in a timely manner.

2.2.1.2. In the course of development ICAO defined 11 Key Performance Areas (KPAs) in it’s ATM Global Operational Concept (Doc 9854). They are in alphabetical order:

• Access/Equity
• Capacity
• Cost Effectiveness
• Efficiency
• Environment
• Flexibility
• Global Interoperability
• Participation of ATM Community
• Predictability
• Safety
• Security

2.2.1.3. The main change laid in moving away from prescribing solutions to specifying desired performance criteria.

2.2.1.4. ICAO is using the following terminologies:

Focus Area

Focus areas may be defined as areas where performance must be addressed in a any given KPA. For example, in the safety KPA, focus may be in such areas as CFIT accidents, runway incursions, or mid-air collisions for general aviation aircraft.

Performance Objective

• Each expectation should be reached through a set of specific, measurable, achievable, relevant and timely (SMART) performance objectives.
• Performance Objectives define – in a qualitative way – a desired trend from today’s performance (eg improvement), within a well specified ATM planning environment. In other words it is a high level statement of outcome that satisfies ATM community expectations. Example : Enhance terminal airspace capacity.

Performance Indicator

Indicators are defined when there is a need to numerically document current performance levels and progress in achieving an objective. It is a measure of achievement of a performance objective. Example: increased number of aircraft in a defined terminal airspace.

Performance Target

A set of agreed numerical values of related performance indicators, representing the minimum performance levels at which an objective is considered to be ‘achieved’. Performance targets can only be specified after indicators have been defined. Example: Y number of additional aircraft in terminal airspace.

Performance Metric

A generic definition of what can be measured, how it can be measured and in which context and scope this should be done. It also defines the units in which the measurement is to be expressed. Metrics are a quantitative measure of system performance – how well the system is functioning The data needs to be collected to calculate values of performance indicators. Example: Traffic density, number of city pair flights.

2.2.1.5. The following diagram shows ICAO’s performance measurement approach:

2.2.2. ICAO EUR/NAT

2.2.2.1. ICAO’s EUR Region performance framework applies to a much larger geographical scope than the SES Performance Scheme. It’s use could be viewed by non-SES States as a low-effort first step towards adoption of a performance oriented approach, based on the knowledge and experience gained through regional processes in place and in particular the SES performance scheme. In that respect due consideration was given to the ICAO provisions as well as to the more stringent implementation of performance regulation like for example the SES Performance Regulations of the EU. Therefore the EUR Region initiative has a far bigger geographical reach, whereas the SES Performance Scheme shows a measurement scope that is extending much further.

2.2.2.2. After the need for a harmonized performance framework adaptable to the different contexts in the EUR Region was identified, areas where improvements are realistically possible were defined.

2.2.2.3. A list of useful, realistic and measurable indicators that are applicable in the whole EUR Region was set up, making sure that they can be reported by all States without huge efforts and can be implemented through a stepwise approach.

2.2.2.4. Out of the 11 ICAO KPAs, a task force identified a first set of KPAs and related objectives, KPIs and metrics that are selected for the implementation of the first phase of the Regional performance framework. These are:

• Safety
• Capacity
• Efficiency and Environment
• Cost-effectiveness
• Participation by the ATM Community

2.2.3. European Commission

2.2.3.1. Under the Single European Sky (SES) the European Commission launched the SES Performance Scheme. This was described in Art 11 of the 2nd package of the SES proposed amendment to Framework regulation (Regulation No 549/2004). Commission Regulation (EU) No 691/2010 contains the performance scheme for air navigation services and network functions.

2.2.3.2. As opposed to ICAO’s 11 Key Performance Areas the European Commission decided to consider only 4:

• Safety
• Capacity
• Environment
• Cost-Efficiency

2.2.3.3. The indicator for safety was not deemed mature enough at the time, which is why subsequently targets were only set for 3 KPIs.

2.2.3.4. Reference Period 1 (RP1) covers the first years under performance regulation, 2012 – 2014. The European-wide targets for this period were set at the following:

2.2.3.5. In the context of the Regulation, safety is mentioned at the forefront, but when it comes to indicators, it is falling behind for lack of maturity.

2.2.3.6. One part of the SES is the establishment of Functional Airspace Blocks (FABs), which are multinational groupings of ANSPs to provide services regardless of national boundaries in order better allow for more efficient routes and procedures. KPIs are also established for these FABs, as well as for each individual country as laid down in the national performance plans.

2.2.3.7. The baseline for RP1 and RP2 is 2009 traffic figures, when anticipated growth until 2014 was forecast at 15%, at ±8% for the high and low scenarios (source STATFOR – Eurocontrol Statistics and Forecast Service). This has been superseded by current developments. The European Commission has not taken these latest changes in traffic figures into account to change RP1 targets in the process.

2.2.3.8. Reference Period 2 (RP2) for the years 2015 – 2019 has been under intense discussion. While RP1 was meant as a start to performance regulation, RP2 will tighten the targets considerably.

2.2.3.9. On 4th February 2014 the Single Sky Committee agreed to the RP2 EU-wide cost efficiency target to be set at a 2.1% reduction per annum throughout the whole period. This is based on the low scenario STATFOR forecast of 1.2% annual traffic growth for the same period. It accumulates to a 3.3% reduction in the EU average determined unit rate. So far the cost-efficiency target comprises only enroute. With RP2 this will be extended to Terminal unit rates. As a notional target the PRB is suggesting alignment with the en-route targets, until the calculation of a terminal target in 2017.

2.2.3.10. For safety performance no binding targets are set. It is monitored only and expected to establish a target at the end of RP2.

2.2.3.11. The capacity target for the whole period 2015-19 is set at 0.50 minutes of delay per flight.

2.2.3.12. The environment target is divided into two sub-parts, horizontal en-route flight efficiency of the actual trajectory (KEA) and horizontal en-route flight efficiency of the last filed flight plan (KEP). KEA is set between 2.50% (lower bound) and 2.75% (upper bound). KEP will fall between 4.10% and 4.40%.

2.2.4. Performance Review Body

2.2.4.1. The Performance Review Body’s (PRB) purpose (European Commission Decision of 29.07.2010 on the Performance Review Body of the Single European Sky; the complete list of tasks is described in Article 3 of Regulation 691/2010 and Regulation 390/2013.) is amongst others to assist the European Commission in the implementation of the performance scheme. Two of the PRB’s key tasks include:

• advising the European Commission in setting EU-wide performance targets and assessing National / Functional Airspace Block (FAB) Performance Plans.
• monitoring the performance of the system in four Key Performance Areas: Safety, Capacity, Environment and Cost-Efficiency.

2.2.4.2. Thus the PRB enjoys a key function. Not only does it suggest EU-wide performance targets, but is also monitoring and publishing those in it’s annual Performance Review Reports and monthly dashboard updates.

2.2.4.3. As such it has recently released it’s suggestions for RP2, which are considerably more challenging compared to the RP1 targets.

2.2.5. IFATCA

2.2.5.1. Since the Toulouse Conference in 1998 IFATCA has continuously developed it’s policy on Performance Indicators. Current policy (WC 1.1.8) as adopted in Hong Kong in 2004 reads as follows:

2.2.5.2. When this policy was adopted in 2004, the establishment of performance schemes was on the horizon. Setting indicators and targets however was still far off. IFATCA foresaw that performance schemes bear certain risks for the front end users. By applying the policy above such risks should be avoided, or at least diminished.

2.2.5.3. Further publications by IFATCA on this subject are the “Statement On The Future Of Global ATM (27. Feb. 2007), SESAR: Mission Possible? SESAR expectations from the perspective of the ICAO Concept and IFATCA’s Statement on the future of Global ATM (12. Mar. 2007) and the IFATCA letter presented to Commissioner Jaques Barrot and the EU High Level Group in 2007, looking at the performance element from IFATCA’s view. More recently, IFATCA in a joint letter with ATCEUC, ECA and IFATSEA issued their statements to the Vice-President of the European Commission and Transport Commissioner Siim Kallas (11. June 2013).

2.2.6. Unions (ATCEUC, ETF)

2.2.6.1. ATCEUC sees the RP2 targets as unrealistic and meanwhile overambitious in view of unrealistic traffic forecasts. The environment target is considered as putting more pressure on the ATCOs without bringing any added value to the environment. The approach towards Safety KPA is even regarded as buying time without measuring the trade-off between safety, costs, capacity and the environment. They suggest the inclusion of staff representatives into the cnsultation process. The call is more for incentives rather than penalizations.

2.2.6.2. ETF fears that RP2 will jeopardize safety, number and quality of jobs in Europe. They reject a “performance scheme dominated by unrealistic targets and forecasts, which destabilizes the economic viability of a majority of ANSPs, and by cost reduction that mainly aims to cut on jobs”. Instead ETF wants jobs respected and created, whilst working conditions be improved. They see the aim in conituing cost reduction, where safety is not considered the highest priority. The pressure put on workers is seen as becoming unacceptably high. Social aspects are not addressed enough.

2.2.7. CANSO

2.2.7.1. CANSO (Civil Air Navigation Services Organisation) developed it’s own policy with regards to the SES Performance Scheme for RP2. In it’s position papers it is committing itself to the Single European Sky. Beginning the last position paper stating CANSO views it says:

“CANSO is committed to the Single European Sky:

• CANSO Member air navigation service providers (ANSPs) perform a vital role in the aviation value chain, providing fundamental services to deliver safe, punctual, cost- effective and more environmentally friendly air traffic flows and operations.
• The services provided by ANSPs rely on the skills and involvement of professional experts such as air traffic controllers (ATCOs), air traffic safety electronics personnel (ATSEPs) and technicians.
• We fully support the objectives of the Single European Sky: defragmenting European airspace to reduce delays, improve safety, reduce costs for airspace users and minimise the impact of aviation on the environment.”

2.2.7.2. As has to be expected from such an organisation it is also warning of overregulation and it’s effects, at the same time urging all players and their analyses to be respected. A further point of criticism is the work on RP2 had already started before lessons from RP1 could be learned and experience integrated.

2.2.7.3. CANSO expresses its view in the “Vision 2020” paper. It also developed a benchmarking report, which is however not publicly available.

2.2.7.4. Further criticism stems in the way cost targets are calculated. DFS for example is stating that the complete cost of training as well as the company pension scheme contribution are fully creditable towards the unit rate, while in other countries, where training is not done in-house, this is not contained and neither is pension as a matter of the state. This is distorting the basis of the calculation.

2.3. Aims for Setting Performance Targets

2.3.1. ICAO’s strategic vision for the ATM community was to foster the implementation of a global air traffic management system for all users during all phases of flight that would meet agreed levels of safety, provide for optimum economic operations, is environmentally sustainable, and meets national security requirements.

2.3.2. After its inception in 2004 it was to develop from the then ground-based air navigation system into a performance-based global ATM system by 2008. This was to be achieved through progressive, cost effective and cooperative implementation of air navigation systems worldwide. Consequently the ICAO Global ATM Operational Concept was developed and 11 KPAs defined.

2.3.3. In Europe since 1998, Performance Review Reports (PRR) have been established by the independent Performance Review Commission.The PRC was set up by Eurocontrol to “introduce a strong, transparent performance review and target setting system to facilitate more effective management of the European ATM system, encourage mutual accountability for system performance and provide a better basis for investment analyses.” Three KPAs were selected: safety, delay and cost-effectiveness.

2.3.4. With regards to the Safety KPA the PRR 1998 states, that “there is an urgent need to introduce a harmonised approach to national ATM safety performance reporting…”. At this stage, 15 years on, there still is no Safety KPI connected to the KPA.

2.3.5. Four years of experience with the Single European Sky taught the European Commission (EC), that performance could not be delivered if left to the discretion of the individual Member states, said former Director General Air Transport Daniel Calleja in 2009.

2.3.6. Therefore with the introduction of the second package of the SES, the European Commission strengthened the importance of the KPAs by underlying three of the four KPAs with clear targets. The non-achievement of those targets are to be penalized.

2.3.7. Along with their approval of the enlargement of the SES the EU transport ministers state the major goals in improving air safety, increasing the capacity of the airspace, reducing the cost of travelling, and reducing the negative environmental impact of avaition emissions. The fragmentation in Europe, they say, lags behind the U.S., resulting in approximately 5 billion EUR of damage.

2.4. Advantages and Risks

2.4.1. Advantages For Setting Performance Targets

2.4.1.1. There is a marked difference between Europe’s and the U.S.’s ATM infrastructure. Whereas there is a single provider in the FAA, Europe is, despite the European Union, divided into many countries, cultures, and subsequently many providers. Every state has its own interest to maintain its sovereignty over its own airspace (e.g. defence issues). Over the years every state developed its own model of operation and service provision.

2.4.1.2. This made streamlining, even a comparison very difficult. Every states respective ANSP saw it’s own interest at the forefront. The importance lay in the company’s own figures, whereas the neighbours’ were looked at with less emphasis.

2.4.1.3. Creating a performance scheme enabled a supra-national organisation, here the European Commission, to build a common base for national improvements, to better compare performance under varying economic systems which ANSPs operate under, and to enable a roadmap, along which the overall system is supposed to improve without regard to national boundaries.

2.4.1.4. Crucial to setting the targets right is a proper traffic forecast. Getting the expected figures wrong would make the targets either too easily achievable or unachievable. Neither way leads to the desired effect.

2.4.2. Risks Created By Performance Targets

2.4.2.1. There is a number of risks created by the setting of performance targets. The aim mainly focuses on satisfying the target. The performance in a majority of areas could be significantly better, but the concentration lies on fulfilling one target. Target achievements could be shifted. For example if one target gets satisfied easily, it’s performance could be sacrificed for the accomplishment of another.

2.4.2.2. An ANSP may possibly not survive a major negative event. 9/11 and it’s subsequent extreme downturn in traffic could leave providers seriously cashstrapped or insolvent. Investments could be delayed or cancelled altogether. A company in need to reduce expenditure will have to consider various ways of achieving this. Projects will need to be screened for their timing or necessity.

2.4.2.3. Staff incentives may be used for better motivation. This can be seen as positive, but is associated with a certain hazard. Staff – in this case ATCOs – might feel a desire to push too hard for their own benefit. In a worst case scenario this can lead to incidents, or even more serious, accidents.

2.4.2.4. Some cost efficiency initiatives may take too long to pay back. This can lead a provider to put off respective longer-term savings projects for the benefit of those with short-term effects.

2.4.2.5. A serious concern for the providers is the inability to influence some of the factors that are set in the KPIs an incentives. This is a view shared by CANSO and the Director General of Eurocontrol, Frank Brenner. Instead of setting the directions, targets are setting people in the direction of meeting a numerical target, while the desired system state does not necessarily get achieved. The system can’t produce an intended result, if the goal is defined badly. Targets can set a system in a direction that nobody wants. The assumption that people need an external motivator in the form of a target could end up in motivating for the wrong sort of behaviour.

2.4.2.6. Targets allow comparison, but experience shows that it often allows comparing false, manipulated or meaningless data. Unintended consequences of targets have been documented in various fields, as long as hundreds of years ago. (Hindsight 17: Target Culture: lessons in unintended consequences, Tony Licu & Steven Shorrock)

2.4.2.7. The conditions in ATM are subject to continuous change. The difficulty lies in timely adaptation. KPIs may not offer the very best understanding of the process. (Performance metrics and scenarios in ATM, Dr. Andrew Cook, University of Westminster, 10 January 2012)

2.4.2.8. As far as delay is concerned, the PRR 2010 states that:

“Despite … the large share of almost 50% reactionary delay, there is presently only a limited knowledge of how airline, airport and ATM management decisions affect the propagation of reactionary delay throughout the air transport network.”

(Performance Review Report 2010)

It warns of ATM not operating in a vacuum, dynamic context. Moreover possible scenarios over the lifetime of a Regulation, which have already been experienced, include the ash cloud of 2010 and severe winter disruptions. Another concern is for best practices being completely ignored in today’s performance measurement. An example are doorstep to destination times as vital elements in measuring. (Performance Review Report 2010)

Conclusions

Recommendations

References