48TH ANNUAL CONFERENCE, Dubrovnik, Croatia, 20-24 April 2009WP No. 84Description of What Constitutes PerformancePresented by TOC |
Summary
A performance based approach to Air Traffic Management (ATM) is increasingly affecting all areas of ATM. All controllers need to be aware of how performance management affects them and have a basic understanding of performance management, including underlying concepts such as metrics, functions and systems. Expert knowledge of performance is required by some controllers in order that changes can be argued for (or if necessary against) using performance methods and terminology. This paper recommends policy to ensure controllers are involved in performance management that affects controllers, including assessment of the validity of models and in the selection and use of metrics.
Introduction
1.1 This paper addresses three questions:
- What is performance in regard to Air Traffic Management (ATM)?
- How will performance measurement in ATM affect me as an air traffic controller?
- What should be IFATCA’s view on ATM performance?
Information is provided in general terms with the objective of providing clarification through information material that should be of benefit to all controllers.
1.2 This paper recommends that IFATCA establishes new Policy to ensure that the expertise of controllers is actively included in ATM performance management.
1.3 This paper also reviews the IFATCA Policy that states:
| “the measurement of performance shall reflect the impact of environmental constraints”. |
1.4 This working paper does not review the ICAO Manual on the Global Performance of the Air Navigation System (ICAO Doc 9883 Manual on Global Performance of the Air Navigation System) as this is being done this year by the Professional and Legal Committee (PLC). See also last year’s PLC working paper on “Performance Based ATM” Agenda C.6.10 WP 170.
Discussion
2.1 What is Performance?
2.1.1 There are three areas where performance is applied. The first is requirements for a particular performance in order for the system to operate as designed; for example that a particular performance navigation capability is required from aircraft, for example for determining spacing of navigation routes. This can be viewed as performance input to ATM system design. The second is management of change using performance; for example determining what changes will reduce delays or in comparing two service providers to determine which has the better service. This can be viewed as measure the performance outcomes or output. These are the major applications and are often dependent on each other; in other words performance requirements (inputs) often affect performance changes (outputs). In addition as the perspective and level of aggregation of components changes, a performance outcome (for example the number of aircraft interactions a controller can safely process) becomes an input to another aspect of system design (number of sectors required). Another aspect of performance management is the use of Performance Incentives.
2.1.2 There is much on-going work in relation to performance, and so the full meaning and consequences of what performance in ATM is all about will only be revealed over time. However there is no doubt that a paradigm shift has occurred and a performance based approach is being applied to all aspects of ATM. The ICAO Doc 9854 Global ATM Operational Concept states “a key tenet of the operational concept is performance orientation” (Appendix F 1.1). There is no longer any discussion about if performance management will be used; all discussion is about how performance management will be applied.
2.1.3 Safety has not been forgotten and is still the most important part of ATM performance – however it is becoming less common to discuss safety separately to other performance measures. (This is part of a systems approach, see paragraph 2.2.4.) ICAO lists eleven ‘key performance areas’ (KPAs) but still states “safety is the highest priority”. The ICAO list is contained in the Appendix.
2.1.4 ATM performance is a subject that is often mentioned however it is likely that it still means different things to different people.
2.1.5 There are many publications on performance, both generally and within ATM and there are plenty of introductions as to what performance-based management (or other performance issues) is about.
2.1.6 Despite all this, many controllers (and others) remain unclear as to what all this discussion on performance means for them. It is the intent of this paper to provide information that will assist controllers and others (for example ATM engineers, managers without controller expertise, etc.) when reading about and discussing ATM performance related issues.
2.2 Metrics, Functions and Systems
2.2.1 Performance can apply to just about everything in ATM. However, there are three broad underlying principles that need to be kept in mind whenever working on performance; these are metrics, functions and systems.
2.2.2 Performance is based around measurement. Measurement of present state, measurement of change from a previous measurement (improvement or regression), average performance over time (statistics), etc. A strong underlying assumption (most insist it is a requirement) is that this measurement is objective and not subjective, that is fact and not opinion. In other words a metric that represents the performance. The problems associated with metrics are discussed in paragraph 2.6.
2.2.3 A companion concept, often included when discussing performance, is the concept of functionality rather than specifics. For example, instead of insisting via regulation that the aircraft carries an inertial reference system (IRS) for navigation, the regulation is that a certain navigation performance is met (the functionality) irrespective of what equipment is used to meet that standard. This has been widely applied outside ATM and is only of more recent times being applied within ATM. For example ICAO Doc 9613 Performance Based Navigation (PBN) Manual adopts this concept.
On page iii it states:
“Under PBN, generic navigation requirements are defined based on the operational requirements. Operators are then able to evaluate options in respect of available technologies and navigation services that could allow these requirements to be met.”
2.2.4 The third and final principle (or companion concept) is a systems approach. That is performance of a function is not in isolation, for it is part of a system. For example, if you only concentrate on one issue (for example capacity) then it can affect other performance areas (such as safety). This has led to systems where several metrics are monitored simultaneously so that the overall consequences on the system can be monitored.
2.2.5 There are also systems within systems, both within ATM and external systems affecting ATM; for example ATM cannot act as if it existed separately from the rest of the world, for example legislative requirements from States (not ATM regulators), local governments, etc. regarding environmental performance.
2.2.6 Within ATM there are systems within systems, for all aspects of performance management (including collaboratively setting performance objectives). For example there can be consideration of just at an airport level; however if there are complex interactions for example on a high-density short-distance route a wider approach may be required and so each airport and the intervening airspace may need to be considered together as a “system”.
2.2.7 By keeping in mind the three underlying principles of metrics, functions and systems then many misunderstandings and pitfalls of performance work can be minimised.
2.3 Performance Changes (and Comparisons)
2.3.1 Performance-based management is widely used. Oak Ridge Associated University defines performance-based management as:
“a systematic approach to performance improvement through an ongoing process of establishing strategic performance objectives; measuring performance; collecting, analysing, reviewing, and reporting performance data; and using that data to drive performance improvement”
2.3.2 ICAO Doc 9883 Manual on Global Performance of the Air Navigation System on page ii states:
“The performance-based approach (PBA) is based on the following principles: strong focus on desired/required results through adoption of performance objectives and targets; informed decision making, driven by the desired/required results; and reliance on facts and data for decision making. Assessment of achievements is periodically checked through performance review, which in turn requires adequate performance measurement and data collection capabilities”.
In other words, measure where you are, determine where you want to be, identify the difference, make the changes, and then measure again to see if you have achieved what you wanted. The cycle then repeats.
2.3.3 The performance is not just the service provider performance but on the performance of all within the ATM community. For example airspace user performance can affect service provider performance and vice-versa.
2.3.4 For this working paper, performance comparisons are included within the area of performance changes. This is because ATM performance comparisons should not be simply about seeking to establish who (or what) is best, but rather to establish a means of ensuring the best ATM system (that is performance change for the better).
2.3.5 The Civil Air Navigation Services Organisation (CANSO) has a working group dedicated to benchmarking.
According to the CANSO web site (https://www.canso.org):
“One of CANSO’s key strategic goals is to support harmonization of global ANS and improve ATM service delivery. Global benchmarking of ANSP economic, safety, environmental, and quality performance measurements remains one of the most effective management tools to support improvements in ANS service provision.” and “The aim of the (CANSO Global Benchmark Working Group) is to establish effective and robust processes to support the production of a timely annual global performance report covering a comprehensive set of meaningful performance measures. These annual reports provide a reasoned comparison between and among the world’s ANSPs and also facilitate the identification of best practices among ANSPs so that all CANSO members can directly benefit from this resource.”
2.3.6 Some regions and states have made significant progress in measuring ATM performance. The Performance Review Commission (PRC) was established in 1998.
The PRC:
“monitors the performance of ATM in Europe and makes recommendations for action to the Governing Bodies of EUROCONTROL. It main tasks are to monitor and analyze ATM performance, target setting and guidelines for economic regulation”.
More information on the PRC, including reports, is available from their web site: https://www.eurocontrol.int/prc.
2.3.7 Whether it is for performance changes or performance comparison, a comprehensive set of metrics that can be applied at all levels (global, regional, state and lower levels) and over time is needed. This is so one set of performance results can be compared against another set of performance results. In other words, to ensure that as the saying goes “you are comparing apples with apples”.
2.3.8 It is expected that the process of setting strategic performance objectives and selecting what changes are needed will be part of collaborative decision making within ATM. In other words, where the setting of a performance objective affects a member of the ATM community then it is expected that the affected member would participate in the setting and assessment of that objective. For example, if it is determined that a particular capacity is required, then it will not simply be an imposition for example of a more precise aircraft navigation performance but rather an assessment of all possible solutions to increasing capacity and assessments of their consequences (performance cases).
2.3.8.1 Note: Collaborative objectives are used where the subject is the performance of the whole ATM system. It will still be possible for performance based management to apply within individual members of the ATM community (for example service providers, airspace users, etc.) as part of their internal management and so the measures and decision will be internal data and not necessarily shared. For example, if a service provider desired to increase their own ability to provide capacity independently to other collaboratively agreed performance objectives then this would not have to be done collaboratively (provided collaboratively agreed objectives were not affected). The service provider would not be required to share this information with other service providers, etc. (for example some data may be considered commercially sensitive) – once again provided it does not affect any collaboratively agreed data sharing agreements. The service provider of course can choose to share data – but this note is stressing the fact that only certain data is required to be shared and only common objectives need to be collaboratively agreed and the rest of the data may or may not be shared depending on the ATM community member’s choice. This relates to performance management data as well as any other data.
2.3.9 Eleven KPAs have been identified for ATM based on ATM community expectations from ICAO Doc 9854 Global ATM Operational Concept. This illustrates the system nature of ATM and that changes in one area can affect one or more other areas of ATM. The IFATCA Statement on the Future of Global ATM gives examples of how each of these Performance Areas may start to be addressed.
2.3.10 It is significant to note that the issue of metrics is an on-going issue and that the ICAO Global Performance Manual at this stage only has examples of metrics. This work is currently assigned to the ICAO ATM Requirement and Performance Panel (ATMRPP) but is currently not being progressed due to other work priorities. On its current program the ATMRPP will not resume work on metrics until 2011.
2.4 Performance Requirements
2.4.1 Performance requirements are for a particular performance in order for the system to operate as designed. For example, in designing air routes there may be requirements on navigation by the airspace users.
2.4.2 An example is the Performance Based Navigation (PBN) Manual (ICAO Doc 9613).
The Executive Summary states:
“The PBN concept represents a shift from sensor-based to performance-based navigation. Performance requirements are identified in navigation specifications, which also identify the choice of navigation sensors and equipment that may be used to meet the performance requirements.”
This is the functions (or functional) approach to performance requirements.
2.4.3 Another example is the Manual on Required Communications Performance (RCP) (ICAO Doc 9869).
Paragraph 2.2.1 states:
“The RCP concept characterizes the performance required for communication capabilities that support ATM functions without reference to any specific technology and is open to new technology.”
2.4.4 One view of the ATM system has been ‘Communication Navigation Surveillance / Air Traffic Management’ (that is CNS/ATM). At one stage it was considered that the performance of the whole ATM system could be defined as the aggregate of the required performance of each section, in other words required communication performance, required navigation performance, required surveillance performance and even Required ATM System Performance (RASP) that would lead to a Required Total System Performance (RTSP). The task of defining RASP and RTSP was considered as part of developing ICAO Doc 9854 Global ATM Operational Concept (Appendix F). It is now unclear if further work will continue with this approach.
2.4.5 What will continue is the functional approach to performance requirements allowing the ATM system to be designed and evolve using global defined performance measures that are not dependent on a particular type of equipment but rather the functional performance of that equipment. This will enable new equipment to be introduced without necessarily requiring a re-design of the ATM system.
2.4.6 ICAO Doc 9854 Global ATM Operational Concept defines the ATM system as:
“A system that provides ATM through the collaborative integration of humans, information, technology, facilities and services, supported by air and ground- and/or space-based communications, navigation and surveillance”.
Required performance could be applied to each of these elements of the ATM system.
2.4.7 In relation to required human performance, ICAO Doc 9854 Global ATM Operational Concept Appendix F paragraph 2.5.23.m states:
“human performance: humans within the ATM system must demonstrate appropriate competence at all levels. Such competence should be continuously monitored and frequently tested, and competence regimes should be established sufficient to provide an assurance of human performance. Appropriate mechanisms may include licensing, certificates of competence, technical qualifications, training certificates or similar”.
2.4.8 An emerging concept is ‘Capability Levels’. Capability level could be defined as:
“a set of functional requirements necessary to be able to supply or use a particular service”.
2.4.9 A capability level represents a set of functions. The functions will have to eventually be associated with required levels of performance for each of the functions. This set of functional performance requirements will be applied to all affected members of the ATM community (that is applying to service providers as well as airspace users). Examples are included in the IFATCA Statement on the future of Global ATM Appendix 14.
2.4.10 The European ATM modernisation program, the Single European Sky – ATM Research (SESAR) project, introduced the concept of capability levels in its deliverable D3 and these have been translated into six levels of service in deliverable D5. Service level 1 is “preparing for trajectory based operations” and progresses to service level 5 “accommodating full 4D trajectory management based on user preferred trajectories”.
2.4.11 “Capability Levels” are a way of expressing systems within systems. There is the overall ATM system. This overall ATM system can be broken down into separate individual functions. However these individual functions are rarely applied in isolation, and so a set of functions can be grouped together to form a ‘system within a system’. For example consider navigational accuracy – you can have horizontal, vertical and time as separate functional requirements but really navigational is a collection of each of these functions. (It is interesting to note that ICAO Doc 9613 Performance Based Navigation Manual has yet to address the issue of time.)
2.4.12 The example of navigation alone is not enough for capability levels as navigation requirements are just one part of ATM requirements. For example sharing of information is very important in future ATM concepts, so the ability to share this information with others (both service providers and other airspace users) may become part of a defined ‘Capability Level’ (or functional set). In other words, it is not enough just to meet navigation performance requirements, as there is also a need to be able to concurrently meet the communication requirements too. So the particular capability level must include within the one set of requirements the navigation and communication performance requirements.
2.4.13 Performance requirements should in the future arise from detailed ATM system design. In the past performance requirements were supposedly based on a design requirement – but there are many examples where this can be questioned. For example the requirement to carry Mode S transponders – was this because of need or because of the availability of new technology that might be able to address some needs (or wants). Those who claim it is based on needs should be able to answer the question “if it was needed by a date, then why is the requirement only on the airspace users and not on the service providers too?”. In other words if the system needs it, then both airspace users and service providers need to use it by the date.
2.4.14 So performance requirements will be not because something is possible or is better (for example simply more accurate) but because there is a benefit to the ATM system from achieving the performance required. This is where there is overlap and synergy from performance requirements and performance changes (performance-based ATM). In other words, it will be possible to measure the benefit (or not) of the change in a performance requirement.
2.4.15 A final note on ATM system design. It would be wrong to think of ATM system design as creating a single solution, which is one homogeneous state (even at the functional level). Future ATM concepts are about mixed modes and tailoring services to individual aircraft performance. Therefore the functional requirements for performance will vary depending on aircraft and services delivered. This is true in a very limited way today (performance requirements can change between controlled airspace, uncontrolled airspace, between VFR and IFR rules) however we can expect many more variations in future based on varying metrics – hopefully all within a comprehensive ATM system design. It may even be that the future services will be based on the functional performance of the aircraft – and that the performance requirements will be on the service providers (based on what the aircraft can do)!
2.5 Performance Incentives
2.5.1 Performance incentives can take a number of forms; for example there can be salary payments or working conditions based on performance or there may be benefits for airspace users (or others) for achieving a particular performance level early in a process (that is “an equip early incentive”).
2.5.2 IFATCA policy is that:
| “Performance indicators as published and used by Air Navigation Service Providers must not be linked in any way to the pay and/or working conditions of individual Air Traffic Control Officers”. |
IFATCA considers that those in operational positions should make decisions based on the facts and not have their judgement potentially compromised by pay or working conditions considerations. This applies in both positive and negative senses – that is neither reward nor punishment for a particular performance indicator.
2.5.3 The issue of incentives for particular performance levels can apply to airspace users, service providers, aerodrome operators, etc. This is performance in relation to a capability level and is usually for reaching a capability level sooner rather than later. However, in future ATM it is expected that aircraft will be able to use services as they become capable (rather than waiting for a particular implementation date for all users) and so there will be benefits even without special incentives. This will require the service provider to change first, which is before the first airspace user wishes to use the service.
2.6 The problem of metrics
2.6.1 ICAO Doc 9854 Global ATM Operational Concept requires that metrics are S.M.A.R.T. – that is Specific, Measurable, Accurate, Reliable and Timely.
2.6.2 ATM metrics is a very big problem. It is very difficult to select metrics and it is problematic to apply them.
2.6.3 Air traffic control has evolved as a craft and is slowly transforming into the science of air traffic management. What this means is that a lot of ATM has been done by judgement, whether it is for major system changes or in daily operations. There has been accumulated experience, but few definitive measurements. In other words, things are considered significant to controllers (based on experience and judgement) that cannot yet be expressed as a number (or metric). Performance orientation of ATM is the transforming agent – but it will be a long and difficult process. A series of examples follow that will illustrate this.
2.6.4 Consider the case that controllers readily accept that 5 Nautical Miles (NM) by radar is a safe and acceptable separation standard. This has been used for a considerable time. However how do you prove (by mathematics) that it is safe? What are the underlying assumptions that are not explicitly stated? For example is 5 NM by radar equally safe for opposite direction traffic as same direction traffic? When is it safe to reduce to 3 NM? What is necessary for using 1 NM separation (for example definite passing by radar)?
2.6.5 Consider the case where traffic wishes to offset, that is fly parallel to a route to avoid flying over the centreline with other traffic on the route. The ICAO Separation and Airspace Safety Panel (SASP) decided that in oceanic procedural airspace a 1 NM offset was safe but a 2 NM offset was unsafe – despite track spacing of 60 NM. This position was subsequently revised, so that 1 and 2 NM offsets are possible. The problem lies within the modelling that is used. The modelling involved is a mathematical process and it is not intended to debate the merits or issues of the modelling involved – however in any modelling assumptions are made which means that the model does not completely reflect the real world in all aspects. George E. P. Box (Professor Emeritus of Statistics at the University of Wisconsin) is often quoted as saying “essentially all models are wrong – but some are useful”. The importance of modelling is to ensure that within the scope in which the model is used that it correctly reflects what is being modelled. The role of the controller in such situations is not as an expert mathematician or modeller, so the controller’s task is not to argue how the model is constructed and calculated. The role of the controller is as an expert in ATM and able to comment on how the system is observed to behave. It is then up to the mathematicians and modellers to adjust their models so that the model reflects the behaviour of the ATM system. In the quoted example above it was not considered reasonable that the modelling showed that a 2 NM offset was unsafe.
2.6.6 Consider the simple case of whether a separation minimum is infringed or not. Automatic recorders were installed at Heathrow, however a problem was identified. The aircraft were accurately spaced in trail, but when the automatic system measured the distance between an aircraft on final and an aircraft on base the distance was less than the radar standard (even though the distance to run between the aircraft was the radar standard). So even “simple” performance measures have unexpected consequences.
2.6.7 Consider the case where a politician promises to “improve ATM safety by 10 times”. Is the ATM system necessarily safer when this 10 times metric is achieved? The performance measure may become so difficult to achieve that new equipment cannot be certified, and so the safety benefits of the new equipment are never installed. Also existing equipment and standards may be deemed to be “no longer safe” and so the controller is forced to work with a more limited set of equipment using larger separation standards, and so the controller is overloaded. If the metrics do not correctly factor in the controller workload then even a metric that “proves” the system is 10 times safer does not in reality provide a safer system – in fact safety may decrease.
2.6.8 These examples reflect real problems. In investigating the safety of Reduced Vertical Separation Minima (RVSM) in Europe, questions were raised as to the safety of using 1000 feet below Flight Level (FL) 290. If a safety standard is set for RVSM, should it be any lower for other vertical separations?
2.6.9 Consider the issue of controller workload. A common metric is the number of aircraft in the sector. However the number of aircraft is not the critical factor, it should be the number of controller interactions required. For example if a sequence of aircraft is going through a sector and the slower aircraft transfer to the sector before faster following aircraft then the controller’s workload would be higher for the same number of aircraft than if the reverse occurred. Another problem with the metric is the time period, if the workload is managed so that there are x number of aircraft (or x number of controller interactions) in y minutes, then it can still create a workload problem if say 90% of x aircraft/interactions occurs in the first 10% of y time. So it is not just the number of interactions but also the time available in which to do them.
2.6.10 Does this mean we should not have metrics? No. We should in fact have more metrics to help us understand what is happening in ATM and to make informed decisions. For example, it is surprising that there are requirements for the carriage of Aircraft Collision Avoidance System (ACAS) and mandatory requirements to follow resolution advisories – however there are no statistics kept on the total number of Traffic Advisories, Resolution Advisories, number of aircraft (or total time) operating without ACAS (using the delay allowed before it must be functioning again), etc.
2.6.11 Wikipedia states “Lies, damned lies, and statistics” is part of a phrase attributed to Benjamin Disraeli and popularized in the United States by Mark Twain: “There are three kinds of lies: lies, damned lies, and statistics.” The semi-ironic statement refers to the persuasive power of numbers, and succinctly describes how even accurate statistics can be used to bolster inaccurate arguments”. So even just getting accurate statistics is not enough. The problem is that the accurate statistics can still only report on part of the system, and so what is needed is ‘the whole truth’.
2.6.12 The IFATCA Statement on the Future of Global ATM states:
“Measuring ATM performance at this time seems to be both a science and an art – for getting the right measures and interpreting them correctly is difficult”.
2.7 Performance Based not Driven
2.7.1 ICAO Doc 9854 Global ATM Operational Concept does not specifically refer to terms performance based or performance driven. The Concept refers often to performance and that a key tenet is “performance orientation”.
The Concept states:
“it is crucial that the evolution of the global ATM system be driven by the need to meet the expectations of the ATM community” (includes safety as the highest priority) and “the ATM system performance requirements should always be based on the key understanding that the ATM system is the collective integration of services, humans, information and technology”.
2.7.2 The term ‘performance driven’ is often used by those seeking to change ATM. Basically this means that if the metric is not achieved then the ATM must change – that is the metric drives or demands the change. IFATCA should use, and insist others use, the term “performance based”. This reflects that performance should be measured and will strongly influence decisions. However the term “based” is used to indicate that the overall objective is not a particular metric, in other words that the metric is actually intended to support an outcome other than just the metric. As the examples above indicated, the selection and application of metrics is difficult. If a particular metric is not met, then performance based means that the suitability of the metric can be reconsidered, including how this metric is considered within a system sense – that is how other metrics are affected too.
2.7.3 A major concern of performance driven approach is when a particular mathematic model or metric “proves” that something is safe. The premise is “proving” something is safe is a major application of performance orientated ATM so surely it can be trusted. The answer is that models and the results from modelling should only be trusted so far and never trusted without question. Precise mathematical proofs can appear convincing – but the real issue is how accurately the model or metric reflects the real world, the integration of all the parts and allowance for all the implied conditions. There are several techniques for testing models including sensitivity to changes (either being too sensitive to change, for example 2 NM offset was unsafe although 1 NM was safe in the offset example above – or by not being sensitive enough and so no appreciable change even with significant variation in values). So it is possible to test if models respond in a way controllers observe the ATM system behaves without too much difficulty or complexity. Controllers need to be part of the process of establishing and verifying ATM models and will require the support of experts, such as mathematicians.
2.7.4 This use of models and metrics becomes even more significant with increasing ATM automation and controller support tools. The models and metrics seem ideal resources for creating software. What IFATCA needs to be cautious of is a premature software application that has operational consequences based on an incomplete or inappropriate model.
2.8 What will this mean for controllers?
2.8.1 Data gathering will increase significantly. This can be manual or automatic collection of data. Manual collection has workload implications. Automatically collected data may still require some manual processing.
2.8.2 As performance management will affect all areas of ATM, all controllers will need to be able to understand basic concepts of performance management and be able to communicate using performance management terms. This has implications on training for controllers, both initial training and training for existing staff. There will be a need for some controllers to become experts in performance management.
2.8.3 All the data will require interpretation and controllers will need to be able to assist with this.
2.8.4 Controllers can expect to have their performance reviewed (individual and collective performance) using statistics (performance data), and not only by the service provider. While it is unlikely (at this stage) that organisations outside the service provider will be able to monitor individual performance, it will increasingly be possible for performance of ATC sectors or units using automated data gathering and analysis software. For example, there are already systems being sold to airlines for monitoring aerodrome activity. They can be multilateration or ADS-B surveillance. The analysis of the gathered data is usually much more than is used by service providers. It can track both company and non-company traffic, giving taxi times, speeds and routes, where aircraft were held on the ground and for how long, etc. So whether or not service providers gather the data, airspace users (or noise lobby groups, etc.) may collect and analyse data and use it to push for changes in ATM.
2.8.5 With increasing automation in ATM there will be more attempts to create models of controller behaviour. Controllers will need to be able to assist and review how this modelling is done.
2.8.6 In regard to performance changes, controllers will need to be able to explain the reasons for changes being required (or changes being inappropriate) in terms of performance based management. As mentioned earlier, with experience and judgment having been the hallmarks of air traffic control the paradigm shift to being able to explain what is important and significant in terms of metrics is going to be very challenging. For example, consider how controllers will feel when something is implemented that the mathematics proves is safe but that many controllers feel is unsafe. This is not easy because sometimes the mathematics/model/design may be correct and the controllers wrong. For example, consider the initial reluctance of pilots to accept fly by wire aircraft and the change in some fundamental beliefs of the pilots to accept fly by wire. This is not an argument to accept change without question – questioning is important especially if it is contrary to your understanding however accept that in a changing world sometimes it is ourselves who must change.
2.8.7 As performance requirements are increasingly based on functionality rather than equipment, controllers will not need to be concerned about exactly what equipment is used, only the performance required. This will result in a simplification of data that needs to be displayed and a reduction in “mixed mode” combinations in the near to mid- term. This assumes that the controllers will not have to take any action or make any decision based on what particular equipment is on the aircraft or what the flight crew are authorised to use (that is the controllers will act based on the functional performance as opposed to specific equipment). However in the long term it is expected that automation will make use of individual aircraft performance so that the services delivered will be tailored for that particular aircraft (which will create more mixed mode operations).
2.9 What should IFATCA’s position be?
2.9.1 IFATCA should support the performance-based approach to ATM. At the very least, it will provide a means of communicating the needs of air traffic control to other professions that make decisions affecting ATM. The need to be able to communicate is significant as important decisions regarding ATM are being made by politicians, lawyers, accountants and other professions that do not have detailed understanding of ATM. This is already true in airlines where for example management decisions are not just by the Chief Pilot. IFATCA can use the effort being put into performance orientated ATM to better explain the importance and role of controllers.
2.9.2 IFATCA should insist on controller involvement in the setting of metrics that measure ATM performance, especially those relating to controller performance. This includes reviewing how well models reflect real-world ATM behaviour.
2.9.3 IFATCA should insist on controller involvement in the interpretation of collected data, and be ready to identify where data is misleading because it is incomplete or incorrectly applied.
2.10 Policy Review
2.10.1 IFATCA policy is:
| “the measurement of performance shall reflect the impact of environmental constraints.” |
2.10.2 The reason for this policy is not explicitly stated in the original working paper (Cancun 2002 – working paper 91), however various problems with imposed environmental constraints are mentioned.
2.10.3 This policy is unique in the discussion on ATM performance as it can refer to the requirements of external systems on the ATM system, and external systems effect on performance is not reflected in ICAO documents yet.
2.10.4 The policy can also refer to actions that the ATM system takes in order to behave responsibly in regard to environmental issues. Environment is one of the key performance areas used in ATM performance management and its effects are balanced by ATM with the other key performance areas (such as safety). This is mentioned in ICAO documents (for example, ICAO Doc 9883 Manual on Global Performance of the Air Navigation System).
2.10.5 The value of IFATCA Policy is therefore in highlighting the effect of external systems on the ATM system. An example is where government (not the ATM regulator) imposes requirements on noise reduction. If these regulations require actions that go beyond the actions that ATM would otherwise have done to behave responsibly then there is value in highlighting the additional consequences of this external to ATM requirement.
2.10.6 The ATM Regulator is considered part of the ATM community and so environmental requirements established by a regulator can be seen as an internal ATM requirement. It may well be that the regulator imposes regulations that are from pressure from other sections of government, but it is hoped that the regulator would be aware of the consequences on ATM, especially safety, of any environmental or other regulations.
2.10.7 An example of the types of external requirements on ATM is where a local government (for example a city or an area) imposes requirements on ATM, for example when runways can be used, without due understanding of the consequences on ATM.
2.10.8 Of course ATM exists in the wider world and so cannot expect to be independent from external (wider community) requirements. However, if the consequences of external requirements are made visible in the measurement of ATM performance then ATM can respond. For example, if a local government imposes requirements that mean that a runway cannot be used at a particular time due noise, then if this is the runway with an instrument approach with glide slope guidance and the alternative runway that must be used only has a non-precision approach then ATM can lobby (educate) the local government of the consequence and get the requirement changed – or more likely decide to install glide slope guidance on the other runway (as ICAO now requires).
2.10.9 It is also true that ATM is in a process of continual learning and adjustment. It was once accepted that 25 knots crosswind was acceptable for noise abatement, and now ICAO has stated 15 knots crosswind as acceptable. One issue that arose was that aircraft manufacturers advised that even if an aircraft type was certified for a particular maximum crosswind it was not considered that this maximum would be routinely and repetitively used. For example the landing gear may weaken with repeated exposure to maximum crosswind landings. This again highlights the problems of metrics, that a stated figure can have certain implied conditions associated with it.
2.10.10 The Policy is specifically about the environment, but there are other external system requirements possible. For example requirements on cost recovery/profit, on competition for provision of ATM services, etc.
2.10.11 It is recommended that the Policy be reworded to reflect the measurement of all of external requirements on ATM performance.
The proposed wording is:
“the measurement of performance of the Air Traffic Management System shall reflect the impact of any external-to-ATM constraints, including external environmental constraints”.
2.10.12 The IFATCA Technical and Professional Manual currently has no separate key word for performance, however this is a subject of much current and future work. It is therefore recommended that a new key word “Performance of the Air Traffic Management System” be established under the classification Air Traffic Services (ATS).
Conclusions
3.1 Performance orientated ATM is a key tenet of ICAO Doc 9854 Global ATM Operational Concept and is the subject of much on-going work. IFATCA should strongly support this work and use it to explain the importance and role of controllers.
3.2 Performance is mainly in relation to performance managed change and design performance requirements. Another application is performance incentives. IFATCA needs to be able to argue for (or against) change using performance terminology and cases. IFATCA Policy on performance incentives remains valid.
3.3 Performance should be considered bearing in mind three underlying concepts or principles, these are metrics, functions and systems. Functions can be grouped into capability levels. IFATCA should ensure that work on performance correctly incorporates these concepts.
3.4 The selection and use of metrics (and any associated mathematical models) is still both a science and an art. IFATCA should ensure that selection, interpretation and use of metrics that affect controllers includes controllers. Controllers should use their experience and judgement to ensure that the metrics and models reflect the actual way the ATM system operates.
3.5 It is important that IFATCA continues to stress through Panel representatives and presentations that a global set of metrics for Air Traffic Management needs to be established for application at global, regional, state and lower levels.
3.6 IFATCA should challenge any outcome from modelling that is not consistent with the controller’s understanding of how ATM functions.
3.7 IFATCA Policy on environmental constraints should be revised to reflect the effect of any external systems on ATM performance.
Recommendations
It is recommended that;
4.1 IFATCA Policy is:
Global metrics for the performance of the Air Traffic Management System be developed through ICAO processes as soon as possible.
And is included on page 4 1 1 8 of the IFATCA Technical and Professional Manual.
4.2 IFATCA Policy is:
Controller expertise must be used in the establishment and settings of metrics that measure the performance of the Air Traffic Management System.
And is included on page 4 1 1 8 of the IFATCA Technical and Professional Manual.
4.3 IFATCA Policy is:
Controller expertise must be used in establishing and reviewing models used for determining performance of the Air Traffic Management System to ensure that the models accurately reflect how the ATM system functions.
And is included on page 4 1 1 8 of the IFATCA Technical and Professional Manual.
4.4 IFATCA Policy is:
Controller expertise must be used in the interpretation of data used to assess the performance of the Air Traffic Management System to ensure that data is not misleading because it is incomplete or incorrectly applied.
And is included on page 4 1 1 8 of the IFATCA Technical and Professional Manual.
4.5 IFATCA Policy is:
The measurement of performance of the Air Traffic Management System shall reflect the impact of any external-to-ATM constraints, including external environmental constraints.
And is included on page 4 1 1 8 of the IFATCA Technical and Professional Manual.
4.6 IFATCA Policy on page 3 2 3 25 of the IFATCA Technical and Professional Manual:
In the operation, maintenance and development of the ATM system:
– when balancing the requirements of safety, efficiency and the environment, the level of safety shall always be maintained or improved.
– the measurement of performance shall reflect the impact of environmental constraints.
is amended to read:
In the operation, maintenance and development of the ATM system when balancing the requirements of safety, efficiency and the environment, the level of safety shall always be maintained or improved.
Appendix
ICAO Doc 9854 Global ATM Operational Concept Appendix D lists eleven ATM Community Expectations which have become the basis of Key Performance Areas (KPAs) for ATM.
“ICAO Doc 9854 Appendix D
EXPECTATIONS
Key to the operational concept is a clear statement of the expectations of the ATM community. The expectations for the global ATM system have been discussed among members of the ATM community in general terms for many years. These expectations stem from efforts to document ATM “user requirements”. The expectations hereafter are interrelated and cannot be considered in isolation. Furthermore, while safety is the highest priority, the expectations are shown in alphabetical order as they would appear in English.
Access and equity
A global ATM system should provide an operating environment that ensures that all airspace users have right of access to the ATM resources needed to meet their specific operational requirements and that the shared use of airspace by different users can be achieved safely. The global ATM system should ensure equity for all users that have access to a given airspace or service. Generally, the first aircraft ready to use the ATM resources will receive priority, except where significant overall safety or system operational efficiency would accrue or national defense considerations or interests dictate that priority be determined on a different basis.
Capacity
The global ATM system should exploit the inherent capacity to meet airspace user demands at peak times and locations while minimizing restrictions on traffic flow. To respond to future growth, capacity must increase, along with corresponding increases in efficiency, flexibility and predictability, while ensuring that there are no adverse impacts on safety and giving due consideration to the environment. The ATM system must be resilient to service disruption and the resulting temporary loss of capacity.
Cost-effectiveness
The ATM system should be cost-effective, while balancing the varied interests of the ATM community. The cost of service to airspace users should always be considered when evaluating any proposal to improve ATM service quality or performance. ICAO policies and principles regarding user charges should be followed.
Efficiency
Efficiency addresses the operational and economic cost-effectiveness of gate-to-gate flight operations from a single-flight perspective. In all phases of flight, airspace users want to depart and arrive at the times they select and fly the trajectory they determine to be optimum.
Environment
The ATM system should contribute to the protection of the environment by considering noise, gaseous emissions and other environmental issues in the implementation and operation of the global ATM system.
Flexibility
Flexibility addresses the ability of all airspace users to modify flight trajectories dynamically and adjust departure and arrival times, thereby permitting them to exploit operational opportunities as they occur.
Global interoperability
The ATM system should be based on global standards and uniform principles to ensure the technical and operational interoperability of ATM systems and facilitate homogeneous and non- discriminatory global and regional traffic flows.
Participation by the ATM community
The ATM community should have a continuous involvement in the planning, implementation and operation of the system to ensure that the evolution of the global ATM system meets the expectations of the community. The ATM community is more fully defined in Appendix A.
Predictability
Predictability refers to the ability of airspace users and ATM service providers to provide consistent and dependable levels of performance. Predictability is essential to airspace users as they develop and operate their schedules.
Safety
Safety is the highest priority in aviation, and ATM plays an important part in ensuring overall aviation safety. Uniform safety standards and risk and safety management practices should be applied systematically to the ATM system. In implementing elements of the global aviation system, safety needs to be assessed against appropriate criteria and in accordance with appropriate and globally standardized safety management processes and practices.
Security
Security refers to the protection against threats that stem from intentional acts (e.g. terrorism) or unintentional acts (e.g. human error, natural disaster) affecting aircraft, people or installations on the ground. Adequate security is a major expectation of the ATM community and of citizens. The ATM system should therefore contribute to security, and the ATM system, as well as ATM- related information, should be protected against security threats. Security risk management should balance the needs of the members of the ATM community that require access to the system, with the need to protect the ATM system. In the event of threats to aircraft or threats using aircraft, ATM shall provide the authorities responsible with appropriate assistance and information.”
Last Update: September 29, 2020