Industry Engagement Request: Questionnaire 1 - Assessment Methodology

Meeting with Companies: May 03, 2013

Purpose

• To provide companies with details on the assessment methodology that will be used in assessing your responses to Questionnaire 1
• To provide clarifications and provide answers to questions
• To provide companies with information on next steps
• We are committed to an open, fair and transparent process

Industry Engagement Request Questionnaire 1

• Canadian fighter mission and threat information across two time horizons (2020-2030 and 2030+).
• Responses to include defined mission configurations:
  • Air-to-Air;
  • Air-to-Ground; and
  • Air-to-Surface.
• Responses to include capability developments (planned upgrades).
• Responses to include descriptions of capability, production and supportability.

Principles

The following Principles will guide all participants involved in the assessment of the responses:

• Consistency
  • Team leads will ensure that the assessment is well documented, including all qualitative and professional military judgement aspects.
  • Individual aircraft are being assessed against each Canada First Defense Strategy (CFDS) mission.
  • No aircraft will be screened out. All options will remain on the table. Final product will be a high level risk assessment.
• Objective
  • The National Fighter Procurement Secretariat will provide oversight through all steps of the assessment process
• Use of Third Parties
  • Responses will be shared with the Independent Review Panel (i.e. Panel) members and any significant differences of opinion between assessment teams will be brought to the attention of the Panel.

Overview of Q1 Assessment

• The "Capabilities" questions of the Industry Engagement Request will be assessed using a 3-step assessment process:
  • Step 1 – Assessment of Measures of Performance (MOP);
  • Step 2 – Assessment of Measures of Effectiveness (MOE); and
  • Step 3 – Assessment of Aerospace Capabilities weighted against Canada First Defense Strategy (CFDS) Mission and criticality to result in "Operational Risk" assessment of each aircraft for each of the missions.
• The "Production and Supportability" questions will be assessed using a 2-step assessment process:
  • Step 1 – Assessment of responses by technical Subject Matter Experts teams on Military Strategic Assessment Factors – Impact Statements; and
  • Step 2 – Assessment of responses by the Royal Canadian Air Force Senior leadership level from a missions perspective to result in "Strategic Risk" assessment of each aircraft for each of the missions
• Both the "Operational Risk" and "Strategic Risk" will then be consolidated into an overall risk profile for each aircraft against each mission.

Step 1 of Capabilities Assessment

• Assessments being performed using responses to the 17 areas of "Capabilities" contained in the Industry Engagement Request.
• National Defence teams assessing specific areas, with oversight from the National Fighter Procurement Secretariat.
• Qualitative assessment - independent of missions.
• Results in Measures of Performance assessment for each aircraft.

• Sensors
  • Radio Frequency Sensors
  • Electro-optical or Infrared Sensors
• Weapons
  • Air-to-Air Weapons
  • Air-to-Ground Weapons
  • Air-to-Surface Weapons
  • Non-Kinetic Weapons (Electronic Attack)
• Self Protection
  • Radio Frequency Self Protection
  • Infrared Self Protection
  • Countermeasures
• Avionics
  • Data Link
  • Pilot Workload
  • Communications
  • Sensor Integration
• Performance
  • Radio Frequency Signature
  • Infrared Signature
  • Engine/Airframe
  • Combat Radius/Endurance

Evaluation Grid for Step 1

• Performance rating of individual systems
• Rating scale from 0-10 to allow sufficient fidelity
• Classified systems information represented by a number (supported by classified references)

Rating

• 9-10: Excellent - Without appreciable deficiencies
• 7-8: Very Good - Limited only by minor deficiencies
• 5-6: Good - Limited by moderate deficiencies
• 3-4: Poor - Limited by major deficiencies
• 1-2: Very Poor - Significantly limited by major deficiencies
• 0: Non-Existant

Step 2 of Capabilities Assessment

• Three Royal Canadian Air Force teams (four members each) performing this assessment, with oversight from the National Fighter Procurement Secretariat:
  • Including fighter pilots and at least one Air Force Air Weapon Controller on each team.
• Significant differences of opinion between assessment teams will be brought to the attention of the Panel.
• The following inputs will be used in this assessment:
  • Scores and input from the results of Step 1;
  • Companies responses to the Industry Engagement Request, including the mission configuration systems;
  • Vignettes detailed in Appendix "A" of the Industry Engagement Request;
  • Detailed threat capabilities from Canadian Forces Threat Assessment;
  • Detailed fighter task lists; and
  • Other information available to Government.
• Fighter Measures of Effectiveness areas of consideration detailed in the Industry Engagement Request.

Information on Missions as per the Canada First Defence Strategy

1. Conduct daily domestic and continental operations, including in the Arctic and through NORAD (Vignette 1)
2. Support a major international event in Canada, such as the 2010 Olympics (Vignette 2)
3. Respond to a major terrorist attack (9/11-type scenario under CFDS Mission 1 through NORAD) (Vignette 5)
4. Support civilian authorities during a crisis in Canada such as a natural disaster (No Vignette)
5. Lead and/or conduct a major international operation for an extended period [including complex peace enforcement operation, and coalition state-on-state war-fighting] (Vignettes 3 & 4)
6. Deploy forces in response to crises elsewhere in the world for shorter periods (Vignette 6)

Of the 6 Canada First Defense Strategy missions above, only 5 relate to the 6 vignettes. Mission 4 (Support civilian authorities during a crisis in Canada such as a natural disaster) will be analyzed together with Mission 2, understanding the fighter's role in this mission is generally minor.

Information on Aerospace Capabilities

1. Defensive Counter Air (DCA)
2. Offensive Counter Air (OCA)
3. Strategic Attack
4. Close Air Support (CAS)
5. Land Strike
6. Tactical Support to Maritime Operations (TASMO)
7. Intelligence, Surveillance, and Reconnaissance (ISR)

Information on Fighter Measures of Effectiveness

1. Awareness
2. Survivability
3. Reach and Persistence
4. Responsiveness
5. Lethality
6. Interoperability

Step 2 of Capabilities Assessment Overview

• Step 2 will first generate "raw" scores for Measures of Effectiveness (MOEs).
• Raw results will then be weighted against the applicable mission to come up with an Operational Level Aerospace Capability Score for each aircraft against each mission over the 2 timeframes.
• Mission weightings are sensitive and not included as part of this presentation.

Evaluation Grid for Step 2 Raw Scores

• Effectiveness rating of contributing system performances.
• Multiple systems will contribute to each measure of effectiveness.
• Will vary across all missions.
• Rating scale from 1-10 to allow sufficient fidelity.
• Qualitative, and thus requires Subject Matter Expert input.
• Will result in "Raw" scores for Step 2.

Rating

• 9-10: Excellent - No appreciable limitations in delivering the desired effect
• 7-8: Very Good - Only minor limitations in delivering the desired effect
• 5-6: Good - Moderate limitations in delivering the desired effect
• 3-4: Poor - Major limitations in delivering the desired effect
• 1-2: Very Poor - Severe limitations in delivering the desired effect

Step 2 Example

Image Description

This chart is an example of an overview of the aerospace capabilities being assessed against the threat information and missions as per the Canada First Defence Strategy (CFDS).

The left column shows the capability being assessed; Defensive Counter Air, and the threat information across two time horizons; 2020-2030 and 2030+

The right column shows information on missions as per the CFDS categorized by Vignette 1 and tasks.

This step will generate raw scores for Measures of Effectiveness (MOEs)

Step 2 Example – Assessment of Raw Scores

Image Description

These two charts (side by side) are an example of how the assessment of raw scores works.

The first chart (on the left side) shows example aircrafts X and Y's score against each of the Measures of Effectiveness' (MOEs) categories under Defensive Counter Air (the example capability being addressed) using Vignette 1 in the 2020-2030 timeframe. The MOEs are; Lethality, Survivability, Responsiveness, Reach and Persistence, Interoperability and Awareness.

The second chart (on the right side) shows the raw scores for example aircraft X and Y against the Tech Area (Sensors, Weapons, Self Protection, Avionics and Performance) and the Contributing System for each Tech Area.

Step 2 Example – Capability Raw Scores

Image Description

This chart shows example aircraft X' scores (numbers and colour coded) on Aerospace Capability: Defensive Counter Air, under Vignette 1, against each MOEs. It shows an example average total score (in numbers and colour coded). The MOEs are; Lethality, Survivability, Responsiveness, Reach and Persistence, Interoperability and Awareness.

Image Description

This chart shows example aircraft Y's scores (in numbers and colour coded) on Aerospace Capability: Defensive Counter Air, under Vignette 1, against each MOEs. It shows an example average total score (in numbers and colour coded). The MOEs are; Lethality, Survivability, Responsiveness, Reach and Persistence, Interoperability and Awareness.

Step 2 – Capability (Weighted)

Applying weighting function to each measure of effect allows capability to be properly quantified within that Mission

Image Description

This chart shows example aircraft X's colour coded scores (no numbers) on Aerospace Capability: Defensive Counter Air, under Vignette 1, against each Weighted MOEs for the timeframe 2020-2030. It shows an example average total score in numbers and colour.

Image Description

This chart shows example aircraft Y's colour coded scores (no numbers) on Aerospace Capability: Defensive Counter Air, under Vignette 1, against each Weighted MOEs for the time horizon 2020-2030. It shows an example average total score in numbers and colour.

Step 3 of Capabilities Assessment

• This last step of the Capabilities Assessment takes into account: a) the weighting of each Fighter Aerospace Capability within a particular mission and b) criticality as follows:
  • DCA – XX%, OCA – XX%, TASMO – XX%, ISR – XX%;
  • Mission Critical (MC), Mission Essential (ME), or Mission Routine (MR);
• The Operational Level Score obtained at the end of Step 2 is aggregately weighted and assessed by Royal Canadian Air Force against the mission criticality matrix.
• Aerospace capability limitations will carry risk forward dependant on their mission criticality (translation matrix on slide 22 expands).

Step 3a – Operational Mission Weighting

Image Description

This slide shows how operational mission weighting is applied.

The Table on the top left of the slide shows how example aircraft “Y” scored for each of the applicable capabilities under Vignette 1 (DCA, TASMO and ISR).

The Table directly to the right shows how these scores result in a single operational level score for the 2020-2030 timeframe before mission criticality is applied.

The table on the bottom left of the slide shows the resulting scores after the weighting of fighter roles/CFDS Missions and mission criticality is applied.

Finally on the bottom right of the slide, a table collates the weighted scoring to arrive at an overall operational risk for example aircraft “Y” for Vignette 1 for the 2020-2030 timeframe.

Step 3b – Mission Criticality Translation Matrix

Image Description

This slide shows how the mission criticality translation matrix works.

On the left of the slide are the following definitions:

Mission Critical: A capability that delivers a direct effect as part of its primary function. This effect is evaluated as critical to mission success and will pose risk to mission success if it is not employed.

Mission Essential: A capability that is an essential enabler to Mission Critical Capabilities; the lack of one or more Mission Essential Capabilities will pose risk to mission success.

Mission Routine: A capability that is required for the mission but either as a routine supporting function or a very low likelihood of employment. Only in cases where multiple Mission Routine Capabilities constitute a systems failure will any significant risk be posed to the mission.

The right side of the slide shows a series of columns that when combined result in the maximum operational risk assessment.

The first column is for the Weighted Capability of each Aerospace Capability. The scale range is as follows:

Dark green – Insignificant limitations
Light green – minor limitations
Yellow – moderate limitations
Orange – moderate limitations
Red – severe limitations

The second scale corresponds to the criticality of the aerospace capability. The scale range is as follows:

Mission Critical
Mission Essential
Mission Routine

The third column is the result of the first two columns. Depending on the combination of weighted capability (column 1) and criticality (columns 2) we arrive at a Maximum Operational Risk Assessment. The scale range is as follows:

Dark green – Low operational risk
Light green – Medium operational risk
Yellow – Significant operational risk
Orange – High operational risk
Red – Very high operational risk

Definitions

Mission Critical

A capability that delivers a direct effect as part of its primary function. This effect Is evaluated as critical to mission success and will pose risk to mission success if it is not employed.

Mission Essential

A capability that is an essential enabler to Mission Critical Capabilities; the lack of one or more Mission Essential Capabilities will pose risk to mission success.

Mission Routine

A capability that is required for the mission but either as a routine supporting function or a very low likelihood of employment. Only in cases where multiple Mission Routine Capabilities constitute a systems failure will any significant risk be posed to the mission.

Step 3 – "Operational Risk" Assessment

Rating

• 9-10: Insignificant limitations in accomplishing the mission
• 7-8: Minor limitations in accomplishing the mission
• 5-6: Moderate limitations in accomplishing the mission
• 3-4: Major limitations in accomplishing the mission
• 1-2: Severe limitations in accomplishing the mission

Definition

• Low operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
• Medium operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
• Significant operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
• High operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.
• Very high operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Image Description

This slide is a summary of the previous two slides. It shows how an Operational risk rating for the example aircraft is derived.

On the far right of the slide are two charts, one on top of the other.

The first chart shows the scale that is used for Operational Level Scoring (Before Mission Criticality) in the 2020-2030 time-frame.

The Scale is as follows:

Scores 10-9 are depicted as Dark green – Insignificant limitations in accomplishing the mission
Scores 8-7 are depicted as Light green – Minor limitations in accomplishing the mission
Scores 6-5 are depicted as Yellow – Moderate limitations in accomplishing the mission
Scores 4-3 are depicted as Orange – Moderate limitations in accomplishing the mission
Scores 2-1 are depicted as Red – Severe limitations in accomplishing the mission

The second chart shows the scale that is used for Operational Risk.

The scale is as follows:

Dark Green – Low operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Light Green – Medium operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Yellow – Significant operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Orange – High operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Red – Very high operational risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

On the top left of the slide is a chart that shows the Operational Level Scoring of the example aircraft for CFDS Mission 1, Vignette 1 in the 2020-2030 timeframe. This scoring is before mission criticality is applied. The example aircraft has received an operational level score of light green – Minor.

Below this chart is an arrow showing the Operational Level Scoring put through the Mission Criticality Translation Matrix that was described on the previous slide.

To the right of the Mission Criticality Matrix is the chart depicting the End State - Operational Risk for the 2020-2030 time-frame. The example aircraft has received an Operational Risk rating of yellow – Significant, after Mission Criticality has been applied.

Assessment of the "Production and Supportability" Questions

• Step 1 – Review of responses by technical Subject Matter Expert teams – no scoring.
• Step 2 – Assessment of responses by the Royal Canadian Air Force Senior leadership level from a Canada First Defense Strategy missions perspective to result in "Military Strategic Risk" assessment of each aircraft against the missions.
• Both Steps will be overseen by the National Fighter Procurement Secretariat.
• Significant differences of opinion between assessment teams will be brought to the attention of the Panel.

Factor Analysis through Deductive Reasoning

Where does the Industry response fit inside the framework?

Image Description

This slide shows how the “strategic assessment” teams will be using professional military judgment based on a standard deductive reasoning process to analyze source documentation on each fighter aircraft under consideration.

The top of the chart provides a general overview of this process. Factors feed into consideration (step 1) and Deductions (step 2).

Given a set of Factors, the teams will apply their experience and subject matter expertise to run through a “so what” analysis of considerations. This will be the output of Step 1.

The Step 2 team will then deduce the Strategic Risk to accomplishing the mission through further analysis of the considerations (Likelihood of occurrence, impact statements and mitigation measures).

The responses to the Industry Engagement request questionnaire are one of the source documents feeding into this deductive reasoning process. The others include, Open source information, Government to Government Information and other products from the evaluation of Options.

Acquisition

• This Strategic Assessment Factor (SAF) is an assessment of factors, other than cost, that would affect acquisition of aircraft types. This includes:
  • planned production periods of aircraft type;
  • the manufacturer's ability to continue or re-establish production periods;
  • the ability of the manufacturer to offer a complete "cradle to grave" program, including; training, in-service / life cycle support and disposal;
  • the manufacturer's successful completion of any required developmental work before the aircraft are ready for acquisition; and
  • the ability of aircraft type to be certified for airworthiness under Canadian Forces regulations.

Supportability and Force Management

• An assessment of the overall / long-term supportability and aspects related to management of the fighter force. This includes factors such as:
  • the required quantity of aircraft;
  • managed readiness postures and sustainment rations;
  • the training system and production of appropriately trained pilots and maintenance / support personnel (including use of simulators);
  • training considerations for maintenance / aircrew personnel;
  • aircraft mission availability rates and aircrew / technician workload;
  • long-term availability of components needed for aircraft maintenance;
  • implementation and sustainment of supporting infrastructure;
  • the ability to operate from required locations;
  • weapons compatibility and support; and
  • the suitability of a reprogramming capability including its flexibility and responsiveness for new threats and/or theatres of operation or new capabilities.

Integration

• This Strategic Assessment Factor assesses broad interoperability within the Canadian Forces and with allied forces. This includes interoperability with air-to-air refuelling services (other than Canadian Forces), common ground/spares support with allies and their supply lines, training systems, data sharing, communications, STANAGs, and the ability to feed data into Canadian Forces and Government of Canada networks taking into consideration national security requirements.

Growth Potential

• This Strategic Assessment Factor assesses the growth potential and technological flexibility to respond to unforeseen future advances in threat capabilities, to implement required enhancements to fighter technology, and to evolve as needed to meet the Canadian Forces' needs. Factors include analysis of the architecture of aircraft types, power and cooling capability for new systems.

Image Description

The box on the top left of the slide lists the documents and factors that will be used to conduct the assessment of the 4 Strategic Assessment Factors shown in the middle of the slide.

The documents/factors include the following:

Response to IER
Other reports resulting from evaluation of options
Future security environment
Open source/other information and government to government information

The Strategic Assessment Factors are as follows:

Supportability and Force Management
Acquisition
Integration
Growth Potential

Below this list is a strategic risk scale that shows the colour assigned to each risk category.

The scale is as follows:

Dark Green – Low strategic risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Light Green – Medium strategic risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Yellow – Significant strategic risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Orange – High strategic risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Red – Very high strategic risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Listed on the top right of the slide, are the Step 1 team reviews that will be conducted. They will identify possible risks, likelihood and impacts of risk, as well as potential mitigation strategies. The work of the Step 1 teams will be overseen by the NFPS.

Listed on the bottom right hand side of the slide, are the factors that the Senior Air Force staff will consider when reviewing the analysis provided in Step 1 and which will be used to determine risk ratings (or scores) for each of the 4 strategic components, against each mission and in the 2 time horizons that will be completed in Step 2.

This assessment will be done through the CFDS missions' lens and it will be performed in a group discussion environment with NFPS' oversight. It is at that point that a strategic risk assessment will be assigned to each of the 4 strategic factors and for each mission.

The “End State – Strategic Risk” box at the bottom of the slide basically takes all 4 strategic factors and assigns an overall risk rating per mission. This step will be done by the same team that performed the Step 2 assessment and will be based on professional military judgement.

Final Stage – Overall Mission Risk Assessment

This slide shows how strategic risk and operational risk both contribute to the end state: mission risk.

Image Description

On the top left corner of the slide is a table showing Operational Risk for example Aircraft “Y” for CFDS Mission 1, Vignette 1 in the 2020-2030 time-frame. The example aircraft is assessed as yellow.

At the bottom left of the slide is a table showing the Strategic Risk for example Aircraft “Y” for CFDS Mission 1, Vignette 1 in the 2020-2030 timeframe. The example aircraft is assessed as light green.

End State - Mission Risk is the fusion of Operational and Strategic Risk.

Mission Risk will be determined by a Working group of senior Air Force personnel, who will consider the Operational Risk and the Strategic Risk using a mission risk translation matrix as a decision support tool.

In the middle of the slide is a table that show the Mission Risk for example Aircraft “Y” for CFDS Mission 1, Vignette 1 in the 2020-2030 time-frame. The example aircraft is assessed as yellow.

In the far right of the slide is a scale showing the risk colours and what they represent. The scale is as follows:

Dark Green – Low risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Light Green – Medium risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Yellow – Significant risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Orange – High risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Red – Very high risk to the RCAF ability to successfully fulfill the fighter contribution to CFDS mission X.

Finally, the slide notes that the Chief of the Air Force will endorse the Final Report.

Mission Risk

• Mission Risk will be an outcome of the fusion of Strategic and Operational risk by the same RCAF senior leaders assessment teams who took part in step 2 of the Strategic Assessments.
• Decision support tools, such as but not limited to a mission risk translation matrix and the DND/CF Integrated Risk Management Guidelines will be used to determine a rating.
• Final assessment will be informed by, but not limited to the decision support tools provided. Assessors will be allowed to deviate from the mission risk translation matrix used by + or – one risk level.
• Rationale behind a final assessment that deviates from the mission risk translation matrix used will be provided to the Independent Review Panel for review.

Summing Up

• We are committed to an open, fair and transparent process
  • IER and methodology posted publicly on Government website
  • All questions and answers during IER shared simultaneously with all companies
  • Details of assessment methodology shared with companies in advance of IER response deadline
  • NFPS overseeing entire assessment process
  • Significant differences of opinion between assessment teams will be brought to the attention of the Panel
  • The process will result in an assessment of benefits and risks associated with each aircraft against each mission. Canada will release a summary report that respects commercial sensitivities and classified information restrictions.