Statement by the Independent Review Panel regarding the Royal Canadian Air Force Report to Ministers on the Evaluation of Options to Sustain a Canadian Forces Fighter Capability

February 28, 2014

Our Mandate

In April 2012, in response to a report by the Auditor General of Canada, the Government of Canada announced a Seven-Point Plan in support of its decision on how to move forward with the replacement of the CF-18 fleet. A key element of the Plan was a commitment that "the Department of National Defence would evaluate options to sustain a Canadian Forces fighter capability well into the 21^st century". The Independent Review Panel (IRP) was created in December, 2012 to oversee this evaluation and, once it was completed, to provide assurance on the integrity of the evaluation process. This is our statement.

The four members of the IRP combine experience in the academic world with experience in different sectors of government, including National Defence, the Treasury Board Secretariat, the Office of the Comptroller General, and the Privy Council Office. Keith Coulter is a former fighter pilot and later a Deputy Minister. Philippe Lagassé is an associate professor of public and international affairs with particular expertise in national defence matters. Jim Mitchell is a former senior official in the Privy Council Office and Treasury Board Secretariat who leads a consulting firm that provides strategic advice to government. Rod Monette is a former Comptroller General of Canada and Deputy Minister.

Statement

This Panel statement addresses the conduct of the evaluation and the resulting report presented to ministers by the Royal Canadian Air Force (RCAF). The RCAF report contains a risk-based analysis of four modern fighter aircraft available on the market. The RCAF evaluation was not a selection process, and the RCAF report contains no recommendations or direct comparisons. Rather, it presents a summary of risk-based assessments based on the ability of each of the candidate aircraft to successfully conduct missions under the Canada First Defence Strategy.

We should stress that the Panel's mandate is one of assurance. That is to say, our role in the review process, as defined in the Terms of Reference for the Evaluation of Options, has been to "assess the methodology used and the analyses performed, and participate at key milestones in the work". On that basis we are to assure users of the evaluation that the resulting report of the RCAF evaluation team "is both rigorous and impartial", and that the results "are comprehensive and understandable."

It has not been our responsibility as a Review Panel to reach conclusions about the merits of particular fighter aircraft, or to compare one aircraft against another, or still less to provide advice on which aircraft may be best for Canada. Responsibility for drawing conclusions from the work of the RCAF evaluation team, and for providing advice to Ministers, rests with the Deputy Ministers Governance Committee (DMGC) which was set up to oversee implementation of the Seven-Point Plan including the evaluation of options.

Our Work

Over a period of 14 months, we met on more than thirty occasions with the National Fighter Procurement Secretariat (NFPS) in Public Works and Government Services Canada (PWGSC) and with experts from the RCAF, usually for a full day each time. We also met regularly as a Panel by ourselves and we have stayed in active contact with one other during the entire period.

Throughout our work, we have been engaged in a continuing, active dialogue with the RCAF evaluation team and the NFPS on every aspect of the evaluation process. This engagement has involved detailed briefings by military and civilian officials on every dimension of the evaluation. It has meant our reading thousands of pages of documentation on threats, mission needs and fighter aircraft capabilities. We have provided detailed, written comments on most of this material and on all of the review products, at all stages of their development, always with a view to satisfying ourselves that the evaluation process has been rigorous, impartial, comprehensive and understandable.

In the course of our review work we also met with senior officials from PWGSC, the Canadian Armed Forces, National Defence, the Privy Council Office and Industry Canada. As part of our assurance function, we also met on five occasions with DMGC, three times with the Chief of Staff to the Minister of PWGSC and once with the Minister of PWGSC and her staff. All those sessions were substantive and always thoroughly professional.

It is worth noting what the members of the Panel did not do in the course of our work. First, we did not question the policies and missions outlined in the Canada First Defence Strategy (CFDS). This statement of government policy served as the framework for defining the kinds of missions the new fighter aircraft would be expected to undertake and thus as the reference point for our work and that of the RCAF evaluation team.

Second, we did not meet with representatives of the aircraft manufacturers who participated in the evaluation of options. Relations with industry were conducted by the NFPS and, to a lesser extent, by officials from Industry Canada on the issue of industrial benefits. Third, although we were familiar with the initial Statement of Requirements that was set aside as part of the evaluation of options, our work did not entail a review of analyses that informed decisions made prior to the announcement of the Seven-Point Plan. The present evaluation of options carried out by the RCAF was part of a true reset of the process to replace the CF-18s, as was our review work.

In conducting this review and fulfilling our assurance responsibilities, the Panel was given a clear mandate to act independently, and we did. We also decided among ourselves not to comment publicly about our work on the evaluation of options until we felt it was appropriate. Most importantly, we can attest that we received no direction from anyone on any aspect of our work throughout the review process.

We wish to express our appreciation for the expertise and professionalism of the RCAF officers with whom we have been working over the past 14 months. We are also grateful for the work of the staff of the NFPS, led by Executive Director Rachel Wernick. Their support to the Panel, their expertise in procurement and their skill bringing the evaluation process to completion is a credit to the department and to the Public Service.

On the basis of our work over the past 14 months, we are now in a position to offer the following opinion in fulfillment of our mandate.

Our Opinion

1. With respect to rigour

We are entirely satisfied with the rigour of the evaluation process.

In this regard, we would refer specifically to the transparency of the Seven-Point Plan as a whole, and to that of the evaluation process itself. We observed that the four companies whose aircraft were evaluated (Boeing, Dassault, Eurofighter and Lockheed-Martin) were effectively engaged at the initial stages of the process to ensure they understood it, and they were given every opportunity to provide input to the information-gathering step in the methodology. This allowed them to present their aircraft in a manner that best represented its strengths and capabilities.

Throughout the evaluation process, the Panel played a significant challenge role, posing questions and providing input and advice that required additional work by the RCAF to address the issues we raised. While this sometimes meant some delays, it also resulted, in our view, in a more rigorous process and outcome.

We would also note the strength of the six-step evaluation methodology for the evaluation process. While the methodology was developed by the RCAF and defence scientists from Defence Research and Development Canada, members of the Panel provided input and advice as it was being developed and refined. Indeed, the methodology was subject to an adjustment at one point when the Panel realized a need for an additional step in the assessment process. (This was at the point when the results of the operational and critical enabling assessments were subjected to risk mitigations and brought together in the final phase of the evaluation.) That adjustment in the methodology shows the Panel's determination to ensure the evaluation was rigorous, fair and transparent, and that all relevant factors were being properly weighed in arriving at conclusions.

We would also note that great care was taken in the evaluation process to give full credit to each candidate aircraft for its declared capabilities and upgrade path, and to ensure that each was evaluated in a manner that reflected Canadian practices in employing fighter aircraft and Canadian mission needs. All this was done with a view to ensuring that no aircraft was disadvantaged or subject to institutional bias. Again with a view to fairness, reasonable assumptions were made regarding the Government's capacity to mitigate some risks that would otherwise fall outside the mitigation capacity of the RCAF or Department of National Defence (DND).

Under this heading, we would further note the role played by the Chief of Force Development and his staff, who are responsible for assessing the future capabilities required by the Canadian Armed Forces, and also the role of the Chief of Defence Intelligence, who carries responsibility for describing the current and future threat environment. Their work served as an essential reference point for the options analysis and provided a broader context with respect to what capabilities Canada's future fighter aircraft would contribute to operations under the CFDS, which requires the Canadian Armed Forces to be ready to conduct domestic and international operations in six mission areas.

We would point to the fact that the work of the evaluation team involved bringing together teams of operational and subject matter experts (SMEs) with knowledge and experience in procuring, flying and maintaining modern fighter aircraft. Their work was then checked and re-evaluated by a senior RCAF team, all with a view to ensuring that the assessments of the four aircraft were objective, fair, reasonable and accurate, and were properly considered in keeping with the way in which the RCAF employs fighter aircraft.

The evaluation began with an assessment of the risks associated with conducting operations with each aircraft in six mission areas. This was followed by an assessment of the risks associated with "critical enabling factors" that relate to Canada's ability to acquire these aircraft in a timely fashion and maintain their operational relevance over the longer term. The evaluation then concluded by integrating the results of these two assessments into an overall assessment of risks for each aircraft in the six mission areas.

The key steps in the evaluation process, particularly the assessment of risks associated with critical enabling factors, necessarily involved the application of professional military judgment to arrive at findings that were inevitably qualitative in nature. This was, in our view, entirely appropriate. There can be no algorithm, no matter how sophisticated, that can substitute for informed, professional judgment based on a comprehensive assembly of facts and analysis.

2. With respect to impartiality

Impartiality was a crucial consideration for the Panel and for the evaluation process as a whole. Under the Seven-Point Plan, the RCAF was asked to go back and re-examine the marketplace for modern fighter aircraft and to conduct a risk-based assessment of candidate aircraft based on the policies and missions set out in the Canada First Defence Strategy. Part of the backdrop for this re-examination was the fact that the Government, on the advice of the RCAF, had previously selected one of the candidate aircraft - Lockheed-Martin's F-35 Lightning. In this context, it was our duty to ensure that there was no bias in the current evaluation process or its results. It was essential to the credibility and utility of the evaluation that all four candidate aircraft were assessed objectively and that no personal or institutional prejudices crept into the assessment or the resulting conclusions.

The Panel took great care to ensure the impartiality of both the evaluation process and the eventual product. We watched particularly to ensure that each aircraft was assessed on its own merits - i.e., that each assessment was based solely on the capabilities of that aircraft to conduct the defined missions and was not influenced negatively by perceptions of the particular advantages of any other aircraft in present or future threat environments. In this regard, it is worth noting that all four aircraft have specific advantages and capabilities that served them well in particular mission scenarios.

In the end, we are confident in saying that the evaluation process was conducted professionally, that it was not biased in favour of any of the four aircraft, and that the resulting report to ministers by the RCAF evaluation team is therefore fair, objective and impartial in all material respects. Indeed, it is our view that the same result would have been obtained through the application of this methodology to the same set of aircraft by a different set of military experts.

3. With respect to comprehensiveness

The Panel understood the term 'comprehensive' in two senses. The first was in terms of ensuring that all relevant candidate aircraft were brought within the scope of the evaluation.

It is widely known that our process began with five aircraft, but that after several months, one manufacturer - SAAB - decided not to continue, stating that they would consider re-entering if a formal competition were held. The four remaining candidates - Boeing's F-18 E/F Super Hornet, the Eurofighter Typhoon, Dassault's Rafale and Lockheed-Martin's F-35 Lightning - represent the only other aircraft offered by Western manufacturers as candidates to replace the CF-18 fleet.

The second sense in which we sought comprehensiveness was in the scope and depth of the evaluation itself. Here, the evaluation methodology described in the RCAF report is key. Briefly, that methodology was designed to assess fighter aircraft against the six missions set out in the Canada First Defence Strategy and against all of the criteria relevant to the successful employment of those aircraft in well-defined operational scenarios for each of those missions. It is important to bear in mind that those assessments are an assessment of risk to the successful completion of the mission rather than risk to pilots or aircraft. Essentially, the question in each scenario was, "what are the risks that I will not be able to complete this mission using this aircraft in this setting"?

In this regard, we believe the Panel also played a role in ensuring the comprehensiveness of the methodology by satisfying ourselves that both design and implementation were made as effective as possible throughout the review process, taking into account all relevant considerations that emerged along the way. As a result, when subsequent discussion raised questions about previous work, we 'doubled back' as necessary to an earlier point to ensure the foundation was a solid as possible at each stage. In the most general sense, we used our collective judgment to provide input, advice and guidance, and the RCAF always responded in a constructive manner.

We are satisfied that the RCAF report is comprehensive in both of these senses, and we therefore believe that it provides a useful, credible and comprehensive assessment of risks for each of the four aircraft that participated in this evaluation of options.

4. With respect to understandability

As with any exercise in which the views of military experts are being conveyed to a wider audience, the question of 'understandability' has proven to be a challenge in the evaluation process. Military people are trained to write for military audiences. Moreover, the subjects addressed in this evaluation are themselves complex and highly technical.

In the most basic sense, the task facing the RCAF evaluation team was to bring together hundreds of pages of detailed analysis in a manner that enabled them to make accurate assessments about capability and risk, and to present those findings in a straightforward manner for consideration by senior officials and Ministers. In this process, it was essential that complexity and detail not obscure the basic questions at stake. It was also essential that the separate results of the risk assessments related to operations and critical enabling factors, as well as how they are combined in the overall results, be presented clearly and fairly.

The appropriate disclosure of each of the risk components is important for full understandability. On these critical issues the Panel provided considerable advice, and we are now satisfied that the report presents the risk components and results of this complex evaluation in a way that is clear, fair and understandable.

Finally, since the primary purpose of the evaluation was to assess risks to missions, it is important to understand that every military mission carries risk; it is for commanders and, ultimately, the Government of Canada to determine what degree of risk is tolerable. No piece of military equipment, and certainly no fighter aircraft, can do every job perfectly or in a manner that carries no risk.

Against this backdrop, and bearing in mind that readers will have to make up their own minds about the clarity of the RCAF report, we are satisfied that the issues at stake in this evaluation have been presented in a way that can be understood by an interested reader.

This is an unqualified opinion. We have no reservations or qualifications to add.

Keith Coulter, PhD

Rod Monette, Fellow Chartered Professional Accountant -
Fellow Chartered Accountant (FCPA-FCA)