Technical Briefing Remarks on the evaluation of options' process

National Fighter Procurement Secretariat

June 12, 2014

Check against delivery

Setting the context (Tom Ring)

Good day everyone and welcome to our technical briefing. My name is Tom Ring and I am the Assistant Deputy Minister of Acquisitions at Public Works and Government Services Canada.

The information that is provided today is focussed on the methodology and process used to conduct the evaluation of options, a key element of the Seven-Point Plan for the replacement of the CF-18 fleet.

With me here today are Brigadier General Stephan Kummel representing the Royal Canadian Air Force, and members of the Independent Review Panel:

• Mr. Keith Coulter, former fighter pilot and former deputy minister;
• Mr. Philippe Lagassé, assistant professor of Public and International Affairs at the University of Ottawa;
• Mr. Rod Monette, a former senior public service executive, Comptroller General of Canada and a Fellow Chartered Accountant.
• Mr. James Mitchell, founding partner of the Ottawa consulting firm Sussex Circle is joining us by phone today from Québec city.

Today's briefing will be structured in four parts:

• First, I will provide a few introductory remarks that set the context and clarify what we will be covering;
• Second, Brigadier General Kummel will brief you on the process used for the evaluation of options;
• I will then turn it over to the Independent Review Panel to add a few remarks about the process; and
• Finally, we will open the floor to questions.

In May 2013, we provided a technical briefing on the methodology that was used to assess the information provided by companies as part of the evaluation of options.

Over the past several months, the Department of National Defence and the Royal Canadian Air Force have conducted a tremendous amount of work and in-depth analysis.

This work involved a comprehensive assessment of four available fighters against each mission in the Canada First Defence Strategy. The four participating fighter jet suppliers included: Dassault, Boeing, Eurofighter and Lockheed Martin. SAAB was also invited to participate in the market analysis; however they decided not to do so for business reasons.

The Independent Review Panel, whose members have joined us here today, was established to ensure the rigour and fairness of the assessment process including the methodology.

The assessment used information we sought directly from companies, and was supplemented with open source and government to government information as needed. Companies were provided with a full and fair opportunity to put forward information on their aircraft

In keeping with its commitment to openness and transparency, the National Fighter Procurement Secretariat, shared draft versions of the capability, price and industrial benefit questionnaires with the companies to provide them the opportunity to comment on the scope and structure of the questionnaire before it was finalized.

The Secretariat further held face to face briefings with the companies on all three questionnaires, including a detailed briefing on the methodology that would be used for the assessment. On the website you will find both the draft and final versions of the questionnaires, as well as the documents used for the briefings.

Throughout the process, the Secretariat has regularly posted information on its website, including the Summaries of Discussion for all of the Panel's meetings with the Royal Canadian Air Force (RCAF) and other officials. I encourage you to visit the Secretariat's website, we have had extensive positive feedback from Canadian industry on how informative it is.

Our work was aimed at ensuring that Ministers have the best possible information to inform their decision-making.

Ministers are now reviewing that information, including reports on fighter mission needs and capabilities, industrial benefits, costs, and other factors related to the decision to replace Canada's CF-18 fleet.

Information that is not commercially sensitive or classified will be released. We have also committed to the companies involved in the evaluation of options that they will have the opportunity to review the reports in advance of their release to ensure no commercially sensitive information is being released.

To be clear, the Government has not yet made a decision.

Today we are here to provide you with an overview of the methodology used to assess each aircraft against the missions outlined in the Canada First Defence Strategy and to allow the Independent Review Panel to speak to their role in overseeing the process.

I will now turn it over to Brigadier General Kummel to provide an overview of the assessment methodology.

Overview of the assessment methodology (BGen Kummel)

Thank you Tom.

The methodology report you received outlines the four guiding principles of our assessment: comprehensiveness, impartiality, understandability and robustness.

I will speak about how we were guided by each of these principles in turn. Allow me to first outline the key steps of the methodology which will demonstrate our commitment to comprehensiveness.

The methodology was designed to articulate the risks associated with each aircraft's ability to successfully execute the six missions outlined in the Canada First Defence Strategy.

We started by assessing the raw technical abilities of each aircraft, for example, radar or speed. We then adjusted these ratings where appropriate by applying mitigation measures that could be used in an operational context.

Next, we assessed all the things needed to acquire, support and upgrade the aircraft over time. This assessment looked at aspects such as the ability to access spare parts and pool investments in upgrades to ensure the aircraft remains operationally relevant over its full service life.

Finally, these risks were integrated into an overall risk rating for each aircraft and its ability to successfully conduct each of the six missions in the Canada First Defence Strategy.

Next I would like to speak to impartiality. You will hear shortly from the Independent Review Panel that was established by the Government to ensure the highest standard of impartiality was applied to the assessment.

Whenever any significant issues arose during the assessment, they were brought to the attention of the Independent Review Panel for their guidance. The Panel played a key role in ensuring the methodology was adjusted as appropriate as we completed the process.

As an additional level of oversight, observers from the National Fighter Procurement Secretariat participated at each phase of the assessment to ensure that the methodology was applied consistently across all aircraft.

In terms of understandability, we made sure that the findings could be easily traced back to the appropriate source.

The results associated with each of the steps of the process are fully documented. In this way, the facts, judgements and supporting data behind each risk rating for each aircraft across both timeframes can be traced and easily understood.

Finally, with respect to robustness, we made sure that the assessment was robust by ensuring that it could be applied under a wide range of conditions without sacrificing rigour.

In real terms this means designing and implementing an assessment that could be repeated if necessary with different assessors, or under different conditions. To do this, all assessment team members were provided identical information in order to conduct the assessment.

To be clear, the assessment is one of capabilities and enablers to achieve the missions over the duration of the life of the aircraft. It is a risk assessment, and is not designed to compare one aircraft to another.

In summary, the process used to conduct the evaluation of options was consistently guided by the principles of comprehensiveness, impartiality, robustness and understandability.

The evaluation of options was a major undertaking for the RCAF that took 40,000 hours of concerted staff effort over 14 months.

In all my years of experience, I have never been involved in such a thorough evaluation exercise, no stone was left unturned. I am confident that our work meets the highest bar of due diligence and rigour that could be applied.

Thank you.

Introducing the Independent Review Panel (Tom Ring)

I will now turn it over to the Independent Review Panel to say a few words.

Independent Review Panel Remarks (Keith Coulter)

Thank you Tom and good morning everyone.

You now have the statement of assurance that the four of us have written and signed. This statement communicates our opinion on the evaluation of options and the resulting RCAF Report provided to Ministers for their consideration. We believe this statement is self-explanatory.

However, before we respond to any questions you might have, we would like to highlight a few important things about our mandate and the way we delivered on it. I will start with a few introductory comments on behalf of the four of us, and my three colleagues will then each speak briefly as well.

Let me start with three quick points about our mandate:

• First, we did not have a mandate to make recommendations. The RCAF report presents in detail the results of a risk-based assessment of the ability of fighter aircraft available on the market to fly missions the RCAF might be asked to conduct. Neither the report nor the panel made recommendations. It is the committee of Deputy Ministers set up to oversee implementation of the Seven-Point Plan that is responsible for providing advice to Ministers.
• Second, what we did have a clear mandate to do was to provide independent assurance on the conduct of the evaluation and the resulting report. As outlined in the terms of reference, our role was to help ensure this exercise was “rigorous and impartial”, and the results were “comprehensive and understandable”. Our opinion in the statement is therefore structured around these four key pillars – rigor, impartiality, comprehensiveness and understandability.
• Third, we believe we have fulfilled this mandate. As you would expect for an important and complex undertaking like this, our formal statement is quite long. However, the simple bottom line is that we have provided ministers with assurance that the evaluation was rigorous and impartial, and the results are comprehensive and understandable.

With respect to the way we delivered on our mandate, I would like to stress four key points:

• First, the four of us were deeply involved in this evaluation. The centrepiece of our engagement was a regular schedule of face-to-face panel meetings with officials involved in this exercise. These meetings usually lasted a full day and took place every couple of weeks. Beyond preparing for and attending these meetings, we regularly participated in conference calls, provided written and oral comments on a wide range of documents and very actively engaged in many other ways. This evaluation was a big part of our lives for a period of more than 14 months.
• Second, we worked together closely as a team. Each of us brought to this exercise very different reference points and perspectives, and we took full advantage of this. In addition to the scheduled panel meetings, we stayed in active contact with each other throughout the evaluation and often met by ourselves to compare notes and discuss issues. We believe this approach enabled us to be much more effective in delivering on our mandate.
• Third, we acted independently. We were careful to keep the right professional distance. We did not receive any direction from anyone on anything, and we were not lobbied by any individuals with military, bureaucratic, commercial or political interests. We had a clear mandate to act in a way that was fully independent, and that is precisely what we did.
• Fourth, we saw it as our fundamental job to challenge. We constantly asked hard questions, often insisted on more in-depth analysis where we thought this was needed or might help, and always provided frank input, feedback and advice. This sometimes slowed the process down, but we felt so much was at stake that we were not willing to live with any significant gap in our understanding or with any significant discomfort about results. So we were assertive in performing a very real challenge function.

Those are my introductory comments. The statement we have signed obviously provides a more detailed assessment against each of the key elements of our mandate.

I would now like to turn it over to my panel colleagues who will make a few additional introductory points.

Philippe Lagassé

Thank you, Keith.

I would like to discuss the methodology that was applied during the evaluation of options. This methodology is further detailed in the documents made public this morning.

The methodology applied in the evaluation of options was designed to identify the risks that four fighter aircraft would face in completing the missions outlined in the Canada First Defence Strategy.

In the interests of comprehensiveness and rigour, these risk assessments were divided into two timeframes, 2020-2030 and 2030 plus.

The first timeframe allowed for an assessment of mission risks in the near term, which also coincides with the retirement of the existing CF-18 fleet and introduction of a replacement aircraft, while the second timeframe allowed for an assessment of mission risks over the majority of a new fleet's operational life.

In order to guarantee the impartiality of the analysis, the methodology allowed the four aircraft to be assessed in the best possible light, during both timeframes.

Indeed, as part of our challenge function, the Panel worked with the RCAF to ensure that the methodology offered a fair assessment of all four aircraft.

Although each of the four aircraft are highly capable modern fighters, the methodology allowed the different risks associated with each of the aircraft to be properly analyzed and understood.

Moreover, the methodology produced traceable results, meaning that it is possible for ministers to go back and see how each step in the analysis led to a particular mission risk assessment.

Since the risk assessments necessarily involved a mix of technical calculations and professional military judgements, we stressed that the results should be understandable and we are satisfied that the findings are replicable, meaning that a different set of assessors would arrive at the same results.

We are therefore confident that the methodology produced results that will allow ministers to make a well-informed, evidence-based decision on replacing the CF-18s.

Finally, we are entirely satisfied with the rigour of the evaluation process and with the methodology that was applied to this evaluation.

Rod Monette

Thank you colleagues and good morning all.

As Keith has said, the job of our panel was to provide an independent opinion. This is commonly called assurance, and it is intended to give decision makers confidence in the information they are receiving.

Our opinion, found in our statement and based on our mandate, is on the evaluation work done and on the RCAF Report to Ministers. It is an opinion on whether the work done is rigorous and impartial, and whether the RCAF Report to Ministers is comprehensive and understandable. In other words, ‘are Ministers getting the quality of information they need to make an informed decision on replacing Canada's CF-18s'?

An opinion can be unqualified, which means that we have no concerns with the information going forward to Ministers, or it can be qualified, which means that there are elements of the work with which we are not satisfied.

We have delivered an unqualified opinion. We believe that in all material respects, in other words, in all respects important to decision-makers, that the work is rigorous and impartial, and that the RCAF Report is comprehensive and understandable.

I think it is important to stress that, after 14 months of providing our frank input, feedback, and advice, we are comfortable that we have achieved our mandate as an independent panel regarding the RCAF's work and their final report. If we had any material concerns, we would have strongly communicated this to Ministers in our statement. As such, we have no reservations or qualifications.

Thank you.

Jim Mitchell

I would like to say a few words about the way in which we did our work.

First, I should point out that in our work we did not question the government's defence policy as set out in the Canada First Defence Strategy. That document sets out the six missions which the RCAF is expected to be able to undertake, both now and in the future. It served as the foundation and reference point for the review and for our work as an independent review panel.

Second, a word about independence. The review process necessarily involved close engagement with the RCAF review team. They had their job to do, and we had ours. Our task was to ask questions throughout the process, to challenge assumptions, to ensure the right questions were being asked and answered, and to satisfy ourselves that the results of the process could be understood by ministers and senior officials, and ultimately by the interested public.

Throughout the entire process we were impressed not only with the expertise of the review team, but also with the professionalism and objectivity with which they approached their task.

As Rod has indicated, the panel was asked to provide assurance to ministers and to Canadians that the review process was "rigorous and impartial" and that the results of the process are "comprehensive and understandable".

We have no hesitation in saying that it was.

Concluding remarks (Tom Ring)

Thank you, Gentlemen. I will now turn it over to Annie to lead the Question period.