Evaluation of the Cost and Profit Assurance Program (final report)

Main points

i. The Cost and Profit Assurance Program (CPAP) is delivered through the Acquisitions Branch (AB) of Public Services and Procurement Canada (PSPC) and provides assurance services on domestic and international contracts for both the Government of Canada and foreign governments. The program provides assurance on the integrity of pricing and payments of government contracting and supporting federal institutions and internal partners; and, provides advice and analysis to support innovation in procurement policies and practices. Though the majority of the assurance work conducted by the program is on defence contracts, the program may also conduct assurance work on non-defence contracts on a fee-for-service basis, when requested.

ii. The evaluation noted clear authorities governing the conduct of cost audits and recoveries of overpayments for defence contracts through the Defence Production Act. However, as this act does not apply to non-defence contracts, we noted a lack of clarity in authorities for the program to conduct this work in relation to non-defence contracts.

iii. The evaluation found the program is relevant. There is a continuing need to undertake cost audits of defence contracts. The program identified potential overbillings and identified needed improvements in a number of supplier accounting systems. The program also meets the continuing need to discharge Canada's obligations outlined under the Canada – U.S. Defence Production Sharing Agreement.

iv. The evaluation also found the program aligns with federal priorities and PSPC's strategic outcomes. The CPAP's activities related to helping to ensure fair and reasonable cost and profit coincides with federal priorities as outlined in the 2014 Speech from the Throne and in the 2015 Public Services and Procurement Ministerial Mandate Letter emphasizing the importance of transparent and open government and fiscal prudence of public funds. The program's activities also align with the stewardship elements of CPAP's strategic outcome "to deliver high quality centralized programs and services that ensure sound stewardship on behalf of Canadians and meet the program needs of federal institutions."

v. The evaluation found the 'at risk' universe and the majority of cost audit effort is limited to defence contracts. This is consistent with the funding received under the Special Purpose Allotment (SPA) and the authorities vested in the Minister of PSP under the Defence Production Act. While the authority for conducting cost audits is not explicitly stated in the Department of Public Works and Government Services Act or any other legislation, nor is the current demand for service high, analysis of non-defence contracts indicates the potential risk of overbilling. This potential risk provides a measure of evidence of potential value of cost audit activities in relation to non-defence contracts.

vi. Overall, the evaluation found the program is achieving its immediate outcomes and contributing to intermediate outcomes, although improvements were identified. Although the program is achieving its outcomes related to the conduct of objective and credible assurance engagements, it was noted that the timing and timeliness of these engagements could be improved. Furthermore, the evaluation found assurance engagements undertaken by the program contribute to the identification of potential overbillings. To properly measure performance of the program, a more appropriate performance measurement framework would be of benefit to the program. This framework would more accurately reflect the strategic and value added accountabilities of the program, particularly in relation to recovery of overpayments and system improvements. The evaluation also found the CPAP supports Canada in meeting its international obligations although some stakeholders noted a desire for improved timeliness and reporting. With respect to the Non-core activities of the program related to providing advice and insight to support procurement, the evaluation found that although this work is limited in scope, it did provide value to PSPC's procurement functions.

vii. With respect to program economy, CPAP has worked towards managing its structural deficit by performing work on a fee for service basis, although the current blended funding model creates a risk of incurring deficits. The program has worked to produce a similar level of program outputs compared to its United States equivalent. With respect to program efficiency, decreased program labour rates over time indicate improvements in resource utilization. The evaluation however was not able to conclude on program efficiency due to a lack of data against which to benchmark the program.

viii. The current delivery model is limiting the program's effectiveness and the department's ability to appropriately manage relationships with stakeholders. This includes greater engagement with stakeholders throughout the lifecycle of the contract. The benefits of greater engagements include better selection of assurance engagements, the opportunity for assurance engagements to contribute to rate negotiations, and greater likelihood of recovery of identified potential overpayments. We were informed the program is implementing a number of changes that clarify its role to improve its impact. Decisions regarding the program's mandate and who its primary client is are required to inform future funding and best organizational fit and design.

Management response

Program management and the AB Assistant Deputy Ministers (ADM) agree and accept the evaluation report's recommendations.

Since 2011, when the program delivery arm was brought in-house within the AB, program management have looked for opportunities to integrate its work in direct support to procurement processes and contract management. Management continues to pursue opportunities to support acquisition activities to maximize the value derived from the program, especially the provision of insight into the practices of key contractors and ensuring open book access to financial information used to negotiate the price and finalize payment for non-competitive contracts. Both AB – ADM's strongly support and endorse the approach taken by program management.

Furthermore and as noted within this report, the CPAP has been instrumental to informing management regarding areas of improvement within procurement processes. In conjunction with the CPAP operations, over the past couple of years, the Price Support Directorate has invested significant efforts in reviewing Canada's environment with respect to pricing sole source and/or negotiated procurement and contrasting it to other similar jurisdictions. The resulting action plan that we are pursuing has the propensity to address most of the required actions in respect of the recommendations below.

AB is committed to act on these recommendations in conjunction with the action plan from the Procurement Modernization Initiative on Cost and Profit Policy (PMI-CPP). In fact, the resources to address the PMI-CPP, have been linked to the CPAP renewal proposal that is currently under consideration by TBS.

We are proposing the following Management Action Plan to the recommendations in this report and, where applicable explained the linkage with the PMI-CPP. We have included a background document on the PMI-CPP as an Annex.

Recommendations and management action plan

Recommendation 1

To maximize contribution of the program to outcomes and broader procurement modernization objectives in both defence and non-defence contracting, the ADM, Defence and Marine Procurement and the ADM, Procurement should clarify its mandate and continue to implement initiatives that support the program in adapting to a more strategic and value-added role. This clarification should consider the nature and timing of CPAP's assurance work in the contract life-cycle, as well as greater engagement with AB management, contracting officers, industry and stakeholders in support of the achievement of expected audit results.

Management action plan 1: As part of procurement modernization, the Branch will be examining the complementary role of the CPAP, building on the contribution made from working collaboratively with Procurement and DND.

Specifically, the PMI-CPP includes adoption of a program management structure to enable AB to more effectively manage pricing and payment risks associated with non-competitive contracts. This includes consideration for adoption by Canada of best practices and management frameworks found elsewhere, such as the Single Source Regulations Office (SSRO) from the United Kingdom and proactive risk management of pricing and payment risk exposures.

Recommendation 2

To clarify authorities for non-defence contracts so that risks of overpayment can be mitigated, the ADM Procurement in collaboration with the Treasury Board Secretariat, should clarify accountabilities for non-defence procurement contract management, including conduct of risk assessments, determination of assurance requirements, authority to re-assess costs and profits, and authority to recover overpayments. Once clarification is complete, the ADM Procurement should consider the appropriate framework and funding mechanisms to support assurance work on non-defence contracts.

Management action plan 2.1: PSPC will obtain direction from TBS on accountabilities and establish necessary practices to ensure the Minister meets her FAA obligations, beyond those specific to PSPC's current program accountabilities.

Management action plan 2.2: Once the accountabilities for assurance work on non-defence procurement is clarified, PSPC will consult with TBS on the appropriate framework and funding mechanisms in support of program delivery.

Recommendation 3

To facilitate achievement of outcomes related to assurance work, the ADM, Defence and Marine Procurement and the ADM, Procurement should establish mechanisms to support cooperation of suppliers, in particular in relation to requirements for timely provision of access to documents and support for resolution of disputes.

Management action plan 3:  A number of measures have already been put in place to complement standard clauses on Canada's right of access to records and address the primary source of disputes related to a contractor's costing practices.

These include:

• creation of the Price Support Directorate, which provides a basis for obtaining consensus within PSPC on compliance of a contractor's costing practices with Canada's standards
• introduction of the contract audit protocol to explain and clarify Canada's rights and the contractor's obligations
• administrative change to formally assess and explicitly approve acceptability of a contractor's costing practices, as a separate process and precondition to rate negotiations
• adoption of administrative arrangements to resolve differences of opinion on appropriateness of costing practices and accounting treatment of value propositions

Preliminary proposals on Canada's formal assessment of a contractor's costing practices and dispute resolution process are planned to be presented to stakeholder community in September 2016. The stakeholder community is composed of representatives from industry, key client departments and senior procurement officials. This work is conducted under the PMI-CPP within the framework of the Sustainment Initiative, a component part of Defence Renewal. PSPC's proposals draw on recent experience and advice from legal counsel. Canada's standard clauses, including audit clauses, will be revised to support introduction of these administrative changes.

The Branch expects that as a result of changes in approach, Canada's pricing provisions will be clearer and therefore facilitate finalization of contract claims and resolution of differences of opinion on matters related to contract pricing.

Recommendation 4

To demonstrate value of the program, the ADM, Defence and Marine Procurement and the ADM, Procurement should establish a performance measurement strategy and report regularly on results achievement. This framework would more accurately reflect the strategic and value added accountabilities of the program.

Management action plan 4.1: In conjunction with the program renewal proposal to TBS, the Branch intends to review the appropriateness of current measures used to assess program performance and amend or adopt new measures that serve to reinforce integration and the achievement of broader procurement modernization objectives, as established by the ADM AB. New measures are expected to directly assess program effectiveness and economy and build on the insights gained through this program evaluation.

Management action plan 4.2: The program is in the second year of implementing a new management information and control system.  Executive dashboards are planned to reinforce the program's contribution, relevance and reach and support both continuous and on-demand reporting.

Recommendation 5

To facilitate the recovery of overpayments, the ADM, Defence and Marine Procurement and the ADM, Procurement should establish processes to support consistency in the approach to recoveries and consequences for suppliers for non-reimbursement of overpayments.

Management action plan 5:  While the Supply Manual provides guidance on disposition and resolution of potential overclaims, the PwC Review of Canada's Pricing Framework drew attention to inconsistency of practices within PSPC in the administration of matters related to pricing. Inconsistency relates to outdated guidance and has contributed to a divergence of interpretation of mechanics of Canada's cost-based pricing regime.

The administrative changes underway and revision of audit clauses and supporting standard clauses related to pricing are expected to provide greater clarity on the construction of a cost-base for pricing. Further work is required to address the current void on guidance, which exists in the administration of negotiated pricing. This work is being undertaken as part of the PMI-CPP.

Revision of guidance, training and promulgation of changes are expected to foster consistency of practice within PSPC and shared understanding with contractors and client departments on mechanics of cost-based pricing, use of incentives and performance-based pricing provisions. This work is planned to occur over the next 2 fiscal years and be completed by March 31, 2019.

Introduction

1. This evaluation examined the relevance and performance of the Cost and Profit Assurance Program (CPAP), which is administered by the Acquisitions Branch of Public Service and Procurement Canada (PSPC). The CPAP is not identified in the PSPC Program Alignment Architecture (PAA), but by virtue of its position within the AB and its program activities, the CPAP is seen as supporting the Acquisition Program's expected result of "Fair, open and transparent acquisition that provides best value to Canadians and is delivered effectively and efficiently to the satisfaction of government and Canadians". The CPAP's activities also support sub-sub program 1.1.1.1, Acquisition policy and strategic management, in the "management and continuous improvement of government acquisitions, the provision of acquisition-related advice, guidance and oversight, including the development and application of standards and guidelines".

Profile

Background

2. Canada expects suppliers to deliver goods and services in accordance with the terms and conditions of the contracts they enter into with the Crown. These terms and conditions include considerations for the timing (at the time requested), scope (consistent with the specifications requested), and budget (at the price negotiated) for the goods or services which the supplier is expected to deliver. Federal departments are responsible for acknowledging that goods or services were received in accordance with the terms and conditions of contracts, and are expected to certify that this has occurred in accordance with Section 34 of the Financial Administration Act. To support best value in contracting, as well as fair, open, and transparent contracting, any failure of the supplier to deliver in accordance with the terms and conditions of the contract should be managed. PSPC offers the Government two primary tools to manage a failure to adhere to the terms and conditions of contracts: poor performance can be managed through the Vendor Performance and Corrective Measures Policy, while incorrect or inappropriate billings can be managed through the CPAP.

3. The CPAP has a dual role of delivering cost audit services on domestic and international contracts for both the Government of Canada and foreign governments. Since the 1930s, the core function of the CPAP has been conducting defence procurement cost audits and this remains the case today. Until 2011, cost audit services to the Government of Canada were offered on a cost recovery basis by Audit Services Canada, a separate operating agency within PSPC. In 2011, Audit Services Canada was disbanded and its auditors transferred to AB in PSPC.

4. In 2012, CPAP received a Special Purpose Allotment (SPA) of $3M per year for five years ($15M in total) to provide cost audit services in support of defence contracting with the Government of Canada (i.e. domestic) and to test program design changes that support the program in providing better value by engaging through the contract life cycle.

5. The CPAP also offers cost audit services on a fee for service basis to other government departments. In the period from April 1, 2012 to March 31, 2016, it earned fees of $1,755,973 from the Canadian Commercial Corporation (i.e. international) and $1,151,488 from the Real Property Branch of PSPC (i.e. domestic). As the Real Property Branch has established its own cost audit function modeled after the CPAP, this revenue stream will not exist in the future.

6. A number of the CPAP activities, including price and payment risk-assessments, formulating assurance strategies, policy advisory services and meeting Canada's international obligations for assurance services, primarily related to the United States Department of Defense are not covered by the SPA or fee for service work. These are covered by A-Base from the AB and amounted to $1,497,529 during the period from April 1, 2012 to March 31, 2016 (i.e. international).

7. The CPAP cost audit function focusses on domestic, sole source defence contracts awarded by PSPC to Canadian suppliers. These contracts are negotiated based on the cost of production and contain reasonable allowances for profit. These contracts present the greatest risk for overbilling because actual billings rely on supplier transparency during contract negotiations, when the supplier establishes the cost of production, as well as the accuracy and integrity of the suppliers' accounting system which produce the actual billings. Accordingly the Crown's audit rights can serve an important role in the contract management process because they allow the Crown to validate actual cost of production and profit allowance applied. Over the last 5 years, PSPC has issued approximately $10B worth of these types of contracts. Risks in competitive defence contracts are also assessed, but because they are sourced competitively, the risks related to basis of payment are mitigated to some extent. Over the last 5 years, PSPC has issued approximately $25B worth of these types of contracts.

8. Risks in the non-defence contract universe are not routinely assessed as part of the CPAP's audit planning. Over the last 5 years, PSPC has issued approximately $48B worth of these types of contracts. This is consistent with the program funding as the SPA funds are intended for cost audits for defence contracts. In addition, the authorities provided to the Minister of PSP under the Defence Production Act do not extend to non-defence contracts. As such, audit work for non-defence contracts is conducted only upon request by other government departments through memoranda of understanding. As a result, the proportion of non-defence contract work that the CPAP undertakes is modest.

9. In performing its functions, the CPAP undertakes three types of cost audits on these contracts:

• system reviews:  Representing roughly 5% of the work conducted in CPAP, a system review includes determining whether the supplier's internal controls over its accounting systems are designed and implemented to provide reasonable assurance of performing and reporting billing activities in conformance with applicable professional standards. Results of systems reviews are used to improve the billing activities of the supplier. The CPAP has completed 16 domestic systems reviews since the fiscal year starting on April 1, 2012 and ending on March 31, 2013 (the 2012 to 2013 fiscal year)
• contract audits:  Representing approximately 35% of its work, contract audits include a validation of contract costs and a review of the percentage of profit realized on contract costs to ensure that they are in accordance with the terms and conditions of the contract. Results of contract audits are used to recover excess costs and profits. The CPAP has completed 44 domestic contract audits with a contract value of $12.7B in the 4 years since the 2012 to 2013 fiscal year
• overhead audits:  Representing roughly 60% of its work, results of overhead audits are used primarily to support negotiations of refreshed overhead rates on long-term contracts. The CPAP has completed 40 domestic overhead audits with a contract value of almost $1.5B in the 4 years since the 2012 to 2013 fiscal year

Authority

10. Federal contracting is governed by a number of authorities. In terms of contracting authorities that relate to price certification, right to audit, and settlement of final claims, the following authorities apply. The Financial Administration Act assigns responsibilities to the Treasury Board of Canada for setting general administrative policy in the federal public administration for procurement. The Department of Public Works and Government Services Act provides authority to the Minister of Public Service and Procurement (PSP) to procure goods and services on behalf of federal organizations. The Financial Administration Act, section 34 and the Treasury Board Directive on Account Verification, provide authority to the client department to certify price. The Government Contracting Policy and Government Contracting Regulations, assign roles and responsibilities to PSPC and client departments in relation to contracting, including the right for the Crown to audit suppliers through the inclusion of audit clauses in the contract. Lastly, the PSPC Supply Manual also assigns accountabilities to PSPC and client departments, including recognizing the settlement of final claims as a joint accountability of the two parties.

11. Defence contracting is also governed by the Defence Production Act which provides unique powers to the Minister of PSP specifically related to price certification, right to audit, and settlement of final claims on defence contracts. The Defence Production Act provides the Minister of PSP with broad and exclusive authority for matters related to the procurement of defence supplies, including the authority to procure defence supplies, to ensure that costs and profits paid to defence suppliers are fair and reasonable and in accordance with the contracts terms and conditions, and to recover any overpayments identified through the reassessment of costs and profits. In this context and through this authority, the program has the authority to conduct cost audits and PSPC contracting officers have the authority to pursue recoveries. As a result, authorities and accountabilities are very clear.

12. It should be noted the authorities vested in the Minister of PSP under the Defence Production Act do not extend to non-defence contracts. The Department of Public Works and Government Services Act does not provide explicit authority to the Minister of PSP to re-assess costs and profits or to recover identified overpayments. As noted, the Financial Administration Act assigns authority for price certification to client departments. Other guidance documents assign accountabilities to both. As such, it is not clear who owns the authority to re-assess costs and profits for non-defence supplies. The authority to pursue recoveries is less clear. To manage this situation, the CPAP uses memoranda of understanding, which clearly outline the understood and agreed assignment of responsibility between the CPAP and client department to support both the right to audit and right to recover overpayments.

13. Under the Canada-U.S. Defence Production Sharing Agreement, the program supports Canada's international obligations to provide audit services of Canadian defence suppliers to the U.S. Government. As part of reciprocal agreements with international allies (North Atlantic Treaty Organization members), the program also undertakes audits and other assurance engagements on behalf of other governments, primarily the Government of the United States of America.

Roles and responsibilities

14. The Price Support Directorate is responsible for the delivery of the program. The program is located in the AB reporting to the Director General of the Procurement Business Management Sector, who in turn reports to the Associate Assistant Deputy Minister AB. The CPAP is delivered using in-house cost audit expertise. The CPAP has a regional presence in Toronto, Vancouver, Winnipeg, Halifax and Montreal to facilitate access for Government of Canada auditors to supplier records. The regional delivery model also serves to distribute work outside of the National Capital Region and allows for economical business operations through the reduction of travel costs.

15. Within the Price Support Directorate there are two groups involved in the delivery of the CPAP:

• Assurance Services Group:  Thirty four (34) FTEs conduct system reviews, contract audits and overhead audits, and facilitate resolution of audit findings
• Professional Practice Group:  Four (4) FTEs analyze audit results and work in other jurisdictions to identify best practices, develop tools and guidelines for the Pricing Framework, and provide advice on contract terms and conditions related to pricing and price verification

Resources

16. The program currently has three (3) sources of funding. In 2012, CPAP received one-time funding of $15 million over 5 years from the fiscal framework as a SPA for cost audits and related activities of defence contracts. This amount includes the costs related to employee benefit plans, leaving a net amount of approximately $2.3M per year for salaries and operating and maintenance. This funding sunsets in March 2017. Other government departments are served on a fee for service basis via Memoranda of Understanding. A number of the CPAP activities, including price and payment risk-assessments, formulating assurance strategies, policy advisory services and meeting Canada's international obligations for assurance services, primarily related to the United States Department of Defense, are not covered by the SPA or fee for service work. These are covered by A-Base from the AB. There were 34 employees (full time equivalents) working for the program between April 1, 2015 and March 31, 2016 (the 2015 to 2016 fiscal year) and the program budget was $3.581M.

Logic model

17. A logic model is a visual representation that links a program's activities, outputs and outcomes; provides a systematic and visual method of illustrating the program theory; and shows the logic of how a program is expected to achieve its objectives. It also provides the basis for developing the performance measurement and evaluation strategies, including the Evaluation Matrix.

18. As part of the evaluation, a logic model for the program was developed based on a detailed document review, meetings with program managers and interviews with key stakeholders. It was subsequently validated with program staff. The logic model is provided in Exhibit 1.

Exhibit 1: Logic model for the Cost and Profit Assurance Program

Program activities

19. As illustrated in the logic model developed for the program, it performs two main activities to accomplish its objectives. The first activity is core to the program, while the second is Non-core:

• providing assurance on the integrity of pricing and payments of government contracting and supporting federal institutions and international partners
• providing advice and analysis to support innovations in procurement policies and practices

20. The CPAP conducts cost audits to help ensure fair and reasonable cost and profit on contracts with the federal government. This is accomplished through the conduct of assurance work related to systems, contract costing, and overhead costing. In addition to contributing to managing the risk of overpayments, these activities also support the government in discharging international obligations. This is the core activity of the program and the majority of resources are focussed on cost audits of defence contracts.

21. Activities related to providing advice and analysis involve applying the insight gathered from assurance work to identify the risk universe of non-defence contracting, contributing to policy development, and contributing to price negotiation. As this is non-core, the funding for and extent of these activities is limited.

Focus of the evaluation

22. The objectives of this evaluation were to determine the program's relevance and to assess its performance in achieving its expected outcomes in an economical and efficient manner, in accordance with the Treasury Board Policy on Evaluation. The evaluation assessed the program for the period of April 1, 2012 to March 31, 2016. This is the first formal evaluation of the CPAP. This evaluation was conducted to satisfy a requirement identified in the program's funding and authority document. Although this requirement was specific to the program's defence-related assurance work, the decision was made to expand the scope of the evaluation to include an analysis of the program's other activities. As such, the evaluation considered the program's activities in relation to non-defence assurance work and included an analysis of the program's Non-core activities. The evaluation did not include activities outside the accountabilities of the CPAP, such as rate negotiations or recoveries of potential overpayments.

Approach and methodology

23. Five lines of evidence were used to assess the program. These were:

• program data and document review: Data and documents were reviewed to provide information on the program and its context as part of the planning phase of the evaluation, and in determining the answers to the evaluation questions
• literature review: Literature was reviewed to obtain information on the program and its context as part of the planning phase of the evaluation, and in determining the answers to the evaluation questions
• financial analysis: Financial data related to the program's budgets, revenues, expenditures, and staff resources was reviewed
• independent review conducted by internal auditors from the Office of Audit and Evaluation: An auditor, supported by the A/Director of Procurement Audit, reviewed 12 engagement files to assist the evaluation team in identifying whether the CPAP products were timely, credible and objective. As well, the auditor reviewed the risk methodology used by the CPAP to select their engagement activities
• interviews: Interviews were held with 36 key stakeholders from PSPC, other federal organizations, suppliers and defence industry organizations that use, or have in depth knowledge of, the program

24. More information on the approach and methodologies used to conduct this evaluation can be found in the About the Evaluation section at the end of this report.

Findings and conclusions

25. The findings and conclusions below are based on multiple lines of evidence that were used during the evaluation. They are presented by evaluation issue (relevance and performance).

Relevance

26. Relevance assesses the extent to which the program addressed a continuing need, was aligned with federal priorities and departmental strategic outcomes, and was an appropriate role and responsibility for the federal government.

Continuing need

27. The evaluation assessed the extent to which the CPAP continues to address a continuing need. The evaluation found that the need to undertake cost audits of defence contracts persists. The program's cost audit function supports the Minister's authority under the Defence Production Act to re-assess contracts to ensure fair and reasonable cost and profit in defence procurement. Risks of overbilling of excess costs and profits vary depending on a number of factors, however, cost audit, as a tool to validate the accuracy of billings is value added in the government context. The program helps manage both the Department's and Canada's overall risks of overpayment on high risk defence contracts.

28. Through the conduct of 44 domestic contract audits and 40 domestic overhead audits on contracts with a total value of $14.2B in the four years between April 1, 2012 and March 31, 2016, the CPAP has identified potential overbillings of excess costs and profits. These findings were subsequently communicated to both the suppliers and the contracting officers in consideration of resolution/recovery. In addition, through its system review work, the program has identified internal control issues that could have resulted in potential overbilling if not corrected. Of the 16 system reviews conducted between April 1, 2012 and March 31, 2016, the program identified five supplier systems that were deemed non-compliant. As there is no other federal organization providing cost audit services, without the CPAP, these potential overbillings and systems issues would not be identified.

29. Stakeholder interviewees generally expressed strong need for the cost audit function to help ensure fair and reasonable cost and profit in federal government management of defence contracts. Stakeholders also supported the importance of cost audits of large, non-defence sole source contracts, but recognized the program does not have resources for this work and that client departments are not taking advantage of the fee for service option available for these types of contracts. The stakeholders indicated the existence of the program and the audit clauses included in contracts, which provide the government with both the expertise and right to audit, can be sufficient to influence a supplier's behaviour and reduce the risk of excessive costs and profit charged in sole source contracts.

30. The program also meets a continuing need related to Canada's international obligations under the Canada-U.S. Defence Production Sharing Agreement. In providing cost audit services on a fee-for-service basis to the Canadian Commercial Corporation, the CPAP contributes to fulfilling Canada's obligation for ensuring that a fair and reasonable price is paid for defence contracts between Canadian suppliers and the United States Department of Defense.

31. In addition, the evaluation found a demonstrable need for the CPAP with respect to its advisory function. Interviews and documentary analysis indicate that the group has developed in-depth expertise related to contract costing and profit issues that has been useful in informing the scope and approach of assurance engagements and in assisting contracting officers on complex issues related to rate negotiations in sole source contracts. The group has also demonstrated value in supporting the development of new departmental policies related to cost and profit. In 2015, the program worked closely with a team of consultants on a review of Canada's Contract Cost Principles and the Department's Profit Policy. This review provided insight into contracting practices primarily related to defence contracts, both in Canada and abroad. The group is developing an action plan in response to the review's recommendations, which supports the creation of a new PSPC Profit Policy.

Alignment with federal priorities and Public Service and Procurement Canada's Strategic Outcomes

32. The evaluation assessed the extent to which the CPAP is aligned with federal priorities and the PSPC Strategic Outcome. Based on these criteria, the evaluation found the CPAP is aligned with the PSPC Strategic Outcome and federal priorities.

Alignment with federal priorities

33. The 2014 Speech from the Throne committed the government to ensuring open and transparent government. The Prime Minister's 2015 Ministerial Mandate Letter to the Minister of PSP also emphasizes the importance of transparent and open government and fiscal prudence of public funds. Specifically, the letter emphasized that the Department's procurement activities should reflect modern best practices and deploy modern comptrollership.

34. Given the CPAP's activities related to helping to ensure fair and reasonable cost and profit, it demonstrates its support for transparency, fiscal prudence, and best practices related to procurement. Interviews with key stakeholders also emphasized the important role the program plays in helping to ensure fair and reasonable cost and profit, and that the program's existence and involvement in the contracting process supports the prudent stewardship of public funds.

Alignment with the Public Service and Procurement Canada's Strategic Outcome

35. The program aligns well with PSPC's identified Strategic Outcome, which is "to deliver high-quality central programs and services that ensure sound stewardship on behalf of Canadians and meet the program needs of federal institutions." The CPAP's role is seen as supporting the stewardship aspects of the Department's Strategic Outcome.

36. Further, the CPAP is well-aligned to the expected results identified for PSPC's Acquisitions program, element 1.1 on the Department's Program Alignment Architecture. The Acquisitions Program's expected results are to deliver "open, fair and transparent acquisition that provide best value to Canadians in contracting and are delivered effectively and efficiently to the satisfaction of the government". In this context, CPAP's primary objective of assessing costs and determining whether profits are reasonable supports the realization of AB's expected results.

Alignment with federal government and Public Service and Procurement Canada roles and responsibilities

37. The evaluation examined to what extent the responsibility for ensuring fair and reasonable cost and profit aligns with the roles and responsibilities of the federal government and to what extent these responsibilities could be assumed by the private sector or another level of government.

38. The legislated authority for the CPAP derive from the Defence Production Act, the Financial Administration Act, and the Department of Public Works and Government Services Act. As these are federal authorities, the evaluation concluded discharging the federally legislated responsibility for ensuring fair and reasonable costs and profit and contributing to help ensure best value, fairness, and transparency in government contracting should rest with the federal government. This accountability can't be devolved to the private sector or another level of government.

39. With respect to the PSPC's delivery of the CPAP, the evaluation concluded that PSPC was the appropriate organization for the delivery of the function. Under the Defence Production Act, the Minister of PSP is responsible for defence procurement, and has authorities under the act to re-assess the amount paid on contracts to ensure fair and reasonable cost and profit in defence procurement. Given the Minister's clear legislative authority, the evaluation found sound rationale for PSPC to be responsible for the conduct of cost audit functions on defence contracts.

40. Roles and responsibilities related to non-defence contracts are less clear. As noted earlier, authorities vested in the Minister of PSP under the Defence Production Act do not extend to non-defence contracts. The Department of Public Works and Government Services Act does not provide explicit authority to the Minister of PSP to re-assess costs and profits or to recover identified overpayments. Individual departments are responsible for certifying the accuracy of payments per their obligations under Section 34 of the Financial Administration Act and as such, could rely on their own cost audit functions. However, the evaluation found that, given the resident expertise of the CPAP auditors, and that the program already conducts work on certain non-defence contracts on a fee-for-service basis, the CPAP may be an appropriate organization for the provision of cost audit services for non-defence procurement.

41. In terms of delivery of the cost audit function, the evaluation concluded it is appropriate for the federal government to use federal civil servants to conduct cost audits. The program authorities state the cost audit functions performed by the program should not be outsourced to the private sector due to risks of conflicts of interest between the private sector auditor and the supplier, supplier confidentiality being violated by providing access to private sector auditors, and the administrative burden posed by acquiring these particular assurance services from the private sector. As well, with respect to work conducted for international governments, the Canada-U.S. Defence Production Sharing Agreement states that all suppliers under contract through the Canadian Commercial Corporation are "to be placed in accordance with the practices, policies and procedures of the Government of Canada covering procurement for defence purposes."

Conclusions: relevance

42. There continues to be a need for the CPAP for defence contracts as the program aligned with PSPC's authority under the Defence Production Act, identified potential overbillings, and aligned with the federal priorities and PSPC's Strategic Outcome. There is strong federal legislation supporting the federal accountability for re-assessment of cost and profit for defence contracts, as well as support for federal civil servants to deliver cost audits and supporting activities related to defence procurement.

Performance

43. Performance assesses the extent to which the program has achieved its objectives and the degree to which is it able to do so in a cost-effective manner that demonstrates economy and efficiency.

Outcome achievement

Defining the audit universe

44. To deliver on the outcomes of the program, the audit universe used for planning purposes must be sufficiently well defined. The evaluation assessed how well the program defines its 'at risk' contract universe and how the universe informed audit selection and planning. We found the methodology for defining the universe is reasonable based on the program's delivery model.

45. The 'at risk' universe includes defence contracts awarded by PSPC on behalf of Department of National Defence and Department of Fisheries and Oceans in the previous five years. As noted, over the last five years, PSPC has awarded $10B in sole source defence contracts and over $25B in competitive defence contracts. The list of defence contracts is sorted and ranked by supplier based on the largest total dollar value of contracting. It is then further sorted based on sourcing method, i.e. sole sourced or competitive. Contracts are generally selected for audit based on their materiality and sourcing method, although supplier history can also factor into the selection decision. As materiality of contracts and sourcing method are high risk factors and selecting based on materiality provides the greatest coverage, this methodology seems reasonable given the program's delivery model.

46. The universe is defined and audits are selected by the program largely independent of the contracting officers. Further, according to PSPC's Supply Manual, the Department of National Defence and the Department of Fisheries and Oceans determine whether a particular requirement represents a defence or a non-defence contract. This could potentially result in defence contracts considered high risk for reasons other than materiality or sourcing method being excluded from the risk universe.

47. The 'at risk' universe does not include non-defence contracts that are awarded by PSPC on behalf of other client departments. As noted, over the last five years, PSPC has awarded approximately $48B in non-defence contracts. This figure does not include contracts awarded by other government departments under their own contracting authority. The CPAP may undertake cost audits of non-defence contracts on a fee-for-service basis at the request of the client department under a memorandum of understanding. Unlike defence contracts that are issued under the Defence Production Act, the primary responsibility for price certification of non-defence contracts awarded by PSPC rests with the client department, as project authority, and the responsible entity for accepting the price of the goods and services received under section 34 of the Financial Administration Act. This distinction limits the CPAP's ability to examine these contracts, and consequently the program's ability to mitigate risks of overbilling in non-defence contracts.

48. The program included a Departmental Assessment which identified contracts potentially at riskfor the top 25 departments/agencies in its for the period from April 1, 2013 to March 31, 2014 (the 2013 to 2014 fiscal year). This assessment was provided to the Treasury Board Secretariat for information purposes through CPAP's annually reporting exercise. No action was requested of the Treasury Board Secretariat and no additional follow-up was undertaken by the CPAP. The assessment indicated that there are potential unmitigated risks of excess costs and profit on contracts not included in the 'at risk' universe. Although it is outside the CPAP's role to conduct assurance engagements on non-defence contracts without a memorandum of understanding and agreement for recovery for services, the Departmental Assessment provides valuable information for the PSPC contracting authority as well as other government departments. While the authority for conducting cost audits is not explicitly stated in the Department of Public Works and Government Services Act or other legislation, nor is the current demand for service high, the identification of potential risk of overbilling for non-defence contracts provides a measure of evidence of potential value of cost audit activities in relation to non-defence contracts.

49. The CPAP also does not assess the U.S. 'at risk' contract universe for audit planning purposes as it undertakes assurance engagements on a fee for service basis upon request by the Canadian Commercial Corporation and at no cost to the United States Defense Contract Management Agency.

Conducting timely, objective, and credible assurance engagements (system reviews, cost and overhead audits)

50. The evaluation assessed the timeliness, objectivity and credibility of the three types of assurance engagements. Overall, the evaluation found the timing of where in the contract life cycle of assurance engagements are undertaken is being improved, and the outcomes related to timeliness (length of time to complete), objectivity and credibility of the assurance engagements were generally achieved.

51. Formal system reviews have represented a small portion of the CPAP's workload and the number has declined over the past four years from 7% in the 2012 to 2013 fiscal year to 5% in the 2015 to 2016 fiscal year. However, as a regular practice, the program also undertakes assessments of supplier accounting systems as part of contract and overhead audits. These assessments are informal and are used to inform contract and overhead audit scope. As noted previously, contract and overhead audits represent 95% of the assurance engagements conducted by the program.

52. In terms of timing of systems reviews, ideally, for cost-reimbursable and firm price and firm unit price contracts, a system review of a supplier's accounting system takes place early in a contract to help ensure negotiated rates are properly billed. This helps ensure a common understanding of costing principles and criteria required for the costing system to produce proper billing. In terms of timing of contract and overhead audits, the program has made significant effort to focus more on active contracts rather than undertaking post-contract audits as had previously been the practice. Approximately 75% of its contract audits since April 1, 2012 were undertaken during active contracts. In increasing its focus on active contracts, the program is better positioned to be able to support Acquisitions Branch's current and future contract negotiations and improve the ability of the Crown to recover any potential overpayments. Interview evidence suggests that stakeholders strongly support early CPAP engagement for all three types of assurance engagements to better manage risks, reduce the need for recovery, and support relationships with suppliers.

53. As part of the evaluation, the Office of Audit and Evaluation's internal audit staff reviewed a sample of systems reviews, and contract and overheads audits completed by the program during the evaluation period. This review was intended to determine whether CPAP processes were reasonable in the context of the program's objective that its audits/reviews should provide credible, objective and timely assurance. Overall, the auditors found that credible and objective assurance was being provided by CPAP system reviews, contract and overhead audits. It was also noted that the timeliness of CPAP's engagements could be improved. On average, engagement reports are finalized three months after the target deadline (as indicated in the audit planning documents).

54. A significant contributing factor to the timeliness of completion of audit work is due to the suppliers' lack of availability or unwillingness to provide information. Further, the CPAP has limited ability to demand cooperation due to the lack of formal mechanisms in contracting documents to ensure timely access to suppliers' records for audit purposes. While the CPAP has an Engagement Protocol that is shared with suppliers at the beginning of an audit, it is not part of the formal contract management process.

55. The CPAP assurance engagement reports are generally viewed as highly credible and objective, and the vast majority of stakeholder interviewees indicated that, in their view, CPAP auditors are knowledgeable and have the requisite expertise to undertake assurance engagements. Some issues related to the timing and timeliness of work conducted by the program were noted in stakeholder interviews. The following represent the common themes for both domestic and U.S. assurance engagements identified from stakeholder feedback:

• audits need to be done earlier in the contract lifecycle to better manage risk of overpayments and avoid compromising rate negotiations and relationships with suppliers (timing of audits)
• audits take too long and findings are sometimes presented after rate negotiations have been completed (timeliness of audits)
• In the case of Canadian Commercial Corporation, contracts are shorter in duration/low value and audits need to be completed within the first six months (timing and timeliness of audits)
• while findings in audit reports are generally clear, some stakeholders found the reports to be too technical and lacked a clear rationale and sufficient detail to support discussions of findings with suppliers

56. In interviews with stakeholders, the suggestion emerged that the program conduct horizontal audits – assessments of multiple contracts with a single supplier, rather than focusing on individual contracts or continuous audits that are narrower in scope. Stakeholders believe this approach would create a more comprehensive view of a supplier, which would benefit those contracting officers responsible for awarding multiple contracts with one supplier.

Facilitating improved integrity in accounting systems and recoveries

57. The evaluation assessed the extent to which systems reviews contributed to improvements in accounting systems and the contract and overhead audits contribute to the identification and recovery of potential overbillings. The evaluation found the assurance engagements completed by the CPAP contribute to improvements in accounting systems and identification of potential overbillings. It also found recovery of potential overpayments was low, however, this is not linked to the timeliness, credibility, objectivity or quality of the assurance work. As noted above, timing can make recoveries more difficult, but does not preclude them from happening. Recoveries are outside the control of the CPAP.

System Reviews (Improvement in accounting systems)

58. As noted above, the program undertakes both separate, formal system reviews with stand-alone reports and recommendations for improvement, as well as informal system reviews as part of contract or overhead audits. Between April 1, 2012 and March 31, 2016, the program undertook a total of 16 independent system reviews of which five were deemed non-compliant. Of those five, one system has been fixed and confirmed compliant, and three other systems are in the process of being addressedThe fifth system review was conducted at the request of a client department, but the client did not request follow-up action, so none was undertaken by the program.

59. Stakeholders indicated that, in their view, the practice of conducting system reviews has led to improved integrity in suppliers' accounting systems and a consequent reduction in the risk of overbilling. In some cases, following an audit, a supplier has identified its own system errors that have resulted in potential overbillings, and has taken measures to correct them.

Contract and Overhead Audits (Identification and recovery of potential overbillings)

60. Between April 1, 2012 and March 31, 2016, the CPAP undertook 44 contract and 40 overhead audits. As noted in Table 1 below, the program identified $108M in potential overpayments (approximately $92M in defence contracts and $16M in non-defence contracts). With program costs of just under $15M, this represents a ratio of 7:1 of potential overpayments identified to program costs incurred. Over the four year period, recoveries have been $8M, resulting in a 4 year average recovery to cost ration of 0.5 to 1.

61. As a condition of the one-time funding provided by the Treasury Board in 2012, the program committed to achieving a target of 2:1 ratio of recovery to program costs. This target reflected the fact the program was not yet mature. It was also influenced by the U.S. metric of 5:1, adjusted for the fact the U.S. metric includes savings on active contracts (which increases the return on investment ratio), while the Canadian metric does not. With a recovery ratio of 0.5:1, the program was not able to achieve the target recovery ratio of 2:1.

62. As indicated in Table 1, there is a significant gap between potential overbillings identified by the program and amounts recovered to date. It is important to note that efforts to recover amounts are on-going.  However, it is important to note that, while the CPAP helps to facilitate recoveries, and devotes three FTEs to these efforts, it does not have the authority to recover overpayments. For defence contracts issued under the Defence Production Act it is PSPC's contracting officers who are responsible for negotiating recoveries with suppliers who have potentially overbilled the Department. For non-defence contracts, it is not clear where this authority lies – PSPC contracting officers or client departments.

63. The gap between potential overpayments and amounts recovered is significant and points to the need for a more comprehensive strategy to pursue recoveries to support the Minister in discharging her authorities under the Defence Production Act. Opportunities for the CPAP to contribute to greater recoveries are discussed in greater detail in the Program Design and Delivery section, below.

64. The performance target of 2:1 for recoveries to program costs does not align with the program's authorities, thus representing a performance standard for the program that is outside the program's control. More importantly, the evaluation found that this performance metric does not accurately reflect the value proposition of the program. To properly measure the performance of the program, a more appropriate performance measurement framework is required. Ideally, this framework would capture both the impact of the program's assurance work and how the program leverages risk intelligence from the assurance work to help inform ongoing and future procurements. These measures would demonstrate the program's contribution to supplier management by the Crown. Potential measures might include the ratio of overpayments identified relative to program costs (for which, over the four year period, the ratio was 7:1) or cost savings as a result of the assurance engagements. The latter would provide a quantifiable measure of the program's impact and a more tangible demonstration of the program's value proposition.

Understanding and taking action by Stakeholders based on assurance results

65. The evaluation assessed the extent to which Stakeholders understand the results of the assurance engagements and take action based on those results.

66. As noted earlier in the report, since the 2012 to 2013 fiscal year the program undertook a total of 16 independent system reviews. The results of those system reviews indicate that the majority of the suppliers reviewed had adequate systems in place demonstrating that they had put in place appropriate accounting systems. Of the suppliers whose systems were deemed non-compliant by the system reviews, four of the five have resolved the issues or are in the process of doing so. The remaining system did not require CPAP follow-up as the work was conducted by request from a client department.

67. Suppliers consulted as part of the evaluation recognized the audit rights of the CPAP, including access to supplier documents. As noted earlier in the report however, timely access to supplier documents can create delays in rendering completed audit reports. As well, although the evaluation noted suppliers recognize the rights and duties of the program to perform audits, the recovery of a significant amount of potential overbillings and excess profits remain outstanding, which could indicate a lack of understanding of the rights of the Minister of PSP under the Defence Production Act.

Ensuring Canada meets its international obligations

68. The evaluation assessed the extent to which the CPAP supports Canada in meeting its international obligations. The evaluation found the CPAP does support Canada in meeting its international obligations.

69. Canada has obligations to audit Canadian companies on behalf of foreign governments to comply with the Canada–U.S. Defence Production Sharing Agreement and other international agreements. Program Management informed us it has not received any formal complaints from the U.S. or other foreign governments with respect the CPAP not fulfilling any international obligations. Program management indicated to the U.S. Government in June 2012 that they would no longer undertake discretionary audits requested by Defense Contract Management Agency. The U.S. government expressed concern, and indicated that there could be consequences to the Canadian supplier base should CPAP cease to conduct this work on behalf of the U.S. Government. Canadian industry also expressed concern with the Department's plan and indicated their view that a significant amount of trade was being put at risk for a very small cost issue for Canada. However, an agreement was subsequently reached to avoid these consequences and continue to provide assurance services at no cost to the U.S. Government.

70. One of the obligations under the Canada-U.S. Defence Production Sharing Agreement relate to the conduct of cost audits on behalf of the Canadian Commercial Corporation. A memorandum of understanding supports the conduct of this work on a fee-for-service basis. During the period from April 1, 2012 to March 31, 2016, the CPAP completed 19 engagements for the Canadian Commercial Corporation, as well as another 16 engagements which were deemed low risk by the program and for which no activity was undertaken due to this reduced audit requirement. Interviews with Canadian Commercial Corporation stakeholders indicated overall satisfaction with the CPAP audit reports, but indicated the need for reports to be timelier and to provide clearer explanations to support findings. Canadian Commercial Corporation stakeholders also expressed the desire to be consulted on the planning, timing and selection of audits, as well as a desire for greater clarity on how its funding was being allocated for the work performed.

71. Similarly, the U.S. Defense Contract Management Agency official consulted expressed overall satisfaction with the work done by the CPAP for the other obligation under the Canada-U.S. Defence Production Sharing Agreement. However, there is a desire for more information on the CPAP role, types of audits and what to expect. Similar to PSPC officials, Defense Contract Management Agency officials felt that CPAP reports did not provide sufficient information on audit methodologies and rationale for findings, making it difficult to explain audit findings to U.S. buyers.

Providing Timely and Credible Advice and Guidance

72. The evaluation assessed the extent to which the CPAP provides timely and credible advice and guidance. The evaluation found this work to be Non-core to the CPAP. It also found that such advice and guidance has the potential to be value-added to the procurement function and procurement modernization. Further, it found that work in this area is currently limited, but that the program is investing increasing levels of effort in this area.

73. The program has performed work related to timely assessments to identify unmitigated risks to departments with respect to payment and pricing. The program undertook a Departmental Assessment for the top 25 departments/agencies, which identified potential unmitigated risks of excess costs and profits in contracts awarded on behalf of those departments.

74. To support improvements and innovation of procurement policies and practices internally and externally, the CPAP provided assistance and insight towards a review of Canada's Contract Cost Principles and PSPC's Profit Policy. The goal of this review was to identify the issues and provide recommendations to help address issues related to the Principles and Policy. A report was rendered to PSPC in December 2015 and CPAP prepared an action plan at address the recommendations in the report. Although this work is outside of the core work conducted by the CPAP, insight learned from conducting assurance engagements was provided to the team conducting the review, and is expected to be used to help improve procurement policies and practices. As well, the CPAP provided insight and advice in support of DND's Sustainment Initiative which aims to "institutionalize ways to optimize performance, flexibility, value for money and economic benefits through the implementation of sustainment best practices that leverage the capabilities of the Government of Canada and Industry."

75. Stakeholders indicated that timely advice provided by the program could be beneficial during contract negotiations, but that it is not a regular part of the contract negotiation phase. Cost analysts working on rate negotiations are encouraged to consult the auditors if a need arises, but the evaluation found that this type of consultation is ad hoc and infrequent.

Conclusions: performance

76. The evaluation found the 'at risk' universe and the majority of cost audit effort is limited to defence contracts. This is consistent the funding received under the SPA and the authorities vested in the Minister of PSP under the Defence Production Act. While the authority for conducting cost audits is not explicitly stated in the Department of Public Works and Government Services Act or any other legislation, nor is the current demand for service high, analysis of non-defence contracts indicates the potential risk of overbilling. This potential risk provides a measure of evidence of potential value of cost audit activities in relation to non-defence contracts. Overall, the evaluation found the program is achieving its immediate outcomes and contributing to intermediate outcomes, although improvements were identified. The program is achieving its outcomes related to the conduct of objective and credible assurance engagements, it was noted that the timing and timeliness of these engagements could be improved. Furthermore, the evaluation found assurance engagements undertaken by the program contribute to the identification of potential overbillings. To properly measure performance of the program, a more appropriate performance measurement framework would be of benefit to the program. This framework would more accurately reflect the strategic and value added accountabilities of the program, particularly in relation to recovery of overpayments and system improvements. The evaluation also found the CPAP supports Canada in meeting its international obligations although some stakeholders noted a desire for improved timeliness and reporting. With respect to the Non-core activities of the program related to providing advice and insight to support procurement, the evaluation found that although this work is limited in scope, it did provide value to PSPC's procurement functions.

Economy and efficiency

77. Demonstration of economy and efficiency is defined as an assessment of resource utilization (ie. program inputs) in relation to the production of outputs. Economy refers to minimizing the use of program inputs. Efficiency refers to the rate at which resources are used in the production of an output, with greater efficiency realized when the same level or a lower level of resources are used to produce a given level of output. A program has high economy and efficiency when financial resources inputs are minimized while outputs are maximized.

Economy

78. The evaluation assessed the extent to which CPAP has managed budget and resources in an economical manner. The evaluation concluded that the program's funding is not sufficient to cover its activities, as demonstrated by the annual structural deficit and its actual deficits. Despite this, the program has been successful in achieving similar economy to its United States equivalent with slightly lower costs per outputs for the period from April 1, 2014 to March 31, 2015 (the 2014 to 2015 fiscal year).

79. The $2.3M in one-time funding provided in Budget 2012 was intended to represent 50% of total program costs. The balance of funding was to come from PSPC appropriations and fee-for-service memoranda of understanding with other departments. In the 2015 to 2016 fiscal year, the program budget was $3.581M for 34 FTEs. Budget 2012 funding currently covers approximately two-thirds of program costs, resulting in annual structural deficit of $1.3M.

80. Ideally, the program would cover the structural deficit of $1.3M via revenue recovered from fee-for-service work under memoranda of understanding. However, as noted previously, client department demand for non-defence cost audits has not materialized. Further, the nature of fee-for-service revenues is such that it will always vary from year to year, and does not constitute a stable or predictable source of funding. This creates a risk of managing staffing levels, as it is challenging to match ad hoc memoranda of understanding requests, which are not in the program's control, with the permanent FTE/salary liability.

81. The program has incurred budget deficits for the last 4 years (see Table 2). The range in fee-for-service revenue, and the resulting deficits, illustrates the unpredictable nature of fee-for-service funding from client departments. For example, in the 2015 to 2016 fiscal year, the program had anticipated $317K in funding from fee-for-service work which did not materialize, resulting in a significantly larger deficit than anticipated. In addition, the Real Property Branch's recent creation of its own audit function modelled after the CPAP represents a lost revenue stream for the program. The average annual deficit over the past four years has been $345K. A key component of the deficits is the Defense Contract Management Agency-related work which is not funded by the Defense Contract Management Agency.

*Includes SPA funding and revenue from fee-for-service in the following amounts (the 2012 to 2013 fiscal year: $1,177,229; the 2013 to 2014 fiscal year: $869,402; the 2014 to 2015 fiscal year: $985,290; the 2015 to 2016 fiscal year: $535,202)

82. Despite unpredictable funding and systemic deficits, the program has demonstrated similar economy compared to its U.S. equivalent, the Defence Contract Audit Agency (Table 3), which provides audit and financial advisory services of non-competitive defence contracts to the U.S. Department of Defense and other entities. In the 2014 to 2015 fiscal year the CPAP averaged 0.8 audits per FTE, while U.S. Defence Contract Audit Agency averaged 0.9 audits per FTE. Similarly, the CPAP used 1.3 FTEs per assurance engagement, while U.S. Defence Contract Audit Agency used 1.1 FTEs. The average CPAP cost per audit was $80K, while the average Defence Contract Audit Agency cost was $112K.

83. The Defence Contract Audit Agency operates under the authority and direction of the Under Secretary of Defense/Chief Financial Officer. While its methodologies differ from Canadian practice, its objectives are similar: to ensure contract costs are reasonable and best value in procurement is achieved. Like CPAP, the Defence Contract Audit Agency has no direct role in determining which company is awarded defense contracts but provides support to contracting officers to support them in negotiating prices and settle final payments.

84. Table 4 shows the volume of work produced by CPAP by type from April 1, 2012 to March 31, 2016, including contract value, total costs to produce the outputs and the average cost to produce the outputs.

85. The average annual cost to produce all assurance engagements was $80K over the period April 1, 2012 to March 31, 2016. The cost of audits (contract audit and overhead audit combined) varied widely from $47K for U.S. related work to $91K for Canadian audits.

Efficiency

86. The evaluation assessed how well the program utilized its resources to manage business volumes. The assessment found the program saw their hourly labour rates decrease in the period from April 1, 2012 to March 31, 2016. Despite this increase in program efficiency, the evaluation was unable to determine the achievement of program efficiency due to a lack of comparable data to benchmark the program against.

87. Over the four year period outlined in Table 5, CPAP devoted roughly 80% of resources/effort to domestic activity and 20% of resources/effort to U.S. activity. The program's projected labour rates have decreased from a high of $126 per hour in the 2012 to 2013 fiscal year to a low of $99 per hour in the 2014 to 2015 fiscal year. Although the program saw an increase in labour rates last fiscal year, CPAP's trend prior to the 2015 to 2016 fiscal year was indicative of improved resource utilization and efficiency, despite budgetary pressures.

88. The level of effort devoted to cost and overhead audits and system reviews has remained relatively stable over the 4-year period. The level of effort and resources devoted to U.S. work (Canadian Commercial Corporation and Defense Contract Management Agency) has increased – from 20.8% in the 2012 to 2013 fiscal year to 24.2% in the 2015 to 2016 fiscal year. U.S.-related work now represents nearly 25% of total program spending. If the trend continues, there will be continued and increasing pressure on resources and costs required for U.S.-related work.

Conclusions: economy and efficiency

89. With respect to program economy, the CPAP has worked towards managing its structural deficit by performing work on a fee-for-service basis, although the current blended funding model creates a risk of incurring deficits. The program has worked to produce a similar level of program outputs compared to its United States equivalent.

90. With respect to program efficiency, decreased program labour rates over time indicate improvements in resource utilization. The evaluation however was not able to conclude on program efficiency due to a lack of data to benchmark the program against.

Program design and delivery

91. Design and delivery assesses the extent to which the design of the delivery model supports the achievement of outcomes.

92. The current program delivery model is a blend of advisory and oversight functions. Although it currently operates primarily as an oversight function, the program has been moving towards an advisory role. Where it provides advisory type services, these are not always linked to the results of its oversight activities. The evaluation concluded the program may benefit from clarifying the delivery model that best aligns to its expected results and obtaining support and funding to implement that model. It was noted that more recently the program is implementing a number of changes that clarify its role to improve its impact.

93. As an oversight function, contribution would begin late in the contracting process. The primary client would be the Deputy Minister. The program would independently define the audit universe and audit selection. Results would be reported to the Deputy Minister for action and follow-up. As advisors, the program would be an integral part of and contribute through the entire life cycle of the contract. The primary clients would be the contracting officers and senior AB management. It would provide non-binding strategic advice and insight that could inform contracting officers at all phases of the contracting processes. Exhibit 2 outlines the activities of the two models.

Exhibit 2: Impact of Cost and Profit Assurance Program intervention during contracting lifecycle

Current state

Pre-Award

94. Under the current model, engagement at the pre-award stage is limited, basically following the oversight model. Only in exceptional circumstances do contracting officers request assistance from the CPAP at this stage. As a result, the opportunity to provide advice and insight into RFP development, including appropriate audit clauses and rate negotiation clauses is lost.

Contract negotiations

95. Under the current model, engagement at the contract negotiation stage is also limited, again following the oversight model. Although cost analysts from within the Branch participate in the determination of rates, this is done largely based on Canada's Contract Cost Principles and Profit Policy and is not generally informed by the assurance work performed by the CPAP. As a result, the opportunity for information and insight from assurance engagements to inform rate negotiations is lost.

During contracting: Audit planning

96. There is currently little to no engagement between the CPAP and contacting officers or senior branch management in defining the 'at risk' universe or in establishing the timing and selection of audits, following neither the oversight model nor advisory model. As previously described, the program defines the 'at risk' universe based on its own analysis, which it uses to develop an Annual Operating Plan. The Plan outlines planned audits and resource allocation, but it is used primarily as a budget and work planning tool and is not shared with PSPC contracting officers or formally approved by senior branch management.

97. It was noted in interviews that the lack of program engagement with contracting officers and senior management in defining the 'at risk' universe and influencing the timing and selection of audits limits the program's ability to maximize its value added potential. Audits selected by the program are not necessarily focused on those contracts that contracting officers feel could provide the most value added. It could also result in higher risk contracts not being selected. Further, the opportunity for assurance results to contribute to rate negotiations on specific contracts is also limited, due to the limited influence of contracting officers' views on timing and selection. Finally, appropriate timing may not be fully considered, resulting in audits that are finalized after contract negotiation is complete or after contracts have ended, at which point it becomes more difficult to recover overpayments.

98. CPAP officials have acknowledged that the current approach is transactional and there is a need for more strategic engagement with the PSPC contracting function. Contracting officers who are aware of the program have expressed the desire to have audit results before rates are negotiated in order to help ensure best value is achieved. In keeping with program renewal commitments made in 2012 to conduct audits while contracts are still active, the CPAP is devoting greater attention to audits of active contracts. Because of the lack of engagement in audit planning, when a stakeholder who is aware of the program issues an ad hoc requests they are not included in the Annual Operating Plan and consequently cannot always be addressed, or are delayed due to a lack of available resources.

During contracting: Audit conduct

99. There is also little to no engagement of contracting officers in the audit conduct phase, following the oversight model. Engagements are undertaken directly with the supplier and departmental contracting officers may not be advised in advance. Once an audit is completed, a notification letter is sent to the contracting officer advising of the audit findings and requesting a management action plan to address issues and recoveries within 30 days of receipt.

100. Contracting officers indicated challenges for them as contract managers when they are not involved in the audit conduct phase. They explained the notification letter and request for a management action plan may be the first time they become aware of the audit and they may not have the knowledge or experience required to handle the audit findings in an efficient and timely way. They may also be limited in what action is available for them to take if the contract is already closed and the final payment has already been made. CPAP officials have acknowledged the need for the program and contracting officers to work more collaboratively.

Post contracting

101. As noted above, the results of the assurance work is provided to the contracting officer with a request for a management action plan, that identifies the plan for resolving audit issues, including recovery, to be submitted to the CPAP within 30 days of the audit report. The program monitors the management action plan and briefs AB's Executive Committee twice annually. This model is a mix of advisory and oversight.

102. The Supply Manual (10.70) outlines the process for recovery. However, there is no clear articulation in the Supply Manual of PSPC legal authorities to determine reasonable cost and profit in defence contracts or to undertake audits pursuant to the Defence Production Act. Furthermore, contract terms and conditions reviewed as part of the evaluation did not explicitly outline PSPC authorities to pursue recovery under the Defence Production Act. As a result, most contracting officers are unaware of their responsibilities to seek recovery of potential overpayments as part of their contract management responsibilities.

103. As previously noted, the CPAP has identified recoveries of $108M during the period from April 1, 2012 to March 31, 2016. Recoveries on those findings has amounted to $8M during the same period. Further, where there were recoveries, the evaluation noted inconsistency in the approach to recovery. Rather than recovery of the overpayment, overpayments can be "offset" against future payments with the same supplier, are negotiated down to a different amount, or are abandoned altogether.

104. Contracting officers within PSPC who had previous experience negotiating recoveries felt they were in a conflict of interest, as they were responsible for negotiating the contract rates in the first place. They indicated wanting greater flexibility to offset excess profit against future payments or against other contracts managed by the same supplier for contracts with the same client department, in lieu of requesting re-payment from suppliers, which would allow recoveries to benefit the client department. They also noted challenges to recoveries, such as the negative impact it can have on their relationship with the supplier.

105. Suppliers noted concerns with the fact they do not see a single departmental position for pursuing recoveries or approach to recovery and raised questions of consistency, fairness and transparency. Stakeholders indicated that the recovery process has no "teeth" to require recovery and there is a lack of standardized criteria for pursuing recovery. There are no formal incentives or penalties to repay excess costs and profits in a timely manner. In addition, the Supply Manual is not binding, and therefore frequently suppliers argue there is no contractual obligation to repay.

106. There is no defined dispute resolution mechanism in the event a supplier or client department disagrees with the identified recovery amount. Contracting officers have no mechanism to require recovery. The Defence Production Act provides that if a supplier disagrees with the recovery amount, the supplier and/or the Crown can seek resolution through the Courts, however, in practice contracting officers have not pursued recoveries through the courts. To avoid such escalation, current practice is to search for workarounds (for example offsets against future payments and accepting partial or no recovery).

Defining program mandate and responsibilities

107. The PSPC Supply Manual (s. 10.70) intended the program to work closely with contracting officials, stipulating that "throughout the assurance engagement, there is close collaboration and regular communication among all key stakeholders, specifically the Assurance Advisor, Cost Auditor, PSPC contracting officer, client department representative and the Cost Analyst". In practice however, the CPAP has been operating largely independently of contracting officers and client departments.

108. The majority of interviewees felt the CPAP should remain in PSPC given the authorities provided to the Minister under the Defence Production Act. Further, stakeholders also supported the importance of cost audits of large, non-defence sole source contracts. Many also indicated mixed views on what the program's role should be - an advisory function to contracting officers as part of contract management or an oversight function on behalf of the Deputy Minister in the traditional sense, as with Internal Audit.

109. Although the Defence Contract Audit Agency reports independently to the U.S. Under Secretary of Defence, it works closely with U.S. contracting officials to establish timing, audit priorities and plans. Its model aligns more closely with that of an advisory role. Canadian private sector cost auditors also acknowledge that client involvement in audits is essential to ensure weaknesses are understood and relationships with suppliers are well managed, which would also align more closely with an advisory role.

Conclusions: design and delivery

110. The current delivery model is limiting the program's effectiveness and the department's ability to appropriately manage relationships with stakeholders. This includes greater engagement with stakeholders throughout the lifecycle of the contract. The benefits of greater engagements include better selection of assurance engagements, the opportunity for assurance engagements to contribute to rate negotiations, and greater likelihood of recovery of identified potential overpayments. We were informed the program is implementing a number of changes that clarify its role to improve its impact. Decisions regarding the program's mandate and who is its primary client are required to inform future funding and best organizational fit and design.

Recommendations and management action plan

Recommendation 1

To maximize contribution of the program to outcomes and broader procurement modernization objectives in both defence and non-defence contracting, the Assistant Deputy Minister (ADM), Defence and Marine Procurement and the ADM, Procurement should clarify its mandate and continue to implement initiatives that support the program in adapting to a more strategic and value-added role. This clarification should consider the nature and timing of CPAP's assurance work in the contract life-cycle, as well as greater engagement with AB management, contracting officers, industry and stakeholders in support of the achievement of expected audit results.

Management action plan 1: As part of procurement modernization, the Branch will be examining the complementary role of the CPAP, building on the contribution made from working collaboratively with Procurement and DND.

Specifically, the PMI-CPP includes adoption of a program management structure to enable AB to more effectively manage pricing and payment risks associated with non-competitive contracts. This includes consideration for adoption by Canada of best practices and management frameworks found elsewhere, such as the Single Source Regulatory Office (SSRO) from the United Kingdom and proactive risk management of pricing and payment risk exposures.

Recommendation 2

To clarify authorities for non-defence contracts so that risks of overpayment can be mitigated, the ADM Procurement in collaboration with the Treasury Board Secretariat, should clarify accountabilities for non-defence procurement contract management, including conduct of risk assessments, determination of assurance requirements, authority to re-assess costs and profits, and authority to recover overpayments. Once clarification is complete, the ADM Procurement should consider the appropriate framework and funding mechanisms to support assurance work on non-defence contracts.

Management action plan 2.1: PSPC will obtain direction from TBS on accountabilities and establish necessary practices to ensure the Minister meets her FAA obligations, beyond those specific to PSPC's current program accountabilities.

Management action plan 2.2: Once the accountabilities for assurance work on non-defence procurement is clarified, PSPC will consult with TBS on the appropriate framework and funding mechanisms in support of program delivery.

Recommendation 3

To facilitate achievement of outcomes related to assurance work, the ADM, Defence and Marine Procurement and the ADM, Procurement should establish mechanisms to support cooperation of suppliers, in particular in relation to requirements for timely provision of access to documents and support for resolution of disputes.

Management action plan 3:  A number of measures have already been put in place to complement standard clauses on Canada's right of access to records and address the primary source of disputes related to a contractor's costing practices.

These include:

• creation of the Price Support Directorate, which provides a basis for obtaining consensus within PSPC on compliance of a contractor's costing practices with Canada's standards
• introduction of the contract audit protocol to explain and clarify Canada's rights and the contractor's obligations
• administrative change to formally assess and explicitly approve acceptability of a contractor's costing practices, as a separate process and precondition to rate negotiations
• adoption of administrative arrangements to resolve differences of opinion on appropriateness of costing practices and accounting treatment of value propositions

Preliminary proposals on Canada's formal assessment of a contractor's costing practices and dispute resolution process are planned to be presented to stakeholder community in September 2016. The stakeholder community composed of representatives from industry, key client departments and senior procurement officials. This work is conducted under the PMI-CPP within the framework of the Sustainment Initiative, a component part of Defence Renewal. PSPC's proposals draw on recent experience and advice from legal counsel. Canada's standard clauses, including audit clauses, will be revised to support introduction of these administrative changes.

The Branch expects that as a result of changes in approach, Canada's pricing provisions will be clearer and therefore facilitate finalization of contract claims and resolution of differences of opinion on matters related to contract pricing.

Recommendation 4

To demonstrate value of the program, the ADM, Defence and Marine Procurement and the ADM, Procurement should establish a performance measurement strategy and report regularly on results achievement. This framework would more accurately reflect the strategic and value added accountabilities of the program.

Management action plan 4.1: In conjunction with the program renewal proposal to TBS, the Branch intends to review the appropriateness of current measures used to assess program performance and amend or adopt new measures that serve to reinforce integration and the achievement of broader procurement modernization objectives, as established by the ADM AB.  New measures are expected to directly assess program effectiveness and economy and build on the insights gained through this program evaluation.

Management action plan 4.2: The program is in the second year of implementing a new management information and control system.  Executive dashboards are planned to reinforce the program's contribution, relevance and reach and support both continuous and on-demand reporting.

Recommendation 5

To facilitate the recovery of overpayments, the ADM, Defence and Marine Procurement and the ADM, Procurement should establish processes to support consistency in the approach to recoveries and consequences for suppliers for non-reimbursement of overpayments.

Management action plan 5:  While the Supply Manual provides guidance on disposition and resolution of potential over-claims, the PwC Review of Canada's Pricing Framework drew attention to inconsistency of practices within PSPC in the administration of matters related to pricing. Inconsistency relates to outdated guidance and has contributed to a divergence of interpretation of mechanics of Canada's cost-based pricing regime.

The administrative changes underway and revision of audit clauses and supporting standard clauses related to pricing are expected to provide greater clarity on the construction of a cost-base for pricing. Further work is required to address the current void on guidance, which exists in the administration of negotiated pricing. This work is being undertaken as part of the PMI-CPP.

Revision of guidance, training and promulgation of changes are expected to foster consistency of practice within PSPC and shared understanding with contractors and client departments on mechanics of cost-based pricing, use of incentives and performance-based pricing provisions. This work is planned to occur over the next 2 fiscal years and be completed by March 31, 2019.

About the evaluation

Authority

The Deputy Minister for Public Services and Procurement Canada (PSPC) approved this evaluation, on recommendation by the Audit and Evaluation Committee, as part of the April 1, 2013 to March 31, 2018 Risk-Based Audit and Evaluation Plan.

Evaluation objectives

This evaluation examined the Cost and Profit Assurance Program (CPAP). The program is administered by the AB of PSPC. The CPAP is not identified in the PSPC Program Alignment Architecture. However, located within the AB, CPAP is supporting the Acquisition Program expected result (Section 1.1 of PSPC's 2015 to 2016 fiscal year Program Activity Architecture) of "Fair, open and transparent acquisition that provides best value to Canadians and is delivered effectively and efficiently to the satisfaction of government and Canadians." The evaluation had two objectives:

• to determine the relevance of the program: the continued need for the program, its alignment with governmental priorities and departmental outcomes and its alignment with federal roles and responsibilities
• to determine the performance of the program: the achievement of its intended outcomes and a demonstration of economy and efficiency by the program

Approach of the evaluation

The evaluation was conducted in accordance with the Standard on evaluation for the Government of Canada. The evaluation took place between October 2015 and April 2016 and was conducted in four phases: pre-planning, planning, examination, and reporting. The evaluation Directorate completed a risk-based calibration assessment to determine the best approach for the conduct of this evaluation. An Evaluation Matrix was developed using the Treasury Board's five core issues to be addressed in evaluations. The following data sources were used to inform the lines of evidence to answer the questions in the Evaluation Matrix:

• program data and document review: The program document review involved an assessment of primary documents that provided information on the program and its context for the planning phase and also provided information for the assessment of the evaluation questions. The primary documents included program administration, monitoring and reporting documents, such as reports, meeting minutes, studies, as well as financial records
• literature review: The initial literature review involved an assessment of secondary documents that provided information on the program and its context for the planning phase and also provided information for the assessment of the evaluation questions. The secondary documents included: departmental documents (such as annual Reports on Plans and Priorities and Departmental Performance Reports), government documents (such as legislative and policy documents), and external publications
• financial analysis: Financial data related to the program's budgets, revenues, expenditures, and staff resources were reviewed
• interviews: Twenty nine structured interviews with a total of 36 participants were held with key stakeholders from PSPC officials, other government departments, suppliers and other federal organizations
• independent review conducted by internal auditors from the Office of Audit and Evaluation (OAE): An auditor, supported by the A/Director of the Procurement Audit Directorate within the OAE, reviewed 12 engagement files in order to assist the evaluation team in identifying whether the CPAP products were timely, credible and objective. As well, the auditor reviewed the risk methodology used by CPAP to select their engagement activities

Limitations of the evaluation

While the evaluation relied upon a broad range of information sources, both qualitative and quantitative, it is important to note that there are limitations to social science research. Key evaluation issues were targeted during the evaluation to respond to TB Policy on evaluation questions, whereas the program's overall performance could have been measured in a number of other ways. In addition, as the evaluation team was unable to directly compare the program's performance against an identical program in another jurisdiction, and every jurisdiction bases their cost-audit functions on different principles, it was not possible to benchmark the program against a similar program

Reporting

Based on the lines of evidence, the evaluation team prepared Director's Draft Observations and Conclusions for Validation. These observations and conclusions will be reviewed by the Quality Assessment and Practice Improvement Directorate within the OAE and by the Chief Audit and Evaluation Executive. Subsequently, it will be sent to the Director General responsible for the delivery of the CPAP for review and comments. Validation of facts and any other comments on the Director's Observations and Conclusions for Validation will be assessed and, where appropriate, incorporated in the Chief Audit and Evaluation Executive's Observations and Conclusions for Validation. This report will be sent to the ADM, Procurement for final acceptance. The draft Final Report will be tabled at the Audit and Evaluation Committee for the Deputy Minister's approval in October 2016. The Final Report will be submitted to the Treasury Board of Canada Secretariat and posted on the PSPC website.

Project team

The evaluation was conducted by evaluation employees of the Office of Audit and Avaluation, supported by members of the Procurement Audit team, and overseen by the Director of Evaluation and under the overall direction of the Chief Audit and evaluation Executive. The evaluation was reviewed by the Quality Assessment Practice Improvement function of the Office of Audit and Evaluation.