System Under Development Audit – eManifest

Redacted

Table of Contents

• Executive Summary
• 1.0 Introduction
  • 1.1 Background
  • 1.2 Risk Assessment
  • 1.3 Audit Objective and Scope
  • 1.4 Approach and Methodology
  • 1.5 Audit Criteria
  • 1.6 Statement of Assurance
• 2.0 Audit Opinion
• 3.0 Findings, Recommendations and Action Plans
  • 3.1 Business Requirements
  • 3.2 Procurement
• Appendix A: Audit Criteria
• Appendix B: List of Acronyms

[ * ] An asterisk appears where sensitive information has been removed in accordance with the Access to Information Act and Privacy Act.

Executive Summary

Background

eManifest is the third phase of the Advance Commercial Information (ACI) program. When fully implemented in 2014, it will improve the Canada Border Service Agency's (CBSA's) ability to gather commercial data on all modes of transport before commercial shipments enter Canada. This information will enable the CBSA to detect shipments that could pose a risk to the safety and security of Canadians.

The audit-planning work that the Internal Audit and Program Evaluation Directorate (IAPED) completed in December 2008 recommended three separate audits to provide assurance on the management control framework for the eManifest project. The first audit, which focused on the project management framework, was completed by the IAPED in September 2009. This audit — the second — examined the processes for defining business requirements and for procuring Information Technology (IT) goods for the project. It was conducted in Headquarters between February and October 2010 and reviewed project activities carried out from November 2009 to October 2010. The third audit, currently planned for initiation in fiscal year 2011–2012, will focus on the implementation of new CBSA business processes resulting from eManifest, and the training of CBSA users in these new processes. As well, an IAPED program evaluation of eManifest that started in fall 2010 will assess the quality of communications and consultations with stakeholders, and identify any communications issues.

Significance of This Audit

This audit should be of interest to management, first because this is a large project valued at an estimated $396 million. Second, eManifest is a key element in facilitating trade between Canada and the United States, to which the government has given high priority. Accordingly, it is important that senior management have assurance on the adequacy of controls to plan and manage the development of business requirements and procurement, two areas which are key factors to the successful delivery of the eManifest project.

Objective and Scope

The objective of the audit was to provide assurance to senior management that the processes for defining business requirements and managing procurement for the eManifest project were adequate and effective.

The scope included the processes for involving users in defining the project's business requirements (what eManifest should be able to do). It also included the processes for ensuring that approved business requirements would fall within the scope of the project, that requirements were adequately documented and that any changes to approved requirements were adequately tracked and managed.

Regarding procurement, the scope included the processes for managing and controlling the procurement of goods (software and hardware) so that the right products would be available when needed for developing and implementing eManifest. The procurement of some goods needed for eManifest that was managed under a separate project (the IT Pre-requisites Project) was beyond the scope of this audit.

Audit Opinion

The audit concluded that the methodology and tools used to define and manage business requirements were adequate. The audit noted weaknesses in the ability of eManifest to [*], and in managing the quality of requirements in terms of their completeness and usefulness to the project's software developers. Project management was aware of these concerns and was working to address them.

The audit found that there was a strategy for managing and controlling procurement activities, but that it had not been followed in some respects. Areas of concern included finalizing eManifest's technical design and technology-acquisition plan, and the implementation of standard processes for monitoring and controlling procurement. Project management was also aware of these concerns and was working to address them.

Statement of Assurance

The audit engagement was planned and conducted in accordance with the Internal Auditing Standards for the Government of Canada.

Key Findings

The processes for defining and documenting eManifest's business requirements were generally adequate. Mechanisms existed to ensure that requirements fell within the project's scope as originally approved. However, some requirements had not been documented in enough detail to guide the design and development of the system.

[*] Interim CIG is part of the existing CBSA technology that eManifest uses to receive data from highway carriers. [*]

While the audit found that there was a strategy for planning and managing procurement, weaknesses in planning the procurement of certain hardware for eManifest were noted. Key decisions about what hardware the system would use to support particular software had not been made, which created the risk of delays in two of eManifest's critical components. Both are central to strengthening the Agency's ability to identify the risks associated with commercial shipments entering Canada.

Responsibility for managing and coordinating procurement was divided between the Programs Branch and the Information, Science and Technology Branch. Accordingly, a single focal point for managing procurement activities was lacking. Of note is that there was no integrated schedule that would alert procurement staff as to when to order and deliver items in time to meet project development milestones.

The audit found weaknesses in monitoring and reporting on the procurement budget status. The financial information that was available did not indicate estimated costs to complete the procurement over time. Therefore, senior management could not tell whether procurement would be completed within its budget. Of concern is that the audit team noted that cost overruns were a possibility for procurement of hardware needed to strengthen the Risk Assessment component of eManifest.

Recommendations

The audit has made recommendations on:

• [*]
• expediting hardware decisions needed to support two of eManifest's critical components;
• preparing an integrated procurement plan and schedule to help mitigate potential procurement delays; and
• improving assurance that the eManifest procurement budget is appropriate to complete remaining procurement.

Management Response

The Programs Branch, Information, Science and Technology Branch and Comptrollership Branch agree with the recommendations in this report.

1.0 Introduction

1.1 Background

This audit was identified as a high priority audit in the Canada Border Services Agency's (CBSA's) Three-year Risk-based Audit Plan for Fiscal Years 2007-2008 to 2009-2010, which was approved by the Audit Committee on October 24, 2007. The Internal Audit and Program Evaluation Directorate (IAPED) completed its audit-planning work in December 2008. At that time, it recommended three separate audits to provide assurance on the management control framework for the eManifest project.

The first audit was completed in September 2009 and focused on the project-management framework. This audit — the second — examined the processes for managing the definition of business requirements and the procurement of Information Technology (IT) goods for the eManifest project. Both of these processes are critical in large application-development projects to prevent delays, re-working and cost overruns. The third audit, currently planned for initiation in fiscal year 2011–2012, will focus on the implementation of new CBSA business processes resulting from eManifest, and the training of CBSA users in these new processes. As well, an IAPED program evaluation of eManifest that started in the fall 2010 will assess the quality of communications and consultations with stakeholders, and identify any communications issues.

eManifest is Phase III of the Advance Commercial Information (ACI) program. When fully implemented in 2014, it will further strengthen the CBSA's ability to gather commercial data on all modes of transport before commercial shipments enter Canada. This information will enable CBSA to detect shipments that could pose a risk to the safety and security of Canadians.

eManifest received Effective Project Approval on November 29, 2007, as a Major Crown Project with an estimated cost of $396 million and a completion date in the fourth quarter 2011–2012. In February 2009, the project completion date was revised to 2014 to reflect a funding shift for the eManifest project as part of the CBSA's reduction of Reference Levels for two years in support of Canada's Economic Action Plan.

The eManifest budget included funding for the procurement of some IT goods that was managed under the separate IT Pre-requisites Project.

eManifest is a multi-year project. During the period under audit review, a major focus for the project was to enable highway carriers to submit cargo and conveyance data using Electronic Data Interchange (EDI) or a secure portal.

The IAPED carried out the audit in Headquarters between February and October 2010. It included the review of project activities undertaken during the period of November 2009 to October 2010.

1.2 Risk Assessment

To update the risk assessment for the areas in the scope of this audit, the audit team interviewed the project's sponsor and representatives from the project team, and reviewed project documentation. This work identified the following key risks:

• Business Requirements: The complexity of the project and numerous changes to the project's schedule create a potential risk that the original business requirements approved for the eManifest project will not be met, and that the project will miss its milestones for implementation.
• Procurement: An estimated procurement schedule of 18 to 24 months, the need to ensure that procured items comply with the CBSA technology standard and a lack of coordination in managing and coordinating procurement activities increase the risk that procurement will not be completed in time to meet the eManifest development and deployment schedule.

1.3 Audit Objective and Scope

The objective of the audit was to provide assurance to senior management that the processes for defining business requirements and managing procurement for the eManifest project were adequate and effective.

The scope included the processes for involving users in defining the project's business requirements (what eManifest should be able to do). It also included the processes for ensuring that approved business requirements would fall within the scope of the project, that requirements were adequately documented, and that any changes to approved requirements were adequately tracked and managed.

Regarding procurement, the scope included the processes for planning and scheduling the procurement of goods (software and hardware) so that the right products would be available when needed for developing and implementing eManifest.

Procurement managed under the separate IT Pre-requisites Project was beyond the scope of this audit.

1.4 Approach and Methodology

The audit examination used the following approach and methodology:

• Interviewing eManifest project managers, project directors, team leaders and team members, sponsor branch representatives, IT Security, IT Architecture, IT Engineering, IT Solutions, and Departmental Security Officer (DSO) representatives to identify and assess the framework for managing the areas in the audit scope.
• Reviewing a sample of business requirements documents to assess the processes for developing requirements, reviewing quality, and approving the requirements.
• Reviewing the development of requirements and procurement schedules to assess the extent to which they were integrated and supported efforts to meet the project's milestones.
• Reviewing the procurement-planning process, including procurement approval decisions and compliance of planned procurements with CBSA technology standards.
• Reviewing change-control documentation to assess the application of the change control process.
• Reviewing project risk and issues logs to assess the process for identifying, tracking and mitigating risks relating to business requirements and procurement.

1.5 Audit Criteria

Please see Appendix A for a list of audit criteria.

1.6 Statement of Assurance

The audit engagement was planned and conducted in accordance with the Internal Auditing Standards for the Government of Canada.

2.0 Audit Opinion

The audit concluded that the methodology and tools used to define and manage business requirements were adequate. The audit noted weaknesses in the ability of eManifest to [*], and in managing the quality of requirements in terms of their completeness and usefulness to the project's software developers. Project management was aware of these concerns and was working to address them.

The audit found that there was a strategy for managing and controlling procurement activities, but that it had not been followed in some respects. Areas of concern included finalizing eManifest's technical design and technology-acquisition plan, and the implementation of standard processes for monitoring and controlling procurement. Project management was also aware of these concerns and was working to address them.

3.0 Findings, Recommendations and Action Plans

3.1 Business Requirements

Business requirements for eManifest represent or specify what it must be able to do in order to contribute to meeting the project's objectives and mandate. A significant area of activity in large IT-enabled projects is the analysis, definition and approval of business requirements. Diligence and care in establishing requirements are essential to avoid the need to rework the requirements and system components already under development, which could cause delays and increase costs.

Requirements Definition

The audit criterion for this area requires organizations to have a formal process for analyzing, defining and documenting the needs and expectations of the users of a system.

The use of a standard process facilitates communication between all parties involved in requirements definition, and contributes to ensuring that the requirements are complete and accurately specified. We found that the Agency had met this criterion, and that it had processes in place to define, analyze, manage and approve business requirements for the functional areas covered by this audit. The process included various methodologies and tools for defining requirements, and project members had been trained to use them. As required, consultants had been engaged to support any eManifest team members who were not experienced in using the methodology, or to assist with developing requirements.

Reviewing the quality of requirements

According to our criteria, requirements definition activities should include a quality-review process to verify that requirements have been clearly documented.

If requirements are not completely and accurately specified and documented, there is a risk that additional downstream work will have to be done to correct or clarify the requirements as documented so that they may be useful in guiding those who are responsible for designing and building the system.

We found that the quality-review mechanisms for requirements needed to be improved. As documented, they did not always provide software developers with enough of the right type of information on requirements to proceed with their work. However, project management had recognized this fact, and before the audit team had completed its work, the problem was being resolved. One of the changes implemented was a more stringent review of requirements documentation before software development work starts.

Change Control

The audit criterion for this area requires that where business requirements change, any changes must be managed and controlled.

Approved business requirements are the basis for planning and delivering future project activities. Consequently, any changes in requirements must be carefully considered to assess their impact, if any, on project schedule, cost, scope or risk. The audit found that the Agency had an adequate process for managing changes to requirements and assessing their downstream implications and risks for the project's scope, cost and complexity.

Specifically, the documentation that the audit team reviewed indicated that all changes had been assessed in terms of their overall impact on the project, and most had been subjected to an analysis of their impact on scope, cost and schedule.

Security Requirements

The audit criterion relating specifically to this area says that any requirements for data security and privacy must comply with related government security policies and standards.

As part of defining requirements, the technical performance needed to meet these requirements must be considered in order to identify any impacts on the software and hardware solution. The audit focused on those aspects of security relating to protecting the CBSA's data where highway carriers have access to the Agency's systems and data through eManifest. eManifest enables highway carriers to report information to the CBSA on the cargo they are carrying before they arrive at the Canadian border. The Agency uses this information to assess the risk to health and safety that any commercial shipments entering this country may pose.

Since October 31, 2010, highway carriers have been able to use a component of eManifest known as "EDI Highway Carrier Reporting." This component receives data on cargo and the method of conveyance (i.e., by truck) via Internet. [*]

Recommendation:

1. The Vice-President, Information, Science and Technology Branch, in consultation with the Vice-President, Comptrollership Branch, and the Vice-President, Programs Branch [*]

Management Action Plan	Completion Date
The Information, Science and Technology Branch has developed a plan to identify any remaining risks with the current CIG platform. This plan involves reviewing the implementation of the Secure Portal design for eManifest and assessing if there are any measures that could be implemented in the CIG [*].	September 2011
Modifications to the CIG, if required, will be implemented post the deployment of the Secure Portal, which is planned to be delivered in summer 2011. Any changes will likely be coupled in the next Commercial release, which is currently targeted for the fourth quarter of 2011-2012.	March 2012
The Interim Authority to Process has been extended until March 31, 2012 [*].	March 2012

3.2 Procurement

The eManifest project depends largely on the procurement of hardware and software products, and must follow complex government procurement processes. Procurement planning and management are important to ensure procurement activities are integrated with the eManifest project plan and schedule, and to minimize problems affecting schedule and cost.

Procurement Planning

The criteria for procurement planning call for a process for planning and managing procurement so that appropriate products are available when they are needed.

The audit found that procurement was generally well-planned and managed. This included using a procurement task force to mitigate risks associated with complex government procurements. The process ensured that appropriate products would be acquired at competitive prices. However, the audit noted some weaknesses relating to coordinating the timing of purchases to meet the schedules of the software developers.

The audit found that decisions on what hardware to purchase for two components of eManifest were not being made far enough in advance so that the project could meet its planned implementation dates for various components of eManifest. These components were Risk Assessment (RA) and Business Intelligence/Data Warehouse (BI/DW), which includes a repository of information needed to manage the risk relating to goods entering Canada. Regarding BI/DW, at the time of the audit, various questions remained to be answered about the design of the hardware. Until these questions have been dealt with, procurement will remain stalled. The BI/DW component could proceed with the software-development phase using existing hardware. However, once the outstanding questions have been answered, more time and effort may be needed to make the necessary changes to transfer the software to the new hardware.

With respect to the RA component, the Agency had still not decided whether to use the existing or new hardware platform to implement commercial software that will be used to support eManifest risk assessment activities. [*]

A contributing factor to the delays in making the hardware-related decisions noted above was the lack of a single focal point for managing and coordinating the technical and non-technical aspects of the procurement of hardware between the Programs Branch and the Information, Science and Technology Branch.

We reviewed documentation relating to these delays which indicated that senior management had been briefed on the value of designating a dedicated project manager within the Information, Science and Technology Branch. This person would monitor and control procurement schedules to ensure that they are made in a timely manner. This measure would also help in establishing an integrated procurement schedule that would alert procurement personnel as to when items must be ordered and delivered in order to meet the project's milestones. We found no evidence of such a schedule.

Recommendations:

2. The Vice-President, Information, Science and Technology Branch, in consultation with the Vice-President, Programs Branch, should expedite outstanding hardware decisions.

Management Action Plan	Completion Date
The Information, Science and Technology Branch (ISTB) and Programs Branch have commenced a comprehensive review and analysis of eManifest requirements to extract and consolidate all business intelligence (query, reporting, decision support) and data warehouse related requirements.
A senior-level multi-disciplinary technical team is elaborating these requirements and translating them into the end-state technical functionality, capacity and service availability requirements.	August 2011
In the first quarter of 2011–2012, work will start to prepare a Request for Proposal with a projected timeframe to achieve "Ready for Use" status (i.e. delivered, installed and certified) of end-state hardware and software by mid-to-late 2013–2014. At this point, the end-state platform is ready to accept the transition plan of the interim solution/data.	December 2013
For the Risk Assessment component, a decision was taken at the ISTB Director General Review Forum on November 17, 2010, to extend the current technology platforms and use the existing Architecture process to determine placement of software components until such time as funding a new technology platform is viable. This decision was communicated to ISTB directors at the subsequent Architecture Review Board meeting.	Completed – November 2010

3. The Vice-President, Programs Branch, in consultation with the Vice-President, Information, Science and Technology Branch, should put an integrated schedule in place to help mitigate potential procurement delays.

Management Action Plan	Completion Date
The eManifest project team is currently reviewing the project's schedule and deployment plan. Given the size, scope and nature of the project, several phases of review are required.
A Target State Task Force was established to define the end state for eManifest. Their work was completed in November 2010.	Target State Task Force November 2010 – Completed
A Strategic View Task Force, composed of members from business and IT, has been established to validate eManifest's deployment sequence considering dependencies, constraints, risks, etc. The work was completed in February 2011.	Strategic View Task Force February 2011 – Completed
The Information, Science and Technology Branch will take the results of the Strategic View Task Force (further informed by the work of the Target State Task Force) and assess the remaining work to complete eManifest from a technological perspective. This exercise is expected to be completed in August 2011.	ISTB Target State Task Force – August 2011
Once this impacting is complete, the eManifest schedule will be updated to reflect all of the validated information and is expected to be available in November 2011.	eManifest Schedule – November 2011
The validated schedule will be used to monitor and report on the project's status, including the integrated procurement activities.

Procurement Monitoring and Control

The criterion for this area requires appropriate monitoring and reporting mechanisms to be in place to provide management with accurate information for decision making on the procurement schedule and budget status.

Monitoring and controlling the procurement budget requires tracking of costs incurred to date. It also involves estimating the costs needed to complete remaining procurement activities in order to identify any variances from planned total procurement costs. The audit found that procurement budgets are monitored, and actual expenditures are tracked in relation to the total budget. Although management could tell how much has actually been spent, the budget information did not indicate whether any items (software and hardware) still needed to complete the project could be purchased within the remaining funds. For example, the current budget-status information does not take potential expenditures into account. As noted above, a decision had not been made about whether to use existing hardware platform or buy a new one. If the Agency decides to acquire new hardware for eManifest's RA component, it would entail unplanned costs.

The audit team was advised that project management had started a detailed review and realignment of cost projections for all project components, including estimating costs and conducting a variance analysis for the procurement budget.

Recommendation:

4. The Vice-President, Programs Branch, in consultation with the Vice-President, Comptrollership Branch, and the Vice-President, Information, Science and Technology Branch, should improve assurance that the eManifest procurement budget is appropriate to complete remaining procurement.

Management Action Plan

Completion Date

Following the completion of the action plan for Recommendation three, anticipated to be completed by November 2011, the validated integrated schedule will be used for monitoring and reporting of procurement activities including the impacts of the Risk Assessment hardware decisions. The work remaining at this time will be costed and used to identify cost variances in order to validate the procurement budget. Note: Improved financial reporting is dependent upon the Agency's ability to track project costs at an appropriate level. It is anticipated that a new cost tracking method will be introduced on April 1, 2011, in order to improve financial reporting.

New Cost Tracking Method – April 2011 Completion of Previous Action Plan – November 2011 Completion of Recommendation – December 2011

Appendix A: Audit Criteria

Line of Enquiry	Audit Criteria[ 1 ]
1. Business Requirements	1.1 There is a process in use to define, analyze and manage business requirements.
	1.2 The business requirements definition process includes steps to link requirements to business and project objectives, and trace them throughout the project life cycle.
	1.3 Business requirements include the quantified and documented needs and expectations of the sponsor branches, customers and other stakeholders.
	1.4 IT performance and availability requirements have been mapped to the expected business outcomes.
	1.5 Business requirements are verified with the project sponsors, and formal acceptance of the requirements is obtained from the sponsors.
	1.6 Once base-lined, business requirements are subject to change management and control.
2. Procurement	2.1 There is a procurement management plan that describes how procurement processes will be managed from developing procurement documents through contract closure.
	2.2 Lead times to purchase procurement items are coordinated with eManifest overall schedule development.
	2.3 Procurement risks are identified and managed through the project risk management process.
	2.4 Procurement items are described in sufficient detail to allow prospective sellers to determine if they are capable of providing the items.
	2.5 Appropriate monitoring and reporting mechanisms are in place to provide eManifest project management with accurate procurement schedule and budget status information for decision-making.
	2.6 IT Infrastructure acquisitions are planned in accordance with the CBSA technology direction.

Appendix B: List of Acronyms

ACI

Advance Commercial Information

BI/DW

Business Intelligence/Data Warehouse

CBSA

Canada Border Services Agency

COBIT

Control Objectives for Information and related Technology

DSO

Departmental Security Officer

EDI

Electronic Data Interchange

IAPED

Internal Audit and Program Evaluation Directorate

Interim CIG

Interim Corporate Internet Gateway

ISTB

Information, Science and Technology Branch

IT

Information Technology

MAF

Management Accountability Framework

PMI

Project Management Institute

RA

Risk Assessment

Notes

1. Sources for audit criteria were the Project Management Institute (PMI), the Control Objectives for Information and related Technology (COBIT), and the Management Accountability Framework (MAF). [Return to text]