Harmonized Risk Scoring-Advance Trade Data

Internal Audit Report
March 2011

This document is also available in PDF (190 KB)  [help with PDF files]

Table of Contents

• Executive Summary
• 1.0 Introduction
  • 1.1 Background
  • 1.2 Risk Assessment
  • 1.3 Audit Objective and Scope
  • 1.4 Approach and Methodology
  • 1.5 Audit Criteria
  • 1.6 Statement of Assurance
• 2.0 Audit Opinion
• 3.0 Findings, Recommendations and Action Plans
  • 3.1 Software Development
  • 3.2 Technology Implementation
• 4.0 Other Matters of Interest
  • 4.1 Communication
  • 4.2 Funding and Scheduling Issue
• 5.0 Management Response
• Appendix A: Audit Criteria
• Appendix B: List of Acronyms

Executive Summary

Background

The Harmonized Risk Scoring-Advance Trade Data (HRS-ATD) audit is a System Under Development (SUD) audit, approved by the Canada Border Services Agency (CBSA) Audit Committee as part of the Three-year Risk-based Audit Plan for Fiscal Years 2009-10 to 2011-12.

The objective of HRS-ATD is to strengthen the CBSA's ability to identify and assess the risk associated with marine shipments of commercial goods destined for Canada. The "harmonized" aspect of the project refers to harmonizing Canada's risk-assessment and targeting methods with the standards established by the World Customs Organization and the United States Customs and Border Protection.

The HRS-ATD project includes seven components. At the time of the audit, two components had been completed:

• Hybrid 1 (H1), consisting of new risk-scoring algorithms and implemented in September 2009; and
• U.S. Marine In-Transit (USMIT), consisting of a new process for receiving data on marine containers in-transit from the United States to Canada and implemented in October 2010.

HRS-ATD was approved in June 2008 with a scheduled completion date in fiscal year 2010–2011. Its total allocated budget was $31 million. Of note is that the forecast total actual spending at the end of fiscal year 2010–2011 was $22 million, and carryover of the remaining funds to continue project work was not approved.

Funding and scheduling options for completing the remaining HRS-ATD components are being addressed by project management (see Section 4.2).

Significance of this Audit

This audit is of interest because the HRS-ATD project represents a new means of assessing the risk of goods coming into Canada. This activity is central to fulfilling the CBSA's mandate and, accordingly, it is important that senior management have assurance on the extent to which the processes for developing and implementing this project were adequate.

Objective and Scope

The objective of the audit was to assess the adequacy and appropriateness of the CBSA's processes for managing both the development of HRS-ATD and the integration of the software and hardware products flowing from HRS-ATD with the CBSA's existing computer technology.

The H1 component was implemented under a different project organization structure than USMIT, and followed management processes that have since been revised for the remaining components. Consequently, in meeting this audit objective, the audit focused on HRS-ATD's USMIT component as it was more representative of the current development processes used by HRS-ATD. As no changes were needed in the computer hardware associated with USMIT, the audit included examination of the processes to implement USMIT on the existing hardware.

Audit Opinion

The audit found that the processes which the HRS-ATD project followed for managing the development and implementation of the USMIT component were adequate and appropriate.

Key Findings

HRS-ATD's processes for developing USMIT were consistent with the CBSA's Major Project Governance Framework, and USMIT was successfully developed and implemented.

The USMIT component used standard technology and did not affect the CBSA's other computer technology or processes.

Of the nine criteria considered relevant to this audit, the project met seven and partially met the other two.

Our observations were mainly positive, so we make no recommendations in this report.  However, the audit team did note opportunities for improvement. These matters posed little risk to USMIT. However, they warrant management's attention as they could affect the scope and delivery schedules for the future components associated with HRS-ATD. Accordingly, they are covered under "Other Matters of Interest."

Statement of Assurance

This audit engagement was planned and conducted in accordance with the Internal Auditing Standards for the Government of Canada.

1.0 Introduction

1.1 Background

The Canada Border Services Agency (CBSA) Audit Committee approved an audit of the Harmonized Risk Scoring-Advance Trade Data (HRS-ATD) project as part of the Three-year Risk-based Audit Plan for Fiscal Years 2009-10 to 2011-12.

The CBSA is looking to effectively "push the border out" to ensure that serious threats to Canada's health, safety and security are identified and intercepted before they approach or cross Canada's physical border. To this end, the CBSA has carried out various complementary activities, including the HRS-ATD project. Accordingly, the objective of HRS-ATD is to strengthen the CBSA's ability to identify and assess the risk associated with commercial marine shipments bound for Canada, and to target high-risk shipments to prevent them from entering this country. The "harmonized" aspect of the project refers to harmonizing, to the extent possible, Canada's risk-assessment and targeting methods with the standards established by the World Customs Organization (WCO) and the United States Customs and Border Protection (U.S. CBP).

HRS-ATD has been built on the successful marine component of the Advance Commercial Information program. This program requires carriers to electronically transmit data on marine cargo to the CBSA 24 hours before that cargo is loaded onto a ship in a foreign port. HRS-ATD will improve the Agency's ability to identify and target risky commercial marine shipments by:

• harmonizing, as noted above, risk-assessment and targeting methods with the standards established by the WCO and the U.S. CBP;
• incorporating an expanded set of risk indicators; and
• incorporating new sources of advance trade data.

The HRS-ATD project consists of seven components. At the time of the audit, two components had been completed:

• Hybrid 1(H1), consisting of new risk scoring algorithms and implemented in September 2009; and
• U.S. Marine In-Transit (USMIT), consisting of a new process for receiving data on marine containers in-transit from the United States to Canada and implemented in October 2010.

The five remaining components to be developed are:

• Advanced Trade Data (ATD), for capturing Advance Trade Data Set (ATDS). 
• Container Status Messages (CSM), which will increase the data collected in the risk assessment process and will allow targeters to make more informed pre-load and pre‑arrival marine decisions.
• Determining and documenting historical trade patterns using existing and new commercial information to develop new risk indicators. The trade pattern information will include data related to routing, trade chain partners, the commodity, importer and combinations of these elements.
• Establishing contracts with corporate and trade data providers. The corporate and trade profile data will be used where the CBSA has determined that certain trade chain partner involved, such as shippers and consignees, in the importing of the cargo in the marine mode are not known to the CBSA.
• Implementing new scoring algorithms into TITAN and ACROSS to enhance the CBSA's marine container tracking and targeting capabilities, which are comparable to that of the United States.

While not completed, some progress had been made in developing these components. For example, hardware and software to deliver three of the remaining components dealing with continuing analysis of risk indicators had been purchased, and some supporting processes had been developed.

HRS-ATD was approved in June 2008 with a scheduled completion date in fiscal year 2010–2011. Its total allocated budget was $31 million. Of note is that the forecast total actual spending at the end of fiscal year 2010–2011 was $22 million, and carryover of the remaining funds to continue project work was not approved.

Funding and scheduling options for completing the remaining HRS-ATD components are being addressed by project management (see Section 4.2).

USMIT is not a separate, independent application; rather it is a feature being added to four large existing applications.[ 1 ]

1.2 Risk Assessment

The risk assessment carried out when planning this audit identified the following key risks:

Development Process

• Changes in roles and responsibilities for developing the software for HRS-ATD were viewed as a risk that could affect the development of future HRS-ATD components.

Technology Integration

• The Agency has experienced both delays in obtaining data-mining hardware, and difficulty in reaching the service levels necessary to improve its risk assessment for marine shipments. This situation has given rise to the risk that computer hardware will not be available in time to allow the Agency to get the necessary software up and running in a timely manner.

The examination phase of this audit determined that the risk associated with the Development Process (refer to Section 3.1, Criterion 1.3) was not a factor for USMIT. However, the delivery of future components of HRS-ATD could be affected if the role of the sponsoring organization with respect to requirements is not clarified (refer to Section 4.1). For the risk related to Technology Integration, the audit determined that this risk did not impact USMIT as it used existing hardware – not new hardware. Technology Integration risks associated with the five HRS-ATD components still to be completed will be addressed by project management in the revised funding and scheduling plans referred to in Section 4.2.

1.3 Audit Objective and Scope

The objective of the audit was to assess the adequacy and appropriateness of the CBSA's processes for managing both the development of HRS-ATD and the integration of the software and hardware products flowing from HRS-ATD with the CBSA's existing computer technology.

The H1 component was implemented under a different project organization structure than USMIT, and followed management processes that have since been revised for the remaining components. Consequently, in meeting this audit objective, the audit focused on HRS-ATD's USMIT component as it was more representative of the current development processes used by HRS-ATD. As no changes were needed in the computer hardware associated with USMIT, the audit included examination of the processes to implement USMIT on the existing hardware.

1.4 Approach and Methodology

The audit gathered evidence by conducting interviews, reviewing documentation and assessing the HRS-ATD system-development and software-integration process. The audit:

• interviewed selected project personnel to assess the development processes on the HRS-ATD project, specifically to deliver USMIT;
• interviewed technical management personnel to assess the CBSA's readiness to deal with the new technology, the overall technology configuration management, and plans to manage USMIT's impact on computer software and hardware related to this component; and
• reviewed project documents to assess the degree to which HRS-ATD project management practices aligned with industry practices and applicable Treasury Board (TB) and Agency policies.

1.5 Audit Criteria

The detailed audit criteria are presented in Appendix A of this report.

1.6 Statement of Assurance

This audit engagement was planned and conducted in accordance with the Internal Auditing Standards for the Government of Canada.

2.0 Audit Opinion

The audit found that the processes which the HRS-ATD project followed for developing and implementing the USMIT component were adequate and appropriate.

3.0 Findings, Recommendations and Action Plans

3.1 Software Development

This section looks at the extent to which the software for HRS-ATD was developed in accordance with the Agency's standards for this area. Using a formal system-development process with specific milestones is accepted as a best practice and is essential for managing risk in developing software applications. Adhering to a standard means that all projects will follow a similar, proven process; that management will know what to expect; and that terminology will not be confusing to participants.

This section presents the findings relating to audit criteria 1.1 to 1.6, listed in Appendix A.

The "System Development Life Cycle Standard" (Criterion 1.1)

This Standard is an industry term for a well-controlled software-development process. The CBSA Major Project Governance Framework[ 2 ] is the Agency's standard approach to managing major projects. It states that projects are to follow a Project Management Life Cycle approach that includes six phases, with formal approvals after each.

The USMIT component of the HRS-ATD project (a major project) met this criterion, and it followed the Agency's framework in most respects. We noted that in the case of its USMIT component, the project had obtained only informal sign-off approvals at the end of each phase, rather than the formal approvals that the framework calls for. The framework is designed to ensure — among other things — that systems will meet all business requirements. Therefore, any deviation from the standard framework creates a potential risk that the project may not meet future business requirements.

The audit noted that the CBSA had already begun to review and re-develop the governance structure for major projects with a view to strengthening the formal approval process. When the new structure has been approved, HRS-ATD will be required to follow it.

Requirements Definition (Criterion 1.2)

This criterion calls for a formal process for ensuring that business, technical, and security and privacy requirements have been agreed upon and clearly documented. As well, the documented requirements should be traceable to the final software product to ensure that it reflects these requirements. This process helps to ensure that a system will contribute to fulfilling an organization's business requirements.

The audit found that the project met these criteria. Requirements had been clearly specified, documented and met. In examining and tracing a sample of business requirements, we found that they were reflected in the design of the software. Staff from the project team had validated the way in which USMIT software had met documented business requirements. All problems had been resolved before implementation.

Communication among stakeholders (Criterion 1.3)

A key means of controlling risk when following a standard system-development approach is to determine where and when essential communication must take place among all stakeholders. According to this criterion, systems developers and clients should communicate clearly with each other on the design of a system, and project management, users and the project sponsor should sign off on the design.

The project generally met this criterion. The audit found that communication with respect to defining requirements for the USMIT component was adequate. Subsequently, the organizational structure under which the Agency had defined the requirements for USMIT changed. While these changes resulted in a lack of clarity on the respective role of the sponsor and project management for USMIT development and implementation, this lack of clarity did not affect USMIT.

In order to ensure that future HRS-ATD components meet all business requirements, it would be beneficial to clarify the role of the sponsoring organization. Please refer to Section 4.1 for further discussion.

Software Construction (Criterion 1.4)

Once requirements have been determined, software should be developed that reflects the agreed-upon and documented business requirements, and meet relevant standards.

The project met this criterion. The development of business requirements and software conformed to the Major Project Governance Framework.

Software Assurance (Criterion 1.5)

The criterion for this area requires a Quality Assurance (QA) mechanism for software which includes comprehensive testing and ensures that this software will ultimately meet business and security requirements. The assurance process includes testing to verify, most importantly, that the software does what it is intended to do (functionality) and also that the software meets performance and other requirements.

The project partially met this criterion. The audit was advised that work was in process to strengthen software assurance controls.

Testing was well managed and was successfully completed. The scope of testing included software functionality and other areas such as the time it takes for the software to respond to enquiries, the time taken to recover in case of failure, and performance.

We were advised that the Agency's Information Technology (IT) quality management organization is implementing a more rigorous process to ensure the readiness of all systems, not only from a testing perceptive, but, in terms of ensuring appropriate documentation is completed and approved.

A standard assurance process includes ensuring that proper security requirements have been documented, and certifying that they have been met. The audit found that a Threat and Risk Assessment (TRA) for USMIT had not been prepared. Information from a TRA is an input to certifying that a system has met all security requirements. As USMIT uses existing hardware and software, its TRA analysis was dependent on a review and update of the TRA analyses for the existing systems. This work had yet to be done. In the meantime, an interim authority to operate USMIT had been approved. This authority was scheduled to expire in January 2011. We note that the risk had been assessed as "Medium" until this work could be completed. We were advised that the Agency's IT security organization expects to complete TRA updates by April 2011, and that the interim authority has been extended until September 30, 2011, by which time the work on the TRA will be completed.

Software Implementation (Criterion 1.6)

According to this criterion, an organization should have a process for implementing software applications which ensures that new software is integrated into existing operations in a controlled manner. The CBSA met this criterion, and the audit found that the implementation of USMIT was generally adequate.

3.2 Technology Implementation

Projects must consider how their software could affect other existing computer hardware and software. This section focuses on the degree to which project deliverables conformed to the CBSA's technical standards and processes. The sub-sections below align directly with audit criteria 2.1 to 2.3 listed in Appendix A to this report.

Central Infrastructure Processes (Criterion 2.1)

This criterion would require HRS-ATD to adhere to the Agency's processes and standards to ensure that any new hardware and software is implemented in accordance with established acquisition and maintenance processes.

The project met this criterion. We found that the Agency had a process for managing the implementation of new hardware although, in the case of USMIT, no new hardware was involved. The criteria for this audit included two others (Criteria 2.4 and 2.5), which related to planning for, acquiring, and implementing and maintaining the technological infrastructure. Since, as noted above, USMIT did not require any new hardware, these two criteria were not applicable.

IT Capacity and Disaster Prevention (Criterion 2.2)

IT deliverables have the potential to affect either the capacity or security of the Agency's existing computer systems. Therefore, according to this criterion, the HRS-ATD project should adhere to the Agency's IT security policies and standards.

The project partially met Criterion 2.2. With respect to capacity, in implementing USMIT, HRS‑ATD followed the Agency's standard processes. Since USMIT is not an independent application, it was not necessary to change the technology architecture. However it was necessary to analyse whether the existing computer operations had enough capacity to handle the increased workload that would result from integrating USMIT with existing systems. The CBSA did this analysis, which indicated that there was no need for any specific acquisitions to support USMIT because of the additional volume of data.

Regarding security, a key element is disaster prevention. The requirements for USMIT included one relating to restoring the application should a disaster occur. However, there was no disaster-recovery plan in place for at least one application (ACROSS) relating to USMIT. The recent CBSA audit of Business Continuity Planning resulted in a management plan to address weaknesses in this area. The Agency's Data Centre Recovery Project, started in fiscal year 2010–2011 and expected to be completed in fiscal year 2013–2014, will look at the CBSA's ability to recover from a disaster and maintain business continuity. This project is expected to be completed in four years' time. In the meantime, however, there is a continuing residual risk that a disaster or other significant event could interrupt essential CBSA services. Management has accepted this risk.

Development and Test Environments (Criterion 2.3)

To minimize the potential risk to ongoing CBSA computer programs and operations, the HRS‑ATD project should be able to carry out efficient, effective tests of its computer software and hardware.

This criterion was met. The CBSA has various test environments for thoroughly testing new applications and infrastructure. The approach to testing was designed to minimize any potential impact during implementation.

4.0 Other Matters of Interest

While carrying out this audit, the audit team noted certain issues that did not directly affect the delivery of the USMIT component of HRS-ATD. However, in our opinion, these issues were potentially important enough to warrant management's consideration.

4.1 Communication

As noted under the heading "Communication among stakeholders," a new organizational structure had replaced the one in effect when USMIT was under development. Under the original structure, business requirements had been developed, documented and approved by a committee of sponsors from all stakeholder branches within the CBSA. While a new sponsor had been identified under the new structure, the respective roles of the sponsor and project management were unclear in this area. Consequently, some confusion existed over responsibility for defining business requirements.

The audit team found that the HRS-ATD project team and the sponsor representative each believed that it played a key role in defining and approving business requirements. In interviews, staff from the new project sponsor indicated they were unclear of their role in ensuring that the project would ultimately meet the requirements.

This lack of clarity did not affect USMIT because its requirements had been developed and approved under the previous organizational structure. However, if this lack of clarity persists, there is a risk that future components may not meet all business requirements.

4.2 Funding and Scheduling Issue

The audit was advised by HRS-ATD project management that approximately $22 million of the $31 million HRS-ATD budget would be expended by March 31, 2011, and that carryover of the remaining funds to continue project work was not approved. Management has advised that  a funding strategy and revised schedule for implementing the advanced risk assessment functions, which account for three of the five remaining components, has now been developed, and that work was still progressing on updating scope, funding plans and delivery schedules for the two remaining components (ATD and CSM).

5.0 Management Response

Management acknowledges and thanks the audit team for noting the matters of interest.

With respect to the identified matters of interest noted in Section 4:

• Section 4.1 – Communication: Under the new Agency organization and using the new CBSA Project Governance Framework for Major Projects, a new governance structure framework for projects has been developed and implemented. This governance framework, including sponsorship details, was approved in November 2010 and is followed by HRS-ATD. The Risk Assessment Directorate has been identified as the sponsor for HRS-ATD; and
• Section 4.2 – Funding and Scheduling Issue: Action is being taken to develop options for the CSM component. Analysis is expected to be completed by March 2011 and will be submitted for formal approval via project governance committees. The ATD component will be delivered as part of the eManifest project with implementation targeted for fiscal year 2013–2014. The plan for ATD is being prepared as part of the planning of the eManifest project. The new plan for the eManifest Project will be completed by November 2011.

Also, as noted above, with respect to the security requirements for USMIT (see Section 3.1, Criterion 1.5), the IT security organization expects that TRA updates will be completed by April 2011 and that the interim authority has been extended until September 30, 2011, by which time the work on the TRA will be completed.

Appendix A: Audit Criteria

The audit criteria used for the HRS-ATD audit were: