Internal Audit Report
March 2011
This document is also available in PDF (190 KB) [help with PDF files]
The Harmonized Risk Scoring-Advance Trade Data (HRS-ATD) audit is a System Under Development (SUD) audit, approved by the Canada Border Services Agency (CBSA) Audit Committee as part of the Three-year Risk-based Audit Plan for Fiscal Years 2009-10 to 2011-12.
The objective of HRS-ATD is to strengthen the CBSA's ability to identify and assess the risk associated with marine shipments of commercial goods destined for Canada. The "harmonized" aspect of the project refers to harmonizing Canada's risk-assessment and targeting methods with the standards established by the World Customs Organization and the United States Customs and Border Protection.
The HRS-ATD project includes seven components. At the time of the audit, two components had been completed:
HRS-ATD was approved in June 2008 with a scheduled completion date in fiscal year 2010–2011. Its total allocated budget was $31 million. Of note is that the forecast total actual spending at the end of fiscal year 2010–2011 was $22 million, and carryover of the remaining funds to continue project work was not approved.
Funding and scheduling options for completing the remaining HRS-ATD components are being addressed by project management (see Section 4.2).
This audit is of interest because the HRS-ATD project represents a new means of assessing the risk of goods coming into Canada. This activity is central to fulfilling the CBSA's mandate and, accordingly, it is important that senior management have assurance on the extent to which the processes for developing and implementing this project were adequate.
The objective of the audit was to assess the adequacy and appropriateness of the CBSA's processes for managing both the development of HRS-ATD and the integration of the software and hardware products flowing from HRS-ATD with the CBSA's existing computer technology.
The H1 component was implemented under a different project organization structure than USMIT, and followed management processes that have since been revised for the remaining components. Consequently, in meeting this audit objective, the audit focused on HRS-ATD's USMIT component as it was more representative of the current development processes used by HRS-ATD. As no changes were needed in the computer hardware associated with USMIT, the audit included examination of the processes to implement USMIT on the existing hardware.
The audit found that the processes which the HRS-ATD project followed for managing the development and implementation of the USMIT component were adequate and appropriate.
HRS-ATD's processes for developing USMIT were consistent with the CBSA's Major Project Governance Framework, and USMIT was successfully developed and implemented.
The USMIT component used standard technology and did not affect the CBSA's other computer technology or processes.
Of the nine criteria considered relevant to this audit, the project met seven and partially met the other two.
Our observations were mainly positive, so we make no recommendations in this report. However, the audit team did note opportunities for improvement. These matters posed little risk to USMIT. However, they warrant management's attention as they could affect the scope and delivery schedules for the future components associated with HRS-ATD. Accordingly, they are covered under "Other Matters of Interest."
This audit engagement was planned and conducted in accordance with the Internal Auditing Standards for the Government of Canada.
The Canada Border Services Agency (CBSA) Audit Committee approved an audit of the Harmonized Risk Scoring-Advance Trade Data (HRS-ATD) project as part of the Three-year Risk-based Audit Plan for Fiscal Years 2009-10 to 2011-12.
The CBSA is looking to effectively "push the border out" to ensure that serious threats to Canada's health, safety and security are identified and intercepted before they approach or cross Canada's physical border. To this end, the CBSA has carried out various complementary activities, including the HRS-ATD project. Accordingly, the objective of HRS-ATD is to strengthen the CBSA's ability to identify and assess the risk associated with commercial marine shipments bound for Canada, and to target high-risk shipments to prevent them from entering this country. The "harmonized" aspect of the project refers to harmonizing, to the extent possible, Canada's risk-assessment and targeting methods with the standards established by the World Customs Organization (WCO) and the United States Customs and Border Protection (U.S. CBP).
HRS-ATD has been built on the successful marine component of the Advance Commercial Information program. This program requires carriers to electronically transmit data on marine cargo to the CBSA 24 hours before that cargo is loaded onto a ship in a foreign port. HRS-ATD will improve the Agency's ability to identify and target risky commercial marine shipments by:
The HRS-ATD project consists of seven components. At the time of the audit, two components had been completed:
The five remaining components to be developed are:
While not completed, some progress had been made in developing these components. For example, hardware and software to deliver three of the remaining components dealing with continuing analysis of risk indicators had been purchased, and some supporting processes had been developed.
HRS-ATD was approved in June 2008 with a scheduled completion date in fiscal year 2010–2011. Its total allocated budget was $31 million. Of note is that the forecast total actual spending at the end of fiscal year 2010–2011 was $22 million, and carryover of the remaining funds to continue project work was not approved.
Funding and scheduling options for completing the remaining HRS-ATD components are being addressed by project management (see Section 4.2).
USMIT is not a separate, independent application; rather it is a feature being added to four large existing applications.[ 1 ]
The risk assessment carried out when planning this audit identified the following key risks:
Development Process
Technology Integration
The examination phase of this audit determined that the risk associated with the Development Process (refer to Section 3.1, Criterion 1.3) was not a factor for USMIT. However, the delivery of future components of HRS-ATD could be affected if the role of the sponsoring organization with respect to requirements is not clarified (refer to Section 4.1). For the risk related to Technology Integration, the audit determined that this risk did not impact USMIT as it used existing hardware – not new hardware. Technology Integration risks associated with the five HRS-ATD components still to be completed will be addressed by project management in the revised funding and scheduling plans referred to in Section 4.2.
The objective of the audit was to assess the adequacy and appropriateness of the CBSA's processes for managing both the development of HRS-ATD and the integration of the software and hardware products flowing from HRS-ATD with the CBSA's existing computer technology.
The H1 component was implemented under a different project organization structure than USMIT, and followed management processes that have since been revised for the remaining components. Consequently, in meeting this audit objective, the audit focused on HRS-ATD's USMIT component as it was more representative of the current development processes used by HRS-ATD. As no changes were needed in the computer hardware associated with USMIT, the audit included examination of the processes to implement USMIT on the existing hardware.
The audit gathered evidence by conducting interviews, reviewing documentation and assessing the HRS-ATD system-development and software-integration process. The audit:
The detailed audit criteria are presented in Appendix A of this report.
This audit engagement was planned and conducted in accordance with the Internal Auditing Standards for the Government of Canada.
The audit found that the processes which the HRS-ATD project followed for developing and implementing the USMIT component were adequate and appropriate.
This section looks at the extent to which the software for HRS-ATD was developed in accordance with the Agency's standards for this area. Using a formal system-development process with specific milestones is accepted as a best practice and is essential for managing risk in developing software applications. Adhering to a standard means that all projects will follow a similar, proven process; that management will know what to expect; and that terminology will not be confusing to participants.
This section presents the findings relating to audit criteria 1.1 to 1.6, listed in Appendix A.
This Standard is an industry term for a well-controlled software-development process. The CBSA Major Project Governance Framework[ 2 ] is the Agency's standard approach to managing major projects. It states that projects are to follow a Project Management Life Cycle approach that includes six phases, with formal approvals after each.
The USMIT component of the HRS-ATD project (a major project) met this criterion, and it followed the Agency's framework in most respects. We noted that in the case of its USMIT component, the project had obtained only informal sign-off approvals at the end of each phase, rather than the formal approvals that the framework calls for. The framework is designed to ensure — among other things — that systems will meet all business requirements. Therefore, any deviation from the standard framework creates a potential risk that the project may not meet future business requirements.
The audit noted that the CBSA had already begun to review and re-develop the governance structure for major projects with a view to strengthening the formal approval process. When the new structure has been approved, HRS-ATD will be required to follow it.
This criterion calls for a formal process for ensuring that business, technical, and security and privacy requirements have been agreed upon and clearly documented. As well, the documented requirements should be traceable to the final software product to ensure that it reflects these requirements. This process helps to ensure that a system will contribute to fulfilling an organization's business requirements.
The audit found that the project met these criteria. Requirements had been clearly specified, documented and met. In examining and tracing a sample of business requirements, we found that they were reflected in the design of the software. Staff from the project team had validated the way in which USMIT software had met documented business requirements. All problems had been resolved before implementation.
A key means of controlling risk when following a standard system-development approach is to determine where and when essential communication must take place among all stakeholders. According to this criterion, systems developers and clients should communicate clearly with each other on the design of a system, and project management, users and the project sponsor should sign off on the design.
The project generally met this criterion. The audit found that communication with respect to defining requirements for the USMIT component was adequate. Subsequently, the organizational structure under which the Agency had defined the requirements for USMIT changed. While these changes resulted in a lack of clarity on the respective role of the sponsor and project management for USMIT development and implementation, this lack of clarity did not affect USMIT.
In order to ensure that future HRS-ATD components meet all business requirements, it would be beneficial to clarify the role of the sponsoring organization. Please refer to Section 4.1 for further discussion.
Once requirements have been determined, software should be developed that reflects the agreed-upon and documented business requirements, and meet relevant standards.
The project met this criterion. The development of business requirements and software conformed to the Major Project Governance Framework.
The criterion for this area requires a Quality Assurance (QA) mechanism for software which includes comprehensive testing and ensures that this software will ultimately meet business and security requirements. The assurance process includes testing to verify, most importantly, that the software does what it is intended to do (functionality) and also that the software meets performance and other requirements.
The project partially met this criterion. The audit was advised that work was in process to strengthen software assurance controls.
Testing was well managed and was successfully completed. The scope of testing included software functionality and other areas such as the time it takes for the software to respond to enquiries, the time taken to recover in case of failure, and performance.
We were advised that the Agency's Information Technology (IT) quality management organization is implementing a more rigorous process to ensure the readiness of all systems, not only from a testing perceptive, but, in terms of ensuring appropriate documentation is completed and approved.
A standard assurance process includes ensuring that proper security requirements have been documented, and certifying that they have been met. The audit found that a Threat and Risk Assessment (TRA) for USMIT had not been prepared. Information from a TRA is an input to certifying that a system has met all security requirements. As USMIT uses existing hardware and software, its TRA analysis was dependent on a review and update of the TRA analyses for the existing systems. This work had yet to be done. In the meantime, an interim authority to operate USMIT had been approved. This authority was scheduled to expire in January 2011. We note that the risk had been assessed as "Medium" until this work could be completed. We were advised that the Agency's IT security organization expects to complete TRA updates by April 2011, and that the interim authority has been extended until September 30, 2011, by which time the work on the TRA will be completed.
According to this criterion, an organization should have a process for implementing software applications which ensures that new software is integrated into existing operations in a controlled manner. The CBSA met this criterion, and the audit found that the implementation of USMIT was generally adequate.
Projects must consider how their software could affect other existing computer hardware and software. This section focuses on the degree to which project deliverables conformed to the CBSA's technical standards and processes. The sub-sections below align directly with audit criteria 2.1 to 2.3 listed in Appendix A to this report.
This criterion would require HRS-ATD to adhere to the Agency's processes and standards to ensure that any new hardware and software is implemented in accordance with established acquisition and maintenance processes.
The project met this criterion. We found that the Agency had a process for managing the implementation of new hardware although, in the case of USMIT, no new hardware was involved. The criteria for this audit included two others (Criteria 2.4 and 2.5), which related to planning for, acquiring, and implementing and maintaining the technological infrastructure. Since, as noted above, USMIT did not require any new hardware, these two criteria were not applicable.
IT deliverables have the potential to affect either the capacity or security of the Agency's existing computer systems. Therefore, according to this criterion, the HRS-ATD project should adhere to the Agency's IT security policies and standards.
The project partially met Criterion 2.2. With respect to capacity, in implementing USMIT, HRS‑ATD followed the Agency's standard processes. Since USMIT is not an independent application, it was not necessary to change the technology architecture. However it was necessary to analyse whether the existing computer operations had enough capacity to handle the increased workload that would result from integrating USMIT with existing systems. The CBSA did this analysis, which indicated that there was no need for any specific acquisitions to support USMIT because of the additional volume of data.
Regarding security, a key element is disaster prevention. The requirements for USMIT included one relating to restoring the application should a disaster occur. However, there was no disaster-recovery plan in place for at least one application (ACROSS) relating to USMIT. The recent CBSA audit of Business Continuity Planning resulted in a management plan to address weaknesses in this area. The Agency's Data Centre Recovery Project, started in fiscal year 2010–2011 and expected to be completed in fiscal year 2013–2014, will look at the CBSA's ability to recover from a disaster and maintain business continuity. This project is expected to be completed in four years' time. In the meantime, however, there is a continuing residual risk that a disaster or other significant event could interrupt essential CBSA services. Management has accepted this risk.
To minimize the potential risk to ongoing CBSA computer programs and operations, the HRS‑ATD project should be able to carry out efficient, effective tests of its computer software and hardware.
This criterion was met. The CBSA has various test environments for thoroughly testing new applications and infrastructure. The approach to testing was designed to minimize any potential impact during implementation.
While carrying out this audit, the audit team noted certain issues that did not directly affect the delivery of the USMIT component of HRS-ATD. However, in our opinion, these issues were potentially important enough to warrant management's consideration.
As noted under the heading "Communication among stakeholders," a new organizational structure had replaced the one in effect when USMIT was under development. Under the original structure, business requirements had been developed, documented and approved by a committee of sponsors from all stakeholder branches within the CBSA. While a new sponsor had been identified under the new structure, the respective roles of the sponsor and project management were unclear in this area. Consequently, some confusion existed over responsibility for defining business requirements.
The audit team found that the HRS-ATD project team and the sponsor representative each believed that it played a key role in defining and approving business requirements. In interviews, staff from the new project sponsor indicated they were unclear of their role in ensuring that the project would ultimately meet the requirements.
This lack of clarity did not affect USMIT because its requirements had been developed and approved under the previous organizational structure. However, if this lack of clarity persists, there is a risk that future components may not meet all business requirements.
The audit was advised by HRS-ATD project management that approximately $22 million of the $31 million HRS-ATD budget would be expended by March 31, 2011, and that carryover of the remaining funds to continue project work was not approved. Management has advised that a funding strategy and revised schedule for implementing the advanced risk assessment functions, which account for three of the five remaining components, has now been developed, and that work was still progressing on updating scope, funding plans and delivery schedules for the two remaining components (ATD and CSM).
Management acknowledges and thanks the audit team for noting the matters of interest.
With respect to the identified matters of interest noted in Section 4:
Also, as noted above, with respect to the security requirements for USMIT (see Section 3.1, Criterion 1.5), the IT security organization expects that TRA updates will be completed by April 2011 and that the interim authority has been extended until September 30, 2011, by which time the work on the TRA will be completed.
The audit criteria used for the HRS-ATD audit were:
Line of Enquiry | Audit Criteria | Met/ Partially Met/ Not Met |
---|---|---|
1. Development Process | 1.1. System Development Life Cycle Standard. The HRS-ATD Project has a software development and acquisition standard that is adequate for the complexity of the project. | Met |
1.2. Requirements Definition. A formal process exists to ensure business, technical, and security/privacy requirements to achieve the expected outcomes of the HRS-ATD Project are identified, prioritized, specified and agreed upon. | Met | |
1.3. Communication during the Development and Delivery Process. Business solution designs have been effectively communicated and articulated between systems development and clients, and been signed off by project management, users and project sponsor representatives. | Met | |
1.4. Software Construction. Automated functionality is being developed in accordance with design specifications, development and documentation standards, QA requirements, and approval standards. Software components are seen as configurable items and base-lined. | Met | |
1.5. Software Assurance. Assurance tasks needed to support the accreditation of new or modified systems that meet externally defined requirements for accreditation and/or certification have been identified, including a test environment and user involvement. | Partially met | |
1.6. Software Implementation. The project has an implementation and fallback/backout plan. Processes exist and the authority has been established to approve releases on behalf of or representative of project sponsors. | Met | |
2. Infrastructure and Technology Transition | 2.1. Central Infrastructure Processes. The HRS-ATD Project adheres to CBSA central processes and standards to ensure that installation and maintenance of system software is in accordance with the acquisition and maintenance framework for the technology infrastructure. | Met |
2.2. IT Capacity and Disaster Prevention. The HRS-ATD Project adheres to Agency's IT security policies and standards. | Partially met | |
2.3. Development and Test Environments. There are development and test environments established to support effective and efficient testing of infrastructure components. | Met | |
2.4. Acquisition, Implementation and Maintenance of Technological Infrastructure. The HRS-ATD Project has produced a strategy and plan for the acquisition, implementation and maintenance of the technological infrastructure that meets established business functional and technical requirements. | N/A | |
2.5. Technology Transition Plan. A Transition Plan exists that identifies and documents all technical, operational and usage aspects for implementation of technical components and the transfer of information to stakeholders. | N/A |
Acronym | Description |
---|---|
ACROSS | Accelerated Commercial Release Operations Support System |
ATD | Advanced Trade Data |
CBSA | Canada Border Services Agency |
CECP | Customs Electronic Commerce Platform |
CRSA | Commercial Risk Scoring and Assessment |
CSM | Container Status Messages |
EPA | Effective Project Approval |
H1 | Hybrid 1 Release |
HRS-ATD | Harmonized Risk Scoring-Advance Trade Data |
IT | Information Technology |
SUD | System Under Development |
TB | Treasury Board |
TITAN | New name for the Advance Commercial Information Risk Assessment application |
U.S. CBP | U.S. Customs and Border Protection |
USMIT | U.S. Marine In-Transit |
WCO | World Customs Organization |