This page has been archived.
Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.
October 2007
The audit of Canada Border Services Agency (CBSA) systems under development was identified as a priority in the CBSA’s Risk-Based Multi-Year Audit Plan. This Plan was approved by the Internal Audit and Evaluation Committee on March 17, 2005.
The principal objective of this audit was to provide the necessary assurances to the CBSA that its integrated project management and system development practices and procedures for automated business systems under development are adhering to the internal policies and procedures that have been established by the Agency. A further objective of this audit was to identify any opportunities for improvement to the CBSA’s existing internal policies and procedures for managing its systems under development.
The audit was divided into three phases. Phase 1, reported in February 2007, provided an assessment of the adequacy and effectiveness of the CBSA’s current management control framework over systems under development. Phases 2 and 3 provide an assessment of the degree to which two selected system development projects are effectively and appropriately applying the control framework. This report presents the findings from Phase 2 of the audit that reviewed the management framework for the development of a selected system development project: Advance Commercial Information (ACI)-Electronic Data Interface (EDI) Reporting for Air. The audit was conducted by Interis Consulting and audit fieldwork was conducted between July and October 2006.
ACI-EDI Reporting for Air is one component of the ACI initiative, which is a multi-year multi‑million dollar initiative launched under the Customs Action Plan in 2000 prior to the formation of the CBSA in December 2003. ACI provides the CBSA with pre-arrival data regarding commercial shipments entering Canada to identify high-risk commercial cargo and shipments requiring further scrutiny on arrival. A multi-modal phased approach was taken for the ACI development. ACI has been operational in the marine mode since April 2004 and became operational in the air mode in December 2005 (with a phased-in implementation to July 2006). Implementation of the final modes, highway and rail, is ongoing through the eManifest project.
The audit objective for Phase 2 was to assess the development of the electronic reporting of advance information for the air mode. The project was developed by the Innovation, Science and Technology Branch (ISTB) with sponsorship from the Enforcement, Admissibility and Operations branches.
Based on the audit work conducted in Phase 2, it is concluded that ACI-EDI Reporting for Air was developed in accordance with the CBSA policies, procedures and methodology in place at the time and the functionality of the application is working as intended. Many of the observations made in Phase 1 of the audit were supported by the audit work in Phase 2. In addition, the audit noted that since the time of the ACI-EDI Reporting for Air development in 2004 and 2005, many new processes and governance practices have been developed and introduced for more recent projects.
The audit noted a number of strengths for the ACI-EDI Reporting for Air project including the use of a sound system development approach that included the following activities:
Although many of the findings from the audit of ACI-EDI Reporting for Air are the same as those reported in the Phase 1 audit report, the following additional areas of control were identified in Phase 2 that could be strengthened at the project level:
The audit of CBSA systems under development was identified as a priority in the CBSA’s Risk‑Based Multi-Year Audit Plan. This Plan was approved by the Internal Audit and Evaluation Committee on March 17, 2005.
The principal objective of this audit was to provide the necessary assurances to the CBSA that its integrated project management and system development practices and procedures for automated business systems under development are adhering to the internal policies and procedures that have been established by the Agency.
A further objective of this audit was to identify any opportunities for improvement to the CBSA’s existing internal policies and procedures for managing its systems under development.
To achieve these objectives, the internal audit plan was divided into the following three phases:
The audit deliverable for Phase 1 was the following:
The audit deliverable for Phases 2 and 3 were the following:
The scope of this audit was to assess the development of the ACI-EDI Reporting for Air and not the effectiveness of the application itself.
The audit criteria used for Phase 2 were developed in Phase 1 of the audit and were based on risk elements inherent to system development projects, as well as on public sector trends for the effective governance of system development project lifecycles. A detailed set of audit criteria were established, factoring in the primary influences that can significantly impact system development projects in the Government of Canada (GC): management structure and processes, system development lifecycle policies and standards, as well as planning and acquisition processes and procedures. These detailed audit criteria were used to assess CBSA practices.
It was noted during the initial audit planning that business system development projects at the CBSA were inherently exposed to risk as:
Key control areas addressing inherent risks were identified based on the Treasury Board of Canada Secretariat’s (TBS) Enhanced Management Framework and the Control Objectives for Information and related Technology (CobiT) Framework issued by the IT Governance Institute. The audit criteria developed to assess the CBSA’s overall business practices, general controls and governance processes for its system under development projects were organized into the following six categories:
The audit criteria are presented in Appendix A.
The approach and methodology were risk-based and compliant with the applicable TBS Internal Audit Policy. The audit was conducted in accordance with an audit program that defined the audit tasks to assess each criterion. Through interviews and documentation review, it assessed the current CBSA practices against the criteria and formally assessed the effectiveness of each practice. Interviews were conducted with various representatives of the different organizations within the Innovation, Science and Technology Branch (ISTB) and with representatives of the project sponsors, the Enforcement, Admissibility, Operations and Comptrollership branches.
The audit fieldwork for Phase 2 was conducted between July and October 2006.
The results of the audit from Phase 1 were reported in February 2007. A number of strengths were noted in the controls over systems under development. These can provide the foundation for building a strong MCF over systems under development, namely a governance structure and a project management framework. Given the newness of the agency reorganization, many of the processes were still in early development and had not yet reached a mature state in the organization. New committees were being formed, roles and responsibilities were being clarified and specific deliverables were being developed. These activities are indicators that the Agency is making progress toward enhancing its control over systems under development.
The following key areas of control that should be strengthened were noted in the Phase 1 audit report:
Management reviewed the audit report, provided management action plans to address its recommendations and expected completion dates for each action.
The purpose of this document is to present the findings from Phase 2 of the audit.
ACI-EDI Reporting for Air is one component of the ACI initiative, which is a multi‑year multi‑million dollar initiative launched under the Customs Action Plan in 2000 prior to the formation of the CBSA in December 2003. The Plan introduced a new, comprehensive risk‑management model of program delivery based on the principles of advance information and pre-approval. The Plan consisted of 17 initiatives, with proposed implementation over five years. The ACI initiative (formerly known as “Carrier Re-engineering”) was launched in 2000.
ACI is intended to provide the CBSA with more effective risk-management processes and the tools to more effectively manage commercial clients and their commercial shipments. The initiative provides CBSA officers with pre-arrival data regarding commercial shipments entering Canada. Advance access permits officers to analyze the information, with the assistance of an automated targeting and risk-assessment tool, to identify high-risk commercial cargo and shipments requiring further scrutiny on arrival.
ACI development was separated into two concurrent but distinct components:
The audit looked at the system development project that addressed the second component of ACI, specifically for the air mode. ACI has been operational in the marine mode since April 2004 and became operational in the air mode in December 2005 (with a phased-in implementation to July 2006). Implementation of the final modes, highway and rail, is ongoing through the eManifest [ 1 ] project.
The organizational structure of the ISTB [ 2 ] at the time that ACI-EDI Reporting for Air was being developed is presented on the next page.
CBSA / Innovation Science and Technology Organizational Structure
Under this structure, the ACI-EDI Reporting for Air project was managed by one project manager from the MPDD Directorate and by one IT project manager from the Border Systems Directorate. The project was sponsored by the Enforcement, Admissibility and Operations branches. Throughout the development of ACI-EDI Reporting for Air, there were personnel and organizational changes to the ISTB team, as well as in the branches, creating challenges in the development activities.
At the time of the development, there were many pressures on the CBSA and the ISTB. The operations of the Agency are highly dependent on information systems and technology, which are developed, implemented and managed by the ISTB. The scope of system development activity at CBSA is significant, with annual spending on development and maintenance of systems, infrastructure and related technology estimated to be in the $150-250 million range over the next few years. In response to increased North American security concerns following the terrorist attacks in September 2001, the GC introduced a considerable number of security-related initiatives under the umbrella of the Public Security and Anti-terrorism (PSAT) initiative and the Shared Border Declaration (SBD), resulting in the expedited development of initiatives. This put pressure on a number of system development projects including ACI-EDI Reporting for Air.
Although development work began on the ACI initiative prior to the formation of the Agency, the initial development of the scope for the ACI-EDI Reporting for Air project began in early 2004. At that time, the project scope included both air and rail modes. The rail mode was later scoped out and included in the eManifest project which is currently in development. Initial project planning activity for ACI-EDI Reporting for Air largely began after the marine mode was put into production in April 2004. [ 3 ] The initial planned implementation date (for both the air and rail modes) was May 9, 2005, [ 4 ] which was later revised to December 5, 2005. [ 5 ] The systems for ACI electronic reporting for air mode were put into production on December 12, 2005. [ 6 ] As a phased‑in approach was used, air carriers and freight forwarders that were not fully prepared to report electronically to the CBSA on December 12, 2005, were given until June 26, 2006, to fully comply with ACI reporting requirements.
The ACI initiative was funded from the PSAT and Manley-Ridge funding envelopes, as well as through internal departmental allocations. Although a budget was not officially established for the ACI-EDI Reporting for Air project, development costs were estimated to have been approximately $4.7 million. [ 7 ] This cost estimate includes all CBSA costs related to the development of ACI-EDI Reporting for Air from 2005-2006 and 2006-2007, with some costs related to development work on the marine mode, which could not be separated out for presentation.
Based on the audit work conducted in Phase 2, the audit concluded that ACI-EDI Reporting for Air was developed in accordance with the CBSA policies, procedures and methodology in place at the time. Furthermore, the application is working as intended. However, opportunities exist to strengthen the system development processes used in ACI-EDI Reporting for Air to ensure adequate governance, risk management and control over future system under development projects.
Since the time of ACI-EDI Reporting for Air development in 2004 and 2005, many new processes and governance practices have been developed and introduced for more recent projects. In addition, many of the observations made in Phase 1 were supported by this Phase 2 audit. Management action plans for Phase 1 audit recommendations indicated that many audit areas needing strengthening have recently been addressed or are in the process of being addressed.
The audit findings are described in detail below.
The audit found that the CBSA system development methodology was used across all phases of the project. The following key activities were carried out:
This project was developed using an industry-recognized methodology, the Rational Unified Process (RUP), which is an iterative process to achieve the final design and build [ 8 ]. The audit examined various documents that demonstrated good development practices for the technical solution, as expected in any system development process. The system design was described in a project scope document, an initiation phase process document and a high-level business requirements document. User requirements were clearly described and documented in business use cases. The requirements were then translated into a system component design document and system use cases for use by the system developers. The technical design was documented in an abbreviated technology design document and technical specifications document. Many of the documents prepared for the ACI-EDI Reporting for Air development were developed using templates that have since evolved and been replaced by new templates, but the key objectives of the documents remain the same. Key individuals from the ISTB were involved in the initial design and development phases. The Enterprise Architecture Group interfaced with the Business Architecture Group in the MPDD Direcorate and sponsor representatives to define the requirements and review the feasibility of the technical solution.
User requirements were defined with involvement from key stakeholders. The MPDD Directorate took a large role in determining the high-level requirements and priorities; however, joint application development sessions were held with sponsor representatives to develop the business use cases. The Contraband Program in Enforcement Branch provided functional guidance and subject matter expertise to the ACI-EDI Reporting for Air project. The Electronic Commerce Unit in the MPDD Directorate brought its experience in working with the carriers to the system developers. The audit found evidence of ongoing refinement of the business requirements, largely through e-mail communications and updates to project documents. To improve its process of developing user requirements, the ISTB has begun using “usability architects” who look at the interface and how people use the system. User requirements can then be further enhanced to make the system more "usable" for the users.
Testing of ACI-EDI Reporting for Air was performed in a number of different test environments before implementation. Development testing was performed by the IT group and included data and database integrity testing, function testing, systems integration testing, business cycle testing and user interface testing. Test cases were mapped out, schedules were defined, resources were assigned and test results were documented and summarized. After the development testing was completed, the MPDD Directorate conducted user acceptance (client) testing and operational testing. Individuals outside the directorate were invited to participate in the user acceptance testing.
Typically, once the development and implementation are complete, the system application is moved from development to production and is transferred to the Systems Operations Group in the MPDD Directorate for ongoing support. The move is well controlled through release management procedures. At the time of the audit, the ACI-EDI Reporting for Air project had not yet been transferred to Systems Operations. It is a normal practice within the ISTB to keep a system development project under the control of the development group for about a year after implementation to monitor initial operational problems before it gets moved to Systems Operations.
Regular communication and consultation with the carrier community occurred. Implementation plans were communicated to the carrier community through periodic customs notices. Various working groups were established with the carrier community to share information on the implementation. A formal process was in place for carriers to first plan their implementation and then test their interface before they would be permitted to send data electronically. Testing packages were sent to the carriers for implementation. Conferences were held with the industry to inform it of the process. Implementation status and transmission start dates were regularly monitored by account managers in the MPDD Directorate assigned to liaise with specific carriers.
A formal structure and processes were put in place to support the external carriers. More specifically, carriers reporting electronically for air shipments are supported by the Electronic Commerce Unit (ECU), already in place in the MPDD Directorate. The unit is the first line of support for data transmission problems and if required, it escalates problems to the technology services group responsible for monitoring the external interface. A call centre approach is used to document and resolve problems from carriers. Carriers are provided with toll-free support during regular business hours. Support is critical since the ACI is deemed to be a high-availability application, which requires support outside the normal hours of service.
A project charter was not developed to clearly identify the authorities and accountabilities of the different directorates in the ISTB and the project sponsors. The official sponsor was not clearly identified at the outset of the project. Project managers identified three different branches to the audit team as unofficial project sponsors: the Enforcement, Admissibility and Operations branches.This created an environment where there were many different masters. ACI was a broad Agency initiative that crossed a number of internal organizational boundaries and, as such, the MPDD Directorate took a lead role.
Sponsorship and engagement of the business units is seen as an ongoing challenge. As with all development projects, the ISTB and the sponsors have authorities and responsibilities to make decisions affecting the development project. For ACI-EDI Reporting for Air, the delineation of authorities and responsibilities was not clear. This may, at least in part, be caused by the historical structure whereby the MPDD Directorate was part of, and represented, the business group and held overall sponsor authority for all major system development projects. Since sponsors were not officially engaged at the project outset, interviewees indicated that the MPDD Directorate, out of necessity, assumed authority for many development decisions. Senior managers from the sponsor branches interviewed indicated that they did not sufficiently involve themselves early in the development largely due to competing priorities and other operational commitments. Directors general (DGs) from the sponsor branches were informed of the status of commercial system projects through established committee meetings, but their attendance was not constant. As the project evolved, sponsor branches became more involved in providing policy direction. They became particularly engaged late in the development when questions of policy were raised over when and where targeting would happen -- a fundamental component of ACI. This was a significant policy decision requiring extensive consultation that was not made early in the development, as was needed. A decision was made, just prior to implementation, to pilot two approaches to targeting, which will be analyzed later and a final policy decision made. For ACI-EDI Reporting for Air, roles and responsibilities at the development level in the ISTB were well understood. However, within the ISTB, accountability for successful delivery of the project was not clearly defined at the senior levels. There was no single senior manager responsible and answerable for all aspects of project delivery for the project. During this project, there were two DGs with shared accountability for project delivery:
Despite the lack of clarity around authorities and accountabilities, traditional roles and individual efforts ensured that the system was developed effectively. With the reorganization, IT was combined with the MPDD Directorate so that one DG was responsible for all aspects of project development and delivery for all commercial system projects. This should improve the accountability for system development delivery for future projects.
The audit noted that governance processes are evolving and improving in the CBSA. For more recent projects such as eManifest, the audit team was informed that a DG steering committee was established at the project outset to include senior management from all affected branches, including the regions. In addition, authorities, responsibilities and accountabilities are clearly defined in a project charter for the project. These practices are consistent with the new Major Project Governance Framework being developed. The Framework explains the authority, processes and oversight for all phases of a project including definitions of the roles and responsibilities of sponsors, senior management and managers, as well as organizational committees established to govern project activities.
Clearly defined authorities and accountabilities at the project outset can help to ensure that sponsors are sufficiently engaged in the project to define and monitor their requirements to obtain a project that meets their needs. Similarly, accountabilities for successful development will be clear such that a directorate is held accountable for the successful development of the project (i.e. on time, within budget and meets user requirements).
Management Action Plan | Completion Date |
---|---|
The ISTB, in collaboration with various branches, will implement the new Major Project Governance Framework process. A component of the Framework is the project charter, which describes the authorities, responsibilities and accountabilities of sponsors, stakeholders and project leads for all phases of a project, including governance. | December 2007 |
As noted in the Phase 1 audit report, a project management framework (or lifecycle) to manage systems under development is in development and is being refined with supporting tools and templates for implementation. This framework was not in place at the time of the ACI-EDI Reporting for Air development project. The project did however follow a development methodology with clear phases closely aligned to the framework being developed.
The Phase 1 audit report noted areas where the project management framework could be enhanced or improved. The audit work done in Phase 2 supports those earlier conclusions. The following observations were made on the project management practices for the development of the ACI-EDI Reporting for Air project:
In the Phase 1 audit report, it was recommended that the ISTB should continue with its plans to implement a process to ensure that gates were clearly defined and benefit realization is reviewed and documented at each gate. Management had indicated that such processes would be implemented. It was also recommended that a project status reporting regime and a business case process be developed and management had indicated that such processes would be developed.
Management Action Plan | Completion Date |
---|---|
The ISTB is in the process of further refining the integrated master project schedule, working in consultation with project teams and other stakeholders. | November 2007 |
The ISTB is developing a formal Change Management Strategy, which includes establishing a formal change management control board. | December 2007 |
The ISTB will implement an issue management process that defines standard activities and templates for identifying and ranking issues throughout the project lifecycle. | December 2007 |
A formal project risk management process involves the identification, assessment, treatment and monitoring of risks that could jeopardize the success of the project with respect to product quality, schedule and cost. The potential consequences associated with risks and the tolerance for risk exposure would be discussed to determine the formal risk response (avoid, mitigate, assume or transfer/escalate). The project team would determine whether the risks are acceptable and what can and could be done to remedy or better manage the risk. A formal escalation process would ensure that senior management is engaged in the risk discussions at the appropriate times.
For the ACI-EDI Reporting for Air project, the audit found that risks were informally addressed on a day-to-day basis, but they were not well documented. Interviewees indicated that when risks were identified, they were reported to the relevant project manager during regular meetings. Minutes of meetings were not always kept for the regular project meetings to support the documentation of the risks and actions taken. A standard risk log was not maintained for the project to monitor the risk exposure. Since the time of the project development, a risk log template has been developed to document project risks.
In the Phase 1 audit report, it was recommended that the ISTB should continue with its efforts to implement a risk management strategy to support the Major Project Governance and Project Lifecycle Framework. Management had indicated that such a strategy would be implemented.
A security assessment for the ACI initiative was prepared in the appropriate template to ensure key aspects of IT security are consistently addressed for all projects. From the documentation, it was unclear who reviewed and approved the security assessment. Through interviews, it was noted that the process for conducting security assessments was being reviewed at the CBSA by the IT Security Group in the ISTB and the Departmental Security Office (DSO) in the Comptrollership Branch.
Management Action Plan | Completion Date |
---|---|
The Comptrollership Branch and the ISTB will develop a comprehensive process to ensure that threat and risk assessments of new IT systems/projects include a coordinated security assessment that takes into account the technical systems requirements as well as information and physical security considerations. Once developed, this process will be incorporated into the project requirements and be monitored via the project management process. | First quarter of 2008-2009 |
The audit criteria used for Phase 2 of the audit were:
Control Category | Audit Criteria |
---|---|
Governance |
|
Business Benefits |
|
User Requirements |
|
Project Management |
|
Technical Solution | Technical solution is viable in terms of implementing the new technology:
|
Business Transformation | Appropriate governance processes and procedures in place to address the impacts of changes to users generated by the implementation of new system development projects. For example:
|