This page has been archived.
Information identified as archived is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.
October 2007
The audit of Canada Border Services Agency (CBSA) systems under development was identified as a priority in the CBSA’s Risk-Based Multi-Year Audit Plan. This Plan was approved by the Internal Audit and Evaluation Committee on March 17, 2005.
The principal objective of this audit was to provide the necessary assurances to the CBSA that its integrated project management and system development practices and procedures for automated business systems under development are adhering to the internal policies and procedures that have been established by the CBSA. A further objective of this audit was to identify improvement opportunities to the CBSA’s existing internal policies and procedures for managing its systems under development.
The audit was divided into three phases. Phase 1, reported in February 2007, provided an assessment of the adequacy and effectiveness of the CBSA’s current management control framework. Phases 2 and 3 provide an assessment of the degree to which two selected system development projects are effectively and appropriately applying the control framework. This report presents the findings from Phase 3 of the audit that reviewed the management framework for the development of a selected system development project: Advance Passenger Information/Passenger Name Record (API/PNR) Risk Scoring. The audit was conducted by Interis Consulting.
API/PNR Risk Scoring is part of the Passenger Information System, a multi-year, multi-million-dollar project launched under the Smart Border Declaration to improve the handling of air passengers. API/PNR is intended to provide the CBSA with more effective risk-management processes and tools to assess the risk of air travellers (particularly those posing a high or unknown risk) before they arrive at Canada’s international airports. Development work on API/PNR began before the formation of the Agency as a joint initiative between the former Canada Customs and Revenue Agency (Customs Branch), Citizenship and Immigration Canada and U.S. Customs and Border Protection (CBP). The audit addressed the risk-scoring and data-sharing components that were implemented in June 2006.
Based on the audit work conducted in Phase 3, it is found that API/PNR Risk Scoring was developed in accordance with CBSA policies, procedures and methodology in place at the time and the functionality of the application is working as intended. Many of the observations made in Phase 1 of the audit were supported by the audit work conducted in Phase 3. In addition, the audit noted that since the time of the API/PNR Risk Scoring project development, many new processes and governance practices have been developed and introduced for more recent projects.
The audit noted a number of strengths in the API/PNR Risk Scoring development process, including a strong governance framework and the use of a sound system development approach. Because API/PNR is a joint development program with U.S. CBP, the governance framework and structure, joint user requirements and a memorandum of understanding were formally developed and signed off by senior representatives of U.S. CBP and the CBSA. A privacy impact assessment was completed and shared with the Office of the Privacy Commissioner of Canada. Recommendations regarding privacy concerns were implemented. The system development approach included the following activities that would be expected in any system development project:
Although many of the findings from the audit of API/PNR Risk Scoring are the same as those reported in the Phase 1 audit report, the following additional areas of control were identified in Phase 3 that could be strengthened at the project level:
The audit of Canada Border Services Agency (CBSA) systems under development has been identified as a priority in the CBSA’s Risk-Based Multi-Year Audit Plan. This Plan was approved by the Internal Audit and Evaluation Committee on March 17, 2005.
The principal objective of this audit was to provide the necessary assurances to the CBSA that its integrated project management and system development practices and procedures for automated business systems under development are adhering to the internal policies and procedures that have been established by the CBSA. A further objective of this audit was to identify improvement opportunities to the CBSA’s existing internal policies and procedures for managing its systems under development.
To achieve these objectives, the internal audit plan was divided into the following three phases:
The audit deliverables for Phase 1 were the following:
The audit deliverables for Phases 2 and 3 were the following:
The scope of this audit was to assess the development of API/PNR Risk Scoring Phase II and not the effectiveness of the application itself. Given the multi-phase approach taken for the API/PNR program, much of the planning, scoping and identification of requirements for the entire program (including Risk Scoring Phase II) had been completed in Phase I. The scope of the audit included all development activity that affected Risk Scoring Phase II regardless of the phase when it occurred.
Audit criteria were developed based on risk elements inherent to system development projects, as well as on public sector trends for effective governance of system development project lifecycles. A detailed set of audit criteria were established that factored in the primary influences that can significantly impact system development projects in the Government of Canada (GC): management structure and processes, system development lifecycle policies and standards, as well as planning and acquisition processes and procedures. The detailed audit criteria were used to assess CBSA practices.
It was noted during the initial audit planning that business system development projects at the CBSA are inherently exposed to risk given the following conditions:
Key control areas addressing inherent risks were identified based on the Treasury Board of Canada Secretariat’s (TBS) Enhanced Management Framework and the Control Objectives for Information and related Technology (CobiT) Framework issued by the IT Governance Institute. The audit criteria used to initially assess the CBSA’s overall business practices, general controls and governance processes for their systems under development projects have been organized into the following six categories:
The audit criteria are presented in Appendix A.
The approach and methodology were risk-based and compliant with applicable TBS internal audit policies. The audit was conducted in accordance with an audit program that defined audit tasks to assess each criterion. Through interviews and documentation review, the audit assessed the current practices against the criteria and formally assessed the effectiveness of each practice.
Interviews were conducted with representatives of the Innovation, Science and Technology, Enforcement, Comptrollership and Operations branches.
The fieldwork for Phase 3 was conducted between July and October 2006.
The results of the audit from Phase 1 were reported in February 2007. A number of strengths were noted in the controls over systems under development that can provide the foundation for building a strong MCF over systems under development, namely a governance structure and a project management framework. Given the newness of the agency reorganization, many of the processes were still in early development and had not matured in the organization. New committees were being formed, roles and responsibilities were being clarified, and specific deliverables were being developed. These activities are indicators that the Agency is making progress toward enhancing its control over systems under development.
The following key areas of control that should be strengthened were noted in the Phase 1 audit report:
Management reviewed the Phase 1 audit report and provided management action plans to address the audit recommendations and the expected date of implementation for each action.
The purpose of this document is to present the findings from Phase 3 of the audit.
Following the terrorist attacks of September 11, 2001, Canada and the United States signed the Smart Border Declaration, also known as the Manley-Ridge 30-point plan, in December 2001. The Smart Border Declaration outlines a 32-point Action Plan that provides for ongoing collaboration to identify and address security risks while efficiently expediting the legitimate flow of people and goods across the Canada-U.S. border. U.S. Customs and Border Protection (CBP) has had the lead responsibility for 11 of the 32-point Action Plan initiatives. The CBSA had responsibility to develop a parallel API/PNR program in Canada to that being developed in the United States.
The Smart Border Declaration identified a commitment to using innovative technology to more effectively identify high-risk individuals (High-Risk Traveller Identification Initiative). In addressing this commitment, Canada and the United States agreed to share API and PNR data as well as to explore a means of identifying risks posed by travellers on international flights destined to either country. Managed jointly between the former Canada Customs and Revenue Agency (Customs Branch), Citizenship and Immigration Canada and U.S. CBP, the initiative was composed of two parts:
This audit looked at the system development project that addressed the second component, API/PNR Risk Scoring.
API/PNR was one of the highest priorities for the CBSA and a GC commitment to help protect the border. Although conceived before the events of September 11, 2001, the API/PNR Risk Scoring project became a priority for the GC as a result of those events.
API/PNR is intended to provide the CBSA with more effective risk-management processes and tools to assess the risk of air travellers (particularly those posing a high or unknown risk) before they arrive at Canada’s international airports. This initiative provides CBSA officers with pre-arrival data regarding air travellers destined for Canada. Air carriers provide PNR information (in electronic form) for all flights destined for Canada at the time of departure and, for the United States, within 30 minutes of departure. API, together with other relevant information, permits officers to analyze the information and, with the assistance of the automated targeting and risk-assessment tool, to identify high-risk air travellers requiring further scrutiny on arrival.
API/PNR is part of the Passenger Information System, a multi-year, multi-million-dollar project launched under the Smart Border Declaration to improve the flow of air passengers by providing for the receipt and analysis of travel information on individuals before their arrival in Canada.
The CBSA worked on API/PNR in parallel with U.S. CBP, based on a joint requirements document that outlined the requirements that had to be implemented and followed by each border agency.
At the time that API/PNR Risk Scoring was being developed, the organizational structure of the CBSA and the Innovation, Science and Technology Branch (ISTB) [ 1 ] was as presented below.
CBSA's Innovation Science and Technology Organizational Structure
Under this structure, the API/PNR Risk Scoring project was managed by one project manager from the Major Projects Design and Development (MPDD) Directorate and by one information technology (IT) project manager from the Border Systems Directorate. The project was sponsored by the Enforcement Branch. Through the development process, there were personnel changes to the ISTB team, as well as numerous changes in the other branches.
At the time when API/PNR Risk Scoring was being developed, there were many pressures on the Agency and the ISTB. The operations of the Agency are highly dependent on information systems and technology, which are developed, implemented and managed by the ISTB. The scope of system development activity at the CBSA is significant, with annual spending on development and maintenance of systems, infrastructure and related technology estimated to be in the $150-250 million range over the next few years. In response to increased North American security concerns, following the terrorist attacks in September 2001, the GC introduced a considerable number of security-related initiatives, under the umbrella of the Public Security and Anti-terrorism (PSAT) Initiative and the Shared Border Declaration, resulting in expedited initiative development. This put pressure on a number of system development projects, including API/PNR Risk Scoring.
The API/PNR program was funded from the Manley-Ridge funding envelope. Although a budget was not officially established for the API/PNR Risk Scoring project, development costs were estimated to have been approximately $4.87 million. [ 2 ] This cost estimate includes all CBSA costs related to the development of the API/PNR risk scoring project and data sharing from 2003-2004 to 2006-2007.
API/PNR Risk Scoring Phase II was implemented in June 2006 and is in active use by its primary user, the National Risk Assessment Centre (NRAC). NRAC issues lookouts to ports of entry in both Canada and the United States and this facilitates the expeditious flow of information and intelligence between and within partnering agencies to help identify high-risk travellers before they arrive at the border. Further development of API/PNR will involve building interfaces to improve immigration data exchange and to enhance data analysis functionality.
Based on the audit work conducted in Phase 3, it is concluded that API/PNR Risk Scoring was developed in accordance with CBSA policies, procedures and methodology in place at the time. Furthermore, the application is working as intended. However, opportunities exist to strengthen the system development processes used in API/PNR Risk Scoring to ensure adequate governance, risk management of and control over future system under development projects.
Since the time of API/PNR Risk Scoring Phase II development, many new processes and governance practices have been developed and introduced for more recent projects. In addition, many of the observations made in Phase 1 were supported by this Phase 3 audit. Management action plans for Phase 1 audit recommendations indicated that many audit areas needing strengthening have recently been addressed or are in the process of being addressed.
The audit findings are described in detail below.The audit found that the CBSA system development methodology was used across all phases of the API/PNR Risk Scoring project. The following key activities were carried out:
The API/PNR Risk Scoring project was developed using an industry-recognized methodology, the Rational Unified Process (RUP), [ 3 ] to achieve the final design and build. The audit examined various documents that demonstrated good development practices for the technical solution, as expected in any system development process. The system design was described in a project scope document, an initiation phase process document and a high-level business requirements document. User requirements were clearly described and documented in business use cases. The requirements were then translated into a system component design document and system use cases for the developers. Many of the documents prepared for API/PNR were developed using templates that have since evolved and been replaced by new templates, but the key objectives of the documents remain the same. Key individuals from the ISTB were involved in the initial design and development phases. The Enterprise Architecture Group interfaced with the Business Architecture Group in the MPDD Directorate to define the requirements and review the feasibility of the technical solution.
User requirements were defined with involvement from key stakeholders including U.S. CBP. The MPDD Directorate took a large role in determining the high-level requirements and priorities; however, joint application development sessions were held with representatives of the Enforcement Branch and U.S. CBP to develop the business use cases. The Intelligence Sector and Contraband Program in the Enforcement Branch provided functional guidance and subject matter expertise to the API/PNR Risk Scoring project. The audit found evidence of ongoing refinement of the business requirements, largely through e-mail communications and updates to project documents. To improve its process of developing user requirements, the ISTB has begun using "usability architects" who look at the interface to see how people use the system. User requirements can then be further enhanced to make the system more "usable" for the users.
Testing of API/PNR Risk Scoring was performed in a number of different test environments before implementation. Development testing was performed by the IT group and included data and database integrity testing, function testing, business cycle testing and user interface testing. Test cases were mapped out, schedules were defined, resources were assigned and test results were documented and summarized. After the development testing was completed, the MPDD Directorate conducted systems integration testing, user acceptance (client) testing and operational testing. Individuals outside the MPDD Directorate, including those from NRAC, were invited to participate in user acceptance testing.
Typically, once the development and implementation are complete, the system application is moved from development to production and is transferred to the Systems Operations group in the MPDD Directorate for ongoing support. The move is well controlled through release management procedures. At the time of the audit, the API/PNR Risk Scoring project had not yet been transferred to Systems Operations. It is a normal practice within the ISTB to keep a system under development project under the control of the development group for about a year after implementation to monitor initial operational problems before it gets moved to Systems Operations.
Implementation plans were developed and managed by the MPDD Directorate, in collaboration with NRAC, to support internal users (primarily NRAC) and external air carriers. Processes and resources were put in place to record and track systems issues and specific API/PNR problems. End users were trained in the use of API/PNR Risk Scoring based on formal training documentation prepared in the development process. Training and maintenance of training materials is carried out by NRAC as the primary user of API/PNR Risk Scoring.
Air carriers were required by law to provide the data elements defined jointly by the CBSA and U.S. CBP. These data elements were reviewed with the Office of the Privacy Commissioner of Canada and the European Union to ensure that they complied with Canadian and European law. Communications with the external air carrier community were maintained through the development process of the data acquisition component to ensure that necessary data was provided in the correct format for use in API/PNR Risk Scoring. A formal process was in place for carriers to test their interfaces to send data electronically.
A formal structure and processes were developed to support internal users and external air carriers. A call centre approach is used to document and resolve problems from air carriers. Carriers are provided with toll-free support during regular business hours. Carriers reporting API/PNR data electronically are supported by the ISTB call centre as the first line of support for data transmission problems. If required, the call centre escalates problems to the technology services group responsible for monitoring the external interface.
Authorities, responsibilities and accountabilities for API/PNR Risk Scoring were identified in the project charter for the API/PNR High-Risk Traveller Identification Initiative. The project charter was developed jointly with U.S. CBP over the period from November 2002 to April 21, 2004. The project charter was initially signed off by the CBSA and U.S. CBP in May 2003 with subsequent revisions made to reflect changes that were required as the project evolved in the bilateral development arrangement.
In addition to the project charter, a governance structure was defined for API/PNR in the governance structure for the High-Risk Traveller Identification Initiative, which evolved over the period from July 30, 2003, to March 3, 2004. The governance structure was created based on input from the various relevant organizations including Customs Branch and Citizenship and Immigration Canada (before the creation of the CBSA), Contraband and Intelligence Directorate and the MPDD Directorate. The structure included clear definitions of the executive sponsors, the API/PNR working group (steering committee role), the project management group and the project team members. It also outlined the respective roles and responsibilities of these individuals, decision-making responsibilities, the lead U.S. counterpart for bilateral communications and liaison, and the dispute resolution process. The governance structure was approved by representatives of the following organizations:
The Enforcement Branch was identified as the lead sponsor for the project with the MPDD Directorate having the lead role for development. In the early stages, the Enforcement Branch was heavily involved in the definition of user requirements in business use cases. However, as the project progressed, sponsorship and engagement of the business units decreased.
With multiple conflicting demands on the Enforcement Branch, engaging the sponsor was seen as a constant struggle. As with all development projects, the ISTB and the sponsor have authorities and accountabilities to make decisions affecting the development project. On account of the demands on the sponsor, the MPDD Directorate drove the project to its completion with authority for overall development and the resources to complete it. Under the old historical structure, the MPDD Directorate was part of, and represented, the business group and held overall systems authority for all major system development projects. Interviewees indicated that the MPDD Directorate assumed authority for some development decisions out of necessity to meet the timelines for project delivery. Some senior managers from the sponsor branches interviewed indicated that they did not sufficiently involve themselves throughout the development process largely due to competing priorities and other operational commitments. Decisions were made quickly and sponsors did not always have time to respond. This was done to manage an environment of numerous projects underway at the same time and many incremental demands being placed on the Agency to deliver its mandate.
As the project evolved, sponsor branches became more involved in providing policy direction, (e.g. the policy on privacy concerns and the process for sharing sensitive information with U.S. CBP). As the project progressed, decisions were made in bi-weekly project team meetings and recorded in minutes. Apart from the formal project charter, governance structure and scope documents, sponsors’ approval of decisions at key milestones in the project development were not formally documented. It is important to obtain and document sponsors’ support to ensure that the development work continues to meet sponsors’ requirements and is expected to achieve projected benefits.
In the Phase 1 audit report, it was recommended that the ISTB should continue with its plans to develop processes to better engage sponsoring branches in systems development. Management had indicated that processes would be developed, as well as a change management strategy, which would include obtaining appropriate approvals of the sponsoring branch.
As noted in the Phase 1 audit report, a project management framework (or lifecycle) to manage systems under development is in development and being refined with supporting tools and templates for implementation. This framework was not in place at the time of the API/PNR Risk Scoring project development. The project did, however, follow a development methodology with clear phases closely aligned to the framework being developed.
The Phase 1 audit report noted areas where the project management framework could be enhanced or improved. The audit work done in Phase 3 supports those earlier conclusions. The following observations were made on the project management practices for the development of the API/PNR Risk Scoring project:
In the Phase 1 audit report, it was recommended that the ISTB continue with its plans to implement a process to ensure that gates were clearly defined and that benefits realization is reviewed and documented at each gate. Management had indicated that such processes would be implemented. It was also recommended that a business case process be developed and management had indicated that such a process would be developed.
Management Action Plan | Completion Date |
---|---|
Management is in the process of further refining the integrated master project schedule and is working in consultation with project teams and other stakeholders. | November 2007 |
Management is developing a formal change management strategy that includes establishing a formal change management control board. | December 2007 |
A formal project risk-management process involves the identification, analysis, mitigation and monitoring of risks that could jeopardize the success of a project with respect to product quality, schedule and cost. The potential consequences associated with the project risks and the tolerance for the risk exposure would be discussed to determine the formal risk response (avoid, mitigate, assume or transfer/escalate). The project team would determine whether the risks are acceptable and what can be done to remedy or better manage the risk. A formal escalation process would ensure that senior management is engaged in the risk discussions at the appropriate times.
For the API/PNR Risk Scoring project, the audit found that risks were informally addressed on a day-to-day basis but they were not well documented. Interviewees indicated that when risks were identified, they were reported to the relevant project manager during regular meetings. Minutes of meetings were kept for the regular project meetings to support the documentation of the risks and actions taken. A project issues log was maintained for this project to monitor project management issues; however, a standard risk log was not maintained for the project to monitor the risk exposure across the Agency as a whole. Since the time of the development process, a risk log template has been developed to document project risks.
In the Phase 1 audit report, it was recommended that the ISTB should continue with its efforts to implement a risk-management strategy to support the Major Project Governance and Project Lifecycle Framework. Management had indicated that such a strategy would be implemented.
A security assessment for the API/PNR program was prepared in the appropriate template to ensure key aspects of IT security are consistently addressed for all projects. From the documentation, it was unclear who reviewed and approved the security assessment. Through interviews, it was noted that the process for conducting security assessments was being reviewed at the CBSA by the IT Security Group in the ISTB and the Departmental Security Office in the Comptrollership Branch.
Management Action Plan | Completion Date |
---|---|
The Comptrollership Branch and the ISTB will develop a comprehensive process to ensure that threat and risk assessments of new IT systems/projects include a coordinated security assessment that takes into account the technical systems requirements as well as information and physical security considerations. Once developed, this process will be incorporated into the project requirements and monitored via the project management process. | First quarter of 2008-2009 |
The audit criteria used for Phase 3 of the audit were the following:
Control Category | Audit Criteria |
---|---|
Governance |
|
Business Benefits |
|
User Requirements |
|
Project Management |
|
Technical Solution | Technical solution is viable in terms of implementing the new technology:
|
Business Transformation | Appropriate governance processes and procedures in place to address the impacts of changes to users generated by the implementation of new system development projects. For example:
|