ARCHIVED - The National Rail Export Verification Unit Pilot Project
Evaluation Study

This page has been archived.

May 2007

Table of Contents

• Executive Summary
• Introduction and Context
  • Description of Canadian Rail Exports
  • Description of the REVU Pilot Project
  • Evaluation Purpose
  • Evaluation Methodology
• Key Findings
  • Program Design and Delivery
  • Performance Management
  • Results Achievement (Success)
  • Relevance
• Summary of Key Findings and Recommendation
• Management Response

---

Executive Summary

The Rail Export Verification Unit (REVU) pilot project is a national initiative that was developed in partnership with the Canadian National Railway Company (CN) as a result of CN’s business case for consolidated export reporting presented in July 2003 to the Canada Border Services Agency (CBSA). Based on consultations held with both CN and the Canadian Pacific Railway Limited (CPR) and a subsequent CBSA feasibility study, it was determined that a pilot project, which focused on all activity on A5 outward train reports rather than exports alone, was viable. Furthermore, it was decided that the pilot project would be limited to CN cargo.

The REVU pilot project received formal authority from the CBSA’s Vice-President of Operations in September 2004. The REVU pilot project team began developing policy and procedural documentation the following month. One of the challenges to be addressed through the pilot project was the record-keeping process for declining balances on Canadian Wheat Board (CWB) permits. Full implementation of the pilot project was scheduled to coincide with changes to the Reporting of Exported Goods Regulations coming into force in May 2005. Under the revised Regulations, exporters must report goods destined for export within prescribed time frames and at designated locations, except those goods that are specifically exempted from the reporting requirement. For goods being exported by rail, exporters must submit an export declaration no less than two hours before the railcar on which the goods have been loaded is assembled to form part of a train for export.

Winnipeg was selected as the location for the REVU pilot project because Canada’s two largest railways, CN and CPR, have their customer service/support centres located there. As well, Winnipeg is the location for one of the International Traffic in Arms Regulations (ITAR) units and the CBSA’s Project Winfall units. Winnipeg’s ITAR group is a targeting and strategic export control unit that focuses on a daily sample review of rail traffic destined to a seaport for offshore export. Project Winfall is a national rail-targeting unit for imports. It was logical that the pilot project be built on the strong working partnership and trust that had evolved between CN and ITAR and Winfall targeters.

The REVU pilot project has an annual operating budget of $0.5 million, which includes salaries and benefits for seven CBSA officers.

The key evaluation issues assessed were the following:

• The extent to which the REVU pilot project is being delivered as designed;
• The extent to which performance information being collected, monitored and used for continual improvement;
• The progress of REVU in achieving its expected results and responding to the impact of unexpected results; and
• The extent to which the REVU pilot project continues to be aligned with government-wide and CBSA priorities and whether it addresses an actual need.

Methodology

The plan for this evaluation was completed in October 2006 by the CBSA’s Evaluation Division. The consulting firm of T.K. Gussman Associates Inc. was contracted to conduct the evaluation. The Evaluation Division provided overall guidance and participated in the site visit. Data were collected between December 2006 and January 2007. The following methodologies were used:

• Review of documents and statistics;
• REVU site visit to observe compliance review processes in Winnipeg; and
• In-depth interviews with senior managers at CBSA headquarters (HQ) and regional management, program officials and key internal and external stakeholders, including representatives from CN, CPR and the CWB.

Summary of Key Findings and Recommendation

The REVU pilot project is being delivered as it was conceived. The Letter of Intent with CN clearly defined the basis for cooperation between the two organizations at both the national and local levels, including the conditions for access to databases. Overall, REVU has established open and clear channels of communication and forged a solid partnership with the ITAR unit, CN and the CWB. At the same time, however, the evaluation found that there was limited communication and collaboration with CBSA HQ branches beyond REVU’s interactions with the Export Process program area in the Admissibility Branch on policy requirements for the CBSA and CN. Although the REVU pilot project documents details of all activities and processes, no formalized management strategy was developed to obtain performance data.

The evaluation found that the REVU pilot project has achieved positive and observable results. For CN shipments, the REVU pilot project has led to more consistent and uniform application of and increased compliance with regulations under the D3 (transportation), D19 (other government departments) and D20 (export reporting) series of customs D memoranda. As well, increased compliance has been observed with Foreign Affairs and International Trade Canada and Environment Canada regulations. The results for shippers using CN speak for themselves: export reporting compliance reached 98 percent by July 2006. The REVU pilot project has expanded the CBSA’s available skills and knowledge to identify shipments containing ITAR- or other controlled, regulated or prohibited goods. The evaluation also found anecdotal evidence that spillover compliance improvements have been observed in other areas as a direct result of shippers’ and companies’ knowledge of REVU’s increased scrutiny of outbound shipments via CN.

The pilot project has been delivered effectively on a modest annual budget of approximately $0.5 million. Notwithstanding the fact that this pilot has been successful, there are a number of considerations that support a discontinuation of the project.

The current model of serving only two organizations (CN and the CWB) may place the Agency at risk of complaints from other companies that do not receive this service and attention should the REVU pilot project, as it currently operates, become a permanent program. In addition, the compliance rate achieved for CN since REVU’s inception has been successful to the point that there is no longer a need for the project. Unless other railway companies are participating (only limited interest for participation exists at this time), the benefits to the CBSA by continuing REVU are minimal. 

On the basis of these findings, the following recommendation was made:

1. Introduction and Context

The national Rail Export Verification Unit (REVU) pilot project was identified as a priority for evaluation in 2006-2007 in the CBSA Risk-Based Multi-Year Evaluation Plan 2006-2009, which was approved by the Internal Audit and Evaluation Committee in June 2006.

1.1 Description of Canadian Rail Exports

The annual report for Transport Canada, Transportation in Canada 2005, [ 1 ] provides an overview of Canada’s export activity by mode of transportation. Rail transport accounts for slightly more than 20 percent of Canada’s exports to the United States in terms of both volume and value. Chart 1 illustrates the growth in rail exports relative to all other modes of export to the U.S. Over the past decade, the value of exports has fluctuated between 20 and 22 percent, while the share by volume has grown from less than 18 percent to 20 percent. 

Ontario is the largest contributor to rail exports, originating 23 percent of export volume and 58 percent of export value in 2005. Fort Frances and Sarnia were the main border crossing points for rail exports by volume in 2005, accounting for 20.2 percent (15.4 million tonnes) and 16.2 percent (12.4 million tonnes) respectively. Forest products and chemicals accounted for about 60 percent of rail export volumes through these locations. In terms of value, Sarnia and Windsor accounted for 31.2 percent ($23.9 billion) and 21.8 percent ($16.7 billion) of exports respectively. Automotive products accounted for about 70 percent of rail export value at these locations.

There are five Class 1 rail freight carriers in Canada: Canadian National Railway Company (CN), [ 2 ] Canadian Pacific Railway Limited (CPR), [ 3 ] Burlington Northern Santa Fe (BNSF) Railway, [ 4 ] CSX Transportation Inc. [ 5 ] and Norfolk Southern Railway Company. [ 6 ] Industry representatives estimated that CPR and CN account for approximately 90 percent of rail exports, with each having an equal share.

1.2 Description of the REVU Pilot Project

In July 2003, CN presented a business case to the Export Process program of the Canada Border Services Agency (CBSA) for the consolidation of CN’s export reporting to one CBSA office. Export reporting at the various border crossings was a key concern for CN, and the company argued that centralized reporting would result in a more uniform and consistent application of regulations. Based on consultations with both CN and CPR and a subsequent CBSA feasibility study conducted in the Prairie Region, it was determined that a pilot project, which focused on all activity on A5 outward train reports rather than exports alone, was viable. Furthermore, it was decided that the pilot project would be limited to CN cargo.

The REVU pilot project received formal authority from the CBSA’s Vice President of Operations (VP Ops) in September 2004. The REVU pilot project team began developing policy and procedural documentation the following month. A letter of intent was drafted and signed in March 2005 by the VP Ops and the Regional Director General (RDG) of the CBSA’s Prairie Region. CN became a signatory in April 2005.

Winnipeg was selected as the location for the REVU pilot project because Canada’s two largest railways, CN and CPR, have their customer service/support centres located there. As well, Winnipeg is the location for one of the International Traffic in Arms Regulations (ITAR) units and the CBSA’s Project Winfall units. Winnipeg’s ITAR group is a targeting and strategic export control unit that focuses on a daily sample review of rail traffic destined to a seaport for offshore export. Project Winfall is a national rail-targeting unit for imports. It was logical that the pilot project be built on the strong working partnership and trust that had evolved between CN and ITAR and Winfall targeters.

Full implementation of the pilot project was scheduled to coincide with changes to Canadian export reporting regulations coming into force in May 2005. [ 7 ] Under the revised regulations, exporters must report goods destined for export within prescribed time frames and at designated locations, except those goods that are specifically exempted from the reporting requirement. For goods being exported by rail, exporters must submit an export declaration no less than two hours before the railcar on which the goods have been loaded is assembled to form part of a train for export. The regulations provide exemptions from the reporting requirement for certain classes of goods, such as the following:

• Personal and household effects;
• Commercial goods, unless controlled, regulated or prohibited, having a value of less than CAN$2,000;
• Goods exported by a diplomatic embassy or mission personnel for their personal or official use; and
• Goods for which there is a written arrangement between Canada and another country. [ 8 ]

Under section 7 of the Reporting of Exported Goods Regulations, Canada has an information-sharing agreement with the U.S. under which goods exported to the U.S. do not have to be reported. The exporter is, however, required to ensure that permits, licences or certificates are presented at the place of exit from Canada for all controlled goods being exported to the U.S.

In accordance with a 1987 memorandum of understanding, Canada substitutes import statistics from the U.S. for Canadian exports to the U.S.

The REVU pilot project has an annual operating budget of $0.5 million, which includes the salaries and benefits for seven CBSA officers.

The REVU pilot project’s scope is limited to all outbound CN rail shipments. Intermodal containers moving by rail to a Canadian marine port are the responsibility of the ITAR unit (which monitors all controlled goods being exported), not REVU. Hence, REVU officers look at the waybills of CN trains going to the U.S. only. Exports carried by these trains may have the U.S. as a final destination or they may be destined for a third country leaving the U.S. by land to Mexico or by sea to Europe.

Key Program Activities

The REVU pilot project is delivered through the following key activities.

Development and maintenance of SOPs and coordination with the ITAR unit

REVU’s responsibilities include the identification, tracking and tracing of rail containers that are potentially in violation of Canadian export laws and regulations. REVU has created a standard operating procedure (SOP) for each activity and process that has been established through the course of this pilot. These SOPs are stored on the common G drive on REVU’s computer system and are updated regularly by REVU officers.

REVU officers forward information regarding suspect outbound shipments to ITAR targeters after confirming a hold on the shipment with CN. ITAR targeters perform intelligence checks on all referrals to determine whether an examination is required and, if so, to determine the inspection location for a CBSA export examination team or an examination team from U.S. Customs and Border Protection. The team inspects the shipments in question and seizes any illegal cargo or equipment.

Collaborating and maintaining partnerships

The REVU pilot project was conceived as a partnership between the CBSA and CN. The very nature of the initiative depends on full cooperation and trust, given the access granted to REVU officers to CN’s computer system to monitor all waybills for outbound shipments. Moreover, REVU is predicated on collaboration between the CBSA and other federal regulatory departments/agencies or organizations. Information flows steadily between the CBSA, the Canadian Wheat Board (CWB) and CN, with daily and weekly reports. REVU officials, as part of their regular schedule, meet with officials in these partner organizations.

Within the CBSA, REVU is mandated to collaborate with officials in the ITAR unit. The units have coordinated work plans with a two-way exchange of information to support them in delivering their mandates. Informal mechanisms have been established for regular briefing sessions and information exchange between REVU and the ITAR unit, both at the senior and working levels. Formal guidance flows from the ITAR unit to REVU to enable REVU officers to identify certain shipments on the basis of intelligence identified through the ITAR unit.

Verification processing

Pre-departure and post-departure review

REVU reviews all of CN’s outbound traffic through the eight CN ports of exit [ 9 ] to the U.S. using the company manifest system in real time. This allows REVU officers to provide decisions in advance of a train being assembled. Following the SOPs for pre-departure review, REVU officers scan CN waybills for errors or omissions pertaining to proof of report information or permit or cargo numbers, and they assign either "A" (accept) or "H" (hold) status to any specific shipment, enabling immediate follow-up action with shippers or enforcement officials. When issues are identified, contact is made with the exporter by telephone and in writing to promote future compliance. REVU officers also forward information regarding suspect outbound shipments to ITAR targeters after confirming a hold on the shipment with CN.

Post-departure processing of A5 outward train reports is also REVU’s responsibility. Such processing includes acquitting permits and cargo control documents.

CWB permit processing

The CBSA identified certain challenges associated with the CWB’s reporting processes and record keeping for declining balances on CWB permits. All information on grain exports was recorded manually. The exporting carrier used to fax permits to the former Canada Customs and Revenue Agency (CCRA) at all ports of exit where customs officials would verify shipments as the shipments arrived at the border. A summary would then be faxed to the CWB. The REVU implementation team examined a sample of CN waybills for CWB shipments and collaborated with the CWB to streamline this reporting process.

Under the pilot project’s centralized processing, the CWB faxes REVU a copy of each permit at the time of issue. REVU officers record the permit information on a spreadsheet in Excel and then file the permit by the CWB number. CN transmits to REVU a daily summary report of grain exports for the previous day. The volumes exported are recorded against the corresponding CWB permit on the spreadsheet. REVU sends a summary report of export activity by permit each Friday to the CWB. Once a CWB permit has been depleted or expires, the file is closed by REVU and the permit is faxed back to the CWB.

Expected Results

The evaluation plan for the REVU pilot project identified the following expected project results.

Immediate results:

• Improved skills and knowledge to identify shipments containing ITAR- or other controlled, regulated and prohibited goods;
• More consistent and uniform application of regulations under the D3 (transportation), D19 (other government departments) and D20 (export reporting) series of customs D memoranda;
• Increased compliance with regulations under the D3, D19 and D20 series of customs D memoranda; and
• Increased accuracy of export statistics and reports.

Intermediate results:

• Increased CBSA ability to detect and interdict controlled, regulated and prohibited goods on outbound targeted shipments;
• Increased CBSA ability to identify cargo and export violations and violations under other government departments; and
• Increased CBSA control on outbound rail shipments.

Ultimate outcome:

• Enhanced security of Canada’s border -- enhanced voluntary compliance with border legislation and regulations.

1.3 Evaluation Purpose

The main objective of the evaluation was to assess the progress the REVU pilot project has made in achieving its expected results or the extent to which it is positioned to achieve its expected results, as well as the effectiveness and efficiency of program design, delivery and management. A key issue was whether REVU should be made a permanent program and whether it should be expanded to include other railway companies.

The Evaluation Division of the Strategy and Coordination Branch prepared an evaluation plan, which identified the following key issues to be evaluated:

• Program design and delivery -- To what extent is the REVU pilot project delivered as designed?
• Performance management -- To what extent is performance information being collected, monitored and used for continual improvement?
• Success -- To what extent has REVU achieved its expected results and were there results that were unanticipated?
• Relevance -- To what extent does REVU continue to be aligned with government-wide and CBSA priorities and realistically address an actual need?

1.4 Evaluation Methodology

The Evaluation Division completed the evaluation plan in October 2006. The consulting firm of T.K. Gussman Associates Inc. was contracted to conduct the evaluation. The Evaluation Division provided overall guidance and participated in the site visit. Data were collected between December 2006 and January 2007. Evaluation findings are based on the following methodologies: a review of documents and statistics; a site visit; and in-depth interviews with senior management, program officials and key internal and external stakeholders.

Review of Documents and Statistics

Evaluators reviewed and analyzed planning reports, documents outlining roles and responsibilities, management presentations and other relevant internal correspondence regarding the governance structure of the REVU pilot project.

As well, the evaluators reviewed SOPs for a wide range of pre- and post-departure processes. Some SOPs applied to the Reporting of Exported Goods Regulations under the authority of other government departments, including Foreign Affairs and International Trade Canada (DFAIT), Environment Canada (EC) and Statistics Canada.

Further review involved visiting the CBSA’s and other government departmental Web sites for specific information on export and export reporting regulations.  

The evaluators examined rail industry statistics available from the Railway Association of Canada, Transport Canada and Statistics Canada. The intent was to obtain data that allowed for inter-carrier comparisons. Available data were not disaggregated to enable such analysis. It was possible, however, to observe trends in rail exports on an aggregate level.

Program performance statistics were limited to data on compliance rates and client education activities in aggregate form.

It was determined that Canadian Automated Export Declaration (CAED) statistics would not be useful for evaluation purposes for a number of reasons. First, CAED statistics apply to non-U.S. exports only. As noted above, Statistics Canada uses statistics on U.S. imports from Canada (provided by the U.S.) to get Canadian–U.S. export statistics. These statistics are broken down by mode of transportation (e.g. rail) but not by carrier. The statistics are also broken down by province, but there are inconsistencies in terms of company headquarters and the province from which the exports originate.

Site Visit

Members of the evaluation team visited Winnipeg on December 4 and 5, 2006, to meet with the REVU pilot project team and other stakeholders. The evaluators observed the compliance review processes and procedures in the REVU office to enhance their understanding of how the REVU pilot project is designed and managed. The REVU team provided the evaluators with a visual demonstration of the pre-clearance review process, as well as examples of various types of reports submitted by exporters.

In-depth Interviews

In addition to discussions with the REVU staff during the site visit in Winnipeg, in-person interviews were held with the CBSA Prairie Region RDG, the Director of Planning and Program Integration, the District Programs Officer and the Superintendent of REVU. Discussions were also held with local representatives of Winfall and the ITAR unit. The site visit also afforded evaluators with the opportunity to conduct interviews and group discussions with representatives from CN, CPR and the CWB in their respective Winnipeg offices. These meetings enhanced the evaluators’ understanding of how the pilot project was conceived and is being delivered, and provided insights into its challenges and opportunities.

At CBSA HQ, in-person interviews were conducted with officials from the Agency’s Admissibility Branch, Innovation, Science and Technology Branch, Enforcement Branch and Operations Branch to gain their perspective on the pilot project in the context of their respective mandates. Discussions covered the use of intelligence information, partnerships, communication and best practices.

Contact was made with officials from DFAIT, EC and Statistics Canada. Officials from DFAIT and EC were contacted by telephone to discuss their regulatory responsibilities for exports and export reporting as well as their involvement with REVU. The CBSA and Statistics Canada are partners under the CAED program. A discussion with a Statistics Canada official clarified the nature of export statistics, as explained in the “Review of Documents and Statistics” section.

A total of 34 officials participated in the evaluation as summarized in Table 1.

Table 1: Participation of Officials and Stakeholders

Organization	Number of Officials
CBSA REVU	6
CBSA Prairie Region	5
CBSA HQ	9
CN	5
CPR	3
Other government departments (DFAIT, EC and Statistics Canada) or organizations (the CWB)	6
Total	34

2. Key Findings

2.1 Program Design and Delivery

How effective are the governance structure and the channels of communication for the management, coordination and delivery of the REVU pilot project?

The Letter of Intent with CN clearly defined the basis for cooperation between the two organizations at both the national and local levels, including the conditions for access to databases.

The March 2005 Letter of Intent specified that the agreement would run for a two-year period commencing when CN system enhancements had been completed and SOPs were in place for pre-departure reviews. The evaluation found ample documentary evidence that SOPs were developed and are being followed and updated as required by REVU officers. The procedures are available electronically and in hard copy for access by staff.

Although national in scope, because of its location in Winnipeg, the REVU pilot project receives its budget and is administered through the Prairie Region. The CBSA’s Operations Branch allocates program funding to be administered by the office of the RDG. Functionally, the REVU pilot project delivers program elements for both the Admissibility and Enforcement branches. Status reports to the RDG are provided for briefing purposes and as needed. There does not appear to be regular program reporting of this nature to the extent specified in the Letter of Intent. [ 10 ]

This pilot project does not involve any other formal agreements such as memoranda of understanding.

Overall, REVU has established open and clear channels of communication with its major partners and stakeholders.

Within the Prairie Region, oral or written status reports are available to the RDG at any time on an as-requested basis although there is no regular reporting from REVU. Similarly, there is little ongoing communication between the REVU pilot project and CBSA HQ. Communication from the Region about REVU activities is primarily through briefings provided by the RDG at departmental management meetings.

The evaluation found regular reporting between the CBSA and both CN and the CWB. There is a steady flow of information with reports on both daily and weekly bases. In-person meetings can be arranged with great ease, given the proximity of the offices of all three organizations.

The evaluation found an excellent working relationship between REVU, Winfall and ITAR officials and strong communication links between the three areas. In fact, the work of some CBSA officers transcends these groups. The ITAR regional intelligence officer attends information sessions twice monthly at the REVU office, and the REVU and ITAR superintendents meet once a month to discuss operational issues. As well, there is regular communication with the ITAR unit through lookout sheets (which provide tactical information to alert REVU officers to watch for specific shippers or key words) that are updated regularly.

Communications with exporters and shippers have improved as well under the REVU pilot project. The evaluation found evidence that both the CBSA and CN have assigned a high priority to communicate with exporters to assist them in becoming compliant with export reporting. This view was held by most of the officials participating in the evaluation.

How effective is REVU in collaborating and partnering with the ITAR unit, CBSA HQ, DFAIT, EC, CN and the CWB?

The REVU pilot project has forged a solid partnership with the ITAR unit, CN and the CWB.

One area that must be highlighted in this evaluation is the strong working relationships forged between the REVU team and its business partners. The evaluation found this to be a critical element underlying the effectiveness of the REVU pilot project.

Foremost among the various partnerships is the relationship between the REVU team and CN. Officials from both the CBSA and CN point to the excellent cooperation between the two organizations, particularly at the local level. Fundamental to this partnership is the strong element of trust that has been built through the REVU project. The REVU officers have real-time access to waybills entered into the CN computer system, which has enabled them to detect irregularities in export reporting and to inform exporters who can then correct the errors. Moreover, REVU officers, with the full cooperation of CN, monitor the system for certain “triggers” identified by CBSA enforcement officials and send any such information directly to the appropriate areas for further investigation and action.

The strength of this partnership pre-dates REVU. CBSA targeting officers working within the ITAR and Winfall units have had full access around the clock to the premises and computer systems of both CN and CPR since the late 1990s.

While the REVU pilot project was being formulated, the implementation team examined the regulatory requirements of all other departments affected by trans-border shipments. Among these departments were EC and DFAIT, as well as the CWB, a private sector organization.

The evaluation found that the CWB has benefited from the REVU pilot project. Since REVU began operations, monitoring of CWB licences has been more effective and accurate. CWB officials noted that their relationship with the CBSA has improved. They believed that REVU has enabled the CWB to achieve administrative savings.

Collaboration with EC and DFAIT was important during implementation but is currently less relevant.

In a similar review of CN waybills, the REVU implementation team brought specific concerns to the attention of officials at EC and DFAIT. In both cases, these departments indicated that rail transport comprised a very small percentage of the total volumes of exports under their purview. Once REVU became operational, however, irregularities in export reporting were detected. Information reported by the REVU team enabled both departments to take appropriate corrective actions, as described in the section on unexpected results. Currently, communication between the REVU team and these departments only takes place when required.

Collaboration between the REVU pilot project and CBSA HQ is limited.

The evaluation found little direct contact between REVU and CBSA HQ and no examples of collaborative working arrangements. Most HQ officials indicated little, if any, contact at the working level but were aware that the REVU pilot project was associated with positive results. More senior managers expressed greater familiarity with the work of REVU, primarily on the basis of briefings from the RDG of the Prairie Region. REVU officials indicated that they do not report directly to CBSA HQ.

Does the REVU pilot project make the best use of resources?

The REVU pilot project has been delivered on a modest budget.

The human and financial resources were initially reallocated from a customs postal unit that was being eliminated. Table 2 summarizes the projected resources from September 13, 2004, to March 31, 2007.

Table 2: REVU Pilot Project Resources

	2004–2005a	2005–2006	2006–2007
Full-time equivalents	5	7	7
Salaries and benefits	$178,500	$472,200	$481,800b
Operations and maintenancec	$17,440d	$45,100e	$27,500f

1. Reflects 29 weeks of operations commencing September 13, 2004.
2. Estimated on the basis of year-to-date expenditure data.
3. The Superintendent negotiated an agreement with Canada Post for the first two years at zero occupancy cost, ending September 30, 2006. Negotiations for an 18-month lease, commencing October 1, 2006, were in progress during the evaluation.
4. Includes renovations to create the REVU office.
5. Includes the cost of building a wall to separate REVU from the customs mail centre. 
6. Includes six months’ rent estimated at $20,000.

Internal and external stakeholders directly involved in the REVU pilot project were unanimous in their opinion that REVU has been a very effective program, delivered on a modest budget, that has produced better compliance results than conventional approaches. CN officials reported that the incremental cost to provide the CBSA with real-time access to their system was modest at about $40,000. [ 11 ]

Furthermore, CN officials reported a reduction in operational and compliance costs that they could attribute directly to the REVU pilot project. CN officials have observed a sharp reduction in train delays, which in turn has yielded cost savings to the company, estimated to be as much $2,000 per hour per car. This in turn has brought about additional efficiencies to the company because CN is now able to build longer trains and has reduced its locomotive fleet.

2.2 Performance Management

How effective are the performance measurement systems for the REVU pilot project?

The REVU pilot project documents details of all activities and processes but has not formalized a management strategy to obtain performance data.

It is customary to introduce new programs with a set of performance indicators. Management accountability frameworks are often developed to guide the implementation of new government initiatives. The design of the REVU pilot project, however, did not include a traditional management accountability system. The approach was more pragmatic in nature. It responded to a specific situation where the objectives were clear and the consultation process was comprehensive. The REVU process was fully documented and management formulated detailed SOPs for all steps. Since the inception of the pilot project, REVU officers have been tracking activity and compliance statistics.

REVU was being conceived at the same time that the CBSA was evolving and adapting to its new mandate to provide integrated border services in support of national security and public safety priorities. In the context of modern comptrollership, the Agency encourages its officers to show initiative as part of its culture of continual improvement. This commitment is set out under Strategic Priority 4 in the CBSA’s 2006–2007 Report on Plans and Priorities (RPP). At the corporate level, the Operations Branch continues to develop an Agency-wide strategy for performance measurement and a compliance management plan.

Thus, the planning for the REVU pilot project took place in an environment of change. Human resources had to be reallocated in response to the new mandate. Moreover, regulatory changes were anticipated for rail cargo export reporting. This provided an opportunity for the CBSA, in partnership with CN, to redesign the rail export process. The goal was to replace the cumbersome paper-based, point-of-exit reporting system with centralized monitoring. Both the CBSA and CN stood to benefit through the anticipated improvement in exporter compliance.

REVU does ensure that its procedures are always current and documented. Members of the REVU team meet weekly with the Superintendent to discuss operational issues that have arisen in the context of their work. Any required follow-up actions are delegated to specific officers and the decisions are stored on the common G drive on REVU’s computer system. Through this process, documentation on operating procedures is constantly updated and accessible. As well, written records are kept of all follow-up calls made to exporters whose documentation is not in compliance.

These procedures indicate a commitment to operational excellence and to pursuing ongoing improvements. To ensure accountability to management and for evaluation purposes, REVU could have benefited from a more comprehensive set of performance indicators covering key aspects of its operations.

2.3 Results Achievement (Success)

To what extent is the REVU pilot project achieving its expected results?

The REVU pilot project has expanded the available skills and knowledge to identify shipments containing ITAR- or other controlled, regulated or prohibited goods.

REVU provides important support to the ITAR unit. For the ITAR officers, the implementation of the pilot project offered an opportunity to have “more eyes on the ground” monitoring export activity. The ITAR group is a targeting and strategic export control unit that focuses on a daily sample review of rail cargo destined to a seaport for offshore export. The ITAR unit does not have time to monitor and do the primary screening of all exports. REVU’s universal monitoring of daily CN rail export traffic casts a wider net for the ITAR unit.

The ITAR unit provides REVU with its lookout list, which is updated weekly. To date, REVU has made no referrals to the ITAR unit based on “hard lookouts.” The ITAR unit also develops profiles through its intelligence process and provides a list of indicators to REVU to enable the identification of suspect shipments. For example, REVU officers would flag waybills containing discrepancies concerning the exporter, the consignee, the description of goods or the destination. Similarly, a waybill where the address of the exporter or consignee is not compatible with the nature of the goods would trigger interest. To date, these indicators have enabled REVU officers to identify and refer 70 export waybills of potential interest to the ITAR unit, which then undertook intelligence checks.

Under REVU, reporting is at the time of billing, which could be as much as five days in advance of train assembly. According to the existing Reporting of Exported Goods Regulations, cargo reporting can be done as late as two hours before train assembly. Thus, REVU has expanded the window for enforcement action. The results of the REVU pilot project provide a positive basis to make changes to the carrier reporting section of the Regulations.

The evaluation found the REVU team to be well trained, knowledgeable and enthusiastic in their work environment. Officers were able to explain clearly what the procedures were, demonstrate how they used REVU’s tools and processes, and why the various procedures were important and necessary to the work they were doing.

For CN shipments, the REVU pilot project has led to more consistent and uniform application of regulations under the D3 (transportation), D19 (other government departments) and D20 (export reporting) series of customs D memoranda.

REVU has created an SOP for each activity and process that has been established through the course of this pilot. As noted above, these SOPs are stored on the common G drive on REVU’s computer system and are updated regularly by the REVU customs inspectors.

When the pilot project commenced, REVU officers encountered reporting errors related to the D3, D19 and D20 series of customs D memoranda on a large number of waybills they examined. Under normal circumstances, this would have automatically resulted in fines under the Administrative Monetary Penalty System (AMPS). [ 12 ] As part of the pilot project, REVU officers follow up on issues identified during the pre-departure review with a call to the exporter and an export procedure letter. This client education process is intended to promote voluntary compliance with export reporting requirements for future shipments. Some of these issues involve potential AMPS situations. Because only CN is participating in the pilot project, no other carrier has its waybills scrutinized so closely. For this reason, the current policy in REVU is to delay the penalty for a CN client until the violation has occurred for the third time. The REVU Superintendent reviews all potential AMPS penalties before they are being issued. At the time of the evaluation, no AMPS penalties had been issued against CN exporters for failing to provide correct export report information.

There has been increased export reporting compliance under the Reporting of Exported Goods Regulations and DFAIT and EC regulations.

The evaluation found that export reporting compliance had increased significantly during the pilot project period. In July 2006, the rate of compliance reached 98 percent.

Another indicator of increased compliance is a decrease in the number of issue sheets prepared and the number of client education telephone calls made by REVU officers. These are indicative of a corresponding increase of export reporting on the CN waybills reviewed by REVU officers. Table 3 shows that there has been a downward trend in the number of calls made to exporters since REVU officers began documenting these calls on September 26, 2005, until April 2006. Since then, the number of calls has fluctuated, averaging 16 calls per month.

Table 3: Client Education Calls Documented by REVU Officers

Date	Number of Calls
September 21–30, 2005	21
October 2005	67
November 2005	58
December 2005	69
January 2006	37
February 2006	27
March 2006	25
April 2006	11
May 2006	18
June 2006	21
July 2006	13
August 2006	9
September 2006	11
October 2006	24
November 2006	19

Source: REVU, CBSA, December 8, 2006.

CBSA officials reported that REVU’s universal scrutiny of CN waybills has translated into higher confidence levels in CN trains by border control officers at the various points of exit.

Evaluators attempted to utilize data on AMPS to demonstrate improvements in compliance by comparing AMPS penalties issued against CN and CPR exporters before and after the implementation of REVU. Under the REVU pilot project procedures, exporters are given up to two calls to bring them into compliance before an AMPS penalty is applied. A review of the available information revealed that AMPS data related only to penalties lodged against the rail company, not the shipper, and that the majority of these applied to inbound rather than outbound infractions. Evaluators were told that it would be a “massive undertaking” to identify exporters and the reason they were given a penalty or to link such data to a specific rail carrier.

The evaluation found anecdotal evidence on increased compliance with the Reporting of Exported Goods Regulations of other departments or stakeholders. CWB officials report that they now receive 100 percent of their licences back for grain products shipped with CN. Exporters have been known to take advantage of slow reporting. In some cases, the CWB receives information up to six months after a permit has expired. If the exporter has exceeded the permit, it is too late.

REVU identified and communicated information to DFAIT that exporters of logs were doing so without filing B13-A export permits. [ 13 ] Follow-up action by DFAIT involved contact with those exporters and client education to ensure that the permits were accurately completed and filed. A DFAIT official reported that REVU has been of great assistance in helping to achieve better compliance in that area. The evaluation did not encounter evidence about changes to compliance rates for precursor chemicals or food products, which also fall under this department’s responsibility.

In the same way, REVU, in its early stage, encountered irregularities in export documentation for a particular category of waste products. This was conveyed to the regulatory officials at EC, and the information was passed to that department’s enforcement area. As a result, several investigations were undertaken across Canada, and this eventually led to increased compliance with respect to the specific waste stream identified by REVU.

REVU has achieved observable results in a short period of time.

It was anticipated that the REVU pilot project would improve the accuracy of export statistics and reporting. To this end, the CWB has already observed significant improvements in the timeliness and accuracy of information on grain export permits where CN is the carrier. On a larger scale, it is too early at this time to obtain sufficient data to support a comparative analysis of export reports pre- and post-REVU. Nevertheless, the observed increase in regulatory compliance with export reporting could be considered a reasonable indicator of increased statistical accuracy. [ 14 ]

The general consensus among CBSA officials is that the Agency’s ability to detect and control non-compliant exports has already improved as a result of the introduction of REVU. The positive compliance results observed to date have been achieved with only a small investment in resources. REVU data alone will not be sufficient to assess the degree of impact REVU has had on the Agency’s overall ability to detect and control non-compliant exports. Additional analysis outside the scope of this study on the outbound movement of other rail carriers would be required.

Were there any unexpected results and what were their impacts on the success of the REVU pilot project?

An almost immediate improvement in compliance was not expected nor was a “ripple” effect to other exporters.

Stakeholders were asked to identify any unanticipated outcomes that they would associate with the REVU pilot project. A number of stakeholders commented that they were surprised by the cooperative response of shippers when they learned that they were not compliant with the new regulations. Client education has been a cornerstone of REVU for both the CBSA and CN. Even so, achieving a compliance rate of 98 percent was not expected in either organization.

Neither DFAIT nor the REVU project implementation team anticipated the extent of non-compliance with respect to permits required for trans-border shipments of logs by rail. REVU officers did not see log export permits on the system and contacted DFAIT to verify that logs were indeed being exported. Again, communication between the two organizations and client education resulted in a flurry of applications for the required permits, and REVU officers subsequently observed these numbers incorporated into the waybills.

Early communications between the REVU implementation team and EC concerning export documentation for certain hazardous waste streams triggered several investigations by EC enforcement officers. Officials did not anticipate the “ripple effect” on compliance that these investigations had on companies not implicated and credited the intelligence from REVU officials for the increase in compliance.

Beyond the regulatory outcomes, the evaluation found unexpected implications at the policy level. Both internal and external stakeholders suggested that the interest sparked by this pilot project has paved the way for broader discussions on rail and intermodal policy issues. In addition, the Exports Process program is using the CN pilot process for carrier reporting before departure as a basis for a proposed change to carrier reporting requirements in the Reporting of Exported Goods Regulations.

2.4 Relevance

To what extent is the REVU pilot project consistent with CBSA and government-wide priorities and does it address an actual need?

Although the REVU pilot project’s expected results are aligned with both CBSA and government-wide priorities, the evaluation is questioning the pilot project’s continued relevance. 

Strategic Priority 1, stated in the CBSA’s most recent RPP, is the effective delivery of programs and services to strengthen the CBSA’s ability to identify and interdict goods of high or unknown risk while expediting the flow of low-risk trade. REVU’s compliance monitoring has added greater certainty to the ITAR unit’s ongoing targeting activities. REVU and its partners are using real-time data to monitor CN exports and to verify the accuracy of export reporting. In so doing, REVU has the ability to provide advance notification of irregularities. In that sense, REVU and the ITAR unit have begun to achieve on the export side what is being contemplated under Strategic Priority 2 for imports (“Innovating for the border of the future”). Strategic Priority 3 is to ensure strong internal and external relationships. The CBSA relies on partnerships to achieve its mandate and deliver programs and services. The strength of the CBSA–CN partnership through REVU, the ITAR unit and the Winfall project underscores the relevance of REVU in this context. Furthermore, REVU’s dealings with the CWB, DFAIT and EC have contributed to stronger alliances with these partners.

Given the modest budget involved and the positive results observed in the short term, it is logical to assume that such an initiative makes sense and ought to be made permanent. Several factors must be considered, however, before drawing such a conclusion. Is REVU still needed now that compliance has risen so dramatically? How appropriate is it for the Agency to invest in helping a small number of companies to achieve compliance? Does REVU serve the public interest and is there a legitimate and necessary role for government in this project activity? 

Shippers using CN have been made aware of their reporting obligations and have been educated to the extent that reporting errors have been reduced sharply. There have been additional benefits beyond more accurate reporting by shippers using CN. As noted throughout this evaluation report, spillover benefits have accrued through better reporting from shippers using other rail carriers and in other regulatory departments. In the first case, many Canadian companies use both CN and CPR for rail exporting, and lessons learned through REVU for shipments carried by CN have likely helped to improve their reporting for CPR-carried shipments. As well, officials from both EC and DFAIT observed better reporting compliance from companies other than those targeted by their investigative staff. Information travels quickly within industries. These outcomes arose as a result of REVU.

It can be argued, however, that moving the pilot project into a permanent program as it is currently delivered is not in the public’s interest. Other companies that do not receive REVU's service and attention may be subject to penalties for not complying with export regulations.

Expanding the pilot project by bringing other railway companies into REVU will be challenging. CPR alone represents almost half of the rail export traffic from Canada, but CPR officials remain skeptical of the benefits of participation and are satisfied with the system that existed before REVU. CPR officials do not believe that this type of compliance monitoring has sufficient demonstrable business benefits to warrant any major investment in changes to their computer system. They claim that the modular nature of CPR’s billing system provides greater flexibility to correct errors detected on waybills. CPR officials also firmly believe that their method of train assembly is quite different from that of CN. And yet, other parties see ongoing difficulties with reporting for CPR shippers as compared with CN shippers (post-implementation of REVU). The CWB, for example, reports a large difference between CN and CPR information on permit depletion and the timeliness of reporting.

Increased voluntary participation in REVU is thus unlikely. It would likely be necessary to mandate participation through legislative or regulatory change. Such a process is time-consuming and consultation with affected stakeholders will be necessary. In addition, a decision on expansion also needs to include future consideration of the implications of the Advance Commercial Information and eManifest programs, linked in part to international obligations, but which will focus on imports in the foreseeable term.

The uniqueness of the railway industry sector, and especially CN and the systems it built, made REVU possible. REVU’s success has shown that this is a solid model for enhancing rates of compliance. In fact, this project’s achievements may represent an opportunity for the Agency to monitor and measure compliance in other industry sectors and modes.

3. Summary of Key Findings and Recommendation

The REVU pilot project is being delivered as it was conceived. The Letter of Intent with CN clearly defined the basis for cooperation between the two organizations at both the national and local levels, including the conditions for access to databases. Overall, REVU has established open and clear channels of communication and forged a solid partnership with the ITAR unit, CN and the CWB. At the same time, however, the evaluation found limited communication and collaboration with CBSA HQ branches. Although the REVU pilot project documents details of all activities and processes, no formalized management strategy was developed to obtain performance data.

The evaluation found that the REVU pilot project has achieved significant positive and observable results. For CN shipments, the REVU pilot project has led to more consistent and uniform application of and increased compliance with regulations under the D3 (transportation), D19 (other government departments) and D20 (export reporting) series of customs D memoranda. As well, increased compliance has been observed with DFAIT and EC regulations. The results for shippers using CN speak for themselves: export reporting compliance reached 98 percent. The REVU pilot project has expanded the CBSA’s available skills and knowledge to identify shipments containing ITAR- or other controlled, regulated or prohibited goods. The evaluation also found anecdotal evidence that spillover compliance improvements have been observed in other areas as a direct result of shippers’ and companies’ knowledge of REVU’s increased scrutiny of outbound shipments via CN.

The pilot project has been delivered effectively on a modest annual budget of approximately $0.5 million. Notwithstanding the fact that this pilot has been successful, there are a number of factors that support a consideration of the discontinuation of the project.

The compliance rate achieved for CN since REVU’s inception has been successful to the point that there is no longer a need for the project. The fact that only one organization — CN and its clients (including the CWB) — is participating in this program may raise questions as to why government resources are spent helping only one company. Other companies that are not benefiting from this service may be subject to penalties for not complying with export regulations. Unless other railway companies are participating (only limited interest for participation exists at this time), the benefits to the CBSA by continuing REVU are minimal. 

On the basis of these findings, the following recommendation was made:

4. Management Response

REVU began operating in May 2005 and, as detailed in this evaluation study, it has been responsible for a significant increase in export reporting compliance with CN and its clients during the pilot project period.

REVU has provided insight into carrier reporting before loading on the export side. It has been very successful and has generated useful information for the CBSA’s Commercial Vision project, which explores, among several things, the overall role of the CBSA in export reporting and verification.

Agency management will consider the continued purpose of REVU, whether its continued operation is the best use of Agency resources and its potential application to other initiatives. The decision as to whether REVU should continue, and if so in what form, will be taken by September 30, 2007.