January 2012
This document is also available in PDF (183 KB) [help with PDF files]
The Process Monitoring Framework (the Framework) was created by the Operations Branch in 2007 to replace the former Operational Compliance Check System.
The objectives of the Framework are to ensure that:
The Framework is a list of procedures, or a checklist, for reviewing programs in a structured manner. If adequately conducted, the reviews enable managers to detect and correct deficiencies. The review results are reported upwards to the regional level, who in turn report upwards to Headquarters. The review procedures extensively cover program areas delivered by the Agency offices.
The Framework includes three levels of review:
The Process Monitoring Framework is a key monitoring tool of the Agency and is relied on to detect and correct program delivery deficiencies.
This audit is of interest to management because the Process Monitoring Framework is identified as a key control to address many of the recommendations stemming from findings identified in previous audits.
Weaknesses in the Process Monitoring Framework could result in a failure to detect and correct issues, such as improper or inadequate program delivery, including the processing of travellers and commercial shipments and the safeguard of assets and monies.
The Canada Border Services Agency (the Agency) Audit Committee approved an audit of the Process Monitoring Framework as part of the Three-year Risk-Based Audit Plan 2010/11 -2012/13.
The objective of this audit was to provide assurance on the extent to which the Process Monitoring Framework was working as intended and achieving objectives. More specifically it focused on the extent to which:
The audit also followed up on issues reflected in previous audit reports that related to the Process Monitoring Framework to verify whether these issues were resolved.
The audit scope included all Framework procedures and modules, from inception to 2011.
The audit approach and methodology followed the International Standards for the Professional Practice of Internal Auditing as defined by the Institute of Internal Auditors and the Internal Auditing Standards for the Government of Canada as required under the Treasury Board's Policy on Internal Audit.
This audit provides a high level of assurance that the opinion provided therein is appropriate and reflects conditions that existed at the time of the audit.
The audit concludes that the Process Monitoring Framework was not fully functioning as intended. Deficiencies were noted in the execution of Tiers One and Two of the Framework, and the results of these two tiers were incomplete and unreliable. While Tier Three was providing some useful information, it was in the pilot phase, so its full benefit is not yet known.
Overall, there were gaps in the review procedures and in the reporting of results. The process to address reported issues was inadequate.
This translates to a moderate[ 1 ] risk exposure to the Agency.
The Process Monitoring Framework's procedures for Tiers One, Two and Three were in place, however there were gaps. This audit found that approximately half of the Agency offices visited did not have files to substantiate that Tier One reviews had been carried out. Tier One procedures were extensive and many of the procedures were considered low risk by managers.
The audit noted that deficiencies were not detected by Tiers One and Two. The Tier Three was piloted in the summer of 2011, two years after its expected implementation. The Tier Three reviewers concluded that five of seven offices visited had not effectively conducted their Tier One reviews.
Managers stated that issues reported were not being addressed. The audit found that the Headquarters Process Monitoring Framework Team was not communicating issues to the process owners in a manner that produced answers.
Issues reflected in previous audit reports were found to still be problematic. Six areas were reassessed, and in all six areas the expected changes did not occur.
This report makes four recommendations:
The Operations Branch agrees with the recommendations provided by this audit, and has prepared a management action plan.
The action plan includes streamlining operational monitoring and replacing the Process Monitoring Framework with a risk-based review structure. The review structure that will replace the Process Monitoring Framework will include a verification process in order to ensure information is complete, reliable, and timely, and that reviews are being performed.
An issues management log will be developed for tracking reported issues.
The six deficiencies specifically identified in this audit will be added to the review structure and the Regions will be reminded that these need to be addressed. The Operations Branch will subsequently verify whether improvements occurred.
It is important to note that the Operations Branch Executive Committee meeting made a commitment in September 2011 to replace the Process Monitoring Framework with a new review structure. Consequently, work has already begun on the development of the new process, including initial consultations with process owners.
In fact, in an effort to streamline the Process Monitoring Framework, a Risk Rating Review was initiated in November 2010 that involved consultation with the program/process owners and regional contacts to rate the risk level of Process Monitoring Framework activities, identify a source document for all monitoring activities, and remove activities that were no longer relevant or were being monitored elsewhere. This work will contribute to and facilitate the development of the Process Monitoring Framework replacement in a timely manner.
The Process Monitoring Framework (the Framework) was created by the Operations Branch in 2007 to replace the former Operational Compliance Check System.
The objectives of the Framework are to ensure that:
The Framework is a long list of procedures, or a checklist, for reviewing programs in a structured manner. If adequately conducted, the reviews enable managers to detect and correct deficiencies. The review results are reported upwards to the regional level, who in turn report upwards to Headquarters.
The Tier One Framework (2010-2011) consisted of 16 modules covering 120 program areas, with 697 review procedures. The Tier Two review (2010-2011) covered seven program areas, with 69 review procedures. The Tier Three review (July 2011) covered four program areas, with 33 review procedures.
The Framework includes three levels of review:
Tier One (Local) – A local office manager (supervisor, superintendent, chief or other manager) carries out the Tier One reviews at their local office, on the programs delivered by the employees reporting to them. Corrective action is taken for exceptions[ 2 ] noted. Large offices conduct reviews and report twice per year, and small or seasonal offices once per year. The offices report upwards all issues including those that are locally resolvable, i.e. issues that are not in compliance with Agency policies and procedures. Tier One was implemented in 2007.
Tier Two (Regional) – The regional Planning and Program Integration Divisions' program services officers perform this level of review annually. This review is conducted at the local offices and they focus on the exceptions that local managers have reported as part of the Tier One exercise. The Regions report upwards to Headquarters all issues including those that were resolved at the regional level. Tier Two was implemented in the fall of 2009.
Tier Three (Headquarters) – This tier is an independent assessment conducted on the local offices by the Headquarters Process Monitoring Framework Team (the Headquarters Team). This level focuses on problematic areas identified as a result of audits or other reviews. The purpose of Tier Three is to determine whether the information flowing upwards is complete and accurately describes the issues (i.e., exceptions) that need to be corrected. Tier Three was piloted in the summer of 2011.
In addition to Tier Three, the Headquarters Team, comprised of four resources, has a number of roles. These include updating the Framework procedures, monitoring the regional and local application of procedures and providing support to the Regions of the Operations Branch.
Various stakeholders throughout the Programs, Comptrollership, Human Resources, and Information, Science and Technology branches are involved in developing the Framework modules relating to activities within their respective program areas.
If management does not adequately conduct the Process Monitoring Framework, or does not fully implement the Framework as intended, there will be missed opportunities to:
The Canada Border Services Agency (the Agency) Audit Committee approved an audit of the Process Monitoring Framework as part of the Three-year Risk-Based Audit Plan 2010/11 -2012/13.
The purpose of the audit was to provide assurance on the extent to which the Framework was working as intended and achieving objectives.
More specifically the audit provides assurance on the extent to which:
The audit scope included all Framework procedures and modules, from the Framework's inception in 2007 to the present. The audit team visited the National Risk Assessment Centre and four Regions: the Greater Toronto Area, Southern Ontario, Prairie, and Northern Ontario Regions. The audit team surveyed non-visited Regions by questionnaire to obtain their comments.
The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, as required by the Treasury Board Internal Audit Policy.
The examination phase of this audit was performed using the following approach:
Appendix A lists the audit's lines of enquiry and criteria.
The audit approach and methodology followed the International Standards for the Professional Practice of Internal Auditing as defined by the Institute of Internal Auditors and the Internal Auditing Standards for the Government of Canada as required under the Treasury Board's Policy on Internal Audit.
This audit provides a high level of assurance that the opinion provided therein is appropriate and reflects conditions that existed at the time of the audit.
The audit concludes that the Process Monitoring Framework was not fully functioning as intended. Deficiencies were noted in the execution of Tiers One and Two of the Framework, and the results of these two tiers were incomplete and unreliable. While Tier Three was providing some useful information, it was in the pilot phase, so its full benefit is not yet known.
Overall, there were gaps in the review procedures and in the reporting of results. The process to address reported issues was inadequate.
This translates to a moderate[ 3 ] risk exposure to the Agency.
Audit criteria:
Ports of entry, Regions and Headquarters are discharging their Process Monitoring Framework responsibilities as designed (Tiers One, Two and Three).
Management uses information technology in an efficient way to collect Framework information.
These criteria were partly met.
Tier One reviews were done, however were not adequately executed by many offices.
Tier Two reviews were in place, however deficiencies were missed and thus this tier was not functioning as intended.
Tier Three was still being piloted at the time of the audit. However preliminary results indicate that this tier should function as intended when fully implemented.
The Operations Branch created the Process Monitoring Framework (the Framework) in 2007. The objective of the Framework was to ensure that programs were monitored, functioned as intended, and that departures from policy and procedures were detected and corrected.
The audit team visited four Regions and the National Risk Assessment Centre (44 Agency offices) to carry out interviews with managers who were conducting the Tier One reviews. The audit team also reviewed files from these offices, and surveyed non-visited Regions.
During the interviews, managers stated that they understood the objectives of the Framework and their role. The majority commented that the Framework was a useful monitoring tool. Some interviewees commented that the Framework procedures were particularly useful for guiding new managers in their monitoring responsibilities.
For the most part, managers assigned to conduct Tier One reviews stated that they, or other managers in their office, were carrying out the Framework review procedures, or a portion thereof, and were reporting upwards to the Region. Less than half of interviewees (21 of 44) claimed they were carrying out 100% of the Framework procedures. Managers were not specifically tracking their time spent on the Framework reviews.
To verify whether Framework procedures were carried out, the audit team requested to see their files relating to the Framework reviews. Fifty-three percent (53%) had adequate documentation to support the review. The other forty-seven percent (47%) had no substantive files to provide as evidence. This finding is in line with the Tier Three review conducted in the summer of 2011, where unreported deficiencies were observed at all offices, and where five of seven offices were assessed by the Tier Three reviewers as not having effectively carried out the Framework reviews.
Managers stated that the Framework tool needed streamlining (41 of 44). Most managers conducting Tier One reviews stated that the Framework review procedures were too time-consuming. This was particularly acute for ports of entry.
For example:
Overall, the breadth and lack of clarity of the Process Monitoring Framework review procedures were factors compromising the conduct and completion of the Tier One reviews.
The Tier Two reviews were first implemented in 2009 and two cycles have been carried out. The audit team conducted five interviews with regional program officers who had conducted Tier Two reviews and reviewed their supporting documentation. The audit team also obtained the Tier Two results reported to Headquarters and examined the exception reports.
There were gaps in the Tier Two reviews. The audit noted that this process was not working as intended, as deficiencies noted during this audit were not detected by the Tier Two reviewers. For example, warehouse checks and risk assessments were not provided to Headquarters. Yet, this issue was not identified during the Tier Two review.
In conclusion, while the Tier Two reviews were being completed, they were not producing the intended results as designed.
The Tier Three review was piloted in the summer of 2011. This review focused on previously identified problematic areas such as bond rooms, seizure documentation, lookouts and examination results. The audit team visited seven Agency offices in the Regions during the Tier Three reviews.
The Tier Three review noted multiple deficiencies that were not reported in the Tier One and Two reviews. These deficiencies included mixed usage bond rooms, inadequate bond room inventory control, overdue seizure review work lists, lookouts not properly acquitted, and missing documentation pertaining to the disposition of seized monies. The Tier Three reviewers determined that five of seven Agency offices reviewed were not effectively conducting the Tier One reviews.
In conclusion, the Tier Three pilot review functioned as intended. However, it was implemented two years later than originally planned.
Most of the Framework review procedures have to be performed at a local level. It is important that the ports of entry and other regional offices periodically review their operations, as they are accountable for their performance.
However there are a few areas in which checks could be executed from Headquarters through accessing national electronic systems, such as the seizure system or the commercial import systems. Parallel spot checks conducted by Headquarters, independent of and in addition to the Process Monitoring Framework, would further provide assurance on the quality of the Framework information provided.
Recommendation 1:
The Vice-President of the Operations Branch should review the design of the Process Monitoring Framework to streamline the monitoring process from a risk, timing and technology perspective.
Management Action Plan | Completion Date |
---|---|
The process being developed to replace the Process Monitoring Framework will assess and measure operational adherence of Agency priorities and high risk areas of business identified in internal and external audits, at ports of entry. |
April 2012 - |
The proposed process will streamline the Operations Branch's review structure, by aligning the criteria/review strategy with high risk areas of business and audit priorities that affect the port of entry operations nationally. It will also include procedures to ensure that identified corrective measures are in fact implemented. Program/process owners will be consulted in the development of this new structure. |
August 2012 - Ready for full implementation. |
Audit criterion: The information generated is complete, reliable and timely. Overall, this criterion was not met. Information generated from the Tier One and Two reviews was incomplete and, in some cases unreliable. Reporting of results from Headquarters for management was not timely. A Tier Three review showed more promise, but it has not yet completed a full cycle. |
The Process Monitoring Framework is a key monitoring tool of the Agency. It is important that the information generated be complete, reliable and timely. It is also important that there be a process in place to provide assurance on the quality of the information.
The audit team found that managers and regional program officers expressed doubt about the completeness and reliability of the Framework information generated by Tiers One and Two reviews. Less than half of interviewees were completely confident in the completeness and reliability of the Framework results. This coupled with the audit observation that Tiers One and Two review procedures were not fully and/or adequately conducted in all offices or by all reviewers, led the audit team to conclude that the information generated by the Tiers One and Two reviews were not complete or reliable, and deficiencies was not detected and reported upwards.
The Tier Three reviews were piloted in the summer of 2011, thus Agency offices had not been challenged for the previous four years on the depth and completeness of their Tiers One and Two reviews. The Tier Three review did produce some relevant information.
The Regions were reporting upwards every six months as required. However the Tier Three reviews took four years to implement, two years after the original targeted date.
As well, the Headquarters production of the Tier One Summary Exception Reports for management was a year to a year and a half after the receipt of the information, and several were never produced and/or issued. These reports are discussed further in section 3.3.
Recommendation 2:
The Vice-President of the Operations Branch should implement a process to ensure the information generated by the Process Monitoring Framework is complete, reliable and timely.
Management Action Plan | Completion Date |
---|---|
The review structure that will replace the Process Monitoring Framework will include a verification process to ensure the information generated is complete, reliable and timely. | April 2012 - |
Verification reviews will be performed to ascertain whether the Regions are effectively conducting port of entry evaluations and adequately reporting and resolving issues. The reviews will also verify that functions related to priority audit recommendations and/or high risk areas of business are being performed. The reporting process will also be restructured in a manner that makes it easier for the Regions to identify and report issues using an automated checklist tool (Excel), which will assist in capturing more complete and reliable information. |
August 2012 - Ready for full implementation. |
Audit criterion: Management uses the Process Monitoring Framework information to improve performance.
This criterion was partly met.
Local Agency offices indicated that conducting the Framework procedures was useful in detecting deficiencies within their operations. However, Headquarters process owners were not using the information generated.
The information generated by the Process Monitoring Framework should be helpful in improving operations, both at a local, regional and Headquarters level. Information and issues rolled upwards within the Process Monitoring Framework reporting process should be communicated vertically and horizontally at a Headquarters level, and ideally would be useful in process improvement and decision making.
The audit team surveyed 95 Headquarters process owners. Only a couple of responders indicated that the information generated by the Process Monitoring Framework was used in making process improvements or in decision making.
Most locally based Agency managers (80%) stated that conducting the Process Monitoring Framework reviews had been helpful in detecting and correcting deficiencies at a local level, but was less useful in decision making.
When the Regions had issues requiring a national solution or guidance, they rolled these upwards within the Framework reporting process to the Headquarters Process Monitoring Framework Team. For example, the Regions had asked for assistance with signage and for the expiry of access of former employees to a system controlled by Headquarters. The Regions expected these issues would be rolled over to the Headquarters process owner in the Programs Branch for action. However, managers in the Regions commented that issues were not being addressed, communication back from Headquarters was lacking and that the solution or guidance requested from Headquarters was not received.
The audit team determined that the Regions did not keep a centralized inventory log of the issues reported to Headquarters that required a national solution or guidance. Consequently, issues tracking and follow-up was not as centralized or orderly as could be. Similarly, the Headquarters Process Monitoring Framework Team did not keep an inventory log of the issues reported to them.
Furthermore, the Headquarters Team was not systematically identifying these issues (those requiring a national solution or guidance) from the regional Tier One reports or forwarding these to the Headquarters process owners for action. Nor was it clear to the Headquarters Team that they were expected by the Regions to have the issues resolved.
The Headquarters Team considered the Summary Exception Reports as the primary means to communicate with the process owners. A survey of the Headquarters process owners revealed that the Summary Exception Reports were not used and most could not recall ever having received them. These reports included all reported results, including minor issues, and filtering through the reports would be onerous for the process owners. This approach was not an effective issues management strategy. It was unclear if the production of these reports is good value for the effort invested.
Recommendation 3:
The Vice-President of the Operations Branch should ensure that issues stemming from the Framework reviews are logged and communicated to the appropriate branch. An issues log should be kept.
Management Action Plan | Completion Date |
---|---|
The Headquarters Process Monitoring Framework Team will review recent Tier One and Tier Two Framework reports submitted by the Regions to identify issues requiring a national solution or guidance and send them to the process owner, i.e. Office of Primary Interest, for action. |
January 2012 |
In the review structure that will replace the Process Monitoring Framework, the Regions will be responsible and accountable for the management and resolution of issues at ports of entry and for communicating issues to the Headquarters Process Monitoring Framework Team. The Headquarters Process Monitoring Framework Team will be responsible and accountable for the management of issues communicated from the Regions and for follow-up to ensure corrective action is being taken. |
April 2012 - |
A log will be developed to adequately manage issues and communicate responses and results to the Regions. The log will include who at Headquarters and in the Region were notified regarding issues and resolutions. This log will be available to stakeholders. | August 2012 - Ready for full implementation. |
Audit criterion: Issues that have been reflected in previous audit reports (Internal Audit and Program Evaluation Directorate and the Office of the Auditor General) that relate to the Process Monitoring Framework have been resolved.
This criterion was not met.
A close examination of six recommendations and management action plans revealed that the expected changes did not occur, and that issues noted during audits were still problematic.
Management action plans in response to audit recommendations should result in the correction of deficiencies. Reassessment of these same issues should show improvement over previous performance. Fifteen management action plans that relied on the Process Monitoring Framework as the principle change mechanism were identified.
To determine whether commitments were kept to update the Framework, the audit team examined the latest version of the Framework procedures. Thirteen (of 15) commitments to update the Framework procedures had been kept.
Of the 13, we examined six management action plans closely and reassessed the area as compared to previous audit results. The audit team found that expected improvements had not occurred in all six areas assessed.
For example:
The audit team noted that management action plans did not clearly lay out expected performance, and relied heavily on passive controls, such as the issuance of guidance through memorandums or through updates to the Process Monitoring Framework. The management action plans were not designed as fully formulated control frameworks that would include a combination of various controls.
Recommendation 4:
The Vice-President of the Operations Branch should take immediate steps to correct the deficiencies identified in the previous audit recommendations and confirm the deficiencies are addressed.
Management Action Plan | Completion Date |
---|---|
The Operations Branch will send a reminder letter to the Regions asking them to ensure the six deficiencies identified in this recommendation are addressed. This will also be discussed in a Planning and Program Integration Director conference call. The Operations Branch will subsequently verify whether improvements occurred. | January 2012 |
The six deficiencies identified in this recommendation will be added for further examination to the review structure that will replace the Process Monitoring Framework. | August 2012 - Ready for full implementation. |
Given the preliminary findings from the planning phase, the following criteria and sub-criteria were chosen:
Lines of Enquiry | Audit Criteria |
---|---|
Process | Ports of entry, Regions and Headquarters are discharging their Process Monitoring Framework responsibilities as designed (Tiers One, Two and Three). Management uses information technology in an efficient way to collect Process Monitoring Framework information. |
Quality of information | The information generated is complete, reliable and timely. |
Use of information | Management uses the Process Monitoring Framework information to improve performance. |
Follow-up | Issues that have been reflected in previous audit reports (Internal Audit and Program Evaluation Directorate and the Office of the Auditor General) that relate to the Process Monitoring Framework have been resolved. |
Agency office type | Modules | Program Areas | Review Procedures | |
---|---|---|---|---|
All | 16 | 120 | 697 | |
Ports of entry | Land Border | 7 | 68 | 482 |
Air | 6 | 61 | 448 | |
Marine | 6 | 69 | 490 | |
Rail | 6 | 59 | 424 | |
Postal | 6 | 32 | 215 | |
Trade Services Divisions | 6 | 13 | 72 | |
Intelligence Divisions | 5 | 26 | 145 | |
Inland Enforcement Divisions | 5 | 33 | 219 | |
Criminal Investigations Divisions | 6 | 23 | 153 | |
National Risk Assessment Centre | 4 | 17 | 88 |
Drawn from version 1.7 of the Process Monitoring Framework
Cycle | Modules | Program Areas | Review Procedures |
---|---|---|---|
Cycle One (2009-2010) | 2 | 10 | 41 |
Cycle Two (2010-2011) | 1 | 7 | 69 |
Cycle One contained a module that could be used by the Regions to insert procedure(s) that they thought should be reviewed, but that were not contained in Cycle One.
Offices Visited | Modules | Program Areas | Review Procedures |
---|---|---|---|
7 (July 2011) | 1 | 4 | 33 |
Key stakeholders | Principle involvement |
---|---|
Operations Branch – Headquarters Process Monitoring Framework Team |
|
Operations Branch – Regional program officers |
|
Operations Branch – Managers in Agency offices |
|
Various branches - Headquarters program areas |
|
Abbreviation | Definition |
---|---|
Agency | Canada Border Services Agency |
Regions | The Regions of the Operations Branch of the Canada Border Services Agency |
Framework | Process Monitoring Framework |
Headquarters Team | The Headquarters Process Monitoring Framework Team |