Audit of Training and Learning: Framework to Support Integrity of Financial and Non-Financial Information

March 2012

This document is also available in PDF (183 KB) [help with PDF files]

Table of Contents

• Executive Summary
• 1.0 Introduction
  • 1.1 Background
  • 1.2 Risk Assessment
  • 1.3 Audit Objective and Scope
  • 1.4 Approach and Methodology
  • 1.5 Audit Criteria
  • 1.6 Statement of Assurance
• 2.0 Audit Opinion
• 3.0 Findings, Recommendations and Action Plans
  • 3.1 CBSA Training and Learning Information Needs
  • 3.2 Policies and Procedures for Collecting Training and Learning Information and Compliance
  • 3.3 Assessment of the System's Capability and Capability and Capacity to Collect Financial and Non-Financial Training and Learning Information
• Appendix A – Audit Criteria
• Appendix B – Acronyms

---

Executive Summary

Background

The Canada Border Services Agency (CBSA) prepares Border Services Officers (BSOs) to deliver front-line services through rigorous training programs. Other internal training requirements relate to specific programs such as Enforcement Training, Recourse, Trade Compliance, and non-operational training such as Leadership and Management Training. Given the specialized nature of the training required to fulfil the Agency's mandate, the CBSA delivers most operational training internally. Non-operational training is provided both internally and externally to all CBSA employees.

The CBSA makes significant investments in training and learning. The total training and learning costs incurred by the Agency in fiscal year 2008-2009 were estimated at $91 million[ 1 ] (unaudited). These costs represented 6.3% of the total CBSA expenses of $1.4 billion (unaudited) for the year.

The CBSA's Training and Learning Directorate (TLD), which is part of the Human Resources Branch, is responsible for providing training and development and other activities to the CBSA's workforce. It provides functional authority and leadership on training and learning, develops and monitors the implementation of policies, leads the planning process, and provides guidance and support to all areas related to training and learning.

The CBSA Audit Committee approved an audit of the Training and Learning Program as part of the Three-Year Risk-Based Audit Plan for Fiscal Years 2010/2011-2012/2013.

Significance of This Audit

The CBSA has very specific training and learning needs. Documentation of training is required to demonstrate both that it has provided its employees with the necessary or mandated training set out in laws and regulations, and that the Agency has responded to the training needs that it or the employee has identified. In addition to tracking training histories, the Agency should track and account for its training expenditures to determine whether these funds are managed efficiently and transparently, and whether resources align with organizational priorities.

This audit should be of interest to management because it assesses the extent to which the Agency has structures and processes for ensuring that financial and non-financial information on training and learning is available for managing, decision making and reporting. Weaknesses related to training and learning information will make it difficult for CBSA to develop and manage its training and learning activities, particularly in light of the new functional model, and to ensure that these activities adequately support the Agency's efforts in fulfilling its mandate and meeting its priorities.

Audit Objectives and Scope

The audit objective was to determine the extent to which the Agency had implemented structures and processes in order to ensure the integrity[ 2 ] of financial[ 3 ] and non-financial[ 4 ] training and learning information. The audit focussed on the extent to which training and learning information needs link with overall business requirements; the assessment of the Agency's information systems; and the development and communication of policies, procedures and guidelines related to training and learning information. The audit did not examine financial processes and controls, as this was not a financial audit. Nor did the audit examine the design or controls of the Corporate Administrative System (CAS), as various changes were taking place at the time of the audit. The period under audit included April 2010 through July 2011.

The audit did not specifically examine the Arming Program, and the findings in this report do not pertain to it. The processes and procedures for managing arming-related training and learning information are distinct and unique to the Program. It is managed separately by the Arming Division and was recently moved under the TLD.

Statement of Assurance

The audit approach and methodology followed the International Standards for the Professional Practice of Internal Auditing as defined by the Institute of Internal Auditors and the Internal Auditing Standards for the Government of Canada as required under the Treasury Board's Policy on Internal Audit.

This audit provides a high level of assurance that the opinion provided therein is appropriate and reflects conditions that existed at the time of the audit.

Audit Opinion

For the period under audit, the Agency's structures and processes were not adequate for ensuring that relevant, complete, accurate and timely financial and non-financial information on training and learning was available for managing, decision making and reporting purposes.

This translates to a moderate risk[ 5 ] exposure to the Agency as there are compensating controls such as user-based computer applications, including spreadsheets and databases, which enable the Agency to meet its current reporting requirements. However, these are not always efficient or effective.

Key Findings

The audit found that the Training and Learning Directorate (TLD) had not determined what financial and non-financial information it needs in order to manage the training and learning program effectively. This poses the risk that efforts to collect training and learning information could be inefficient and management may not have access to relevant, complete or accurate information for planning and managing training activities. The Agency's approach tends to be reactive based on information requests from senior management. Recent efforts have been taken by establishing a new coding structure for training and learning expenditures.

A review of corporate documents and a series of interviews confirmed that appropriate TLD policies, procedures, and guidelines for collecting information on training and learning were not available.Hence, the Agency does not follow a consistent national approach to collect this information which affects data integrity and the quality of information available to management.

The audit found that accountability for collecting information on the training and learning program was unclear due to policy weaknesses and issues with the funding allocation model.

New coding procedures for training and learning along with a My Learning module were implemented in 2011. As one can expect with changes of this magnitude, implementation issues were noted with both the My Learning module and the new coding procedures. This resulted in not all employees following the procedures for collecting information related to training and learning, which affects the quality and completeness of information.

While the TLD defined business requirements for the My Learning module prior to its implementation, it has not fully assessed the CAS system for providing management with information on planning, managing and reporting on training and learning. The audit noted that various user-based computer applications were being used which may indicate that the My Learning module and the Corporate Administrative System are not satisfying users' needs.

Recommendations

This report makes three recommendations. These relate to:

• identifying training and learning information needs;
• developing and implementing a change management strategy, for managing procedural and cultural changes resulting from the implementation of the new coding procedures and the My Learning module.
• assessing the Corporate Administrative System to ensure it is effective in providing the information management needs to plan, manage and report on the Agency's training activities.

Management Response

The Human Resources Branch concurs with the recommendations of the audit and will work with all Agency stakeholders to define the financial and non-financial information that is required to effectively manage and report on the overall training and learning program within the Agency. This includes the assessment of the current functionality within the Corporate Administrative System (CAS) to ensure alignment with the training and learning information needed and the development of a change management strategy to support the implementation of the new financial tracking procedures and the implementation of the My Learning module.

1.0   Introduction

1.1  Background

The Canada Border Services Agency (CBSA) provides integrated border services that support national security and public safety priorities and facilitate the movement of legitimate persons and goods. The CBSA's role is to manage the nation's borders by administering more than 90 acts, regulations and international agreements that govern trade and travel.

In large part, the Agency's ability to accomplish these tasks depends on the knowledge and skills of its employees. The Agency's training and learning needs are specialized, and the objective of CBSA's training, learning and development program is to provide employees with those specialized skills and knowledge that they need to carry out their work.

The CBSA makes significant investments in training and learning. The total training and learning costs incurred by the Agency in fiscal year 2008-2009 were estimated at $91 million (unaudited). These costs represent 6.3% of the total CBSA expenses of $1.4 billion (unaudited) for that year. The costs of training and learning for fiscal year 2010-2011 were estimated at $80 million (unaudited). Of that, about $56 million (unaudited) is spent on training which is delivered internally. Figure 1 in Annex A shows a breakdown of the estimated 2010-2011 training and learning costs by category, namely Administration, Design and Development, Infrastructure, and Delivery of training.

The CBSA delivers most operational training internally via a network of a main and satellite campuses. As of July 1, 2011 the Agency employed a total of 15,939 employees of whom 5,414 were active Border Services Officers (BSOs). To help prepare for their role as border services officers, candidates must complete the Port of Entry Recruit Training (POERT) program. The program is divided into three phases of training and for the five year period from 2006-2007 to 2010-2011, CBSA expenditures on BSO recruitment and the Port of Entry Recruit program are estimated to have been $106.6M (unaudited) or an average annual cost of $21.3M (unaudited).

In addition to POERT training, other internal training requirements relate to specific programs such as Enforcement Training, Recourse, Trade Compliance, and non-operational training such as Leadership and Management Training.

The CBSA's Training and Learning Directorate (TLD) of the Human Resources Branch is the functional authority and leadership for training and learning. Its responsibilities include developing and monitoring the implementation of policies, leading the planning process, and providing guidance and support to all areas related to training and learning.

Training within the CBSA is funded through a combination of centralized and decentralized allocations. The core business of the Training and Learning Directorate is funded through A-Base funding. The funds for training and learning activities across the Agency come from operational budgets. The TLD does not have spending authority within these budgets. The Directorate has direct authority over training expenditures managed through TLD, and functional authority for other training within the CBSA.

Tracking employees' training, learning and development progress is part of the Agency's risk- management strategy. The need to comply with laws and regulations, and the risk of liability drive the need for training. Training must be documented in order for the Agency to demonstrate that it has provided employees with the necessary or mandated training set out in laws and regulations. Documentation of additional training is also required to demonstrate that the Agency has responded to the training needs identified either by the employer or the employee.

In addition to collecting information on the training histories for employees, the CBSA must track and account for training expenditures to determine whether its training investments have been managed efficiently and transparently, and whether resources are aligned with organizational priorities.

According to the 2007 Training and Learning Policy, the TLD is responsible for developing and implementing the Agency's policy on training and learning. The policy also identifies the TLD as responsible for tracking (i.e., collecting) and reporting information on learning history and training activities and expenditures each fiscal year through a nationally supported tracking system(s).

The Canada Border Services Agency Audit Committee approved an audit of the Training and Learning Program as part of the Three-Year Risk-Based Audit Plan for Fiscal Years 2010/11-2012/13.

1.2  Risk Assessment

The Agency makes significant investments in training and learning each year. In terms of this audit, there is a risk that the lack of appropriate, accurate, complete and timely information on training and learning could compromise decision making around priorities and the use of resources.

1.3 Audit Objective and Scope

The objective of the audit was to determine the extent to which TLD had implemented a framework, including structures and processes, to ensure the integrity of financial and non-financial training and learning information.

Specifically, we looked at:

• whether information was being collected to meet specific needs;
• the policies, procedures and guidelines for collecting training information;
• the extent to which employees were following procedures and other requirements for collecting training information; and
• whether TLD had assessed the effectiveness of the systems for collecting this information.

The period covered by this audit was April 2010 to July 2011.

We did not examine financial processes and controls, as this was not a financial audit. Nor did we examine the Corporate Administrative System (CAS) design or controls because various changes were taking place during the audit. These areas were omitted because when planning the audit, we noted that training information in CAS was neither current nor complete, and was viewed as being unreliable in reporting useful training information.

During the examination phase of the audit, the Training and Learning Directorate was launching two new initiatives—the internal tracking procedures and the My Learning module of the Employee Self Service Portal. The internal tracking procedures were implemented in order to link the training and learning expenditures to the Program Activity Architecture (PAA), which is being put into practice across the Agency. The My Learning module is being rolled out in phases with each phase providing greater functionality to the user. Both initiatives will affect the quantity and quality of training and learning information since one of the desired outcomes was improved data integrity. Under these circumstances we felt that it would be premature to assess financial processes and controls or the design and controls of the CAS system. Instead, the audit focussed on assessing the framework to support the integrity of financial and non-financial training and learning information.

The audit did not specifically examine the Arming Program, and the findings in this report do not pertain to it. This Program is managed separately by the Arming Division and was recently moved under the TLD. The processes and procedures for managing arming-related training and learning information are distinct and unique to the Program.

1.4  Approach and Methodology

The examination phase of this audit used the following approach:

• in-depth interviews conducted with key Agency personnel and senior management in the TLD, Comptrollership Branch and the Regions;
• a review of relevant processes and documents;
• a review and analysis of training and learning policies, procedures, guidelines, standards and reports; and
• a review of information captured in CAS and the training and learning information that the branches and regions maintained.

1.5 Audit Criteria

Appendix A lists the audit's line of enquiry and audit criteria.

1.6 Statement Of Assurance

The audit approach and methodology followed the International Standards for the Professional Practice of Internal Auditing as defined by the Institute of Internal Auditors and the Internal Auditing Standards for the Government of Canada as required under the Treasury Board's Policy on Internal Audit.

This audit provides a high level of assurance that the opinion provided therein is appropriate and reflects conditions that existed at the time of the audit.

2.0 Audit Opinion

For the period under audit, the Agency's structures and processes were not adequate for ensuring that relevant, complete, accurate and timely financial and non-financial information on training and learning was available for managing, decision making and reporting purposes.

This translates to a moderate risk[ 6 ] exposure to the Agency as there are compensating controls such as user-based computer applications, including spreadsheets and databases, which enable the Agency to meet its current reporting requirements. However, these are not always efficient or effective.

3.0 Findings, Recommendations and Action Plans

3.1 CBSA Training and Learning Information Needs

The Agency must collect data or information on training and learning activities in order to both properly manage its training and learning program, and to meet internal and external reporting requirements. It must determine what information it has to collect according to business and reporting requirements so that the information will satisfy senior management's needs.

We expected the TLD to have defined what information on training and learning it needs in order to ensure that the information aligns with the priorities of the organization and meets external and internal reporting requirements.

The audit found that the TLD had not defined training and learning financial and non-financial information needs. By and large, the Agency's approach to identifying training and learning information needs was reactive as opposed to proactive, and was based on senior management's ongoing requests for information.

Through interviews with various stakeholders, including the branches, regions, Comptrollership and (IBBS)[ 7 ] directors and managers, we noted a general lack of consultation throughout the Agency regarding the financial and non-financial training and learning information needs. As a result, branches and regions were having difficulty responding to requests from senior management for such information because it may not have been readily available or may not have been collected.

CBSA completed a Programmatic Assessment, dated June 8, 2011. In this assessment, TLD demonstrated that it had tried to identify CBSA training and learning information needs by laying out the program objectives along with key performance indicators, measures and sources of information. However, as noted in the document itself, some refinement of the objectives and key performance indicators is required in order to finalize what information will be collected to satisfy CBSA's information needs.

The new internal tracking procedures, which were launched in May 2011, involve coding training and learning activities in the financial system in order to collect timely information and expenses related to training and learning activities. These codes are aligned with the Agency's Program Activity Architecture (PAA) to track and report on how training and learning funds are being spent. This represents an effort to identify what the Agency's financial training and learning information needs entail.

In the absence of defined information needs, resources may not be used efficiently if necessary information is not being collected. As well the Agency's information needs for performance and financial reporting may not be met. This could also impact the Agency's ability to manage by the functional model as program leads do not know how much is being spent on training and learning activities.

Recommendation 1:

The Vice-President of the Human Resources Branch should define, in consultation with key learning stakeholders, what information the Agency needs to manage the Agency's training and learning program and to meet internal and external reporting requirements.

Management Response:

The Human Resources Branch concurs with this recommendation.

The Training and Learning Directorate (TLD) of the Human Resources Branch (HRB) will work with key stakeholders to identify and define the financial and non-financial information that is required to effectively manage and report on the overall training and learning (T&L) program. This will involve reviewing current reporting mechanisms and the development of a new performance measurement framework that will provide senior management with a relevant analysis of the performance of the T&L program.

Management Action Plan	Completion Date
Conduct a review of the existing training and learning information that is being collected and reported in various internal and external documents.	August 2012
Consult with key learning stakeholders (i.e. Branches, Regions and senior management) to determine the financial and non-financial information needs for the T&L program and validate with the T&L functional table.	August 2012
Develop a comprehensive logic model for the T&L program that outlines the key activities, outputs and outcomes from which a performance measurement framework can be developed and validate with the T&L functional table.	September 2012
Develop a performance measurement framework (PMF) that complements corporate reporting processes and provides the full spectrum of financial and non-financial information required to support the reporting requirements and management of the T&L function. Validate with the T&L functional table.	October 2012

3.2 Policies and Procedures for Collecting Training and Learning Information and Compliance

Audit Criterion: The Training and Learning Directorate should have policies, procedures, and guidelines for collecting appropriate information on training and learning, which are approved, up-to-date, and well communicated to ensure a consistent national approach. All employees should follow the procedures for collecting this information, and accountability for collecting it should be clearly defined. Conclusion: This criterion was not met. An accountability gap existed with regards to the collection of and reporting on training and learning information. Implementation issues were noted with both the My Learning module and the new coding procedures, which resulted in not all employees following the procedures for collecting information related to training and learning.

This criterion requires the Agency to have established policies, procedures and guidelines that define the roles and responsibilities of various stakeholders for collecting information on the training and learning program. These policies, procedures and guidelines should be followed by employees in collecting and managing training and learning information to ensure its accuracy and completeness for management decision making. A lack of clear guidance around, and accountability for, collecting training and learning information poses the risk of incomplete or inaccurate information. Key elements of the policies and procedures should be monitored regularly to ensure they are appropriately applied by the organization, and frequently reviewed to ensure they remain current, given the Agency's dynamic environment.

Accountability for collecting information

The collection of training and learning information in a large decentralized organization like CBSA is a collective effort by many parts of the organization. Hence, roles and responsibilities of the regions, the branches and of the functional lead, the TLD, should be clearly defined so that the required data is collected and that it is complete, timely and accurate. As the functional lead, TLD is responsible for reporting on the Training and Learning Program. However, collection of information currently happens at the regional and branch level and, therefore, TLD relies on the regions and branches to provide the information that the Directorate needs for reporting purposes. This poses the risk that the information being collected in the regions and the branches may not satisfy TLD's reporting requirements or that it is incomplete or inaccurate.

The audit found that an accountability gap existed between the regions, the branches, and the functional lead regarding the collection of information and the reporting on the training and learning program.

The Training and Learning Directorate (TLD) is functionally responsible for the Training and Learning Program. The Agency's Training and Learning Policy that took effect in December 2007 guides the Program. The existing policy identifies the TLD as being responsible for tracking (i.e., collecting) and reporting information on learning activities and expenditures each fiscal year through the use of a national tracking system(s). This policy is currently under review, given the recent organizational and management model changes within the Agency.

According to management, the revised policy will identify the accountable party for the collection of training-related information, including defined roles and responsibilities. This is important to ensure that no accountability gaps exist for both collecting information on Agency-wide training and learning activities and expenditures, and monitoring these areas. Funding allocation should also align with that accountability.

During the course of the audit, TLD management raised the issue of tracking and reporting of training and learning spending within the CBSA. They confirmed CBSA could not provide an integrated and comprehensive account of spending on training and learning activity in the Agency, nor specific dedicated training budgets within organizational units.

As a result, the TLD received approval in principle during the course of the audit to fence allocations for all training and learning spending through the establishment of regional and directorate-level allocations. It is expected that dedicated fenced funding allocation for training will enable better tracking of operating training expenditures and allow for more rational decision-making on future training spending.

TLD is developing a detailed plan for review and approval through the Agency's governance process.

Guidance and communication

The GC211 Training Application and Authorization form is being used as a mechanism to show manager's approval for the spending of training and learning funds and to capture information on training and learning history. We found that the form was being used inconsistently throughout the Agency.

Further, no guidance had been issued on the purpose of the form since September 2006. This resulted in the GC211 form not being an effective mechanism to obtain training and learning information. It should be noted that with the implementation of the My Learning module, the use of this form is in the process of being eliminated for training taken internally.

New Coding Procedures

CAS is the Agency's financial system which should capture the costs of internal and external training. The total training and learning costs incurred by the Agency in fiscal year 2008-2009 were estimated at $91 million (unaudited). Of that, the Agency incurred costs of about $9 million (unaudited) for external training for employees who attended courses outside the Agency. These costs were recorded in the CAS via the General Ledger account codes. However, the CAS was not recording all of the costs associated with internal training (i.e., course development, teaching and participants' time.), which are estimated at $82 million (unaudited) for 2008-2009. Many employees did not code their time for training activities since time reporting for training was not considered mandatory. With the implementation of the new financial codes for training and learning in May 2011, all employees, including trainers and coordinators, are now expected to use the codes to report time and expenditures. Their supervisors are responsible for verifying that the codes are correct. This is key for the coding system to be effective. The following observations indicate that the desired outcome may not be met:

• there were issues with the communication strategy for the new coding procedures. The announcement of the new coding procedures was not promulgated at a high enough level in the Agency, which impacted the awareness and use of the new procedures.
• there was a general lack of consultation with branches and regions on the new coding procedures;
• the new coding procedures were not being applied consistently across the Agency. Each region had applied these procedures to varying degrees;
• more than one version of the codes existed during the audit period. Early versions contained errors, and the latest had not been finalized;
• no monitoring procedures were established to monitor employee use of the new coding procedures. This is due to difficulties encountered with the training and learning data extraction from the CAS system, which is currently not configured to facilitate the extraction of the required information, requiring extensive manual sorting and manipulation. Monitoring the use and accuracy of the codes is important to ensure financial and non-financial training and learning information captured is accurate and complete, and to assess the effectiveness of the coding procedures; and
• no risk management plan for the coding changes was prepared to identify risks and appropriate contingency plans.

My Learning

The My Learning module, which is part of the Employee Self Service portal, was the second initiative implemented by the Human Resources Branch at the beginning of the fiscal year and is to be used by employees and managers to capture training and learning activities, transcripts and historical data. It is used by the TLD to collect non-financial training and learning information. Launched on April 1, 2011, it was intended to provide increased capacity to collect, deliver, and report on training information. The module is being rolled out in phases with each phase providing greater functionality to the user. My Learning is changing business processes by eliminating data entry related to training into the CAS by administrative staff. Training requests and approvals, course registration, the development of the Personal Learning Plans, among other activities, are to be completed electronically by all employees on a self-serve basis.

Although this was not a systems implementation audit, some issues were noted that impacted the extent to which employees had adopted the My Learning module, including:

• Many users stated that the self-service portal was not user-friendly, was difficult to navigate and training programs were seen as insufficient; difficulties to complete Personal Learning Plans were noted and users reverted to old, manual forms.
• My Learning was not accessible on weekends or during the night. As these are typically periods of down-time for the BSO community, this is when they would most likely be able to complete learning plans and satisfy learning requirements.
• The TLD has not implemented procedures for monitoring the use of the My Learning module. Monitoring will be important in tracking the TLD's performance in getting employees to adopt the module.
• The TLD did not establish any target for mandatory use of the My Learning as part of their implementation plan. Nevertheless, the TLD expects that 50% of employees will be using the module by the end of this fiscal year, 85% the following year, and 100% in the third year.

In conclusion, TLD is making significant changes to the policies and procedures with respect to the reporting of training and learning information, including the implementation of the new coding procedures and the My Learning module. As one can expect with changes of this magnitude, issues will arise and will need to be addressed during the transitional period. Risks should be mitigated through good planning, monitoring and issues management.

Recommendation 2:

The Vice-President of the Human Resources Branch should develop a change management strategy as part of their implementation plan to effectively manage the procedural and cultural changes resulting from the implementation of the new coding procedures and the My Learning module.

Management Response:

The Human Resources Branch concurs with this recommendation.

The Training and Learning Directorate (TLD) of the Human Resources Branch (HRB) will develop a change management strategy, including the development of training tools and a comprehensive communications and engagement strategy, to support the implementation of the new coding procedures and to improve effectiveness of the My Learning module.

Management Action Plan	Completion Date
Establish and communicate the appropriate financial structures and coding processes for the tracking and reporting of T&L financial information. Establish T&L activity codes for tracking salary spending on T&L activities across the CBSA. Establish Training Trust Accounts and funds commitments to track O&M spending on T&L activities.	June 2012
Develop a change management strategy to support the full implementation of the functional management model for T&L, including the implementation of new coding procedures and reporting requirements associated with a centralized (“fenced”) funding model. This strategy will include the development of business rules and work processes, roles and responsibilities, risk assessment, a monitoring and reporting strategy, and a comprehensive communications and engagement strategy.	September 2012
Validate and refine all elements of this strategy with the T&L functional table.	October 2012
Work with key stakeholders (Canada Revenue Agency - IT team and the CBSA HR Systems team) to develop business processes for CAS My Learning, to increase user understanding of the functionality and to make the system more user-friendly for CAS “back-end” users (i.e. technical users/experts). This will also include the design and implementation of training scripts embedded directly into CAS.	October 2012
Assess the current level of understanding and use of the CAS My Learning module and provide on-site training to My Learning "back-end" users across the CBSA (1 session per branch and region) to ensure adequate understanding of the application of their CAS functions and their associated roles and responsibilities. This will ensure consistency in the application of the My Learning module amongst all users, which is directly linked to improving data integrity and our ability to monitor and report on all training delivery.	November 2012 & ongoing support
Develop a change management strategy for the re-launch of the My Learning module including the development of business rules and work processes, roles and responsibilities, a monitoring and reporting strategy, and a comprehensive communications and engagement strategy.	December 2012
Collaborate with the CBSA HR Systems team in the development of training to increase employees' and managers' understanding of the My Learning module in the Self-Service Portal. This training will include electronic simulations and job aids to provide employees and managers with a user-friendly and interactive reference tool for managing their training and learning requirements.	January 2013

3.3 Assessment of the System's Capability and Capacity to Collect Financial and Non-Financial Training and Learning Information

The Corporate Administrative System (CAS) is a business solution that the CBSA and the Canada Revenue Agency (CRA) use to manage and administer their financial, materiel, and human resources business processes and information. It is the system that the Agency intended to use to collect information on internal and external training. Accordingly, the system must provide TLD with complete and accurate data in order to manage the training program effectively.

We expected the TLD to have fully assessed the CAS system's capacity, capability and limitations with respect to collecting complete and accurate training and learning data that meets the Agency's needs. The assessment of the system's capabilities and capacities is a critical element that will determine whether or not the Agency has the necessary tools and information to manage the training and learning program effectively. If the current system in place at CBSA is not providing management with good information for planning, managing and reporting on training, then resources could be misallocated in the form of time and money spent on duplicate recording of information, cleansing data, matching and consolidating data from multiple systems, and/or maintaining multiple systems.

The TLD did assess the system when developing the My Learning module within CAS in July 2009. Business requirements deemed necessary to build the Training and Learning functionality were set out in the Detailed Fit/Gap & Analysis Document. However, we have some concern with the process followed. Without first having defined its training and learning information needs as noted in Section 3.1, it would be difficult for the Agency to assess the system's ability to provide management with the necessary information for planning, managing and reporting on training as what is considered ‘necessary information' is unknown.

Our audit noted certain issues in this area. For example, a number of user-based computer applications were being used throughout the Agency to collect training-related information. These included databases and spreadsheets, which were being used to collect financial and non-financial training and learning information that was not available in CAS. Such information included re-certification requirements and trainer to trainee ratios—key information that is used to determine training effectiveness.

The use of other systems is an indication either that the current system is not meeting users' information needs, or that users are not comfortable with operating CAS, or a combination of both. Using user-based computer applications (as well as CAS) to collect training and learning information represents a duplication of effort and an inefficient use of resources.

Recommendation 3:

The Vice-President of the Human Resources Branch should assess the Corporate Administrative System to ensure that it is effective in delivering on the Agency's requirements for training and learning information needs.

Management Response:

The Human Resources Branch concurs with this recommendation.

The Training and Learning Directorate (TLD) of the Human Resources Branch (HRB) will assess the current functionality of the Corporate Administrative System (CAS) with the objective to identify potential improvements and to reinforce the TLD's reporting capacity on training and learning financial and non-financial information.

In order to address the full scope of this recommendation, key stakeholders (i.e. Canada Revenue Agency, Comptrollership Branch, and Information, Science and Technology Branch) will be involved to assess the feasibility of the proposed improvements and to secure their partnership in the implementation phase.

Management Action Plan	Completion Date
Conduct a review and assessment of the current functionality within CAS (for both non-financial and financial data) and identify gaps and/or areas for improvements that will ensure alignment with the identified T&L information needs from recommendation #1.	November 2012
Develop a partnership strategy with key systems stakeholders (i.e. CRA-IT, HR Systems, Comptrollership Branch, and ISTB) to assess the feasibility of addressing identified improvements in both the financial and non-financial CAS modules.	January 2013

---

Appendix A – Audit Criteria

Given the preliminary findings from the planning phase, the following criteria and sub-criteria were chosen:

Lines of Enquiry	Audit Criteria
1. Framework to support the availability of useful information on training and learning	1.1 The Agency has structures and processes for ensuring that relevant, complete, accurate and timely financial and non-financial information is available for managing, decision making, and reporting.
	Sub-Criteria: 1.1.1 Training and learning information needs are determined based on business requirements.
	1.1.2 TLD has policies, procedures and guidelines for collecting appropriate information on training and learning, which are approved, up-to-date, and well communicated to ensure a consistent national approach. All employees should follow the procedures for collecting this information, and accountability for collecting it should be clearly defined.
	1.1.3 TLD has assessed the capability and capacity of the system for providing management with information on planning, managing, and reporting information on training and learning.

 

Appendix B – Acronyms

Acronym	Definition
Agency	Canada Border Services Agency
BSO	Border Services Officer
CAS	Corporate Administrative System
CBSA	Canada Border Services Agency
CRA	Canada Revenue Agency
HRB	Human Resources Branch
IBBS	Integrated Branch Business Services
PAA	Program Activity Architecture
PMF	Performance Measurement Framework
POERT	Port of Entry Recruit Training program
Regions	Regions of the Canada Border Services Agency
TLD	Training and Learning Directorate

---

Notes

1. As per the findings of the financial analysis presented to the Human Resources Committee (HRC). (Arming accounted for $13.5 million of the estimated costs.) [Return to text]
2. Integrity of information is defined as its completeness, accuracy, consistency, relevance, and timeliness. [Return to text]
3. Financial information includes budget, actual expenditures, direct and indirect costs, salary, the cost of backfilling, overtime, etc. [Return to text]
4. Non-financial information includes type of courses (e.g., technical course vs. leadership course), coding, and history. [Return to text]
5. Moderate Risk: A key control does not exist, is poorly designed or is not operating as intended and the related risk is more than inconsequential. However, a compensating control exists. Corrective action is needed to avoid sole reliance on compensating controls and/or ensure controls are cost effective. [Return to text]
6. Moderate Risk: A key control does not exist, is poorly designed or is not operating as intended and the related risk is more than inconsequential. However, a compensating control exists. Corrective action is needed to avoid sole reliance on compensating controls and/or ensure controls are cost effective. [Return to text]
7. IBBSs are the Integrated Branch Business Services units. Their role is to provide the Agency with support services such as accommodations, training, finance, workforce development and intranet web development. There are separate IBBS units in each branch who operate independently from one another. [Return to text]