CBSA Evaluation Charter

May 2011

Table of Contents

• 1.0 Purpose of the Charter
• 2.0 Mandate for the Evaluation Function
• 3.0 Mission, Vision and Values of CBSA's Evaluation Function
• 4.0 Scope of Work
  • 4.1 CBSA Five-Year Evaluation Plan
  • 4.2 Results-Based Management (RBM)
• 5.0 Governance
  • 5.1 President
  • 5.2 Executive Evaluation Committee
  • 5.3 Agency Management
  • 5.4 Director General, Internal Audit and Program Evaluation
  • 5.5 Director, Program Evaluation
• 6.0 Nature of the Evaluation Function: Operational Process
• 7.0 Operating Standards
  • 7.1 Independence, Objectivity and Integrity
  • 7.2 Competency, Proficiency and Due Professional Care
  • 7.3 Information Management, Privacy and Confidentiality
  • 7.4 Professional and Courteous Conduct
  • 7.5 Reporting
  • 7.6 Timeliness
• 8.0 Effective Date

1.0 Purpose of the Charter

The Government of Canada's Policy on Evaluation[ 1 ] took effect on April 1, 2009 and replaces the 2001 Policy on Evaluation. This renewed policy has the objective of creating a comprehensive and reliable base of evaluation evidence to support policy and program improvement, expenditure management, Cabinet decision-making and public reporting. It introduces a number of changes intended to improve the neutrality, timeliness and quality of evaluations. These include a focus on core issues of value for money (i.e., program relevance and performance) and expanded coverage to include evaluation of all program spending every five years, effective March 31, 2013.

The Policy on Evaluation requires that credible, timely and neutral information on the ongoing relevance and performance of direct program spending is:

• available to Ministers, central agencies and deputy heads and used to support evidence-based decision-making on policy, expenditure management and program improvements; and,
• available to Parliament and Canadians to support government accountability for results achieved by policies and programs.

The policy directly supports the Expenditure Management System of the Government of Canada by ensuring comprehensive and systematic information on program relevance and performance is available to support decision-making and the strategic review process. It also supports a requirement of the Financial Administration Act (as amended by the Federal Accountability Act) to evaluate all ongoing grants and contribution programs every five years.

This charter explains how the Canada Border Services Agency (CBSA) will implement the Policy on Evaluation. It summarizes the mandate, governance structure, operating standards and significant relationships of CBSA's evaluation function. It outlines the adherence of the evaluation function to professional standards for evaluation, as set out in the policy and by other recognized professional bodies, such as the Canadian Evaluation Society.

This charter has been prepared to inform all CBSA employees about the evaluation function, and the support and co-operation expected from them as evaluation work is undertaken.

2.0 Mandate for the Evaluation Function

The CBSA is a department of the federal government within the meaning of Schedule 2 of the Financial Administration Act. As such, it is subject to the application of the Treasury Board Policy on Evaluation issued pursuant to paragraph 7(1)(a) of the Financial Administration Act[ 2 ]. The policy requires that deputy heads establish “a robust, neutral evaluation function in their department” and ensure “that their department adheres to this policy and to its supporting directive and standard”.

The Treasury Board's Directive on the Evaluation Function supports the objectives of the Policy on Evaluation “by operationalizing requirements for departmental staff involved in evaluation and establishing the specific requirements to enable the production of credible, timely, neutral, and cost-effective evaluations to support policy, program and expenditure decision-making”[ 3 ].

3.0 Mission, Vision and Values of CBSA's Evaluation Function

Our Mission is as follows:

To support the achievement of CBSA's strategic priorities by providing senior management with independent, objective and evidence-based information, advice and guidance on program relevance, performance and efficiency for effective decision-making.

Our Vision is:

To be a capable and reliable partner to CBSA management and be widely recognized for the quality and impact of our evaluation work.

Our work is guided by the following CBSA values:

• Integrity
• Respect
• Professionalism

4.0 Scope of Work

CBSA's evaluation function is delivered by the Program Evaluation Division, as part of the Internal Audit and Program Evaluation Directorate. It reports to the Agency's Executive Evaluation Committee (EEC), which is chaired by the President.

The evaluation function's role is to inform management decisions with objective, evidence-based analysis on the efficiency, effectiveness and relevance of CBSA policies, programs and initiatives.

4.1 CBSA Five-Year Evaluation Plan

The Policy on Evaluation requires that the Agency develop a rolling five-year evaluation plan that demonstrates evaluation coverage of all direct program spending over a five-year period. The evaluation universe, based on the Agency's Program Activity Architecture (PAA), provides the cornerstone for evaluation planning within the CBSA.

The CBSA Five-Year Evaluation Plan is presented to the EEC for approval. It provides:

• demonstration of program coverage;
• a schedule of planned evaluation projects;
• a schedule of planned results-based management projects; and
• resource requirements.

Resource requirements identified in the plan form the basis for human resource planning for the Division.

Methodology for the Development of the CBSA Five-Year Evaluation Plan

Alignment of Evaluation Clusters with the Evaluation Universe and PAA

The evaluation universe presents the full set of evaluable programs[ 4 ] within the CBSA based on the Agency PAA. However, the plan does not represent a one-to-one mapping of evaluations to PAA activities or sub-activities. Program activities may be divided or grouped together, as required, to ensure that evaluations are of a manageable size, while at the same time providing adequate coverage to meet the requirements of the Policy on Evaluation.

Establishing Evaluation Priorities

While the evaluation plan must demonstrate coverage of all program spending on a five-year cycle, a number of factors are taken into consideration in determining the timing of each evaluation. They include:

• program risk;
• management interest;
• Treasury Board requirements;
• planned audit activities;
• period since the last evaluation of the program; and
• horizontality (e.g., programs that are delivered across CBSA branches or government departments).

The plan must also reflect the need for flexibility so that the Program Evaluation Division can respond to requests for one-off evaluation studies, and for advice and guidance on results-based management, including the development of performance measurement strategies and special studies.

Consultation with Agency Executives

CBSA Vice-Presidents, direct reports to the President, and Directors General are consulted to validate the proposed evaluation groupings and project timing. These consultations also provide an opportunity to get their perspectives on priority areas and specific issues that need to be addressed by each study.

Approval of the Five-Year Evaluation Plan

The draft CBSA Five-Year Evaluation Plan is presented for review and approval by the EEC on an annual basis.  

4.2 Results-Based Management (RBM)

The evaluation function, through the RBM Unit, provides results-based management expertise and guidance to Agency management and works to foster a results-based management culture. The unit also provides advice and guidance to support research activities undertaken by other parts of the Agency, including post-implementation reviews.

RBM is a life-cycle approach to management that integrates strategy, resources, processes, and measurements to improve decision-making, transparency and accountability. The approach focuses on achieving outcomes, implementing performance measurement systems and reporting on performance. The RBM Unit's activities focus on the following areas:

Lead Work on Evaluations

The unit supports the division's evaluation projects by developing key program evaluation components – the program logic model and related performance measurement framework (PMF)[ 5 ], in advance of conducting an evaluation.

Performance Measurement Strategies (PMSs)[ 6 ]

PMSs[ 7 ] are intended to help program managers continuously monitor and assess program results, economy and efficiency. The RBM Unit provides advice and guidance to program management as they develop and implement these strategies. For Programs Branch, this includes collaboration and consultation with its Performance Monitoring Division. Once developed, these strategies are approved by the Directors General responsible for the delivery of the programs, the Agency's Senior Financial Officer, and the Head of Evaluation.

Horizontal Evaluations

The RBM Unit provides a single window into the CBSA for the coordination of activities associated with evaluations of horizontal initiatives led by other departments and agencies. The vast majority of these evaluations are conducted by consultants, with direction and input provided through advisory or steering committees comprised of representatives from the partner agencies and departments. To ensure that Agency activities are accurately reflected in the evaluation study, CBSA program management is represented on these committees and facilitates the provision of pertinent data and information to the evaluation team. Evaluation officers represent the Program Evaluation Division on these committees, facilitate and participate in CBSA interviews, and review data and draft versions of evaluation reports. The RBM Unit coordinates Agency approval of horizontal evaluation reports, including the CBSA's management response to recommendations.

5.0 Governance

Independence and objectivity in reporting on program relevance and performance are keys to the success of the evaluation function in achieving its objectives and adding value. This requires support and co-operation from several key players. The roles of the principal players are summarized below.

5.1 President

In accordance with Treasury Board policy, the President of the CBSA has the responsibility for establishing a robust and neutral evaluation function and for establishing a departmental evaluation committee comprised of senior officials.

5.2 Executive Evaluation Committee

Chaired by the CBSA President, the EEC membership includes the Executive Vice-President and all Agency Vice-Presidents, Associate Vice-Presidents and direct reports to the President. The evaluation function reports on its activities to the committee, which plays an important oversight role.

Principal responsibilities of the EEC include:

• reviewing, contributing to and approving the CBSA Five-Year Evaluation Plan;
• reviewing, commenting on and approving evaluation reports; and
• approving evaluation recommendations and management responses.

The Program Evaluation Division serves as the Secretariat for the EEC. The following protocol has been established for the committee:

• The EEC meets quarterly, at scheduled intervals. However, additional meetings may be arranged, if required.
• The Secretariat is responsible for:
  • setting the agenda for meetings;
  • preparing bilingual binders of evaluation reports, supporting presentation decks and status updates for committee members, and distributing these to committee members five working days in advance of meetings; and
  • preparing Records of Decision which are forwarded to the Corporate Secretariat for the President's approval within 10 working days of committee meetings.

5.3 Agency Management

A relationship of mutual trust and respect between employees who carry out the evaluation function and other CBSA staff is critical to achieving the objectives of the evaluation function and ensuring effective program management support.

While evaluation projects are being conducted, CBSA management is expected to be co-operative and reasonably available to engage in evaluation activities. Agency management must respond in a timely manner to evaluation staff members' information requests and ensure access to the reliable, timely and objective information needed to conduct their evaluations. All evaluation employees are required to have secret security clearance and, while respecting clearance requirements, they will have unlimited access to all Agency documents.

In addition to these general responsibilities, Agency management is expected to:

• provide comments and feedback on evaluation findings, observations and reports in a timely manner;
• develop management responses to evaluation recommendations;
• incorporate management responses and action plans for improvement into program priority setting, planning, reporting and decision-making processes; and
• respond to call letters from the Professional Practices and Operational Services Division of the Internal Audit and Program Evaluation Directorate, as part of the annual reporting process set out in the CBSA Monitoring Framework for Evaluation Recommendations and Management Responses.

5.4 Director General, Internal Audit and Program Evaluation

The Director General of the Internal Audit and Program Evaluation Directorate is the Agency's Head of Evaluation and is responsible for ensuring an effective evaluation function.

5.5 Director, Program Evaluation

Reporting to the Director General of the Internal Audit and Program Evaluation Directorate, the Director, Program Evaluation Division, is the senior full-time Evaluation Executive in the Agency. This individual has been delegated responsibility by the Head of Evaluation for ensuring that all evaluation activities adhere to applicable policies and standards. The Director is also responsible for the day-to-day operation of the Program Evaluation Division and for verifying overall quality assurance of all evaluation products before they are presented to the Agency's EEC for approval.

6.0 Nature of the Evaluation Function: Operational Process

The following section provides a high-level description of the evaluation function's operational process.

• CBSA Five-Year Evaluation Plan – the Agency's evaluation priorities for the year are identified in a rolling five-year plan, which is revised annually during the Agency's evaluation planning exercise; it employs the methodology described in Section 4.1.
• Results-based Management – based on the evaluation priorities, Agency priorities and commitments to Treasury Board, the Results-Based Management Unit works with program management to develop results-based management tools. These include development of logic models and performance measurement frameworks to identify success criteria and key performance indicators, PMSs and evaluation strategies. RBM activities help program managers to identify and monitor key performance indicators that validate whether the program's activities, processes, products and services are aligned with and contribute to the attainment of its critical success criteria. In some cases, development of RBM tools may be a Treasury Board requirement (e.g., as a condition of a TB decision). For programs that will be evaluated, the logic model and performance measurement framework serve as the basis for developing the evaluation plan.
• Evaluation Planning – the evaluation plan describes the program context and provides a framework for the evaluation with an outline of the evaluation purpose, evaluation issues, methodology, and resource requirements. The Director of Evaluation approves the evaluation plan. Before the plan is finalized, senior executives and program management are consulted to ensure pertinent and relevant questions have been included to meet their information needs.
• Evaluation Research – based on the project plan, research and analysis for the evaluation is undertaken. Each project is calibrated based on risk and other factors such as materiality, previous evaluation findings, and the period of time since the last evaluation or assessment. Calibration is used to determine the approaches, designs, methods and resources required in order to produce a quality evaluation in a cost-effective manner. The collection of data involves data analysis, site visits, interviews with the program stakeholders at Headquarters and in the regions, surveys, and/or other relevant methodologies. In order to ensure quality assurance, an internal peer review is conducted and the Director of Evaluation's approval of the draft evaluation report is required before it is circulated for comments outside of the Program Evaluation Division.
• Reporting – the draft report is shared with program management to communicate the findings and recommendations of the evaluation, and to obtain management's response and action plan. The final draft report, which includes the management responses, is presented to the EEC for approval. Once approved, the report is finalized for release to the public. As part of the process, the Agency's Access to Information and Privacy (ATIP) officials are consulted to ensure the report meets applicable guidelines on disclosure of information. The Program Evaluation Division collaborates with the Communications Directorate in developing communication materials before the report is posted on the Agency's Web site and a link to the report is posted on the Treasury Board Web site. The process from the planning to reporting phases involves approximately 10 months.
• Monitoring – The Professional Practices and Operational Services Division of the Internal Audit and Program Evaluation Directorate conducts follow-ups on the implementation of evaluation recommendations and associated management responses and action plans. The Vice-President responsible for the program approves the program's self-assessment, which is then presented in summary form to the CBSA Executive Committee. Monitoring continues until the recommendations are fully implemented, or are obsolete or superseded.

The following process map provides a graphical representation of the operational process.

7.0 Operating Standards

Employees carrying out the evaluation function have an obligation to perform the work objectively, professionally, and in a spirit of mutual trust and respect. The following sections set out the standards for the evaluation function that all CBSA staff can expect. These standards embrace the well-defined guiding principles set out in the Policy on Evaluation, and by Treasury Board and professional evaluation bodies, in particular, the Canadian Evaluation Society and the American Evaluation Association.

7.1 Independence, Objectivity and Integrity

Evaluation projects will be free from interference from program management in the determination of the scope of work, in the conduct of the work and in the communication of the results.

Evaluators will be objective in performing their work, have an impartial and unbiased attitude, and avoid conflicts of interest. Evaluators will act with integrity in their relationships with all stakeholders and behave in a manner that is consistent with the Values and Ethics Code for the Public Service and the CBSA's values of integrity, respect and professionalism.

7.2 Competency, Proficiency and Due Professional Care

Evaluators will possess the knowledge, skills and other competencies needed to perform their responsibilities. Evaluators will apply due diligence in the performance of their work. Evaluation professionals collectively involves either using the required knowledge, skills and other competencies to complete the work or taking steps to obtain these elements, as required.

7.3 Information Management, Privacy and Confidentiality

Evaluators will comply with government policies related to information collection, use, preservation and dissemination. Stakeholders and participants in an evaluation will be informed of the level of confidentiality and privacy that is afforded them under the Privacy Act[ 8 ].

7.4 Professional and Courteous Conduct

Evaluators will perform their responsibilities professionally and courteously, remaining sensitive to the additional workload imposed on clients by the evaluation.

7.5 Reporting

Evaluation reports will respect all relevant Treasury Board standards, and will be written in a clear and concise manner. Reports will present conclusions and recommendations that flow logically from evaluation findings and will contain clear and actionable recommendations and timing for the action plan.

Completed reports, including management responses and action plans, will be presented to the EEC for discussion and approval. Prior to seeking EEC approval, draft evaluation reports will be provided for comment to senior management responsible for the area under evaluation. Notwithstanding this step in the process, should a serious issue be discovered over the course of the evaluation research, the evaluators would raise it immediately with senior management.

Reports will be considered final after they have been approved by the EEC, and will be released to the public after proofreading and final copy editing by the CBSA Communications Directorate. Reports in both official languages are posted on the CBSA Web site, with a link to the reports posted on the Treasury Board Web site. CBSA ATIP officials are consulted before the reports are released to ensure they respect guidelines on disclosure of information.

7.6 Timeliness

Evaluation Reports

Timely evaluation reports are an important source of information for senior CBSA officials on significant issues of effectiveness, efficiency and management practices. The following time frames have been established for the completion of evaluation reports:

• Under normal circumstances, a program evaluation will take no longer than 15 months, including the planning and publication cycles.
• Management responses to address recommendations in the draft evaluation reports will normally be developed and provided to the evaluation function within 10 working days of receipt of the draft reports.

RBM Advice and Guidance

Unless a PMS is required by Treasury Board, there are no timelines associated with their completion as these are the responsibility of program management. However, the Program Evaluation Division will respond to formal requests from Directors General for such guidance and assistance within five working days to determine how to move the requests forward.

8.0 Effective Date

This charter is issued under the authority of the President of the CBSA and approved by the EEC. The charter came into effect on April 1, 2011.

The Head of Evaluation will recommend updates to the charter as required. Updates will be effective upon approval of the EEC.

---

Notes

1. Policy on Evaluation [Return to text]
2. Financial Administration Act [Return to text]
3. Treasury Board Secretariat, Directive on the Evaluation Function, 2009 [Return to text]
4. The word “program” is used throughout the Five-Year Evaluation Plan to denote programs as well as other evaluable entities such as policies, initiatives, and functions. [Return to text]
5. A logic model is an illustration of the results chain or how the activities of a program are expected to lead to the attainment of the final outcomes. It outlines the intended results (i.e., outcomes) of the program, the activities the program will undertake and the outputs it intends to produce in achieving the expected outcomes. A Performance Measurement Framework (PMF) identifies the indicators required to monitor and gauge the performance of a program, and serves as a plan for the ongoing measurement of the performance of a program. [Return to text]
6. These were formerly referred to as Results-Based Management and Accountability Frameworks (RMAFs). [Return to text]
7. PMSs include a program profile, logic model, PMF and an evaluation strategy. PMSs establish regular processes for the measurement of progress against identified outcomes of programs or initiatives. For low-risk programs, performance monitoring of outcome achievement can provide an adequate barometer of program success and reduce the number of lines of evidence used in the evaluation. Also, for many programs, regular performance monitoring will provide enough evidence to satisfy senior managers that the appropriate results are being achieved. [Return to text]
8. Standard on Evaluation for the Government of Canada, 2009 [Return to text]