Evaluation Plan 2008-2009 to 2012-2013

Commitments to Treasury Board of Canada Secretariat

Presented to and Approved by the
PHAC Evaluation Advisory Committee
May 7th, 2008

Approved by the Chief Public Health Officer
June 10th, 2008

 

1.0     INTRODUCTION

The Centre for Excellence in Evaluation and Program Design (CEEPD) provides corporate leadership in promoting effective, high-quality and consistent evaluation practices across the Public Health Agency of Canada (PHAC). The CEEPD led the development of the Agency’s five-year Evaluation Plan covering evaluations required by the Treasury Board of Canada Secretariat (TBS). This is the Agency’s second 5 year Evaluation Plan.

The Evaluation Plan informs senior management of the scope of the Agency’s evaluation commitments to TBS. The focus of this Plan is to ensure compliance with the requirements of the Federal Accountability Act to evaluate 100% of grant and contribution programs. As well, the Plan also fulfills a TBS requirement to produce an Evaluation Plan each year (Evaluation Policy, 2001).  A copy of the approved Plan will be forwarded to TBS and posted on the PHAC website.

Development of the Plan was a collaborative process. First, CEEPD reviewed existing Treasury Board Submissions and Results-Based Management and Accountability Frameworks (RMAFs) to determine PHAC’s evaluation commitments for the upcoming five-year period. The Agency’s Program Activity Architecture (2008-09) was the framework for analysis. Then, CEEPD issued a call to Directors General across all programs for information to confirm departmental evaluation priorities and commitments.  Programs were asked to confirm existing evaluation plans and provide a detailed description of their planned evaluation activities. They were also asked to conduct a program risk ranking.  Directors General reviewed and approved the information pertaining to their programs. Next, senior executives on the Agency’s Evaluation Advisory Committee reviewed the Plan and recommended its approval by the Chief Public Health Officer (CPHO).  Finally, the Plan was reviewed and approved by CPHO.
           
For information on reports previously completed by the Agency, see http://www.phac-aspc.gc.ca/about_apropos/evaluation-eng.php.

2.0    PLANNED EVALUATIONS

The Plan lists evaluations for 2008/2009 to 2012/2013 which are currently required by TBS (Table 1). It is not an exhaustive list of all evaluative activity in the Agency as it does not include evaluations not required by TBS or evaluations of horizontal initiatives where, PHAC does not have the lead. The estimated budget for each evaluation has been identified (Table 2).

Evaluations (Table 1) have been categorized as formative or summative:

1. Formative evaluations provide information mid-way through a program, policy or initiative by examining the delivery of the program, policy or initiative, the quality of its implementation and the assessment of the organizational context, personnel procedures and inputs.
   
2. Summative evaluations determine the overall impact a program, policy or initiative has had by examining its achievement of outcomes.  The three primary concerns of a summative evaluation are relevance, success and cost-effectiveness.

3.0    LEVEL OF RISK ASSOCIATED WITH PROGRAMS TO BE EVALUATED

All programs to be evaluated were assessed for various types of risk (Table 3). Programs conducted a self-assessment of program risk based on risk types and a ranking schematic.  These tools were based on TBS policy and guidance, approaches used in other departments, as well as PHAC’s own program risk assessment tool from its integrated RMAF/RBAF tool.  The risk types, developed in consultation with program-area and evaluation experts, are:

• Materiality (resource investment/expenditures)
• Alignment with Agency’s strategic objectives
• Political and/or public sensitivities
• Theoretical and practical approaches to ongoing development and/or implementation
• Agency control/influence
• Changing/shifting environment/context
• Agency reputation/ability to deliver results
• Legal liability, litigation
• Ability to prevent, mitigate and/or avoid risk
• Capacity to manage, monitor and evaluate

The risk rankings are used by the PHAC Evaluation Advisory Committee to identify those programs being evaluated which have the most significant risks for the Agency.

The risk rankings are also used to prioritize the work of CEEPD analysts.  Higher risk programs may require additional CEEPD support and guidance to ensure that evaluations meet TBS requirements.  CEEPD also uses the rankings to ensure that key risks to the program are addressed in the evaluation.

4.0     FUTURE DIRECTIONS

In preparing for the implementation of the new TBS Evaluation Policy (anticipated by the end of 2008), CEEPD is reviewing the Agency’s evaluation processes and products. Examples of activities include reviewing the Terms of Reference for the Evaluation Advisory Committee; revising the format and content of the Agency Evaluation Plan; and developing a PHAC-specific Evaluation Policy.

Table 1.  Scheduled Evaluations for 2008/09 to  2012/13

Program Activity	Year to be Completed
	2008/09	2009/10	2010/11	2011/12	2012/13
Health Promotion
SUB-ACTIVITY: Healthy Living
Integrated Strategy on Healthy Living and Chronic Disease •    Healthy Living Program Component	Formative ($300K)
SUB-ACTIVITY: Emerging Priorities in Health Promotion
Population Health Fund	Summative ($100K)
SUB-ACTIVITY: Childhood and Adolescence
Canadian Prenatal Nutrition Program		Summative ($380K)
Community Action Program for Children		Summative ($325K)
Fetal Alcohol Spectrum Disorder Initiative	Summative ($257K)
Chronic Disease Prevention and Control
Integrated Strategy on Healthy Living and Chronic Disease •  Surveillance Functional Component •  Knowledge Development and Exchange Functional Component •  Community Capacity-Building Functional Component	Formative ($100K) Formative ($100K) Formative ($100K)		Summative ($2M)
SUB-ACTIVITY: Cancer
Canadian Breast Cancer Initiative-Community Capacity Building Component	Formative ($25K)
Infectious Disease Prevention and Control
SUB-ACTIVITY: HIV/AIDS
Canadian HIV Vaccine Initiative		Formative ($100K)		Summative ($175K)
Federal Initiative to Address HIV/AIDS in Canada	Formative ($25k)		Summative ($250K)
SUB-ACTIVITY: Pandemic and Avian Influenza Preparedness and Response
Avian/Pandemic Influenza Preparedness				Summative ($2.5M)
Pandemic Influenza Preparedness: Mock Vaccine Development		Summative (Budget to be determined)
SUB-ACTIVITY: Immunization and Respiratory Infections
National Immunization Strategy	Formative ($100K)	Summative ($100K)
SUB-ACTIVITY: Communicable Diseases and Infection Control
Blood Safety Contribution Program [Cells, Tissues, Organs Surveillance System (CTOSS) and Transfusion/Transplantation Injuries Surveillance System (TTISS)]	Summative ($75K)
Hepatitis C Prevention, Support & Research Program			Formative ($100K)		Summative ($150K)
Hepatitis C Undertaking Agreement				Summative ($100K)
Strengthen Public Health Capacity
SUB-ACTIVITY: Building Public Health Human Resource Capacity
Public Health Scholarship and Capacity Building Program		Summative ($150K)
SUB-ACTIVITY: Knowledge-based Information Systems
National Collaborating Centres for Public Health	Formative ($100K)		Summative ($120K)
Across Program Activity Areas
Promotion of Population Health Grants and Contributions		Summative- synthesis¹ ($100K)

¹This evaluation will be submitted as part of the Treasury Board Submission for the renewal of the Terms and Conditions for the Promotion of Population Health Grants and Contributions rather than as a stand-alone evaluation.

Table 2. Estimated budget for scheduled evaluations 2008/09 to 2012/13

Table 3. Scheduled Evaluations for 2008/09 to 2012/13 by Risk Ranking

Legend