Evaluation of the Hepatitis C Prevention, Support and Research Program 1999/2000 – 2005/2006

Previous | Table of Contents | Next

4. Final Program Evaluation

4.1 Evaluation Context

The evaluation demonstrates the progress made toward the Program goals and stated outcomes and identifies the priorities for future work in this area. The report provides an understanding of not only the accomplishments of the Hepatitis C Program, but its contribution to the overall mission, mandate and goals of the Public Health Agency of Canada (PHAC).

The evaluation framework consisted of three key issue areas and tested results and achievements against a range of questions (as indicated in sections 4.3 – 4.5 of this report). Evaluation tools were designed based on these questions and categorized/cross referenced using a data collection matrix.

4.2 Purpose and Scope

The final program evaluation assesses the achievements of results against the Program's stated objectives, goals and outcomes with a primary focus on the activities and outputs for the two one-year extensions as well as those that span the seven years of the Program's existence. The evaluation also identifies continuing gaps and areas of concerns to be addressed within the broad context of the Agency's mandate and future initiatives. Moreover, the evaluation ensures that the Program's achievements and successes are well documented and substantiated, and where possible, attributes the contribution of the Program's efforts in addressing HCV in Canada.

This evaluation will serve to further strengthen federal leadership in broad areas of capacity building for care professionals, researchers, and community organizations, research, surveillance, evaluation and strategic planning including stakeholders, government partners and provincial/territorial governments. Recommendations guide future initiatives to ensure an effective and efficient response to address populations most at risk (Aboriginal peoples, street youth, inmates and injection drug users)including the “hidden” epidemic (those who do not know they are infected); the general public in the area of general awareness/social marketing; and, those infected/affected in need of care and support.

Future programmatic interventions could include a nationally and regionally funded grants and contributions program addressing surveillance data, previous evaluative processes, consultations and progress reports all of which recommend addressing multiple health determinants and disease challenges of diverse populations. These findings demand a coordinated strategy promoting a multi-disciplinary approach, linkages and partnerships.

The results were analysed in the context of reach, results, successes (process, capacity, knowledge development & translation, spin-offs), gaps including lessons learned, challenges including lessons learned in order to form specific and directive recommendations.

The cost-effectiveness of the Program is not addressed, as there is neither significant literature available nor evidence of cost effective figures and comparisons from which to comment. Furthermore, it was determined that the resources (including time and funds) were not sufficient to complete a cost-effectiveness study on the Program. However, evidence collected during this evaluation process can contribute to the larger body of evidence of communicable disease and thus contribute to future cost-effectiveness studies.

4.3 Areas of Enquiry

The data collection matrix aligns the areas of enquiry for the evaluation are acceptable in the context of evaluation methodology. The evaluation issues, questions and lines of enquiry can be categorized as follows:

Areas of Enquiry	HepC Evaluation Issues
Relevance	Operational Review
Implementation	Results/Achievements
Effectiveness	Results/Achievements
Cost Effectiveness	not covered
Efficiency	Strategic Review

4.3.1 Operational Review

The operational review assessed the systems, practices and processes that were implemented to manage the Program and comments on the effectiveness and efficiencies of the Program's ongoing development and implementation.

Key questions included: How did the system (bureaucracy), structure (Program, resources and management practices), processes and support mechanisms contribute to the achievement of the Program's activities and outputs? What challenges were faced and what were the systemic barriers to success? How could these be addressed? How was accountability and transparency incorporated?

4.3.2 Results/Achievements Review

Most significantly, this part of the evaluation identified what results were achieved and what impact those results had on the immediate outcomes, in an attempt to attribute the impact of the results on longer-term (intermediate to long-term) outcomes. Most importantly, the results considered the recommendations from the mid-term evaluation and five-year review to determine if progress was made in addressing these concerns as well as the overall goals of the Program over time.

Key questions included: How did the Program contribute to enhance information and evidence across Canada? How did the Program address stakeholder engagement and support? How broad was the reach of the Program? What partnerships were developed? What new and/or innovative prevention and support projects were implemented? What results were achieved towards the Program's goals?

4.3.3 Strategic Review

This aspect of the evaluation assessed how the program was strategically aligned with the government's (the Agency) mission, goals and objectives. That is, the evaluation comments on the “fit” of the Program's scope, mandate and goals as they relate to the Agency's overall mandate and mission, and discusses how the program design impacted on the achievement of its stated outcomes.

Key questions included: What was the relevance of the Program to the federal government priorities? What was the relevance of the Program to the PHAC's priorities? To what extent did the Program implement a health promotion and population health approach?

4.4 Evaluation Challenges, Study Limitations

As identified in both the mid-term evaluation and five-year review, the Program experienced a number of challenges in the ongoing development and implementation of program components including funding uncertainties, the move to the new PHAC from the Population and Public Health Branch of Health Canada, and, a changing and dynamic epidemic.

A further challenge was the attribution of impacts and effects to the Program itself. That is, any changes in the key result areas – reduction in the number of new cases of hepatitis C or improved care and awareness for those infected with, or affected by, hepatitis C – were difficult to attribute directly to the Program. Since there were many stakeholders and jurisdictions involved in the delivery of services who may influence these results, more emphasis was given to assessing the intermediary steps in achieving the key expected results.

It should be noted that a study limitation exists in the area of the stakeholder survey (see section 4.5.5) in that a total of 126 surveys were sent out, and only 48 or 38% were returned. This represents a margin of error (confidence interval) of +/- 11 given the confidence level is 95%, not considered acceptable in most circumstances.

However, a review of the survey respondents demonstrated a fair representation of all geographic areas (provincial representation), as well as national versus regional/community-based representation (Chart 10). Among these respondents, however, there was a significant lack of representation of those self-identifying as HCV+ and those individuals involved with, or a beneficiary of, Hepatitis C Program activities.

Previous | Table of Contents | Next