Evaluation of the Hepatitis C Prevention, Support and Research Program 1999/2000 – 2005/2006

Previous | Table of Contents | Next

4.5 Methodology

4.5.1 Epi/Surveillance Data

A look at the incidence and prevalence of HCV in Canada, and in specific populations, provided the context in which the Program, as one of the federal government's approaches, was developed to address the complexities of the hepatitis C disease. Key documents and data reviewed for this report included:

• Overview of EHSSS Data, Annual Site Investigators Meeting, March 9-10, 2006
• Report on the Meeting of the Hepatitis C, HIV/AIDS and STI Programs and Evidence – September 26-28, 2005
• Enhanced surveillance of newly acquired hepatitis C virus infection in Canada, 1998 to 2004
• Responding to the Epidemic – Recommendations for a Canadian Hepatitis C Strategy – September 2005
• A Study to Characterize the Epidemiology of Hepatitis C Infection in Canada, 2002

4.5.2 Research Compendium

Significant resources were provided to contribute to HCV research in Canada. The total commitment to these research grants for the period 1999/00 to 2005/06 has been $16.5 million Canadian.

As of March 2006, 64 research-operating grants were awarded, 5 salary awards were provided and 9 training awards were granted. In addition, other strategic and open CIHR programs were established to encourage further study of hepatitis C (e.g., the Strategic Training Program Grants and Social and Behavioural Research Issues in HIV/AIDS, and Hepatitis C: Interdisciplinary Capacity Enhancement Team Grants).

A major initiative of the Program was the establishment of the National Canadian Research Training Program (NCTRP) to: encourage new scientists to the field; provide mentorship opportunities (e.g., 29 mentors were identified); encourage students to work in the area of hepatitis C research (e.g., 18 students were involved); establish new research areas (e.g., psycho-social); ensure research activity from coast-to-coast; and increase financial contributions from outside the Program (e.g., industry contributions were doubled).

Information for the evaluation report was gleaned from the following key documents:

• Turning Research into Action. A Review of the Canadian Institutes of Health Research (CIHR) and the Hepatitis C Program – Research Initiatives on Hepatitis C, 2006.
• Hepatitis C as a Roadmap for Integrated Communicable Disease Prevention and Control. A Strategy for the Renewal of the Health Canada / Canadian Institutes of Health Research (CIHR) Research Initiative on Hepatitis C – November 28, 2004.
• Stakeholder Recommended Priorities for Hepatitis C Research in Canada. An Expanded JAC Meeting held on October 22, 2004.

4.5.3 Document Review

This component of the evaluation was the most substantive given the extent of the Program's existence and the enormity of documentation that has been produced and collected, both at the national level and the regional level. Further, given that a previous evaluation and review have been conducted, and other significant initiatives such as the Legacy Document and the research compendium exist, a larger-than-usual emphasis was placed on collecting, reviewing and analysing the volumes of information available.

For this evaluation, a total of 276 documents were reviewed – 80 from the “nationally funded” jurisdiction and 196 from the “regionally funded” jurisdiction (as shown in Chart 5). The purpose of the document review was to ensure a good understanding of the original Program purpose and plan, document the evolution of the Program and articulate, through both qualitative and quantitative means, the results presented in all documentation.

This data compilation represents a quantitative analysis of the documents reviewed for the Hepatitis C Program evaluation. It is a minimal measurement of 276 documents reviewed in order to provide an analysis of the scope, reach and emphasis that the program placed across three main factors: links to the Program's goals, logic model (activities, outputs and outcomes) and program activity architecture (PAA) of the Community Acquired Infections Division, Public Health Agency of Canada.

These links were categorized according to issue-areas, including:

1. an operational review of the program;
2. results & achievements; and
3. strategic alignment to the Agency and the Federal Government mission & priorities.

Document data also quantified the portion of programs that contributed to evidence development/evidence-base support to the Program; supported Program design, governance and accountability structures; contributed to involvement/engagement and collaboration; and described challenges to Program implementation.

Presented below is the data collected across the entire Program and provided first as combined data sets (National & Regional) followed by a breakdown between National and Regionally funded initiatives.

The documents were further separated by issue area: according to operational elements (i.e., Program systems, structures and processes); results and achievements (i.e., in terms of the Program goals); and/or strategic alignment (i.e., relevance to the Agency's priorities and goals and needs of Canadians). The breakdown according to issue area is illustrated in Chart 6. As expected, most of the documents dealt with results and achievements of the Program (i.e., the extent to which the Program addressed prevention, community-based support, and care and awareness of HCV).¹³

A further breakdown, Charts 7 and 8, shows more than double the percentage of nationally funded than regionally funded documents focused on strategic alignment, where there was proportionately more operational and results/achievement documentation.

4.5.4 Key Informant Interviews

There are a number of key individuals who contributed to the leadership and/or management of the Program. Key information was gleaned from these individuals, particularly in providing evidence in the context of strategic and operational results.

The Key Informant Interview tool and Stakeholder Survey tool were developed in consultation with the Evaluation Advisory group. They are the same tool providing opportunity for comparability between respondent groups. The Key Informant Interview tool was pre-tested using 20 governmental and non-governmental stakeholders asked to provide feedback on the range, scope and purpose of the question within the tool. The test group was provided the evaluation framework for reference and feedback from this group was incorporated into a revised survey tool.

Interviews were held with a pre-selected cross-section of fourteen individuals from government (federal and provincial) and non-government organizations (national and community-based), research institutes and health-care settings located in Nova Scotia, Quebec, Ontario, Alberta and British Columbia. A call for nominations was sent to all national and regional program consultants working in the Hepatitis C Program area, and names were automatically included for those members of the Program Advisory Group (PAG-extended) which included community-based stakeholders.

This data compilation represents a quantitative analysis of the key informant interviews conducted for the Hepatitis C Program evaluation.

The interviews were designed as a tool to enable key informants the opportunity to assess the achievement of results against the Program's stated objectives, goals and outcomes. Key informants included staff of the Program in both the National and Regional offices, advisory committee members and recipients of grant and contribution funding, both national and community-based. The primary focus was the activities and outputs for the two one-year extensions of the Program (2004-05 & 2005-06) but also included outcomes spanning the seven years of the Program's existence (1999-2006). The review also served to identify continuing gaps and areas of concern addressed within the broad context of the Agency's mandate and future initiatives.

Key informant responses ensured that the Program's achievements and successes would be documented and substantiated; identifying challenges, gaps and barriers to success; where possible, attributing the contribution of the Program's efforts in addressing hepatitis C virus (HCV) in Canada; and influencing and informing future efforts to address HCV in Canada.

Informants were encouraged to answer questions from their own perspective based on their personal knowledge, experience and/or expertise and to be completely open and honest in their assessments and were assured that their responses would be kept strictly confidential.

Quantifying this data provides an analysis of the scope, reach and emphasis that the program placed across three main factors: links to the Program's goals, logic model (activities, outputs and outcomes) and program activity architecture (PAA) of the Community Acquired Infections Division, Public Health Agency of Canada.

Key informant interview data also quantified the portion of programs that contributed to evidence development/evidence-base support to the program; supported program design, governance and accountability structures; contributed to involvement/engagement and collaboration; and described challenges to Program implementation.

   

 

This data analysis does not separate out, nor stratify responses between national, regional or community-based funding, but instead, represents all responses combined. Specific comments are presented as provided by the respondents as part of each question and recorded by an independent consultant.

Percentage representation in the following chart illustrates the number of the answers provided in each subsection comparative to the total number of responses for each question.

Key informants indicated their interest/involvement related to HCV in terms of health care provider, friend of HCV+ individual, providing educational services and/or working in a government organization (Chart 9).

4.5.5 Consultation with Stakeholders

As with key informants, certain stakeholders, both within and outside of government, played a significant role in the development and implementation of the Program. These stakeholders provided important quantitative and qualitative data in detailing the evolution of the Program and validating data in the assessment of the overall achievement of Program results. The Key Informant Interview tool and Stakeholder Survey tool were developed in consultation with the Evaluation Advisory group. They are the same tool providing opportunity for comparability between respondent groups.

This data compilation represents a quantitative analysis of the stakeholders surveyed for the Hepatitis C Program evaluation.

This survey was designed as a tool to enable stakeholders the opportunity to assess the achievement of results against the Program's stated objectives, goals and outcomes. Stakeholders included staff of the Program in both the National and Regional offices, advisory committee members and recipients of grant and contribution funding, both national and community-based. All involved stakeholders were invited to participate and were provided the survey tool electronically. The data was collected by an independent consultant to ensure that their responses would be kept strictly confidential and analysis objective.

The primary focus was the activities and outputs for the two one-year extensions of the Program (2004-05 & 2005-06) but also included outcomes spanning the seven years of the Program's existence (1999-2006). The survey also served to identify continuing gaps and areas of concern addressed within the broad context of the Agency's mandate and future initiatives.

Stakeholder responses ensured that the Program's achievements and successes would be documented and substantiated; identifying challenges, gaps and barriers to success; where possible, attributing the contribution of the Program's efforts in addressing hepatitis C virus (HCV) in Canada; and influencing and informing future efforts to address HCV in Canada.

Stakeholders were encouraged to answer questions from their own perspective based on their personal knowledge, experience and/or expertise and to be completely open and honest in their assessments. Stakeholders were assured that their responses would be kept strictly confidential.

A total of 126 surveys were distributed, and only 48 or 38% were returned. This represents a margin of error (confidence interval) of +/- 11. Given that the confidence level is 95%, this is not considered statistically acceptable in most circumstances¹⁴.

However, despite the low response rate, there was a fair representation of respondents from all geographic areas (provincial representation), as well as national versus regional/community-based representation (Chart 10).

Among the survey respondents, however, there was a significant lack of representation of those self-identifying as HCV+ and those individuals involved with, or a beneficiary of, Hepatitis C Program activities. This is consistent, however, with the evaluation methodology which focused primarily on those directly involved in delivery of the program components, as shown in Chart 11, where they accounted for just over 80% of survey respondents.

Percentage representation in the following chart illustrates the number of the answers provided in each subsection comparative to the total number of responses for each question.

4.5.6 Peer Review

This report, with its recommendations, was reviewed by a Peer Review Panel comprised of external stakeholders, in order to provide expert feedback and opinions on the accuracy and validity of the findings of the evaluation. Peer reviewers were selected from the expanded Program Advisory Group which includes non-governmental organizational stakeholders (recipients), HCV health experts, epidemiologists, researchers and individuals living with hepatitis C. The expanded PAG also includes non-HCV organizations concerned and involved in issues related to co-infection (HCV-HIV/AIDS) and other related infectious diseases (HCV, STI, TB).

Peer reviewers were invited to participate in a review of the draft report. Those accepting the invitation were provided the report electronically and provided 3 weeks to return the report with their input. Many peer reviewers opted to provide feedback via a telephone interview.

This report also received input from the Evaluation Advisory Group.

4.5.7 Results Analysis

An analysis of all information collected, including quantitative, qualitative, research and epi/surveillance data is presented in terms of the Program's:

1. Reach;
2. Results;
3. Successes (process, capacity, knowledge development and translation, spin-offs);
4. Gaps including lessons learned;
5. Challenges including lessons learned; and
6. Recommendations for future initiatives addressing HCV.

References

13.	It should be noted that some documents addressed more than one Program element.

 

Previous | Table of Contents | Next