Resource Library

Hepatitis C Prevention, Support and Research Program
Health Canada

Get the Facts: Mid-term evaluation report

V. Program Outcome Achievement

Implementation of Program activities is expected to result in planned Program Outcomes. In this section, outcome achievement (in response to evaluation questions) and Program successes and challenges are discussed.

1. Outcome Achievement

In this section, the evaluation results related to progress toward achievement of the outcomes identified in the Logic Model (page 11) are reported. Progress toward immediate outcomes was assessed, and intermediate outcomes were assessed where possible. The assessments presented here are a synthesis of findings related to Program outcomes.

Knowledge Generation

Evaluation question:
To what extent has knowledge been generated through research about hepatitis C, as a result of the Program?

Research capacity and knowledge has increased

• In partnership with the Canadian Institutes of Health Research, the Program funded 33 research projects and 14 training and salary awards, as of September 2001.

• Respondents to the Researcher Implementation and Outcome Achievement Survey indicated that the Program had attracted new researchers; 24% said that without Program funding they would not have conducted their research projects.

• Researchers said that "the Program stimulated research, increased publications and raised Canada's international stature in hepatitis C research."

• 87% of health experts surveyed identified two areas in which the Program has had an impact: clinical research and basic research.

Research is of high quality

• Funded researchers reported that there had been 18 publications of Program-funded research, most frequently (63%) in peer-reviewed journals.

• Funded researchers reported presenting research findings at conferences and workshops, for a total of 109 reported presentations.

• The breakthrough development reported was development of the "supermouse" model to study vaccines and new treatments of hepatitis C, published in 2001.

Some research on hard-to-reach populations

• A few research projects funded by the Program were completed outside the partnership with the Canadian Institutes of Health Research: six completed and two ongoing research projects on risk factors and injection drug use related issues, four completed surveys and five surveillance-related initiatives were completed in collaboration with the Health Care Acquired Infections Division.

Results

• Some of the projects deal with priority at-risk populations - street youth, injection drugs users and Aboriginal peoples - who are not generally targeted in other research projects.

More social science research is needed

• Gaps in research were noted in the survey results: of 33 grants awarded, only one related to quality of life, which suggests that social science has been overlooked. Health experts surveyed noted that there was a lack of information in Canada in the areas of prevention (51%), care and support (43%) and harm reduction strategies (43%).

Use of Information

Evaluation question:
To what extent has there been increased use of hepatitis C information by people infected with, affected by, or at risk of contracting hepatitis C, caregivers, and organizations providing support and care, as a result of the Program?

Early results

• 66% of funded researchers reported that their research had been completed and 12% knew of cases where their findings had been applied.

Evidence of information use

• Needs assessments were key to the identification of where information needed to be targeted.

• A large number of materials were developed and disseminated; numerous requests were also received for reprints as an indication of reach and level of use.

Obstacles to accessing information exist

• One-third of respondents in the Community Case Study–Primary Client Interview reported difficulty in locating information on hepatitis C. They noted that Internet access and the ability to perform Internet searches seemed necessary.

• Other comments reported were related to language (materials were in English only and/or were American in content), and suggested that text was culturally inappropriate for Aboriginal people, and that the literacy level was too high, especially for inmates.

• Fewer than a third of respondents to the Community Case Study–Secondary Client Interview felt that there was enough hepatitis C information. Topics requiring additional research included prevention, Aboriginal people and youth, disease transmission, risk factors and harm reduction strategies.

• There is a need to further assess the ability of the Program to encourage greater use of hepatitis C information both at the individual and organizational level, and to confirm accessibility of this information.

Development of Partnerships and Collaborative Arrangements

Evaluation question:
To what extent have effective partnerships and collaborations been developed in response to hepatitis C issues as a result of the Program?

Results

Partnerships have been developed at the national level

• To increase Program reach, the Program has formed partnerships and collaborated with many different government departments, agencies and organizations.

Partnerships are part of most Program components

• There is widespread evidence of the development of partnerships and collaborative arrangements.

• Partnerships have been developed for a variety of purposes; the most common was for the purpose of sharing information.

• Program-funded researchers report partnerships with hospitals/health centres, educational institutions, and pharmaceutical companies. Respondents involved in care and treatment support reported a total of 44 partnerships; the majority of community- based support representatives reported partnerships for a total of 474. Community partners included public health units, HIV/AIDS organizations, the Canadian Liver Foundation and the Hepatitis C Society. Most said they planned to continue these relationships.

• Community networks were among the successes reported by community-based support projects, as a result of funding. Examples from case study locations include information exchange, with dissemination of materials and referrals through hospitals, partners working in schools and with street youth, and local community centres, in order to increase reach (Montreal); the importance of learning how to work with the Aboriginal community to develop an adequate response to HCV (Lethbridge); and the realization that partnerships can lend credibility, bring mutual benefits, and foster collaboration for the achievement of common objectives with a shared client group (Moncton).

• Partnerships promote increased capacity to address hepatitis C, but it is unknown whether this improved capacity will survive if funding ends.

Development of Capacity to Address Hepatitis C

Evaluation question:
To what extent have the community-based groups, partners in program delivery, and researchers developed the capacity to address hepatitis C issues, as a result of the Program?

Results

The findings of the regional environmental scans conducted in 1999-2000 found that most community-based groups were at an early stage of development, volunteer-driven and had received little or no funding or formal training.

Survey results have shown that a significant effort has been made across all program components to increase the capacity of communities across Canada to address hepatitis C. There are specific areas where capacity has increased.

Training of professionals and non-professionals

• Training and development opportunities were provided frequently, at varying levels, and in different components of the Program.

• Training was most commonly related to the care and treatment of patients with hepatitis C (basic information on the disease, progression of the disease).

• Training was presented most often to nurses, family physicians/general practitioners, other health professionals, and staff/volunteers, in descending order.

• Training serves to build capacity at both the organizational and individual levels; the impact of training initiatives has yet to be assessed. This information would be valuable at the national level.

• The perception of primary clients and stakeholders is that physicians' level of knowledge and training on hepatitis C remains limited. Areas where this knowledge could be improved include diagnosis, general understanding of the disease and treatment.

Capacity to respond at the community level improved

• The majority of projects responding to the survey reported that they provide training and development opportunities. This was most commonly staff training, health professional training, volunteer training and counsellor/peer support training.

• Projects reported holding workshops or other events on prevention education, partnership development and networking.

• Hepatitis C information dissemination is critical for knowledge at the community level. As such, since receiving funding, the majority of surveyed community groups reported presenting information on hepatitis C to community and project audiences. Nearly 40% presented to regional audiences and 20% to national audiences.

• Innovative approaches to delivering services were observed at the case study sites: examples include a volunteer theatre group; a holistic care and treatment approach integrating primary care, needle exchange counselling and support for marginalized populations; a youth-centred organization using a peer-driven service model; and the formation of a broad-based provincial coalition to address the needs of those infected with/affected by HCV.

Priority populations involved

Involvement of priority populations in project activities is an important element of building a strong response to hepatitis C in Canada.

• The majority of project activities involved priority populations through their inclusion on boards, advisory groups or in consultations and focus groups.

• The majority of community-based support organizations reported involving priority populations such as individuals infected with hepatitis C, family and friends of individuals infected with hepatitis C, women, Aboriginal people and people who inject drugs.

• The primary clients interviewed reported receiving valuable services from the projects, including information, support or counselling, the opportunity to get involved, and referrals to other service providers.

Numbers of staff and volunteers increased

• The numbers of staff and volunteers increased - groups responding to the Community-based Support Program survey reported an average of one full-time staff, 1.6 part-time and 8.2 volunteers per project, and 66 of the surveyed community-based groups reported that more than 500 volunteers were added during 2001-2002 for a total of more than 18,000 volunteer hours of service.

• Community-based groups were observed to be employing strategies aimed at long-term sustainability.

• Capacity-building at the organizational level has not clearly translated into more access for primary clients; only 39 out of 149 community-based projects provided information on client access. Further information is required before the Program can make strong association statements between capacity and increased access.

Access to Prevention and Care, and Treatment Support

Evaluation question:
To what extent has there been improved access to prevention and care, and treatment support?

Results

Access to care and treatment is an issue

• About two-thirds of primary clients surveyed commented that there is not enough care and treatment support available in the community; this view was shared by those responding to the other stakeholder interview.

• Other stakeholders identified access to care and treatment or a lack of expertise (among caregivers) as a barrier to care.

Demonstrated effect at the local/regional level

• Before the Program began there were few hepatitis C-related projects. At the time of evaluation 149 local/regional projects were funded. (See Appendix 7 for details on regional environmental scans.)

• The projects responding to the Community-based Support Program Implementation and Outcome Achievement Survey noted that populations targeted were individuals infected with hepatitis C, people who use injection drugs, families and friends of people with hepatitis C, youth, community organizations, people co-infected with HCV/HIV, Aboriginal people, prisoners or ex-prisoners, women, health care providers, and the general population.

• Referrals to care and treatment support resources were another indicator that accessibility may be increasing.

Public Awareness of Hepatitis C

Evaluation question:
What is the level of public awareness of hepatitis C?

Results

The Program commissioned two national opinion polls, in 1999 and again in 2001, to survey Canadians on general awareness of hepatitis C. Six questions appeared in both survey instruments and can be used as a point of comparison. Self-reported knowledge of hepatitis C decreased considerably between 1999 and 2001: in 1999, 53% of people responding said that they were either somewhat knowledgeable or very knowledgeable about hepatitis C; in 2001, 28% said the same thing. The level of concern about hepatitis C also decreased, with 57% reporting in 1999 that they were concerned, as compared with 45% in 2001.

The evaluation did not examine why this decrease in awareness occurred. It is believed that this drop is attributable to a decrease in coverage of hepatitis C by the national media. At the time of the first survey the national media reported extensively on the issue of federal compensation to those infected with hepatitis C from the blood system. It is likely that the presence of hepatitis C in the national media at the time the first poll was taken had an effect on the level of public awareness. Hepatitis C did not have the same media profile at the time the second poll was conducted. In addition to decreased media coverage and within this time period, the Hepatitis C Program did not launch the public awareness campaign as planned and focused most program activities in selected communities.

The evaluation indicated that awareness has been raised on a small scale in selected communities where hepatitis C projects have been implemented. The case studies and several project evaluations have reported an increase of awareness of risk factors, modes of transmission and harm reduction strategies in their targeted populations (which includes incarcerated individuals) as a result of prevention messages and materials distributed in their community or environment. A systematic targeted approach to raise awareness in key groups at risk has not yet been implemented by the Program.

2. Program Successes and Challenges

Key successes and challenges identified in the technical report are listed here for each component: Management, Prevention, Community-based Support, Care and Treatment Support, and Research.

Management Component

Successes

Broad vision, national scope: The Program provides a broad strategic approach to program development at the national level in response to hepatitis C; it is evident that much has been accomplished in each component.

Understanding of population health: The Program structure demonstrates an understanding of population health principles.

Multi-pronged approach: The Program has approached the challenge of hepatitis C through its five components in varied ways.

Broad initial consultation: The extensive initial program consultation process helped identify research, practice, community and personal issues. No group declared that it had been ignored.

Partnerships demonstrated: Ample evidence was noted of established partnerships and their effect on setting direction and priorities.

Flexibility: The Program has been able to adapt to changing facts about hepatitis C and to address issues, especially those of marginalized groups, without diminishing focus on high-priority populations.

Well-managed overall: Funds have been generally well managed.

Challenges

Need for a clear strategic direction: Initial clarity of program design has resulted in success; however, concerns were expressed about future direction.

Coordination and communication functions need to be strengthened: Survey results indicated that there is a need for improved coordination both across regions and between regions and the national level.

Linkages with stakeholder groups not established: The Program Advisory Group provides representation for people infected with, affected by and at risk of contracting hepatitis C. There is no indication, however, that discussions within the Program Advisory Group have an impact on implementation.

Provincial/territorial liaison needs greater focus: More needs to be done to foster cooperation.

Ongoing consultation with priority populations limited: A number of at-risk populations have not received equal attention. Services to Aboriginal groups, for example, tend to be compartmentalized.

Documentation variable: There is a need to streamline and improve reporting and evaluation mechanisms.

Funding cycle issues need more problem-solving: Government fiscal year beginning and end dates are well known, and planning should be possible; yet project funding is often late and still must be spent before fiscal year end.

Research Component

Successes

Amount of funding: The research community sees Program funding as significant.

Training and salary support dollars: Provision of funds for training and salaries indicates an understanding of researchers' needs and serves to support new researchers.

Responsive research agenda: Communication between researchers and government has been achieved, and through feedback from the research community the Program has been able to relay to the Canadian Institutes of Health Research the need for more behavioural and social science research.

Emerging research is ground-breaking: Published research to date has been valuable.

Other potential successes: Several initiatives were noted as having potential for success where the ground has been prepared for important research findings.

Challenges

Lapsed funding: The mismatch between funding cycles of the Canadian Institutes of Health Research and Health Canada has resulted in lapses of significant amounts of funds for research, and must be resolved.

Lack of behavioural and social science research: Many social issues connected with hepatitis C warrant further exploration and should be more prominent in the research agenda.

Research findings should be integrated into Program implementation: Research findings must be communicated and shared so that they can inform and influence practice guideline development and other critical issues; knowledge generation should be accompanied by communication/dissemination strategies.

Care and Treatment Support Component

Successes

Practice guidelines: Practice guidelines, which are critical to treatment improvements, have been developed and produced in several formats.

Capacity building: Professional education activities in the form of self-guided programs, workshops and Web-based training modules have been developed to encourage access by physicians and other health professionals.

First Canadian Conference on Hepatitis C: This event continues to be the landmark in information-sharing and capacity building on a national level.

Patient/community resources: The development of resources such as fact sheets and brochures was identified as a benefit to organizations working in the field, which, prior to Program funding, had limited resources for such materials.

Challenges

Clearer priority-setting and fund-granting processes: There is a need to establish clear priorities.

Address care and treatment support issues for at-risk populations: Additional tools are required in this component: development of psycho-social guidelines, information on disability issues and information on self-care.

Measure the reach of guidelines: Knowing who is using treatment and support guidelines developed to date and what effect that has had on practice are key.

Clearinghouse: The need for a clearinghouse to coordinate and distribute hepatitis C materials is critical to enhance existing distribution methods and to ensure efficient use.

Prevention Component

Successes

Effective collaboration and consultation: Collaboration and consultation has worked well in this component, and has resulted in much needed resource development.

Significant needs assessment findings: National and regional needs assessments were useful in documenting the lack of hepatitis C-related services at the community level.

At-risk populations identified: The Program has demonstrated flexibility regarding target populations and has initiated important work in areas such as injection drug use.

Important national partnerships developed: Relationships with key departments and organizations has led to fruitful collaboration on prevention research.

Challenges

Clear strategy needed to address prevention: Development of a clear national hepatitis C prevention strategy would support collaboration with partners and stakeholder groups.

Need to work more closely with priority at-risk populations: Several populations engaging in high-risk activities (street youth, injection drug users, and inmates of correctional facilities) require a specialized approach to prevention.

Need to explore relationships with institutions: Work has proceeded with national organizations, however, future work should explore the potential of partnerships at the provincial/territorial level with institutions such as public health units and health regions.

Community-based Support Component

Successes

Funding for national non-governmental organizations: Identification of key non-governmental partners facilitated access to those infected with and affected by hepatitis C.

Community-based support projects effective: Funding has resulted in a significant number of projects meeting local needs as identified in needs assessments; the documentation review plus observations at case study sites led to the conclusion that the projects have been well managed by Health Canada.

Projects are the conduit to and from high-priority populations: Information on high-priority population needs and experiences obtained directly from these projects assist decision-makers to be grounded in the reality of life with hepatitis C.

Community-based partnerships are key to sustainability: Strong community networks are critical to ensuring a sustainable community response to hepatitis C.

Challenges

Relationship with national non-governmental organizations needs examination: Issues of duplication and efficient and effective use of resources to achieve Program objectives were outlined both in the document review and the audit of the national non-governmental organizations funded by the Program. Findings suggest that the Program should assess these relationships and explore partnerships with other national organizations.

Community groups need more capacity-building: Smaller community groups need support.

Models of best practice needed: More focus is needed on how successful projects deal with at-risk populations.

[Previous] [Table of Contents] [Next]