Report from the Evaluation Indicators Working Group: Guidelines for Monitoring Breast Screening Program Performance

[Previous] [Table of Contents] [Next]

Background

Introduction

Organized breast cancer screening has been established in Canadian provinces and territories. With the Canadian Breast Cancer Screening Database fully implemented, consistent program data are available for evaluation. The purpose of this document is to define an initial set of performance measures for Canadian breast cancer screening programs. The document will serve as a guide to promote consistent calculation of key performance measures for various monitoring and evaluation efforts across programs and over time. The description of each measure includes a definition, the context in which the measure is relevant (rationale), method(s) of calculation, target objectives, and the current status of the measure under evaluation. The measures presented in this document were developed on the basis of recognized population screening principles, evidence from randomized controlled trials, demonstration projects, and observational studies (see Appendix A for a brief framework of screening principles).

Purpose of Performance Measures

The principal goal of breast cancer screening is to reduce breast cancer mortality and morbidity associated with this disease. Regular mammography screening for women aged 50 to 69 is expected to prevent approximately one-third of breast cancer deaths 7 to 12 years after sufficient participation (70% of women in the target group) has been achieved1. Because reaching a participation rate of 70% among women aged 50 to 69 will be a gradual process, short-term reductions in mortality rates cannot be used to monitor the effectiveness of breast cancer screening. Instead, performance measures that are valid, reliable and feasible to collect within the screening program are required for interim evaluation of breast cancer screening. Furthermore, these measures provide a means to monitor the individual steps throughout the entire screening process in order to ensure that the objectives of a successful screening program are met. This is the first step in ensuring that screening programs continually strive to increase the benefits of screening while minimizing the negative side effects.

Interim measures used for ongoing evaluation of organized breast cancer screening programs at the national level include participation rate, abnormal call rate, cancer detection rate, rate of advanced cancers, tumour size and nodal status (detailed descriptions to follow). Provincial and territorial programs compute additional measures that are not monitored at the national level. In the past, data from Canadian organized breast cancer screening programs have been collected nationally for comparison against targets set by Sweden², the European Union^3,4, the United Kingdom^5,6 and Australia⁷. These comparison are useful, but the uniqueness of Canada's population health context and differences in elements of its programs heighten the need for a set of Canadian core indicators and targets that measure the performance and quality of its organized breast cancer screening programs.

Canada's Unique Population Health Context

Table 1 provides a comparison of selected characteristics of organized breast cancer screening programs in Canada, the United Kingdom, Australia, Sweden and the Netherlands. In Canada, health care delivery is under provincial/territorial jurisdiction; thus, organized screening programs have been developed and implemented independently across the country. Consequently, programs vary in their organization, screening modalities, recruitment methods, ages targeted for screening, and in the arrangements for diagnostic assessment following an abnormal screen. Currently, all 10 provinces and two territories have organized screening programs⁸. To differing degrees, all provinces/territories continue to provide mammography services to asymptomatic women outside the structure of the organized programs. It has been estimated that as much as 80% of bilateral mammography provided in this manner is for screening purposes⁹. Consequently it is referred to as "opportunistic screening".

Table 1

Comparison of Selected Characteristics of Organized Breast Cancer Screening Programs in Canada, the United Kingdom, Australia, Sweden and the Netherlands

Organized Screening in Canada

Organized screening in Canada was initiated in 1988 on the recommendation of a national workshop11. The workshop consisted of expert representatives from government, and key professional and voluntary organizations. The result was a recommendation that women aged 50 to 69 be offered, and encouraged to participate in, an early detection program offered every 2 years, consisting of mammography, physical examination of the breast by a health care professional, and teaching and monitoring of breast self-examination.

In December 1992, the federal government launched the first phase of the Canadian Breast Cancer Initiative (CBCI), with stable, ongoing funding of $25 million over 5 years. Under the Canadian Breast Cancer Screening component of this initiative, Health Canada enabled a federal/provincial/territorial network to collaborate in the implementation and evaluation of breast cancer screening programs in Canada.

Following the November 1993 National Forum on Breast Cancer, membership of the National Committee on Breast Cancer Screening was expanded, and the group became formally known as the National Committee for the Canadian Breast Cancer Screening Initiative (CBCSI). Its activities included fostering the development of high-quality, organized breast cancer screening programs in Canada with the following essential components: a population-based outcome goal; information about the target population; emphasis on hard-to-reach groups; meticulous quality assurance; outcome data and analysis; information systems and linkages; a woman-centred focus; and excellent coordination with high-quality diagnosis and follow-up. The Canadian Breast Cancer Screening Database, derived from provincial breast screening program data, was developed through a collaborative effort of the National Committee, in 1993. The National Committee for the CBCSI continues its work today as a component of Phase II (1998-2003) of the CBCI (see Future Directions).

The first screening program began in British Columbia in 1988, and programs have since been established in all provinces and in the Yukon and the Northwest Territories. Breast cancer screening in all organized programs includes a bilateral two-view screening mammogram. Manitoba, Ontario, and Newfoundland also provide a clinical breast examination (CBE) carried out by a trained health professional, and Nova Scotia and Prince Edward Island provide a modified CBE carried out by a technologist. In addition, most programs provide information and/or instruction on breast self-examination.

In each province or territory, women of the target age are recruited to the screening program through a letter of invitation from the program, physician referral, or self-referral (except in Quebec for the latter). At the screening facility, which may be a mobile unit or a fixed centre, women receive bilateral two-view mammography. In addition to mammography screening, they undergo clinical breast examination or are encouraged to obtain regular clinical breast exams outside of the program from their family physician.

All programs provide screening results to both the woman and her physician. If the screening result is normal, women are recalled by letter for another routine screen generally after 2 years. However, some women are recalled annually because of their age, mammographic findings, family history, or other factors that vary across programs. Women with abnormal screening results are informed, along with their family physician, of the need for further assessment (which is usually coordinated by the women's physician). Women are also encouraged to consult their physician if symptoms develop in the interval between screening episodes.

Provincial and territorial screening programs obtain information on diagnostic procedures and cancer outcomes through a combination of active follow-up with family physicians, information received from assessment centres, and linkage with provincial medical service databases and cancer registries. Data from the Canadian Breast Cancer Screening Database provide a basis for assessing the performance of organized breast cancer screening in Canada. No comparable data are available from screening activities conducted outside the structure of the screening programs.

History of the Evaluation Indicators Working Group

The necessity for a standardized method of evaluation for all Canadian breast cancer screening programs was first recognized at the 1990 Interchange meeting. This need was identified once more at the 1993 National Forum on Breast Cancer, as well as at the 1997 Workshop on Organized Breast Cancer Screening in Canada¹². With the Canadian Breast Cancer Screening Database newly established, the infrastructure required to formalize a set of performance measures was in place. The CBCSI's Quality Assurance Working Group, Database Technical Subcommittee, and Database Management Sub-Committee began the process of identifying performance and quality measures and indicators to fulfill these past recommendations. In order to effectively devote time and human resources to the project, the Evaluation Indicators Working Group (EIWG) was formed in 1999.

The process of developing evaluation indicators began with the identification of a set of general categories, which were derived from the results of two surveys of the provincial/territorial programs. Ultimately, nine categories were selected: recruitment and retention, client experience, technical aspects, mammography interpretation, diagnostic assessment and diagnosis, treatment, survival and mortality, data quality assurance, and program management. These categories were then assigned performance and quality indicators gathered through a review of national documents from various countries, published research literature, Canadian federal documents, and Canadian provincial/territorial screening program annual reports. The review focused on indicators that were currently available for breast cancer screening programs in publicly funded health care systems. The EIWG selected indicators from the initial findings on the basis of outcomes, pragmatic considerations, and efficiency.

In February 2000, the seven-member working group held a national workshop to assemble a group of knowledgeable stakeholders from the provinces/territories to refine the available indicators and evaluate their applicability in Canada. The efforts of this workshop resulted in 30 core performance and quality indicators, target outcomes for some of these indicators, as well as recommendations on practical means to gather and report these data¹³. Subsequent meetings of the Working Group resulted in the following guidelines for reporting a key set of "performance measures".

Performance Measure Development

In order to achieve reductions in breast cancer mortality and morbidity and to minimize the unwanted effects of screening, the delivery of organized screening must be of high quality. The performance measures and targets presented in this document were selected on the basis of their utility for assessing program progress toward these goals. The 11 performance measures detailed here generally met the following criteria:

• Data for the measure were regularly available.
• Data available for the measure were of high quality.
• Meaningful targets could be defined on an evidentiary basis*.
• Measures and targets would be useful for national comparison.
• Monitoring on an annual basis would be valuable.
• Each measure was widely accepted for use in program evaluation.

*No targets were set for in situ cancer detection rate, given the controversy surrounding the natural history of the condition (see Performance Measures Under Review in Future Directions).

Application

Through its monitoring and reporting role, the Database Management Sub-Committee (DMC) of the National Committee for the CBCSI produces a routine biennial report: Organized Breast Cancer Screening in Canada. The purpose of this report is to provide formal feedback to the programs regarding their relative performance and to assess the national picture. The standardized performance established in this document will serve as a consistent template for reporting progress over time, as well as providing a set of initial targets for programs to strive toward.

Data Sources and Collection

The performance measures are calculated using data from the Canadian Breast Cancer Screening Database (CBCSD) along with routinely available national statistics and population estimates. The CBCSD is a national breast screening surveillance system that permits the monitoring and evaluation of organized breast cancer screening across Canada. Established in 1993, it is operated and maintained through the continued collaboration of the provinces and territories and the Cancer Division at the Centre for Chronic Disease Prevention and Control, Health Canada. The CBCSD currently contains screening information from program inception up to the end of 1998 for the following provinces: British Columbia, Alberta, Saskatchewan, Manitoba, Ontario, Quebec, New Brunswick, Nova Scotia, and Newfoundland. Test data from Prince Edward Island and the Northwest Territories are currently being analyzed. Because the Yukon does not have a computerized information system, its data are not currently available to the CBCSD. Through the Canadian Breast Cancer Screening Initiative, the CBCSD is managed by the Database Management Sub-Committee and implemented by the Database Technical Subcommittee.

The monitoring of screening programs requires reliable, standardized information that is comparable across provinces. Some follow-up data must be obtained from external sources, and this complicates the evaluation process. Many, but not all, programs are directly linked to their provincial cancer registries so that cancer outcome data can be obtained. Further complicating the evaluation process is that some programs experience delays in obtaining registry data. In addition, analyses have suggested that prognostic data vary from one program to another because of the different ways in which breast tumours are assessed and staged. This must be taken into account when the results of the performance measures across programs are integrated and compared.

Context of Performance Measures

For the purposes of these guidelines for reporting performance measures, the target population for evaluation is the same as the national target population for organized screening. This population is defined as asymptomatic women between the ages of 50 and 69 years with no prior diagnosis of breast cancer.

The targets and standards established in this document are intended to apply to the programs' target group as a whole. It is recognized, however, that for some evaluation purposes it may be appropriate to further stratify the target group in terms of demographic characteristics, screening history, or referral of abnormal result by modality. When measures are used for comparison among Canadian programs or with programs in other countries, it is necessary to age-standardize the results using the appropriate population as the standard.

Five programs (Manitoba, Ontario, Nova Scotia, Prince Edward Island, and Newfoundland) offer clinical breast examination (CBE) in addition to mammography. A special comparison must be made of programs that offer CBE and those offering mammograpy alone. CBE in addition to mammography may confer an advantage in that some palpable breast abnormalities go undetected with mammography. Consequently, programs providing both screening methods may have higher abnormal call rates, making comparisons and the computation of meaningful national figures difficult. An independent research effort to evaluate programs offering CBE is currently under way.

Many of the performance measures presented here provide meaningful measures of program progress only when considered in a broader context. In some cases, meeting ideal targets involves achieving a balance rather than continually working to increase or decrease a particular rate or measure. For example, while increased participation and retention will always be desirable, targets for measures such as positive predictive value and biopsy yield ratio are set with the realization that we must tolerate some false-positive results in order to maximize cancer detection. At the same time, performance measures and targets are not necessarily meaningful on their own, and must be considered in relation to each other and (in some circumstances) in relation to other relevant data. For instance, the cancer detection rate must be considered in relation to the underlying cancer incidence rate in the general population before programmatic screening was implemented. An illustration to clarify the relations among the performance measures is presented in Figure 1.

[Previous] [Table of Contents] [Next]