Government of Canada
Symbol of the Government of Canada

Summative Evaluation of the Canadian Shellfish Sanitation Program (CSSP)

3. Conclusions

This section of the evaluation report offers several broad conclusions arising from the detailed evaluation findings provided in the previous section. Conclusions are provided for each evaluation objective.

3.1 Program Outcomes

Conclusion 1.
In areas where the Canadian Shellfish Sanitation Program (CSSP) is operational, the CSSP is providing reasonable assurance that commercially harvested shellfish are safe for human consumption.

While there are no agreed upon outcomes or performance indicators for assessing the CSSP success or results, the evaluation concludes that the CSSP is meeting its intent as articulated in program documentation. This conclusion reflects the findings that federal partners are conducting their activities in accordance with the CSSP MOU and the Operations Manual that lays out the requirements for program delivery.

Key evidence supporting this conclusion includes the facts that international trading partners who have conducted independent reviews and audits of the CSSP have continued to keep their markets open to Canadian shellfish products; and Canadian consumers continue to have a high level of confidence that shellfish products are safe for consumption.

However, it is noted that the program is not designed to provide 100% assurance against incidents of shellfish poisoning. Contaminated shellfish have the potential to enter the domestic market because the current CSSP design is intended to provide reasonable assurance that all harvested and consumed shellfish (especially for recreational, subsistence, ceremonial or domestic consumption) are harvested from classified growing areas or processed through certified processing facilities.

It is important to note that the expected outcomes for the program have evolved over time and presently there is some lack of clarity on the extent to which the program is intended to be for food safety or to support commercial trade. While program staff and documentation indicates that the CSSP's purpose is primarily for food safety, the program design and delivery indicates that it is being implemented in a manner more consistent with supporting commercial trade (e.g., the program is not applied equally for all harvesters - commercial, recreational, Aboriginal). Although these two program objectives (food safety and commercial trade) are not in themselves incompatible, the focus needs to be clarified as it affects the design and delivery of the program, as well the priorities for addressing ongoing operational decisions, as evidenced by the current program design and priorities.

3.2 Pressures, Risks and Liabilities

Conclusion 2.
The risks and liabilities faced by the Government of Canada in managing the CSSP are increasing, including risk impacts related to public health, economic opportunity and potential government of Canada liabilities.

The CSSP is currently designed to address the potential risks associated with the harvesting, processing, trade and consumption of shellfish food products. The design of the program has evolved over an extended period of time in response to specific incidents and changing requirements to put a number of processes in place to manage risks. However, there has not been a systematic assessment of all the specific risks to the overall program. While the federal partners implement their activities within some formal/informal risk based frameworks, the CSSP has not systematically assessed and identified a risk management strategy for the entire program design.

Risk Drivers
Key Risk Drivers (i.e. external influences) are growing and domestic food safety concerns are now an increasing priority for the program. In addition, the following drivers are posing increasing risks for the program:

Standards: International standards for food safety, and shellfish products in particular, are increasing. These standards include international requirements as prescribed by CODEX, and operational standards required by trading partners if Canadian shellfish products are to maintain access to markets.

Coastal development: In most coastal areas, increasing commercial, industrial, residential and urban development is placing increased pressure on marine environments, resulting from discharge of contaminants to the environment. Shellfish growing areas are sensitive to such impacts, leading to larger areas being closed to harvesting.

New bacteria, biotoxins and viruses: Over the past few years, sampling has detected species of concern previously not encountered in Canadian waters. The cause of these new occurrences is not fully understood, but there is potential for new species to occur again and they do pose an increased risk to Canadian shellfish.

Growth of industry: Increased growth of the Canadian shellfish industry is putting pressure on the CSSP to expand its area of coverage. This is particularly acute for the aquaculture industry that has grown between 15-20% per year over the past decade. The ability of the CSSP to respond to this increase has been constrained due to the lack of new resources.

Pressure for geographic expansion: The CSSP is well established in many traditional shellfish growing and harvesting areas. However there is a lack of program coverage in many remote areas and areas where commercial harvesting has not occurred in the past. Expansion of commercial activities into new areas and concerns over food safety for recreational and Aboriginal harvesters who are operating in areas not covered by the program is putting pressure on the program to expand into areas not currently covered.

Risks
Examples of identified risks that could occur as result of the drivers identified above are provided below. This list is not inclusive as it is only based on the information provided during this evaluation, and as a systematic process of identifying risks related to the CSSP has not been undertaken.

Strategic Risks

  • Economic Growth: Failure to respond to the increased demand for program coverage in areas with new commercial harvesting (aquaculture and wild stock) could result in lost economic opportunity. These risks would impact smaller remote communities where opportunities for economic development may be limited, and commercial shellfish harvesting may provide one of the few opportunities to support local community economic development.
  • Reputation: The Government of Canada is seen to be responsible for delivery of the CSSP. Failure of the program to prevent serious illness or death from consumption of contaminated shellfish, could result in a negative impact to the government and partner department's reputations.
  • Consumer Confidence: In the event of a serious incident of shellfish poisoning, there is a risk that confidence in the shellfish industry will fall resulting in significant market/economic losses.

Operational Risks

  • Coverage: Failure of the program to provide coverage in all geographic areas and for all shellfish harvesters means that those consuming shellfish products from areas not covered by the program face a higher level of risk than those consuming product from areas covered by the program.
  • Science: Shellfish food safety related issues depend on an understanding of the complex microbiology underlying the program design and delivery. As the scientific basis for identifying, assessing, and setting standards for shellfish related food safety increase, Canada needs to ensure that the basis for CSSP operations continues to address the needs. The CSSP needs to have the scientific capacity to address in a pro-active way, the emerging scientific and technical issues associated with the program. There is a risk that the currently under funded research and scientific effort for shellfish will result in an inadequate response by the program to deal with emerging issues.
  • Funding: There are limited financial resources within the three partner departments and relatively stable funding levels for CSSP delivery over the past few years. As a result, there has been no financial capacity to increase activities associated with program demands. In the face of increasing external pressures, there is a risk that key components of the program will not be addressed. This is evident in areas where program deficiencies are identified and the costs of corrective action require additional funding (e.g. response to the most recent USFDA audit recommendations).

Delivery Risks

  • Illegal harvest: The CSSP assesses and classifies shellfish growing areas, and places restrictions on the harvesting of shellfish from closed areas, and to the extent possible is taking appropriate action to enforce closures. Where illegal harvesting is known to occur, additional enforcement effort is directed to addressing the issue. However, due to the large geographic and dispersed areas where shellfish are potentially harvested, it is not possible to conduct a sufficient level of enforcement patrols and public awareness or outreach activities that may be needed to fully mitigate this risk.
  • Product not caught by the system: The marine environment monitoring and classification of growing areas component of the CSSP program provides some level of protection for consumers in most areas. However, if the harvested product is not intended for inter-provincial trade or export, federal requirements for processing facilities do not apply, and the requirements associated with intra-provincial trade varies from province to province. As a result, there is some shellfish product that reaches the consumer that has not fallen within the purview of the program.

Risk Impacts
As a result of the risks identified above, potential impacts were identified in three key areas. The relative nature or scopes of these risks have not been assessed, but they are potentially significant.

  • Health: There is a potential for risks of shellfish poisoning or death as a result of consumption of contaminated shellfish food products.
  • Economy: There is potential for lost economic opportunity due to the inability of the program to meet the needs of an expanding commercial shellfish industry.
  • Liability: There is potential for Government of Canada liability as a result of incomplete program coverage.

3.3 Governance, Management and Coordination

Conclusion 3.
The current CSSP governance/administration/coordination structures and processes are effective in providing for delivery of the program at the operational level. However, the governance structure is inadequate in ensuring delivery of a cohesive and integrated program across the three federal partners' organizations.

Program Implementation
The CSSP is delivered through the regions, is focussed on operational matters as defined in the Operations Manual, and generally works well at the operational level. Partners deliver their respective responsibilities within their respective mandates and resources.

Planning and Coordination
The NISC and RISCs function as program coordinating mechanisms, however they are limited in their ability to set strategic directions and guidance for the program. There is no agreed upon vision or performance targets for the program and there are no integrated annual plans.

Reporting and Management Review
There is no consistent performance framework against which partners report on their activities, including the lack of financial or cost data. Although it was done in the past, there is currently no internal audit, review or performance verification procedure in place to assess conformance with requirements.

Governance, Policy and Oversight
The MOU provides overall guidance for the activities of each federal partner. However, it is out of date and does not fully reflect the way the program is currently managed. The nature of CFIA's "leadership" for CSSP has changed, but this role has not been clearly defined and is not reflected in the MOU. It was also noted that there is no mechanism in place to set priorities for the program, make program wide resource allocation decisions, or to hold partners accountable for performance and delivery. Other key issues requiring clearer guidance or direction include the engagement and cooperative working arrangements with industry and provincial stakeholders, and decisions on program scope or reach.

Alternative Governance Models
Ranges of program delivery alternatives were reviewed during the course of the evaluation. In considering the history of the program and the mandates of the current federal partners, it is recognized that the CSSP requires the involvement of a number of federal partners with shared responsibility for the relevant activities. The current delivery model has recently emerged following the establishment of CFIA and the transfer of some resources and responsibilities to the new agency.

Along with changes in organization, there are emerging and increasing challenges that need to be addressed within the governance model. The governance framework for the program needs be strengthened to ensure those new roles and new needs are addressed. Since the program is working reasonably well, a radical re-adjustment of the current approach is not required but rather, a re-focusing of management efforts on those issues and matters that will result in the program becoming more than the sum of the individual activities and contributions by the three partners. The management framework needs to be strengthened to allow it to focus on addressing the gaps and weaknesses identified in this report.

3.4 Program Delivery

Conclusion 4.
The dispersed program delivery structure has resulted in inconsistent implementation across the regions and departments, and difficulty establishing effective national policies, coordinated work plans and procedures to guide the program.

The CSSP is currently delivered through a regional program structure and within the organizations of the respective federal partners. The structure varies between the regions and within the departments and agency. Although the program is delivered in a manner that meets the requirements of the Manual of Operations, as a result of the organizational and regional differences, there is some inconsistency in the approach to delivery across the country. This results in resources being allocated or deployed on the basis of regional, and not program wide, needs and priorities.

The roles and responsibilities of the partners are clear, complementary, well defined and understood. The activities of the partners are also consistent with their respective organizational mandates. However, the dispersed program delivery structure has not allowed for effective development of national policies in response to changing circumstances, has difficulty in providing strategic direction or guidance, and does not lead to coordinated program work plans.

It is also difficult to determine the cost of delivering the CSSP. The overall cost is estimated at approximately $14-15 million, but this is a calculated value based on some actual costs as well a detailed analysis of program activities. There is no mechanism in place to plan, allocate, track, and report on the cost of program delivery.

4. Recommendations

The evaluation found that the context in which the CSSP operates continues to evolve, and the risk drivers associated with the program are increasing. Funds available for program delivery have not been increasing and, as a result, there is a need to improve the analysis and rationalize resource allocations. This effort has recently been challenged due to the structure for delivery being split between the three federal partners. With an increased awareness of the food safety mandate of the program, there is a need to identify and implement changes to the program. The current approach is placing constraints on effecting that change, and the following recommendations are provided with the purpose of enabling the government to increase the responsiveness of the CSSP to these emerging challenges.

4.1 Governance

The evaluation has identified a number of areas where improvements could be made in program governance. These were primarily in the cross cutting areas of program management and coordination, as opposed to operational delivery. Some of the impacts are reflected in operational issues, such as inconsistency of delivery across the regions, or differing levels of effort in and across the regions and programs. However these inconsistencies could be reduced through stronger and more responsive policy direction. This will require stronger and pro-active leadership and program coordination then currently exists.

There are also a number of issues or activities that require a more focussed approach, such as addressing enquiries from international trading partners, dealing with international standards and guidelines (e.g. CODEX), coordination of efforts with the provinces, and putting in place appropriate accountability mechanisms to ensure the program is being implemented as intended. To address these needs, we make the following recommendation.

Recommendation 1

The CSSP ADMs Committee should establish a stronger governance framework for the program that consists of:

  • A strengthened role for the ADM Steering Committee that includes clear CFIA leadership, that makes recommendations on resource allocations, approves documented annual CSSP work plans, clarifies policy direction (including scope and reach), oversees implementation, and holds partners accountable for performance;
  • A CSSP Secretariat that includes dedicated CSSP staff and is established within CFIA. The Secretariat should support the ADM Steering Committee, coordinate program audits, identify research needs, coordinate RISCs and NISC, produce annual CSSP work plans and performance reports, develop and maintain operating standards and guidelines, and manage a stakeholder advisory board. (Secretariat staff would not have responsibilities related to internal agency or departmental coordination and activities, they would be dedicated to horizontal program coordination and support); and
  • A Stakeholder Advisory Board that engages provinces and key stakeholders, and reports regularly to the ADM Steering Committee.

(High Priority - within 6 months)

4.2 Program Risks

In order to ensure that the CSSP remains relevant and effective, there is a need to undertake a systematic assessment of the risks associated with shellfish food products. This risk assessment should be completed on a priority basis in order to inform development of the program policy and scope statement. The risk drivers identified in this evaluation could be a starting point for such an analysis, but should also take into consideration the experiences of other jurisdictions involved with managing shellfish risks, and the views of other experts and scientific specialists.

Such a risk assessment would provide ongoing benefits for setting program priorities.

Recommendation 2

The proposed CSSP Secretariat should undertake a program wide risk assessment, develop a risk reduction strategy, and implement a plan to ensure the program activities and resources are directed to those areas of greatest risk.

(High Priority - within 6 months)

4.3 Program Scope and Policy

The evaluation found that there are an increasing number of risk drivers that are putting pressure on the program. These pressures are difficult to address under the current design and policy framework within which the program is delivered. Operational staff are required to make day to day decisions within the mandates of their respective organizations, and within the resources and operational guidance in place for the program. With the lack of clarity around program priorities, some inconsistencies in program delivery have crept into program delivery across the country. Also, with increasing internal resource constraints and external pressures, it is becoming more difficult to meet client group expectations for increased coverage or service from the program.

Recent efforts to redesign the program have also been stalled due to the difficulty in defining exactly what the program is intended to cover or address. In order to move forward, it is critical that the policy framework for the program be clarified. This will require a thorough analysis of the government's legal and other obligations with respect to the shellfish program, as well as development of a clear vision or directional guidance document for the program.

Recommendation 3

The CSSP ADMs Committee should redefine the policy and scope, including vision and guidance framework, for the program, taking into consideration the following factors:

  • The increasing focus on food safety as a priority for the program;
  • The need to clarify the reach of the program, including the extent of coverage intended for different harvesters (commercial, recreational, and Aboriginal);
  • The need to address shortfalls in the awareness and outreach efforts needed for the public and shellfish consumers;
  • The need for stronger cooperation and coordination of effort with the provinces and industry;
  • The need for equitable coverage for a variety of commercial harvesters (wild and aquaculture);
  • The range of geographic coverage, diversity of users and species to be covered by the program; and
  • The economic benefits and opportunities for growth that would benefit all Canadians.

(High priority - within 6 months)

4.4 Program Structure

The CSSP currently is conducted under the guidance of the MOU and the Operations Manual. These documents provide substantive guidance on the operational requirements of the program. The immediate challenges facing the program relate more to the cross cutting management and administrative requirements to ensure the program meets objectives that go beyond the mandates and priorities of the three program partners.

The challenges of managing programs that cut across more then two or three departments and agencies is not unique to the CSSP. Numerous government programs rely on a coordinated effort from a number of partnering departments and agencies. Treasury Board has recognized these challenges, and provided specific guidance to departments and agencies on the preparation of Horizontal Results-based Management and Accountability Framework's (RMAF) to help guide such efforts. In order to address the challenges facing the CSSP, such a horizontal RMAF should be prepared, recognizing that the most significant short term value to be obtained from such an exercise is the review and discussion required to come to a common understanding of how the constituent parts contribute to the achievement of broader program objectives and results.

Recommendation 4

The proposed CSSP Secretariat should develop a horizontal Results-based Management and Accountability Framework for the program

(Medium Priority - within 12 months)

4.5 Research Support

A key challenge in the CSSP is the scientific and biological complexity of the risks involved. Detecting biotoxins or pathogenic organisms, determining the risk they present to shellfish consumers, and putting in place the testing or sampling protocols necessary to reduce those risks is dependant on a high level of scientific capability. Supporting the development of that capacity through research focussed on shellfish is necessary if the program is to maintain its effectiveness. Over the past few years the research effort has dwindled due to resource constraints, and there is a need to redefine the nature of that research effort and determine how it can best be re-focussed on the needs of the program. At present there has been no systematic assessment of what those research needs are, although scientific and technical staff are well aware of many areas of investigation where Canada needs to rely on the work of other countries to stay current with developing science and technology. Additional effort is required to better define the appropriate level of Canadian investment in shellfish related research.

Recommendation 5

The proposed CSSP Secretariat should undertake a research needs analysis and develop a strategy to address the ongoing science-based research needs for shellfish related issues (including new and emerging pathogens and marine biotoxins).

(Medium Priority - within 12 months)

4.6 Performance Tracking and Reporting

In conducting this evaluation, it was evident that there is a significant gap in the information available on resource expenditures and it was difficult to accurately describe the level of activities and the associated resources due to the lack of budgeting and financial tracking capability for CSSP related activities. There is also a gap in the performance assessment framework for the program, hindering the ability to assess performance over any period of time.

Future planning and decision making efforts will require significantly enhanced financial and performance information.

Recommendation 6

The proposed CSSP Secretariat should develop a tracking and reporting system for CSSP costs and performance.

(Medium Priority - within 12 months)

Next page: Appendix A | Previous page: Summative Evaluation of the Canadian Shellfish Sanitation Program