 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
| |
|
||||||||||||||||
| |
|
||||||||||||||||
|
|
A call for action to support best practices
in evaluation of comprehensive tobacco control evaluation strategies Abstract The National Tobacco Control Best Practices Working Group convened a two-day workshop to support best practices in evaluation of comprehensive tobacco control strategies. A Better Practices Model, aimed at developing a self-correcting system for best practices, guided the workshop content and process. Organizers intended to identify a common surveillance and monitoring framework for tobacco control strategies in Canada by first building strong working relationships between 43 decision-makers, practitioners and researchers from 12 Canadian jurisdictions. Participants identified needs and recommendations related to increased understanding and use of uniform evaluation strategies, building capacity, and recognition of the complexity of the task of evaluating comprehensive tobacco control strategies. The workshop highlighted the need for increased communication to facilitate understanding across the different sectors of participants. It also identified the potential benefits of harmonization in evaluation of tobacco control strategies across jurisdictions. Priority actions include forming a national team to agree on a model for evaluation, conducting an environmental scan for indicators, planning evaluation/monitoring and research agendas and determining roles for various stakeholders. Key words: best practices; comprehensive tobacco control; evidence-based medicine; prevention and control; program evaluation; smoking Introduction Effective public health practice requires informed decisions on the best possible actions to take in tackling complex health problems. These actions - or “best practices” - are “those programs and policies of research and interventions that will have the greatest impact on reducing the current and future burden of disease”.1 This paper describes a workshop held in Toronto on March 25-26, 2002 to support best practices in evaluation of comprehensive tobacco control strategies. The Workshop was convened by the National Tobacco Control Best Practices Working Group* (Working Group), a collaboration of Canadian organizations committed to the identification and implementation of best practices for tobacco control programs and policies. Tobacco control is an appropriate focus since the potential health impact is great, and 30+ years of tobacco control practice and research have resulted in an extensive evidence base. Nonetheless, the concepts and processes guiding the Working Group could have equal application to other complex health problems such as sedentary lifestyles and poor nutrition. Context for the workshop A model for identifying, implementing and evaluating better practices The Working Group employs a better practices model2 that specifies three phases: Phase 1: identify recommended practices and tools (review of what we know) Phase 2: disseminate, implement and evaluate effectively Phase 3: use these results to inform future practice Phase 1 takes advantage of the present knowledge base to identify a set of recommendations and tools in a particular topic area. To achieve this end, the model requires the identification of a clear question and filtering of evidence on the question through a scientific lens. Subsequent critical steps within Phase 1 involve the filtering of evidence through a lens reflecting practical experience, including plausibility and expert advice. Phase 1 concludes with the creation of a toolkit that may consist of program or policy interventions, guidelines for new research, or some combination of the two. The belief that both scientific and practice lenses are necessary to identify best practice distinguishes this approach. Most other frameworks have limited their search for optimum interventions to those scientifically validated,3,4 without further consideration of a context based on practice. Steps in Phase 2 require that the dissemination, adoption and implementation processes encourage understanding of the way recommendations fit the new context in which they will be applied. There is, however, a fallacy in assuming that we can transfer recommendations to a new context or time.5 Therefore, the final step in Phase 2 helps users evaluate the extent to which they have achieved a best practice. Phase 3 directs users to incorporate the results of their evaluation back into the planning of future interventions and research (i.e., Phase 1). In doing so, users will create an iterative, self-correcting system. The model uses the term “better practices” rather than “best practices” because recommendations cannot be regarded as permanent, universal gold standards, given that the context constantly changes. The workshop was intended to develop one component of such a self-correcting better practices system on a national scale. That is, organizers wanted to identify a common surveillance and monitoring framework for comprehensive tobacco control strategies in Canada, and encourage effective adaptation of the framework for use in each jurisdiction. At present, Canada lacks a common framework. Other jurisdictions have developed such frameworks (e.g., Massachusetts,6 California,7,8 and Arizona.9 The United States devoted efforts to the establishment of comparability across states through the ASSIST program.10 ASSIST facilitated use of the framework through technical assistance and training. If the 14 Canadian jurisdictions (one national, 10 provincial, three territorial) adopt a common surveillance and monitoring framework, decision-makers could take advantage of the natural experiments across jurisdictions and compare experiences to improve practices. Ongoing surveillance and monitoring of the national, provincial and regional strategies are key to helping direct the types of data needed to facilitate evidence-based decision-making related to the objectives at each jurisdictional level. Linkage of objectives and evaluation helps to ensure outcomes are achieved and dollars are spent wisely.11 Need for evaluation of comprehensive tobacco control strategies Tobacco remains the primary preventable public health concern in Canada,12-16 with epidemiological evidence of its devastation growing steadily.14 While tobacco is most commonly linked with lung cancer, it is associated with a variety of other cancers, cardiovascular disease, COPD, and diabetes as well.17 This also leads to severe economic and social consequences.18-20 In response, government and non-government organizations have directed systematic efforts toward tobacco control. While the prevalence of tobacco use has abated somewhat with 22% of adults smoking,21 tobacco control remains a priority health issue. Evidence accumulated over four decades points to the need for comprehensive tobacco control efforts. Jurisdictions that have demonstrated reductions in population-level smoking rates (e.g., California, Florida, Massachusetts) have employed comprehensive, co-ordinated strategies.7,22,23 Similarly, comprehensive interventions have shifted population health patterns substantially for a variety of diseases (e.g., CVD24 cancer25). While other countries' experiences can inform Canadian practice, our earlier review of the Better Practices Model2 indicates that any strategies will require adaptation, and subsequent evaluation, in Canadian settings. Principles guiding efforts to identify and implement better practices for evaluation of comprehensive tobacco control strategies The collaboration of researchers and potential users of recommended practices is critical to the process of using the Better Practices model. While their respective expertise in science and practice contributes to an improved set of recommendations, this collaboration is not necessarily simple. The Working Group applied its own “better practices” principles, using the Communities of Practice model26,27 to guide efforts to build collaboration. Communities of Practice (CoP) consist of groups of individuals with a common purpose, regularly interacting to develop shared understandings and practices.26-28 Because collaborators attending the workshop came from varied backgrounds (e.g., Nova Scotia and British Columbia; research and practice), the CoP model suggests mutual engagement could help develop trust among participants. Only when collaborators established trusting relationships could the community negotiate a common purpose and hold each other accountable to what this purpose means. With time, collaborations develop a shared way of describing and acting on these purposes. The Working Group hoped to help build a CoP between researchers and decision-makers to develop a common framework for evaluation of comprehensive tobacco control strategies. The Working Group chose a workshop, a face-to-face format, to engage potential collaborators. The Group recognized, however, that the resulting CoP would require some form of electronic communication such as telephone, e-mail and the Internet to further develop and maintain its activities. While geographically dispersed CoPs in large businesses often rely on electronic communication, they benefit from initial face-to-face interactions to establish personal relationships that build trust and confidence between CoP members.29 Dispersed CoPs are more successful when members share similar values, codes, and stories.30,31 Background work Prior to the workshop, background research was completed on existing evaluation strategies, frameworks and resources used in the United States. The workshop was intended to share this information with key leaders in Canadian provinces and territories. Overall, the workshop strove to achieve the best results from these evaluation frameworks and to share them across jurisdictions. This would help decision-makers adjust tobacco control programs using evidence of effectiveness. The vision for the workshop was to
The objectives of the workshop were to
Summary of workshop proceedings The 43 participants included 11 public health practitioners (e.g., provincial voluntary organization tobacco control practitioners), 15 decision-makers (e.g., provincial tobacco strategy coordinators), and 17 researchers representing 10 provinces, one territory and the federal perspective. On Day One, discussion centred around five domains from the background work that was identified as the potential basis of an evaluation framework or logic model. These domains could be used to map inputs and outcomes of: capacity and resources for tobacco control; policy efforts to control tobacco; program efforts to control tobacco; research, monitoring and evaluation of tobacco control; and collaborative partnerships. Discussion covered a broad range of issues, including recommendations for improving current approaches, indicators that would demonstrate the success of activities in each of the domains, and challenges and barriers to demonstrating success. On the second day, discussions focused on the evaluation of activities addressing three of the goals of the National Strategy† (prevention, protection, and cessation. De-normalization, the fourth goal, was not addressed.) and what is needed to implement the evaluation of comprehensive programs addressing multiple objectives. Through a series of small group and plenary exercises, participants identified evaluation needs and strategies from the perspectives of different jurisdictions, decision-makers, practitioners and researchers. These needs centred around three themes: increasing understanding and use of uniform evaluation strategies; building capacity through financial and policy support and tools that facilitate evaluation; and developing strategies that address the complexities of implementing evaluations. The three themes may be summarized as follows: Understanding and use of evaluation strategies
Capacity
Complexity
Recommendations These major recommendations emerged from the identification of these needs
Workshop participants identified the following priority actions to support the development of a framework for evaluating comprehensive strategies:
Discussion The workshop results demonstrate the need to develop actions to facilitate the evaluation of comprehensive tobacco control strategies in Canada and a national system to support best practices in this area. This will require particular attention to the complexity of such efforts, the additional capacity, in terms of resources and tools, and the improved sharing and co-ordination of evaluation. Workshop participants identified recommendations for mechanisms and products to address these needs. The need for, and difficulty in achieving, effective knowledge translation was a common theme throughout the deliberations. For example, participants from the program/decision-maker sector tended to use different language than the research/evaluation sector when referring to similar concepts. The importance placed on concepts also varied by sector. The workshop started a dialogue, but to achieve Wenger's concept of a community of practice with shared understanding and agreement across these different sectors, jurisdictions and organisations will require more interaction among key stakeholders. Greater information exchange (e.g., of logic models) among participants, prior to the workshop, would have aided understanding at the workshop. To pursue the results of the workshop, participants are using various forms of electronic networking. These include QuickPlace™ web-based software to permit creation of member profiles and facilitate development and sharing of expertise and practical tools, such as surveys and protocols related to comprehensive tobacco control evaluation. The workshop also highlighted the lack of guidelines for the evaluation of comprehensive tobacco control strategies. Participants agreed that adopting a common evaluation framework across jurisdictions would lend itself to many benefits. Such coordination would, for example, capitalize on the natural experiments occurring when one or more provinces implement particular policy changes, while others do not. It would also make possible economies of scale, in which tools and protocols developed in one jurisdiction are shared with others. The Working Group anticipates that the actions it identified as priority will contribute to a framework for evaluation, including a statement of purpose, definitions and timelines for interventions, and a standard minimal set of data collected in all evaluations of comprehensive strategies. These actions will also contribute to an update of an overview of existing efforts to monitor and evaluate, a template to compare provincial level best practices and to evaluate evidence that is sensitive to the content and form of decision-makers' needs. Conclusions This workshop made significant progress in building relationships among researchers, practitioners and decision-makers within and between territorial, provincial and federal jurisdictions. In doing so, the workshop achieved its objectives but the tobacco control community faces considerable work to achieve its vision. Workshop participants established, in broad terms, the information required from the evaluation of core elements/interventions to support ongoing decision-making. They expressed strong interest in developing common indicators. Further steps following the workshop will further develop processes to streamline evaluation across jurisdictions and support more efficient information gathering and decision-making. The National Tobacco Control Best Practices Working Group is pursuing funding and other resources to implement the recommendations. Acknowledgements We acknowledge the organizations funding the workshop: Health Canada, the Canadian Tobacco Control Research Initiative, and the Centre for Behavioural Research and Program Evaluation. The workshop would not have been possible without the full and enthusiastic involvement of the participants and the input of the planning work group. References 1. Moyer C, Garcia J, Cameron R, Maule C. Identifying Promising Solutions for Complex Health Problems: Model for a Better Practices process. 11-10-2002. Unpublished Work 2. Maule C, Moyer C, Manske S, Cameron R, Finkle D. A CCS Approach to Better Practices: Learning from the Past, Acting Sensibly in the Present and Contributing to the Wisdom of the Future. 2002. Toronto, ON, Canadian Cancer Society. 3. Agency for Healthcare Research and Quality. Systems to Rate the Strength Of Scientific Evidence. File Inventory, Evidence Report/Technology Assessment Number 47.AHRQ Publication No.02-E016. 2002. Rockville, MD, Agency for Health Care Quality. 7-11-2002. 4. Cook D, Guyatt G, Ryan G. Should unpublished data be included in meta-analyses? current convictions and controversies JAMA 1993; 269:2749-2753. 5. Green L.W. From Research to “Best Practices” in Other Settings and Populations. American Journal of Health Behavior 2001; 25(3):165-178. 6. Celebucki C, Biener L, Koh H. Evaluation: Methods and strategy for evaluation - Massachusetts. Cancer 1998; 83(Supplement 12A):2760-2765. 7. Independent Evaluation Consortium. Interim report. Independent evaluation of the California Tobacco Control Prevention and Education Program: Wave 2 data, 1998; wave 1 & wave 2 data comparisons, 1996-1998. 2001. Rockville, MD, The Gallup Organization. 8. Russell C. Evaluation: Methods and strategy for evaluation - California. Cancer 1998; 83(Supplement 12A):2755-2759. 9. Loeb J. Evaluation: Methods and strategy for evaluation - Arizona. Cancer 1998; 83(Supplement 12A):1766-2769. 10. Stillman F, Hartman A, Graubard B, Gilpin E, Chavis D, Garcia J, Wun L, Lynn W, Manley M. The American Stop Smoking Intervention Study: Conceptual framework and evaluation design. Eval Rev 1999; 23(3):259-280. 11. Health Canada. A Network for Health Surveillance in Canada. Cat. No. H39-497/1999. 1999. Ottawa, Minister of Public Works and Government Services. 12. Advisory Committee on Population Health. Report on the Health of Canadians. 1996. Ottawa, Health Canada. 13. Ellison LF, Mao Y, Gibbons L. Projected smoking-attributable mortality in Canada, 1991-2000. Chronic Dis Can 1995;16:84-89. 14. Holowaty E, Cheong S, Di Cori S, Garcia J, Luk R, Lyons C et al. Tobacco or Health in Ontario: Tobacco-attributed cancers and deaths over the past 50 years … and the next 50. 1-5-2002. Toronto, ON, Cancer Care Ontario, Division of Preventive Oncology. 15. Illing EM, Kaiserman M. Mortality attributable to tobacco use in Canada and its regions. Can J Public Health 1995; 86:257-265. 16. Schabas R. Tobacco: Sounding the Alarm. 1-15. 1-1-1996. Queen's Printer of Ontario. 17. World Health Organization. World Health Report 2002. 2002. Geneva, Switzerland. 18. Conference Board of Canada. Smoking and the bottom line: the costs of smoking in the workplace. 1997. Ottawa, Health Canada. 19. Kaiserman M. The Cost of Smoking in Canada 1991. Chronic Dis Can 1997; 18:13-19. 20. Stephens T, Kaiserman M, McCall D, Sutherland-Brown C. Preventing smoking in Canada: economic costs and benefits. 35. 1998. Toronto, Ontario Tobacco Research Unit. Working Paper Series. 21. Health Canada. Canadian Tobacco Use Monitoring Survey: Annual Results 2001. 2002. 22. Abt Associate Inc. Independent evaluation of the Massachusetts Tobacco Control Program: Seventh annual report. 2002. Boston, MA, Abt Associates Inc. 23. Trapido E. An independent evaluation of Florida's Tobacco Pilot Program: A two- year summary report. 2000. Miami, FL, Tobacco Research and Evaluation Coordinating Center, University of Miami. 24. National Public Health Institute (Finland). http://www.ktl.fi/portal/english/. [accessed 9-20-2002]. 25. Puska P, Tuomileto J, Nissinen A, Vartiainen E, editors. The North Karelia Project: 20 year results and experiences 1995. Helsinki, Finland, National Public Health Institute. 26. Lave J, Wenger E. Situated Learning: Legitimate Peripheral Participation. New York: Cambridge University Press, 1991. 27. Wenger E. Communities of Practice: Learning, Meaning and Identity. New York: Cambridge University Press, 1998. 28. Lesser EL, Storck J. Communities of practice and organizational performance. IBM Systems Journal 2001; 40(4):831-841. 29. Hildreth P, Kimble C, Wright P. Communities of practice in the distributed international environment. Journal of Knowledge Management 2000; 4(1):27-38. 30. McLure M, Faraj S. “It is what one does”: Why people participate and help others in electronic communities of practice. Journal of Strategic Information Systems 2000; 9:155-173. 31. Owen C, Pollard J, Kilpatrick S, Rumley D. Electronic learning communities? Factors that enhance and inhibit learning within e-mail discussion group.: Conference Proceedings: Learning Communities, Regional Sustainability and the Learning Society, 1998. Author References Steve Manske, Centre for Behavioural Research and Program Evaluation, University of Waterloo, Waterloo, Ontario, Canada Catherine Maule, Canadian Tobacco Control Research Initiative, Toronto, Ontario, Canada Shawn O'Connor, Ontario Tobacco Research Unit, Toronto, Ontario, Canada Chris Lovato, Department of Health Care and Clinical Epidemiology, University of British Columbia, Vancouver, British Columbia, Canada and Centre for Behavioural Research and Program Evaluation, University of Waterloo, Waterloo, Ontario, Canada Dexter Harvey, Department of Education, University of Manitoba, Winnipeg, Manitoba, Canada Correspondence: Steve Manske, Research Associate, Centre for Behavioural Research and Program Evaluation LHI, University of Waterloo, Waterloo, Ontario, Canada N2L 3G1; Fax: (519) 886-6424; E-mail: manske@healthy.uwaterloo.ca * Member organizations at the time of the workshop were: the Canadian Council on Tobacco Control (CCTC); the Canadian Tobacco Control Research Initiative (CTCRI); the Centre for Behavioural Research and Program Evaluation (CBRPE); and Health Canada. Health professionals invited to assist with workshop development were drawn from Health Canada, Ontario Tobacco Research Unit, University of British Columbia and University of Manitoba. Funding for the workshop was provided by CBRPE, CTCRI, and Health Canada. † Steering Committee of the National Strategy to Reduce Tobacco Use in Canada. New Directions for Tobacco Control in Canada: A National Strategy. Public Works and Government Services of Canada (1999). ‡ Based on the experience of California and on information from the US Centers for Disease Control and Prevention, a reasonable estimate may be 10% for evaluation of individual community interventions, and an additional 10% for evaluation of the overall comprehensive program.
|
 |
|||
|
Last Updated: 2004-07-29 |  |
|
