 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
| |
|
||||||||||||||||
| |
|
||||||||||||||||
|
Status Report: Evaluation of a Workshop on Public
Education Messages for Reducing Health Risks from Ultraviolet Radiation
Rosemarie Ramsingh and Christina J Mills
Evaluation of a Workshop on Public Education Messages for Reducing Health Risks from Ultraviolet Radiation Abstract The objective of this paper is to assess the impact of the 1994 Workshop on Public Education Messages for Reducing Health Risks from Ultraviolet Radiation (UVR). The target audience was any organization in Canada doing education on the health risks of UVR. A mailed survey with telephone follow-up was distributed to 130 addresses, including workshop participants, recipients of the workshop report and 40 local public health units. The response rate was 62%. Public health messages from the workshop served as an added impetus or helped to initiate activities around UVR in approximately 40% of organizations over the two years since the workshop. The public health messages were used directly in programming by approximately 38% of all organizations responding. However, looking at those who had previously seen the messages, 61% used them directly in programming. Forty percent of those sampled had never seen a copy of the messages. The results suggest the need for improved dissemination of consensus statements. Key words: Canada; consensus development conference; evaluation studies; program evaluation; ultraviolet rays Introduction In 1994, Health Canada convened a meeting of two dozen national organizations involved in the preparation and/or dissemination of public health messages concerning ultraviolet radiation (UVR) to develop consensus on the core messages for reducing health risks due to UVR.1 The workshop was organized in response to a recommendation from the 1992 national Symposium on Ultraviolet Radiation-related Diseases that there be a national collaborative effort to improve the consistency of messages on UVR being disseminated to the public.2 The goal of the workshop was to translate the scientific conclusions reached at the 1992 Symposium into core message content that could be used by intermediaries in preparing materials for the public. Accordingly, the target audience for the workshop consisted primarily of organizations who would be able to use the consensus messages in their disease prevention or health promotion activities. It was understood that the format, the relative emphasis given specific points and the depth to which each was explored would vary according to the ultimate audience intended for a particular product. In September 1996, as part of the Laboratory Centre for Disease Control's ongoing commitment to evaluating and improving its programs, the organizers initiated a mailed survey to determine the impact of the 1994 workshop on the work of the target organizations. Objectives The objectives of this evaluation were to assess the following.
Methods The target audience consisted of public health practitioners and decision makers at all three levels of government, as well as non-governmental organizations (NGOs) and professional associations involved in developing or disseminating health promotion or disease prevention messages, whether to professionals or to the public. A master list was compiled that included workshop participants, the distribution list for workshop products and additional addresses from requests for information on the workshop or on UVR-related health risks generally. We conducted a mailed survey of the target audience in September 1996. Given the small number of Canadian organizations actively involved in producing and disseminating materials on UVR-related health risks, it was feasible to survey the entire list (90 addresses). In addition, we randomly selected 40 public health units from a list of all such units across Canada, since local public health agencies were not systematically included in the development of the original distribution list. The survey was conducted in either English or French. The survey package consisted of a covering letter, a six-question questionnaire (Appendix) and a copy of the public health messages that were produced from the 1994 UVR workshop. The latter was intended to assist respondents in answering questions on the content of workshop recommendations. Respondents were asked not to look at the messages until they had answered the first two questions identifying whether they were aware of the workshop and whether they had previously seen a copy of the messages. After three weeks, personal reminder telephone calls were made to all subjects for whom a telephone number could be obtained. We used SPSS for Windows to calculate simple descriptive statistics, with limited cross-tabulations. Results After eliminating duplicate addresses (two individuals responding from a single organization), returned questionnaires with no forwarding address and US addresses, the denominator for analysis of response rates was 106. Sixty-six questionnaires were completed and returned by the deadline, for a net response rate of 62%. Respondents and non-respondents had similar distributions for language and type of organization, with the exceptions that local health departments were more heavily represented among respondents and health NGOs were more heavily represented among non-respondents (Table 1). Awareness of the Workshop and Consensus Messages Of the 50 respondents who had not participated directly in the workshop, 25 (50%) indicated that they were aware that the workshop had happened and were aware of or had seen a copy of the public health messages emanating from the 1994 UVR workshop. There was no specific type of organization that stood out as having a higher or lower proportion of awareness compared with the others. Almost all organizations who were aware of the workshop messages had undertaken UVR-related activities in the two years following the workshop (15 of 16 direct participants, 22 of 25 non-participants). Only one organization who had been unaware of the workshop messages had conducted UVR-related activities. Of the 25 respondents who had seen the public health messages from the 1994 workshop but were not direct participants, 17 (68%) had seen them in Chronic Diseases in Canada, 8 had heard about them from colleagues and 5 had encountered them at conferences (respondents could select more than one source). Other sources were organizational newsletters, posters, television, office files and other publications, including the Canadian Medical Association Journal and the Public Health and Epidemiology Report of Ontario. Agreement with Scope and Content of Messages Twenty (30%) respondents felt some messages were missing from the public health messages produced; the proportion was the same for workshop participants and non-participants. Most items could be classified in the following eight areas.
Eighty-two percent (54/66) of respondents stated they had no disagreement with the messages, 11% (7/66) did not respond and 8% (5/66) identified at least one point of disagreement. One of the five in disagreement was a workshop participant. The points of disagreement identified were these.
Usefulness of the Messages Overall, 66% (27/41) of respondents who had seen the messages reported that they had served as an added impetus or helped to initiate program activities related to UVR intheir organizations in the two years after the UVR workshop, but this is a weighted average of 81% (13/16) for workshop participants and 56% (14/25) for non-participants. Similarly, while 63% (26/41) said their organizations had used the messages themselves in some format for UVR-related program activity within the two years following the workshop, this represented 75% of workshop participants and only 56% of non-participants. Table 2 lists the activities related to UVR conducted by respondents' organizations in the two years since the 1994 workshop for which the workshop messages served as added impetus and those in which the messages were used in any format.
|
||
|
Discussion Consensus workshops are a commonly used strategy in public health, and publication of proceedings is a major vehicle for their dissemination. Lomas's review of consensus methods and their impact 3 concluded that in North America, consensus recommendations that have been evaluated have had little impact on the behaviour of the intended target population of practitioners. According to Lomas, successful impact requires that the target audience be already receptive to change and that the information be timely and provided by a credible source. Lomas reviewed six consensus guideline impact evaluations that specifically evaluated objective practice behaviours before and after the consensus process; none of these detected any change. Among four impact evaluation studies of self-reported behaviour change, only one found significant impact.3 In 1987, the US National Institutes of Health consensus guideline process was evaluated for four of its guidelines for clinical practice. The study looked at physician practice through chart review in a before-and-after design and found no impact on physician practice, despite success in reaching the target audience.4 We have looked principally at how those who were aware of the messages used them or were motivated to undertake program activities in education about the health risks of UVR, as an indicator of their quality, relevance and usefulness; 63% of respondents said the messages served as an impetus and were used in their programs, and this represents a substantial impact. However, a better indicator of the net impact of the workshop is the proportion of all respondents reporting program activity motivated by the workshop messages and using them for program purposes (38% for both). Since only 62% of respondents were aware of the messages and thus had an opportunity to use the messages, this reflects the completeness of dissemination more than it does the quality of the recommendations. Although we had no baseline data on the target population for this study, the group who had not hitherto seen the workshop messages might be considered to represent the situation had the workshop not taken place. Only one of these organizations undertook UVR-related activities during the period following the workshop, in contrast to the vast majority of organizations who had been exposed to the workshop messages (94% of direct participants and 88% of others). This could reflect either a truly low level of background activity in UVR education or over-inclusiveness in compiling the survey distribution list. There are at least two possible sources of bias that might lead to an overestimate of the impact of the workshop on the organizations for whom it was intended. First, individuals with a particular interest in and already actively working on UVR risk reduction might be more likely to respond than others, and that possibility must be taken seriously. However, there are no striking differences between responders and non-responders on the limited information provided (Table 1), workshop participants were similarly represented among respondents and non-respondents, and the response rate compares favourably to those found in other impact evaluations of consensus conferences.3,5 Second, some respondents may have overestimated the importance of the workshop in order to meet the perceived expectations of the surveyor or, in the case of workshop participants, because they had a vested interest in demonstrating its impact. We were unable to verify self-reported activities systematically, although we did receive copies of some of the materials produced. It is also possible that some organizations had already produced materials consistent with the messages of the workshop prior to 1994 and therefore would not report an effect. There was relatively little disagreement with the consensus messages developed in the workshop, and some of the few points of disagreement mentioned had to do with issues that were, in fact, beyond the scope of the workshop. For example, it was not the function of the workshop to estimate ground level UVB or to develop a scientific position on the immune effects of UVR, but rather to take the scientific consensus achieved on such issues at the 1992 Symposium and agree on core messages for public education purposes. Although it was stated in the workshop report that the consensus recommendations were for core message content that would have to be tailored to suit specific audiences in terms of format, wording, relative emphasis of particular components, etc., some respondents appeared to want fully elaborated text that they could use "as-is" in their materials (e.g. with exact wording pretested for literacy level). This was not an intended product of the workshop, nor would it be possible to do in a workshop format, given the range and complexity of issues involved and the multiplicity of potential target audiences. However, these comments show that not all organizations with a mandate to disseminate educational materials have the necessary range of expertise at hand, and they suggest a need for tools to assist intermediaries to make the best use of such workshop results. The results of this evaluation suggest that there was an important positive impact from the 1994 UVR workshop on the target population of intermediary organizations, among those who had had an opportunity to see the messages prior to the survey. Of these, 66% said they had used the messages in UVR program activities or that they had helped them in initiating such program activities. However, the net impact of the workshop is best expressed by the proportion of the entire target audience using the messages: 41%. While this is certainly respectable, it suggests the need for greater dissemination efforts. Electronic dissemination may eventually assist us in extending the reach of consensus messages; it was not considered in this evaluation because LCDC's World Wide Web site only became operational after the time of the survey. Lomas suggests that there is greater success in modulating behaviour at a local level, where more careful targeting is feasible than nationally or regionally.3 By analogy, targeting to specific sectors (education, occupational health and safety) could improve dissemination effectiveness. It may also be necessary to develop more detailed interpretive guides to accompany such consensus messages to assist intermediaries in tailoring message content to the needs of their audiences. Conclusions There is evidence that, when they reached the target organizations, the consensus workshop messages were helpful to the respondents and they were used in program activities. It appears there is still much work to be done in ensuring that messages reach a greater proportion of the audience for whom they are intended. We will also need to find ways to assist intermediaries in the most effective use of core consensus messages for public education. Finally, some of the same principles applied by intermediary organizations to target their messages to specific populations for greatest effectiveness could usefully be employed by those charged with the dissemination of consensus recommendations to intermediaries, whether they are organizations or individual professionals. Professional publications and direct distribution of consensus documents will continue to be important and useful dissemination mechanisms, but they are far from sufficient.
Author References Rosemarie Ramsingh, Department of Epidemiology and Community Medicine,
University of Ottawa, Ottawa, Ontario
1. Mills C, Jackson S. Public education messages for reducing health risks from ultraviolet radiation [workshop report]. Chronic Dis Can 1995;16(1):33-6. 2. Gibbons L, Anderson L, eds. Proceedings of the Symposium on Ultraviolet Radiation-related Diseases; 1992 Mar 24-26; Ottawa (Ont). Chronic Dis Can 1992;13(5 Suppl). 3. Lomas J. Words without action? The production, dissemination and impact of consensus recommendations. Annu Rev Public Health 1991;12;41:41-65. 4. Kosecoff J, Kanouse D, Rogers W, McCloskey L, Winslow C, Brook R. Effects of the National Institutes of Health consensus development program on physician practice. JAMA 1987;258:2708-13. 5. Hill M, Levine D, Whelton P. Awareness, use and impact of the 1984 Joint National Committee consensus report on high blood pressure. Am J Public Health 1988;78:1190-4. APPENDIX Questions in evaluation instrument 1. Are you aware there was a multi-organizational (Health Canada, Environment Canada, Canadian Cancer Society, etc.) workshop on public health messages related to UV radiation in 1994? 2. Have you seen a copy of or are your aware of the public health messages that were produced from this 1994 UVR workshop? 3. Where did you see a copy or copies of the public health messages from the 1994 UVR workshop? (as many as apply) List of sources provided:
5. Do you disagree with any of these public health messages? 6. (Answered only if yes to question 2). a) In the last two years, which activities has your organization undertaken with respect to UV? b) Did the 1994 UVR workshop messages serve as an added impetus or help to initiate the activity? c) Were the 1994 consensus conference messages themselves used in any format for the activity your organization undertook? List of activities provided:
.[Previous] [table of Contents] [Next] |
||
 |
|||
|
Last Updated: 2002-10-29 |  |
|
