Canadian Medical Association Journal 1995; 153: 901-907
Dr. Carter is associate director, Department of Health Care and Promotion, CMA, Ottawa, Ont. Dr. Battista is professor in the departments of Epidemiology and Biostatistics, Family Medicine and Medicine, McGill University, Montreal, Que. Mr. Hodge is an MD-PhD candidate in the Faculty of Medicine, McGill University. Mr. Lewis is the chief executive officer of the Saskatchewan Health Services Utilization and Research Commission, Saskatoon, Sask. Dr. Basinski is senior scientist at the Institute for Clinical Evaluative Studies in Ontario, Toronto, Ont. Dr. Davis is associate dean, Continuing Education, Faculty of Medicine, McMaster University, Hamilton, Ont.
Reprint requests to: Dr. Anne O. Carter, Department of Health Care and Promotion, Canadian Medical Association, PO Box 8650, Ottawa ON K1G 0G8
Organizational roles: The national specialty societies were felt to have the largest role to play; the smallest roles were assigned to consumers, who were seen to have a role mainly in priority setting, and to industry and government, both of which were seen to have primarily a funding role. Many barriers to collaboration were identified, the solutions to all of which appeared to be better communication, establishment of common principles and clear role definitions.
Priority setting: There was considerable agreement on the criteria that should be used to set priorities for CPG activities: the burden of disease on population health, the state of scientific knowledge, the cost of treatment and the economic burden of disease on society were seen as important factors, whereas the costs of guidelines development and practitioner interest in guidelines development were seen as less important. Organizations were unable to give much information on how they set priorities.
Guidelines implementation: Most of the organizations surveyed did not actively try to ensure the implementation of guidelines, although a considerable minority devoted resources to implementation. The 38% of organizations that implemented guidelines actively listed a wide variety of activities, including training, use of local opinion leaders, information technology, local consensus processes and counter detailing.
Guidelines evaluation: Formal evaluation of guidelines was undertaken by fewer than 13% of the responding organizations. All the evaluations incorporated assessments before and after guideline implementation, and some used primary patient data. Barriers to evaluation included lack of money, time, data or expertise.
CPG network: Most of the respondents felt that all organizations and individuals interested or involved in guidelines should form the membership of the network. The three most important functions of such a network were deemed to be (a) to facilitate collaboration among those involved in the CPG process, (b) to maintain an information centre on CPGs and (c) to provide expertise to the CPG process. It was felt that the network should have some formal structure and communicate through email and print media.
Résumé
Le Comité organisateur d'un atelier sur les guides de pratique clinique (GPC) a effectué, auprès des organisations invitées, un sondage sur leurs attitudes et leurs activités face à cinq sujets qui seraient abordés au cours des ateliers : rôles organisationnels, établissement de priorités, mise en oeuvre des guides, évaluation des guides et création d'un réseau d'intervenants actifs dans le domaine des GPC.
Rôles organisationnels : Les représentants croyaient que les sociétés nationales de spécialités ont le rôle le plus important à jouer et que les rôles les moins importants reviennent aux consommateurs qui doivent intervenir surtout dans l'établissement des priorités, et à l'industrie et au secteur public, auxquels ont attribue un rôle de financement avant tout. On a défini denombreux obstacles à la collaboration, dont l'élimination semble reposer dans tous les cas sur une meilleure communication, sur l'établissement de principes communs et sur la définition d'un rôle clair.
Établissement de priorités : Il y avait un consensus important sur les critères qu'il faudrait utiliser pour établir les priorités relatives aux activités portant sur les GPC : le fardeau que les maladies imposent à la santé des populations, l'état des connaissances scientifiques, le coût des traitements et le fardeau financier que la maladie impose à la société ont été considérés comme des facteurs importants, tandis que les coûts d'élaboration des guides et l'intérêt des praticiens à cet égard a été jugé moins important. Les organisations n'ont pu fournir beaucoup de renseignements sur la façon dont elles s'y prennent pour établir les priorités.
Mise en oeuvre des guides : La plupart des organisations interrogées n'essayaient pas activement d'assurer la mise en oeuvre des guides, même si une minorité importante y consacrait des ressources. Plus de 38 % des organisations qui ont mis en oeuvre des guides ont énuméré un large éventail d'activités : formation, recours à des dirigeants d'opinion locaux, technologie de l'information, processus de consensualisation à l'échelle locale et description détaillée au comptoir.
Évaluation des guides : Moins de 13 % des organisations qui ont répondu procédaient à une évaluation structurée des guides. Toutes les évaluations comportaient une évaluation avant et après la mise en oeuvre du guide et certaines utilisaient des données primaires sur les patients. Les obstacles à l'évaluation comprenaient le manque d'argent, de temps, de données ou de compétences spécialisées.
Réseau de GPC : La plupart des répondants étaient d'avis que le réseau devrait être constitué de toutes les organisations et les personnes qui s'occupent de guides ou qui s'y intéressent. On a jugé que les trois fonctions les plus importantes d'un tel réseau seraient les suivantes : a) faciliter la collaboration entre les intervenants du processus d'élaboration des GPC, b) maintenir un centre d'information sur les GPC et c) fournir des compétences spécialisées au processus d'élaboration des GPC. Les répondants croyaient que les réseaux devraient avoir une structure officielle et communiquer électroniquement et par écrit.
At a November 1992 workshop involving individuals and organizations interested in clinical practice guidelines (CPGs), participants agreed that it would be valuable to hold future workshops on specific topics in the guidelines field.1-7 An advisory committee was organized, comprising representatives of the main stakeholder and funding groups, and another workshop was planned for the fall of 1994 to focus on four of the most difficult issues in the guidelines field: organizational roles, setting of priorities, implementation of guidelines and evaluation. In addition, the committee decided to hold a session to build on the concept, formed at the first workshop, of a network to support and promote guidelines activities. It felt that all five issues would be best dealt with if organizations sending representatives to the workshop were surveyed on their activities and attitudes with regard to these issues. Accordingly, those charged with developing each of the issue sessions designed a survey that was sent to invited organizations. This report summarizes the findings of the survey.
[Table of Contents]
Methods
All organizations invited to the workshop were sent a questionnaire soliciting their views on the topics to be covered: organizational roles, priority setting, implementation, evaluation, and terms of reference for a CPG network. Invitees consisted of all the organizations identified by the CMA Quality of Care Program as being actively involved in or considering active involvement in the CPG field at the national or provincial/territorial level. This identification process involved extensive consultation with and solicitation of information from national and provincial organizations for more than 2 years. No effort was made to identify locally active organizations. Although new organizations were continually being identified through this process, those identified after July 1, 1994, were sent an invitation to the workshop but were not sent the questionnaire because there was insufficient time to incorporate their responses into the analysis.
It was left up to each organization to determine how and by whom the questionnaire would be completed. Organizations were asked to mail completed questionnaires to the CMA. Nonresponding organizations were telephoned close to the response deadline and reminded that the questionnaire was due. Questionnaire responses were analysed in a descriptive manner only.
[Table of Contents]
Results
Questionnaires were sent to 107 organizations along with an invitation to nominate someone to attend the workshop. By the deadline for analysis 55 completed questionnaires (51%) had been received from the following categories of organization: 8 (62%) of 13 licensing authorities, 12 (71%) of 17 governments or paragovernmental organizations, 6 (43%) of 14 provincial and territorial medical associations or other organizations representing physicians provincially, 18 (51%) of 35 national specialty societies and 11 (39%) of 28 other types of organizations. Since the respondents could indicate several answers to many of the questions, the number of responses to those items often exceeded 55. In addition, some of the respondents did not answer some questions, so that the number of responses at times was less than 55.
Of all the respondents 10 (18%) were not currently involved in guidelines activities. These were mainly from specialty societies. Of the remaining 45 respondents 18 (40%) were involved in funding activities, 25 (56%) in priority setting, 24 (53%) in guidelines development, 34 (76%) in guidelines dissemination, 20 (44%) in guidelines implementation, 15 (33%) in guidelines evaluation, 18 (40%) in coordination of guidelines activities, 15 (33%) in endorsing guidelines, 11 (24%) in teaching and 9 (20%) in research involving guidelines. Most organizations were involved in several different types of activities. The following sections summarize the responses to the five issues covered by the questionnaire.
Attitudes toward the appropriate roles of organizations
The respondents completed a matrix indicating their opinions about the appropriate level of involvement (essential, desirable or should not participate) of various types of organization in each aspect of CPG activity, from funding to research. "Essential" responses were assigned a score of 2, "desirable" a score of 1 and "should not participate" and "no response" a score of 0. There was no differentiation in the scoring of the last two response types: some respondents left several cells in the matrix blank, and thus the definition of the zero score had to be expanded. Although doubtless some "don't know" responses were implicit in the blank cells, it is generally assumed that respondents would have identified positive roles. (The complete set of data is available from the authors upon request.)
All of the respondents indicated that specialty societies should have the most involvement in guidelines activities and that industry should have the least (Table 1).
Other than a moderate endorsement of involvement in establishing priorities for CPGs (i.e., choosing subjects), the respondents saw little role for consumers in any CPG-related activities.
Most of the respondents indicated that there are national, provincial and local roles in CPG activities; support for provincial involvement was virtually unanimous. Setting priorities and guidelines development were considered to be national responsibilities by some, provincial by others and joint by several. Almost all of the respondents identified local roles in implementation.
Barriers to collaboration
The respondents were asked to indicate the barriers to collaboration in the CPG process. Responses varied, but the following main themes emerged.
The main solutions proposed included better communication, a set of national principles for CPGs, clearer role identification, more stable funding and a clear focus on evidence-based development.
Criteria for setting priorities
On a 5-point scale from "very important" (1) to "not at all important" (5) the respondents were asked to rate the relevance of each of seven criteria: the health burden on the population, the economic burden of disease on society, the cost of treatment to the health care system, the extent of practice variation, the state of scientific knowledge, the cost of guidelines development and practitioner interest in having guidelines developed. The respondents were not required to rank the criteria and could thus rate all seven the same if they wished.
For each category of organization, average scores for each item were calculated. Between-group differences were not significantly different. Table 2 presents the criteria that were deemed very important or important, by category. Very important items had average scores of less than 2, and important items had average scores of 2 or more. No item received an average score greater than 3.2.
We found no significant or meaningful differences between the priorities of different stakeholders. All of the groups identified the health burden on the population as a very important criterion for setting priorities for CPG activities. Similarly, all of them rated the costs of guidelines development and practitioner interest as being relatively less important.
Methods for setting priorities
Details about how priorities are set proved elusive. The respondents generally described who was in charge of setting priorities but omitted information about how this was done. Most described some form of reactive priority setting rather than a proactive canvassing of membership, the public or other groups. When asked to identify groups consulted during priority setting, the respondents most often identified members of their respective organizations and other professional or specialty societies. Community members were identified least frequently, by only 8 (32%) of 25 respondents.
Dissemination activities
Most (62% [34/55]) of the respondents indicated that their organizations disseminated CPGs. Of these, 82% (28/34) disseminated the CPGs developed by their own organizations; 47% (16/34) disseminated guidelines developed by other organizations.
Active implementation activities
Most (58% [32/55]) of the respondents indicated that they did not actively try to ensure the implementation of guidelines disseminated by their organization; 21 (38%) of the 55 indicated that they did try, particularly if the CPGs had been developed by that organization (17 [81%]).
Resources
Fourteen (25%) of the 55 respondents indicated that staff resources were used to disseminate and implement CPGs, ranging from small percentages of a full-time equivalent to three full-time equivalents. Twenty (36%) reported that dissemination and implementation of CPGs were the responsibility of committees, of which 75% were standing committees and 50% ad-hoc committees. Finally, 10 (18%) of the respondents indicated that their organization gave financial support - in one instance up to $500 000 a year - to the dissemination and implementation of CPGs.
Specific dissemination and implementation activities
Twenty-one (38%) of the 55 respondents indicated their involvement in the following activities, listed in order of frequency.
Evaluation by organizations
Formal evaluation of CPGs is rare. The respondents were asked to identify any guidelines that had been formally evaluated by their organization and the evaluation design. Only seven organizations had formally evaluated CPGs or were doing so, all since 1992. Two of the organizations were provincial medical associations, two were cancer agencies, one was a provincial college of physicians and surgeons, and two were health services or research organizations. All of the evaluations incorporated assessments done before and after dissemination of the CPGs and were based on administrative data such as laboratory test volume. The hospital, cancer agencies and one provincial medical association also included primary patient data in their assessments. The types of CPGs evaluated varied.
Only one organization claimed that it had made any specific changes to CPGs or to its activities on the basis of the evaluation results.
Main purpose of evaluation
Six of the seven organizations currently active in CPG evaluation cited some aspect of quality of care as the main reason for evaluating CPGs. The general thrust of the responses was that the most appropriate and cost-effective care should be provided through a monitoring of outcomes during quality-assessment activities.
Barriers to evaluation
All of the respondents were asked what they considered to be major barriers to the evaluation of CPGs. The most common barrier cited was lack of money or resources (36% [20/55]). The next commonest were lack of time (18% [10/55]), of data or systems (18% [10/55]) and of the organization's expertise (16% [9/55]). Other barriers included lack of clear goals and objectives, lack of formalized processes, lack of commitment, difficulty relating outcome to the intervention and fear of criticism.
Major supports for evaluation
Major supports for the evaluation of CPGs were varied and difficult to categorize. In general, the themes brought forward included organizational priority for evaluation, accountability, leadership, growing stakeholder interest, evolving research and databases, and integration with quality-assurance activities.
Membership
When asked who should constitute the membership of a network, the most frequently mentioned type of organization was the national specialty society (18 [30%] of the responses), but the most frequent response was a large group of organizations or all organizations and individuals interested or involved in guidelines (36 [61%] of the responses).
Communications mechanisms
When asked to specify the communications mechanisms that should be used to maintain the network, email and newsletters or journals were the most popular suggestions (28% and 36% of the suggestions representing 48% and 63% of the respondents respectively). Workshops and meetings were less common (22% of the suggestions), and the remainder of the suggestions were not relevant to the question (e.g., low cost). The pattern of these suggestions was similar for all types of organizations responding.
Structure
When asked about the structure of a CPG network most of the respondents (61% [17/28]) recommended some sort of formal structure, such as a committee, task force or working group. Only four organizations supported a less formal approach, and nine answers were very general. Governments and specialty societies tended to be the strongest supporters of a more formal structure; licensing authorities tended to specify less formal structures or to give more general answers.
Functions
The respondents rated "facilitating collaboration among organizations involved in the CPG process" as the most important suggested function of the network, 73% rating it as very important and 99% giving it a rating between moderately and very important. The respondents rated "developing and maintaining a central information centre on CPGs and individuals/organizations active in the field" as the second most important function, 60% rating it as very important and 100% rating it between moderately and very important. A close third in importance was "providing expertise to the CPG process," 60% rating it as very important and 98% rating it moderately to very important. Research, education and funding were rated as relatively unimportant (by 13%, 8% and 19% of the respondents respectively). Other functions suggested by the respondents included advocacy, consensus building and priority setting.
Most of the ratings of not important assigned to research and education came from government organizations. Most of the ratings of not important assigned to funding were from medical organizations (licensing authorities, specialty societies and associations).
Roles
Of the 43 who answered this item, 31 (72%) gave coordination, cooperation or collaboration, and communication as the roles that the network most likely would fulfil and that current mechanisms do not. Other possibilities were much less popular: standardization and building commitment were mentioned by 10 (23%) each, and expertise, CPG development or implementation, and decreased duplication was mentioned by 9 (21%), 8 (19%) and 8 (19%) respectively. CPG evaluation was stated by 7 (16%). Government organizations showed the greatest interest in coordination and cooperation or collaboration, with 90% (9/10) mentioning it. The licensing authorities were most interested in building commitment, 67% (4/6) giving this as a potential accomplishment.
Resources
When asked what resources they or their organization would be willing to contribute to the CPG network, the 47 who responded cited information sharing (49%), involvement in committees (41%) and expertise (35%). Fewer (12%) were willing to contribute to dissemination and implementation of CPGs, and fewer still were willing to contribute funding (6%). This latter group consisted of one licensing body, one specialty society and one government organization.
Few of the respondents were aware of other successful networks that could be investigated for approaches, ideas and options. Those mentioned included the Canadian Heart Health Network, the Quality Institute of the Conference Board of Canada, the Federal/
Provincial/Territorial Advisory Committee on Health Services, various international societies, big business (particularly their quality-improvement models), the Cochrane Collaboration and the Behavioural Research Network being developed by the National Cancer Institute of Canada.
[Table of Contents]
Discussion
It is clear that the advisory committee was successful at identifying topics that needed to be addressed in the workshop sessions. Many organizations have not addressed the issues in depth, and there is a need for consensus on the future directions to be taken. It is also clear that these organizations would benefit from a network that would facilitate collaboration among them. It is hoped that the workshop will provide the impetus for progress and that the network will provide a framework to continue the efforts begun at the workshop.
[Table of Contents]
References
CMAJ: Oct. 1, 1995
"Guidelines Workshop" articles