Canadian Medical Association Journal 1996; 154: 665-671
Design: Qualitative evaluation by structured focus groups in September and October 1994 to assess the relevance, clarity and practicality of the guidelines, followed by content analysis of the discussions.
Setting: Vancouver, Winnipeg, Toronto, Montreal, Quebec City and St. John's.
Participants: Primary care physicians randomly selected from the CMA database and nonrandomly selected from the Canadian Medical Directory who had limited experience with HIV testing and counselling and who provided an appropriate mix of characteristics in terms of practice type (solo and group), setting (urban and rural), age and sex. A total of 1247 physicians were approached for the study; a convenience sample of 68 were recruited, of whom 56 participated. The average size of each focus group was eight physicians.
Outcome measures: Clinical experience and information sources with respect to HIV testing, reactions to the counselling guidelines, and suggestions for revisions and improvements to the guidelines.
Results: Most (96% [54/56] of the participants had ordered HIV serologic testing for patients in the 6 months preceding the focus groups, and about half of them (52% [28/54]) had at least one patient with a positive test result. Many (59% [33/56]) of the participants had a copy of the guidelines at the time of recruitment; 19 (58%) of them had used the guidelines in the 6 months before the focus groups. The parts of the guidelines most often read were the checklists and inset boxes. Recommendations for revisions in content were for more information on legal and ethical issues, information on new issues (e.g., rapid testing) and guidelines on how best to tell a patient about a positive test result; recommendations for revisions in format included more tables, algorithms, bulleted points and white space, less text, larger type and plainer language.
Conclusions: The focus groups provided detailed, credible and consistent information about the face and content validity of the HIV counselling guidelines. They are a useful qualitative method for evaluating the relevance, clarity and practicality of clinical practice guidelines at the inception or revision stage.
Top of document
Introduction
Clinical practice guidelines (CPGs) have proliferated rapidly in North America over the last decade.[1-7] Defined as "systematically developed statements to assist practitioner and patient decisions about appropriate health care for specific clinical circumstances,"[8] CPGs offer the potential for assuring quality of care, controlling health care costs, enhancing access to care, empowering patients and addressing issues of health care management.[9] Despite the growth and potential value of CPGs, their actual value is not routinely established through formal evaluation procedures.[1,6,10,11]
It is essential to evaluate CPGs to assess and document their usefulness and to ensure continuous quality improvement in health care delivery.[12] Grimshaw and Russell[13] suggested that guidelines improve clinical practice when supported by rigorous evaluations. They reviewed 59 published evaluations of CPGs that used one or more of five "robust" study designs (e.g., randomized) to study the effects of guidelines on process or outcome of care. A total of 24 studies focused on the effects of guidelines on the process of clinical care (e.g., rates of compliance with diabetes treatment guidelines), 27 on the process of preventive medicine (e.g., rates of compliance with prevention guidelines for tetanus vaccination) and 8 on prescribing and the use of support services (e.g., use of x-rays for head injury). In 55 of the studies, the guidelines had a significant positive effect. Of the 11 studies that focused on the effects of guidelines on the outcome of care (e.g., patients not smoking after 1 year), all but 2 showed significant improvement.
Basinski[10] has suggested that there are three stages of evaluating CPGs that may apply to the level of development: (a) inception evaluation, (b) guidelines-program evaluation and (c) scientific evaluation. Inception evaluation is suitable for new or revised guidelines; it is important to establish face and content validity at this stage, when practitioners and the public assess issues such as relevancy, clarity and practicality. Guidelines-program evaluation occurs after dissemination and implementation of the CPGs; it focuses on the process, outcome or efficiency of CPGs in relation to a predetermined goal or benchmark. Finally, scientific evaluation considers various aspects of guidelines development, format, and dissemination and implementation (e.g., the impact of physician participation in guidelines development on later compliance).
Inception evaluation has been "largely ignored in most guidelines development endeavours,"[10] although it is a vital part of the continuous-quality-improvement cycle of guidelines development.[14,15] This stage of evaluation closely corresponds to what program evaluators call formative evaluation[16-18] -- studies to improve a program or product in a timely manner. These types of evaluations are in contrast to summative evaluations, which examine the effectiveness of a program or product and lead to decisions about its continuation or termination.
Quantitative approaches that measure impact or outcome (e.g., provider satisfaction, compliance, patient health status) are commonly used to evaluate CPGs.[10,13,19] However, qualitative approaches that assist in obtaining a rich understanding of the inner workings and intricacies of a program or product[20-23] have much to offer, particularly in the inception stage of guidelines development.[10] Such studies can provide valid and reliable information by using techniques such as triangulation (i.e., collection and verification of information from multiple sources or procedures): one participant's descriptions of an activity are checked against another's of the same activity, and audiotapes and videotapes are used for data collection, when appropriate.[24]
Focus groups are a special type of qualitative method rooted in sociology and used widely in marketing research.[25] They have been adapted successfully to evaluate health care settings and environments.[26-28] There have been no reports of evaluations of CPGs by focus groups.
Top of document
Background information
The CMA's counselling guidelines for HIV serologic testing were first published in 1990.[29] The second edition was published in 1993.[30] As with the first edition, the primary audience of the second edition was practising physicians who have no specific expertise in caring for patients with HIV infection or AIDS and who may be required to test patients for HIV infection and to counsel them. The main areas covered in the second edition included tests for HIV infection, key information on HIV infection and AIDS, prerequisites for testing, pre- and post-test counselling procedures, and HIV testing of children. There was also a physician reference pretest and a post-test checklist.
In the summer of 1994 Health Canada commissioned a third edition of the guidelines to address new developments in HIV counselling and testing (e.g., HIV testing of women, particularly those who are pregnant or are of childbearing age; concerns about testing street people and other hard-to-reach populations; and the advent of rapid testing). To accomplish this task, the second edition was evaluated by seven focus groups in September and October 1994. Their findings were provided to an expert working group, members of which made final decisions on the content for and format of the third edition after consultation with people from other national organizations. There were approximately 40 external reviewers from organizations such as the Canadian AIDS Society, the Canadian Hemophilia Society, the Canadian Public Health Association, the Society of Obstetricians and Gynaecologists of Canada and the Canadian Association of Nurses in AIDS Care.
We report on the focus groups' evaluation of the face and content validity of the second edition of the HIV counselling guidelines. The evaluation corresponded more closely to an inception evaluation than to a guidelines-program evaluation or scientific evaluation, because the primary goal was to explore ways to improve and revise the guidelines rather than to demonstrate expected outcomes or the influence of factors in guidelines development.[10]
We selected a qualitative design that involved seven focus groups held in September and October 1994 in six cities across Canada: Vancouver, Winnipeg, Toronto, Montreal, Quebec City and St. John's. Two focus groups were held in Montreal. Most of the sessions were held in English except for one of the two sessions in Montreal and the session in Quebec City, which were held in French. The focus group in Winnipeg was intended to address issues related to Aboriginal people and rural or isolated physicians. One of the interests in the St. John's focus group was the testing and counselling needs of women. Focus groups were led by one of three facilitators from a consulting group hired to conduct the focus groups, analyse the qualitative results and write summary reports.
A convenience sample of 8 to 11 subjects was recruited by telephone for each focus group (Table 1). There were two telephone recruiters (English and French) associated with the consulting group who were experienced in recruiting physicians for focus groups. Initially, samples from the CMA database of physician members and nonmembers were randomly selected by city and by specialty in general practice (GP) or family practice (FP) medicine. For some groups further nonrandom sampling was done using the Canadian Medical Directory[31] to obtain a sufficient number of physicians who would agree to participate.
Snowball sampling[32] was done for the Winnipeg focus group to obtain a sample of physicians with Aboriginal patients. This sampling technique involved a key informant in Winnipeg who provided a recruiter with names of potential participants known to have Aboriginal patients. Those contacted to participate were asked to provide the names of other physicians with Aboriginal patients, and so on.
Telephone recruiters screened subjects further on a number of criteria: they had to be the first people contacted in each sample who agreed to participate, reported limited involvement with HIV testing and counselling in the 6 months preceding the focus groups and provided a good mix of the following characteristics: practice type (solo and group), setting (urban and rural), age and sex. Subjects were requested to read the second edition of the HIV counselling guidelines before they attended the focus groups.
In consultation with the focus-group facilitators, we developed a discussion guide that covered the main areas of concern and key questions to help structure the focus groups' discussions and maintain consistency of topics discussed across the focus groups. The main areas included in the discussion guide are outlined in Table 2.
A group activity was introduced near the end of the session during which participants were assigned to small groups and given the option of revising the counselling guidelines or the pre- and post-test counselling checklists. The revision exercise was followed by a short debriefing session.
Qualitative data were collected from audiotapes and handwritten notes taken by the facilitators and those of us who attended the sessions. Results were analysed for content. Key themes and findings were generated, and the facilitators of the French and English focus groups each wrote final reports. Information from the screening questionnaire was analysed quantitatively by us.
Procedures for enhancing validity and reliability of data included triangulation of information. Findings from the focus groups were supported by data obtained from other sources such as discussions between members of the expert working group and physicians treating patients with HIV infection or AIDS, and comments from participants at AIDS conferences and workshops.
Other ways in which the validity and reliability were enhanced included the commonality of themes and consistency of opinions across the focus groups, the perceived frankness of the respondents and, according to a member of the expert working group, the fact that the information "came from the mouths of physicians rather than through another channel."
A total of 1247 physicians were approached to participate in the focus groups. A convenience sample of 68 physicians were recruited; 56 participated, for an average focus group size of 8 (Table 1). All of the participants were GPs and FPs.
The demographic characteristics of the participants in the focus groups were compared with those of the general population of Canadian GPs and FPs (excluding interns and residents) (Table 3). The two groups were similar with respect to most of the characteristics (i.e., less than a 10% difference). However, women were slightly overrepresented and rural practitioners underrepresented in the focus groups. The typical focus group participant was a male GP/FP under 40 years of age in practice for 15 years and currently in an urban or suburban practice setting.
Fifty-four (96%) of the 56 participants had ordered HIV serologic testing for patients in the 6 months preceding the focus groups, just over half of them having at least one patient with a positive result (Table 4). Most (80%) of the 54 participants who had tested patients provided counselling for all of them; some of the participants (17%) indicated that they provided counselling only for those who tested positive. Most (65%) reported that they had not tested Aboriginal patients in the past 6 months.
Many (59% [33/56]) of the participants reported having a copy of the guidelines at the time of recruiting. However, a notable proportion (38% [21/56]) reported not having a copy and were mailed one before they attended the focus group. Of those who had their own copy, 58% (19/33) indicated that they had used the guidelines in their practice during the 6 months preceding the focus groups.
The focus group participants found the guidelines to be valuable and agreed that they were a good resource. A new overall approach suggested for the third edition was to shift some of the emphasis from how HIV testing and counselling should be done (knowledge-based) to why it should be done (motivational-based). Main findings regarding relevance, clarity and practicality are summarized here.
The participants reported that most of their patients who came to them for HIV testing and counselling were heterosexual and at "minimum risk" (e.g., people who were starting new relationships or had "strayed" on a business trip). Many of the participants indicated that patients initiated the request for testing and that many patients seemed quite comfortable in doing so. It was also reported that most of these patients had negative test results. Few of the participants had to deal regularly with patients who were at high risk or demonstrated risk behaviour; it was theorized that such patients had their own regular caregivers or used anonymous testing centres where available.
Problems with the format of the guidelines were an important reason why some of the participants had not used them. Checklists and inset boxes tended to be the parts most often read. Text was described as "too wordy." Recommendations included more tables, algorithms, bulleted points and white space, less text, larger type and plainer language.
Practical guidance was required in some areas, particularly legal and ethical issues. For example, what constitutes informed consent? What are the codes most often used to identify a patient (e.g., a patient's birth date or billing number)? What is a physician's responsibility regarding notification of an HIV-positive patient's sexual or drug-using partner(s)?
Other areas requiring clarification included the definition of the "window period" between exposure to HIV and seroconversion, how best to tell a patient about a positive test result, new issues such as rapid HIV testing, and specific issues concerning counselling of people from different cultural backgrounds and living situations (e.g., ethnic communities, street people).
Top of document
Discussion
The focus groups provided detailed, credible and consistent information about the face and content validity of the CMA's HIV counselling guidelines. The guidelines are relevant, particularly to practising physicians with no specific expertise in HIV/AIDS care who are required to test patients for HIV infection and counsel them. To improve clarity, various changes to format and text were recommended. Practical guidance was felt to be needed mostly in legal and ethical areas.
The main limitation of the study is its generalizability. Women were overrepresented and rural physicians underrepresented in the focus groups. This bias may have been due in part because of the nonrandom sampling method used in some of the cities and because the focus groups were held in urban centres only. The inclusion of a survey or interviews in the evaluation design might have helped to reach a broader range of physicians (e.g., those in rural areas) and to assess patients' reactions. However, although some evaluators agree with combining qualitative and quantitative approaches,[33] others are strictly opposed to this.[34]
There are several strengths of the study. An in-depth understanding of the issues was achieved through substantial interaction among the physicians in the focus groups. Innovative ideas surfaced and were tested in a dynamic forum. The participants felt comfortable in sharing practice dilemmas with their peers and in expressing their discomforts about HIV testing and counselling. Finally, the information was gathered quickly (over 5 months), which greatly facilitated the revision of the guidelines. The third edition of the HIV counselling guidelines was published by the CMA in May 1995.[35]
In conclusion, results from this study suggest that qualitative evaluations of CPGs can be useful, particularly at the inception stage of guidelines development, when an in-depth scope of coverage and dynamic group interaction are valuable in identifying improvements. In addition to the involvement of health care providers, consideration should be given to patient participation and the combining of qualitative and quantitative evaluation methods.
We thank the members of the expert working group (Marc Steben, MD [chair]; Catherine A. Hankins, MD [cochair]; Michael V. O'Shaughnessy, PhD; Allan D. Peterkin, MD; and Gerry Bally, MD) for their valuable insights into the use of focus groups to evaluate clinical practice guidelines; Pamela Thompson, MSc, president of INPRO Consulting, and her associates for their work on the focus groups; and, from the CMA Department of Health Care and Promotion, Anne Carter, MD, and Susan Beardall, MHSc, for their helpful reviews and suggestions, and Anita Kothari, MHSc, for performing the literature search.
Financial and staff support for this study was provided from Health Canada and the Canadian Medical Association.