No magic bullets: a systematic review of 102 trials of interventions to improve professional practice

Canadian Medical Association Journal 1995; 153: 1423-1431

[résumé]

Dr. Oxman (principal investigator) is a member of the Health Services Research Unit, National Institute of Public Health, Oslo, Norway; Ms. Thomson is with the School of Occupational and Physical Therapy, Faculty of Health Sciences, McMaster University, Hamilton, Ont.; Dr. Davis is in the Office of Continuing Education and the Department of Family and Community Medicine, Faculty of Medicine, University of Toronto, Toronto, Ont.; and Dr. Haynes is in the departments of Clinical Epidemiology and Biostatistics and of Medicine, Faculty of Health Sciences, McMaster University, Hamilton, Ont.

Paper reprints of the full text may be obtained from: Dr. Andrew D. Oxman, Health Services Research Unit, National Institute of Public Health, Geitmyrsveien 75, 0462 Oslo, Norway

• Abstract
• Résumé
• Introduction
• Methods
• Results
• Discussion
• References

Abstract

Data sources: MEDLINE, SCISEARCH, CINAHL and the Research and Development Resource Base in CME were searched for trials of educational interventions in the health care professions published between 1970 and 1993 inclusive.

Study selection: Studies were selected if they provided objective measurements of health professional performance or health outcomes and employed random or quasi-random allocation methods in their study designs to assign individual subjects or groups. Interventions included such activities as conferences, outreach visits, the use of local opinion leaders, audit and feedback, and reminder systems.

Data extraction: Details extracted from the studies included the study design; the unit of allocation (e.g., patient, provider, practice, hospital); the characteristics of the targeted health care professionals, educational interventions and patients (when appropriate); and the main outcome measure.

Data synthesis: The inclusion criteria were met by 102 trials. Areas of behaviour change included general patient management, preventive services, prescribing practices, treatment of specific conditions such as hypertension or diabetes, and diagnostic service or hospital utilization. Dissemination-only strategies, such as conferences or the mailing of unsolicited materials, demonstrated little or no changes in health professional behaviour or health outcome when used alone. More complex interventions, such as the use of outreach visits or local opinion leaders, ranged from ineffective to highly effective but were most often moderately effective (resulting in reductions of 20% to 50% in the incidence of inappropriate performance).

Conclusion: There are no "magic bullets" for improving the quality of health care, but there are a wide range of interventions available that, if used appropriately, could lead to important improvements in professional practice and patient outcomes.

Source de données : On a cherché dans MEDLINE, SCISEARCH, CINAHL et la Base de ressources en recherche et développement sur EMC des tentatives d'interventions d'éducation auprès des professions de la santé, publiées entre 1970 et 1993 inclusivement.

Sélection des études : On a retenu les études qui présentaient une mesure objective du rendement des professionnels de la santé ou des résultats sur la santé et dont le plan d'étude recourait à des méthodes aléatoires ou quasi aléatoires de répartition des sujets individuels ou des groupes. Les interventions comprenaient diverses activités comme des conférences, des visites de liaison, le recours à des personnalités locales, des vérifications, la rétroaction et les systèmes de rappel.

Extraction des données : Parmi les détails extraits des études, mentionnons le plan d'étude, l'unité d'attribution (comme le patient, le fournisseur, la pratique ou l'hôpital), les caractéristiques des professionnels de la santé visés, des interventions d'éducation et des patients (le cas échéant), et la principale mesure de résultats.

Synthèse des données : Les critères d'inclusion sont ressortis dans 102 études. Parmi les secteurs où s'est produit un changement de comportement, notons les relations générales avec les patients, les services de prévention, les pratiques de prescription, le traitement de certaines maladies comme l'hypertension ou le diabète, et le service diagnostic ou le recours à l'hôpital. Utilisées comme critères uniques, les stratégies axées uniquement sur la diffusion, comme les conférences ou l'envoi par la poste de documents non sollicités, n'ont suscité que très peu de changements en ce qui touche le comportement des professionnels de la santé ou les résultats sur la santé. Les interventions plus complexes, comme des visites de liaison ou le recours à des personnalités locales, ont donné des résultats allant de pauvres à excellents, mais, dans la plupart des cas, elles n'ont eu qu'un effet modéré (diminution de 20 % à 50 % de l'incidence d'une performance non appropriée).

Conclusion : Il n'y a aucune «formule magique» pour accroître la qualité des soins de santé, cependant il existe un vaste éventail d'interventions disponibles qui, si on les utilise comme il faut, pourraient donner lieu à d'importantes améliorations de la pratique des professionnels et des résultats au niveau des patients.

[Table of Contents]

In 1992 Davis and colleagues(21) systematically reviewed 50 randomized controlled trials of the effectiveness of continuing medical education (CME), which they broadly defined as all the ways by which physician learning and clinical practice might be altered by educational or persuasive means. The objective of our study was to update and enhance that body of knowledge by (a) including nonphysician health care professionals, (b) applying explicit criteria for assessing study quality, (c) defining specific types of interventions, described later, (d) analysing other published systematic reviews of interventions targeted at a specific problem (e.g., vaccination delivery(22)) or provider (e.g., nurses(23)), or of a specific type of intervention (e.g., audit and feedback(24)), and (e) applying theoretic frameworks such as total quality management,(25) the PRECEDE (predisposing, reinforcing and enabling constructs in educational diagnosis and evaluation) health behaviour model26 and various social science and adult educational models27 to clarify the findings of trials. We present preliminary data resulting from these processes.

[Table of Contents]

Methods

We added to the search strategies described by Davis and coworkers(21) by searching the following computerized bibliographic databases for the period January 1990 to December 1993: MEDLINE, SCISEARCH, CINAHL and the Research and Development Resource Base in Continuing Medical Education.(28)

We included trials that met the following criteria: the studies involved health care providers (excluding students); outcomes were measured by objective assessments of provider performance in a health care setting or by any health outcome; and the study designs were such that individual subjects or groups were assigned to one of two or more alternative interventions or to a control group by means of random allocation or some quasi-random method of allocation (e.g., alternation). We included the following types of intervention:

• Educational materials: Distribution of published or printed recommendations for clinical care, including clinical practice guidelines, audiovisual materials and electronic publications.
• Conferences: Participation of health care providers in conferences, lectures, workshops or traineeships outside their practice settings.
• Outreach visits: Use of a trained person who meets with providers in their practice settings to provide information. The information given may include feedback on the provider's performance.
• Local opinion leaders: Use of providers explicitly nominated by their colleagues to be "educationally influential."
• Patient-mediated interventions: Any intervention aimed at changing the performance of health care providers for which information was sought from or given directly to patients by others (e.g., direct mailings to patients, patient counselling delivered by others, or clinical information collected directly from patients and given to the provider).
• Audit and feedback: Any summary of clinical performance of health care over a specified period, with or without recommendations for clinical action. The information may have been obtained from medical records, computerized databases or patients or by observation.
• Reminders: Any intervention (manual or computerized) that prompts the health care provider to perform a clinical action. Examples include concurrent or inter-visit reminders to professionals about desired actions such as screening or other preventive services, enhanced laboratory reports or administrative support (e.g., follow-up appointment systems or stickers on charts).
• Marketing: Use of personal interviewing, group discussion (focus groups) or a survey of targeted providers to identify barriers to change and the subsequent design of an intervention.
• Multifaceted interventions: Any intervention that includes two or more of the last six interventions described here.
• Local consensus processes: Inclusion of participating providers in discussion to ensure agreement that the chosen clinical problem is important and the approach to managing it appropriate.

Assessment of trial quality

The following criteria were used to assess the quality of the trials: concealment of random allocation (protection against selection bias); completeness of follow-up; blind outcome assessment; reliable outcome, indicated by two or more raters with at least 90% agreement or achievement of a kappa value greater than 0.8; baseline measurement of performance or health outcomes measured before intervention; and assurance that contamination was unlikely (i.e., that the control group did not receive the experimental interventions). All four of us tested these on a sample of 10 studies, and then one of us (M.A.T.) applied them to each study included in the review.

Data collection

The details extracted from the studies included, but were not limited to, the following: whether the study was a randomized controlled trial (parallel groups), a quasi-randomized controlled trial such as one using alternative allocation, or a crossover study; the unit of allocation (e.g., patient, provider, practice, hospital); the characteristics of the participating providers and patients; the characteristics of the intervention, including the content, format, source, recipient, setting and timing; and a description of the main outcome measure.

We attempted to obtain sufficient information to calculate the mean proportion (per provider) of times that performance was appropriate for each patient or episode of care in each study group. We also tried to obtain the proportion of baseline performance that was adequate and the absolute and relative percent change in performance for each study group.

[Table of Contents]

Results

Effectiveness of interventions

• Educational materials: Most of the studies that used printed materials only(29,30) failed to demonstrate changes in performance or health outcome, a finding that has been associated also with the distribution of guidelines.(31) Conversely, Marton, Tul and Sox(32) found printed materials to be as effective as audit and feedback in altering test-ordering behaviour.
• Conferences: Reflecting the dissemination-only characteristics of unsolicited printed materials, those conferences, rounds and workshops during which no explicit effort was made to determine practice needs or to facilitate practice change failed to demonstrate change in performance or health outcome. However, other, more comprehensive strategies employing workshops as a central focus did effect changes through intra-session practice rehearsal(34) or other patient educational and practice-reinforcing strategies.(35)
• Outreach visits: Outreach visits, also known as academic detailing or counter detailing, were effective in reducing inappropriate prescribing and, to a lesser extent, increasing the delivery of preventive services. Reductions of 12% to 49% in inappropriate prescriptions as a result of counter detailing were reported in four studies.(29,36-38) Second, Cummings and coworkers(39) reported the effectiveness of detailing in smoking cessation. Dietrich and associates(40) reported small increases (5% to 27%) in the delivery of 10 preventive services, 6 of which were statistically significant (p * 0.05). Finally, Putnam and Curry(41) demonstrated a moderate, statistically significant effect of outreach visits combined with audit on essential elements in the management of five common conditions in family practice.
• Local opinion leaders: The effectiveness of opinion leaders ranged from nonsignificant to substantial. Stross and collaborators(42-44) found some improvement in the quality of care of patients with arthritis and respiratory disease. Lomas and colleagues(31) demonstrated a substantial increase in the number of trials of vaginal delivery after previous cesarean section in hospitals in which a local opinion leader was used.
• Patient-mediated interventions: These included patient educational materials used widely in the literature for smoking cessation(47,48) and diabetes mellitus.(49) Changes in performance were demonstrated in multifaceted interventions when patient-mediated interventions were combined with other interventions, such as academic detailing. Patient education effected statistically significant improvements in the management of diabetes mellitus that were probably clinically important.(49) These effects were enhanced when patient education was combined with physician education. Other patient-mediated strategies, which made use of information derived from patients through questionnaires or interviews, demonstrated mixed results.(50,51)
• Audit, feedback and reminders: The effectiveness of audit and feedback or of reminders across different types of clinical behaviour ranged from nil to moderate. The types of participants and the characteristics of the interventions and targeted behaviours varied across the studies. For example, among the trials of reminders 30 (58%) used computerized prompting, 14 (27%) relied on administrative assistance, and 8 (15%) used simple stickers or chart inserts. Using the audit/feedback strategy one study reported an absolute increase of over 40% in the median weighted rate of prescriptions for generic drugs.(45) Although another study achieved only a small change (6.7%) in the mean cost of prescriptions through this strategy, the ratio of savings to costs of the intervention was 50:1.
• Marketing: Studies have demonstrated that learning experiences based on objective practice-needs assessment or knowledge testing can alter at least some aspects of physician performance.(52,53) The evidence is less clear, however, when more subjective needs or practice assessments are performed.(54) Market research was an important component of outreach visits targeted at reducing the incidence of inappropriate prescribing, but it is not possible to separate out the effect of this component of counter detailing.
• Multifaceted interventions and local consensus processes: The use of a variety of interventions, such as audit and feedback, reminders, outreach visits, patient-mediated interventions or opinion leaders, has demonstrated changes in professional performance and, less consistently, changes in health outcomes. The importance of local consensus processes is not clear. For example, Putnam and Curry(41) demonstrated a moderate effect on performance of a local consensus process for generating criteria for optimal care but did not find a significant effect of such a process for essential care.

[Table of Contents]

Discussion

It is helpful to draw an analogy between trials of interventions to improve the performance of health care professionals and drug trials. There are (arguably) no wonder drugs; often several medications are needed, along with lifestyle or environmental changes, to effect clinically important changes in health status. It is the same with the alteration of health professional performance: many interventions have modest or negligible practical effects when used alone. However, when coupled with other strategies the effects may be cumulative and significant.

Furthermore, just as there is a need for rational drug prescribing, there is a need in the area of health professional performance to include appropriate diagnostic strategies (to determine the reasons for suboptimal performance and to identify barriers to change) and to select carefully the interventions most likely to be effective in light of the diagnosed problem. For example, reminders are likely to be effective only if not having the right information at the right time is an important cause of suboptimal performance. These diagnostic strategies bear a direct relation to quality-assurance activities. Closer collaboration of researchers in the areas of health professional performance, health services and quality assurance appears to be both desirable and necessary.(21) Future efforts should aim to improve and standardize reporting of trials of interventions to improve professional performance. A broad framework should be developed for designing and selecting appropriate interventions across the wide range of professional activities in which gaps between evidence and practice are found. An important initiative in this direction is the formation of the Cochrane Collaboration on Effective Professional Practice, which aims to prepare and keep up-to-date systematic reviews of the effects of interventions to improve professional practice.(131)

There are no "magic bullets" for improving the quality of health care. There are, however, a wide range of interventions available that, if used appropriately, could lead to substantial improvements in clinical care derived from the best available evidence.

We thank the following for their financial support: the North Thames Regional Health Authority Research and Development Programme, National Health Service, London, England; the UK Cochrane Centre, Oxford, England; and Health Canada, Ottawa.

[Table of Contents]

References

1. Sherrington AM: An annotated bibliography of studies on the flow of medical information to practitioners. Methods Inf Med 1966; 2 (5): 45-57
2. Chalmers TC: The impact of controlled trials on the practice of medicine. Mount Sinai J Med 1974; 41: 753-759
3. Stross JK, Harlan WR: The dissemination of new medical information. JAMA 1979; 241: 2622-2624
4. Goodwin JS, Foodwin JM: Failure to recognize efficacious treatments: a history of salicylate therapy in rheumatoid arthritis. Perspect Biol Med 1981; 25: 78-92
5. Schade RR, Donaldson RM: How physicians use cimetidine: a survey of hospitalized patients and published cases. N Engl J Med 1981; 304: 1281-1284
6. Rockette HE, Redmond CK, Fisher B: Impact of randomized clinical trials on therapy of primary breast cancer: the NSABP overview. Control Clin Trials 1982; 3: 209-225
7. Garnier HS, Flamant R, Fohanno C: Assessment of the role of randomized clinical trials in establishing treatment policies. Control Clin Trials 1982; 3: 227-234
8. Fisher LD, Kennedy JW: Randomized surgical clinical trials for treatment of coronary artery disease. Control Clin Trials 1982; 3: 335-338
9. Friedman L, Wenger NK, Knatterude GL: Impact of the coronary drug project findings on clinical practice. Control Clin Trials 1983; 4: 513-522
10. Fineberg HV: Effects of clinical evaluation on the diffusion of medical technology. In Committee for Evaluating Medical Technologies in Clinical Use: Assessing Medical Technologies, National Academy Press, Washington, 1985: 176-210
11. Lewis C: Disease prevention and health promotion practices of primary care physicians in the United States. Am J Prev Med 1988; 4: 9-16
12. Bass MJ, Elford WE: Preventive practice patterns of Canadian primary care physicians. Am J Prev Med 1988; 4: 17-23
13. Stross JK: Relationships between knowledge and experience in the use of disease-modifying antirheumatic agents: a study of primary care practitioners. JAMA 1989; 262: 2721-2723
14. Williamson JW, German PS, Weiss R et al: Health science information management and continuing education of physicians: a survey of US primary care practitioners and their opinion leaders. Ann Intern Med 1989; 110: 151-160
15. Lau JL, Antman EM, Jimenez-Silva J et al: Cumulative meta-analysis of therapeutic trials for myocardial infarction. N Engl J Med 1992; 327: 248-254
16. Antman EM, Lau J, Kupelnick B et al: A comparison of results of meta-analyses of randomized control trials and recommendations of clinical experts. JAMA 1992; 268: 240-248
17. Ketley D, Woods KL: Impact of clinical trials on clinical practice: example of thrombolysis for acute myocardial infarction. Lancet 1993; 342: 891-894
18. Lomas J, Haynes RB: A taxonomy and critical review of tested strategies for the application of clinical practice recommendations: from official to individual clinical policy. Am J Prev Med 1988; 4 (4 suppl): S77-S94
19. Eisenberg JM: Doctors' Decisions and the Cost of Medical Care, Health Administration Press Perspective, Ann Arbor, Mich, 1986
20. Eddy DM: Clinical policies and the quality of clinical practice. N Engl J Med 1982; 307: 343-347
21. Davis DA, Thomson MA, Oxman AD et al: Evidence for the effectiveness of CME: a review of 50 randomized controlled trials. JAMA 1992; 268: 1111-1117
22. Gyorkos TW, Tannenbaum TN, Abrahamowicz A et al: Evaluation of the effectiveness of immunization delivery methods. Can J Public Health 1994; 85 (suppl 1): S37-S40
23. Waddell DL: The effects of continuing education on nursing practice: a meta-analysis. J Contin Educ Nurs 1991; 22: 113-118
24. Mugford M, Banfield P, O'Hanlon M: Effects of feedback of information on clinical practice: a review. [review] BMJ 1991; 303: 398-402
25. Murrey KO, Gottlieb LK, Schoenbaum SC: Implementing clinical guidelines: a quality management approach to reminder systems. [review] QRB Qual Rev Bull 1992; 18: 423-433
26. Green LW, Eriksen MP, Schor EL: Preventive practices by physicians: behavioral determinants and potential interventions. [review] Am J Prev Med 1988; 4 (4 suppl): S101-S107
27. Lomas J: Teaching Old (and Not So Old) Docs New Tricks: Effective Ways to Implement Research Findings. No 93-4 of CHEPA Working Paper Series, McMaster University, Hamilton, Ont, 1993
28. Davis DA, Fox RD (eds): The Physician as Learner: Linking Research to Practice, American Medical Association Press, Chicago, 1994: xiii
29. Avorn J, Soumerai SB: Improving drug-therapy decisions through educational outreach. A randomized controlled trial of academically based detailing. N Engl J Med 1983; 308: 1457-1463
30. Evans CE, Haynes RB, Birkett NJ et al: Does a mailed continuing education program improve physician performance? Results of a randomized trial in antihypertensive care. JAMA 1986; 255: 501-504
31. Lomas J, Enkin M, Anderson GM et al: Opinion leaders versus audit and feedback to implement practice guidelines. JAMA 1991; 265: 2202-2207
32. Marton KI, Tul V, Sox HC: Modifying test-ordering behaviour in the outpatient medical clinic. Arch Intern Med 1985; 145: 816-821
33. Meyer TJ, Van Kooten D, March S et al: Reduction of polypharmacy by feedback to clinicians. J Gen Intern Med 1991; 6: 133-136
34. Maiman LA, Becker MH, Liptak GS et al: Improving pediatricians' compliance-enhancing practices: a randomized trial. Am J Dis Child 1988; 142: 773-779
35. Wilson DM, Taylor W, Gilbert JR et al: A randomized trial of a family physician intervention for smoking cessation. JAMA 1988; 260: 1570-1574
36. McConnell TS, Cushing AH, Bankhurst AD et al: Physician behaviour modification using claims data: tetracycline for upper respiratory tract infection. West J Med 1982; 137: 448-450
37. Avorn J, Soumerai SB, Everitt DE et al: A randomized trial of a program to reduce the use of psychoactive drugs in nursing homes. N Engl J Med 1992; 327: 168-173
38. Soumerai SB, Salem-Schatz S, Avorn J et al: A controlled trial of educational outreach to improve blood transfusion practice. JAMA 1993; 270: 961-966
39. Cummings SR, Coates TJ, Richard RJ et al: Training physicians in counselling about smoking cessation, a randomized trial of the "quit for life" program. Ann Intern Med 1989; 110: 640-647
40. Dietrich AJ, O'Connor GT, Keller A et al: Cancer: improving early detection and prevention; a community practice randomised trial. BMJ 1992; 304: 687-691
41. Putnam RW, Curry L: Impact of patient care appraisal on physician behaviour in the office setting. CMAJ 1985; 132: 1025-1029
42. Stross JK, Hiss RG, Watts CM et al: Continuing education in pulmonary disease for primary care physicians. Am Rev Respir Dis 1983; 127: 739-746
43. Stross JK, Bole GG: Evaluation of an educational program for primary care practitioners on the management of osteoarthritis. Arthritis Rheum 1985; 28: 108-111
44. Stross JK, Banwell BF, Wolf FM et al: Evaluation of an education program on the management of rheumatic diseases for physical therapists. J Rheumatol 1986; 13: 374-378
45. Gehlbach SH, Wilkinson WE, Hammond WE et al: Improving drug prescribing in a primary care practice. Med Care 1984; 22: 193-201
46. Hershey CO, Porter DK, Breslau D et al: Influence of simple computerized feedback on prescription charges in an ambulatory clinic. Med Care 1986; 24: 472-481
47. Cohen SJ, Stookey GK, Katz BP et al: Encouraging primary care physicians to help smokers quit. Ann Intern Med 1989; 110: 648-652
48. Cummings SR, Richard RJ, Duncan CL et al: Training physicians about smoking cessation: a controlled trial in private practices. J Gen Intern Med 1989; 4: 482-489
49. Vinicor F, Cohen SJ, Mazzuca SA et al: DIABEDS: A randomized trial of the effects of physician and/or patient education on diabetes patient outcomes. J Chronic Dis 1987; 40: 345-356
50. Magruder-Habib K, Zung WK, Feussner JR: Improving physicians' recognition and treatment of depression in general medical care. Med Care 1990; 28: 239-250
51. Rubenstein LV, Calkins DR, Young RT et al: Improving patient function: a randomized trial of functional disability screening. Ann Intern Med 1989; 111: 836-842
52. Jennett PA, Laxdal OE, Hayton RC et al: The effects of continuing medical education on family doctor performance in office practice: a randomized control study. Med Educ 1988; 22: 139-145
53. White CW, Albanese MA, Brown DD et al: The effectiveness of continuing medical education in changing the behaviour of physicians caring for patients with acute myocardial infarction. Ann Intern Med 1985; 102: 689-692
54. Putnam RW, Curry L: Physicians' participation in establishing criteria for hypertension management in the office: Will patient outcomes be improved? CMAJ 1989; 140: 806-809
55. Mazzuca SA, Vinicor F, Einterz RM et al: Effects of the clinical environment on physicians' response to postgraduate medical education. Am Educ Res J 1990; 27: 473-478
56. Adams ID, Chan M, Clifford PC et al: Computer aided diagnosis of acute abdominal pain: a multicentre study. BMJ 1986; 293: 800-804
57. Barnett GO, Winickoff RN, Morgan MM et al: A computer-based monitoring system for follow-up of elevated blood pressure. Med Care 1983; 21: 400-409
58. Bass MJ, McWhinney IR, Donner A: Do family physicians need medical assistants to detect and manage hypertension? CMAJ 1986; 134: 1247-1255
59. Berwick DM, Coltin KL: Feedback reduces test use in a health maintenance organization. JAMA 1986; 255: 1450-1454
60. Billi JE, Hejna GF, Wolf FM et al: The effects of a cost-
    education program on hospital charges. J Gen Intern Med 1987; 306-311
61. Britton ML, Lurvey PL: Impact of medication profile review on prescribing in a general medicine clinic. Am J Hosp Pharm 1991; 48: 265-270
62. Bulpitt CJ, Beilin LJ, Coles EC et al: Randomised controlled trial of computer-held medical records in hypertensive patients. BMJ 1976; 1: 677-679
63. Buntinx F, Knottnerus JA, Crebolder HFJM et al: Reactions of doctors to various forms of feedback designed to improve the sampling quality of cervical smears. Qual Assur Health Care 1992; 4: 161-166
64. Buntinx F, Knottnerus JA, Crebolder HFJM et al: Does feedback improve the quality of cervical smears? A randomized controlled trial. Br J Gen Pract 1993; 43: 194-198
65. Chambers CV, Balaban DJ, Carlson BL et al: Microcomputer-generated reminders, improving the compliance of primary care physicians with mammography screening guidelines. J Fam Pract 1989; 29: 273-280
66. Chase CR, Vacek PM, Shinozaki T et al: Medical information management: improving the transfer of research results to presurgical evaluation. Med Care 1983; 21: 410-424
67. Chassin MR, McCue SM: A randomized trial of medical quality assurance. JAMA 1986; 256: 1012-1016
68. Cheney C, Ramsdell JW: Effect of medical records' checklists on implementation of periodic health measures. Am J Med 1987; 83: 129-136
69. Cohen DI, Jones P, Littenberg B et al: Does cost information availability reduce physician test usage? Med Care 1982; 20: 286-292
70. Cohen SJ, Weinberger M, Hui SL et al: The impact of reading on physicians' non-adherence to recommended standards of medical care. Soc Sci Med 1985; 21: 909-914
71. Cohen SJ, Christen AG, Katz BP et al: Counselling medical and dental patients about cigarette smoking: the impact of nicotine gum and chart reminders. Am J Public Health 1987; 77: 313-316
72. Cowan JA, Heckerling PS, Parker JB: Effect of a fact sheet reminder on performance of the periodic health examination: a randomized controlled trial. Am J Prev Med 1992; 8: 104-109
73. Davidoff F, Goodspeed R, Clive J: Changing test ordering behaviour, a randomized controlled trial comparing probabilistic reasoning with cost-containment education. Med Care 1989; 27: 45-58
74. Dickinson JC, Warshaw GA, Gehlbach SH et al: Improving hypertension control: impact of computer feedback and physician education. Med Care 1981; 19: 843-854
75. Everett GD, deBlois S, Chang PF et al: Effect of cost education, cost audits, and faculty chart review on the use of laboratory services. Arch Intern Med 1983; 143: 942-944
76. Garett LE Jr, Hammond WE, Stead WW: The effects of computerized medical records on provider efficiency and quality of care. Methods Inf Med 1986; 25: 151-157
77. Gullion DS, Tschann JM, Adamson TE et al: Management of hypertension in private practices: a randomized controlled trial in continuing medical education. J Contin Educ Health Prof 1988; 8: 239-255
78. Headrick LA, Speroff T, Pelecanos HI et al: Efforts to improve compliance with the national cholesterol education program guidelines. Arch Intern Med 1992; 152: 2490-2496
79. Heale J, Davis D, Norman G et al: A randomized controlled trial assessing the impact of problem-based versus didactic teaching methods in CME. Proc Res Med Educ 1988; 27: 72-77
80. Hershey CO, Goldberg HI, Cohen DI: The effect of computerized feedback coupled with a newsletter upon outpatient prescribing charges. Med Care 1988; 26: 88-93
81. Johnson RE, Campbell WH, Azevedo DJ et al: Studying the impact of patient drug profiles in an HMO. Med Care 1976; 14: 799-807
82. Johnson RE, Campbell WH, Azevedo DJ et al: Examining physicians' drug order recording behavior. Med Care 1978; 16: 408-416
83. Kopelow ML, Schnabl GK, Hassard TH et al: Assessing practicing physicians in two settings using standardized patients. Acad Med 1992; 67 (10 suppl): S19-S21
84. Kottke TE, Brekke ML, Solberg LI et al: A randomized trial to increase smoking intervention by physicians. JAMA 1989; 261: 2101-2106
85. Levinson W, Roter D: The effects of two continuing medical education programs on communication skills of practicing primary care physicians. J Gen Intern Med 1993; 8: 318-324
86. Linn BS: Continuing medical education impact on emergency room burn care. JAMA 1980; 244: 565-570
87. Litzelman DK, Dittus RS, Miller ME et al: Requiring physicians to respond to computerized reminders improves their compliance with preventive care protocols.
    J Gen Intern Med 1993; 8: 311-317
88. Manheim LM, Feinglass J, Hughes R et al: Training house officers to be cost conscious: effects of an educational intervention on charges and length of stay. Med Care 1990; 28: 29-42
89. Martin AR, Wolf MA, Thibodeau LA et al: A trial of two strategies to modify the test-ordering behavior of medical residents. N Engl J Med 1980; 303: 1330-1336
90. Mayefsky JH, Foye HR: Use of a chart audit: teaching well-child care to paediatric house officers. Med Educ 1993; 27: 170-174
91. McAlister NH, Covvey HD, Tong C et al: Randomised controlled trial of computer assisted management of hypertension in primary care. BMJ 1986; 293: 670-674
92. McDonald CJ: Use of a computer to detect and respond to clinical events: its effect on clinician behavior. Ann Intern Med 1976; 84: 162-167
93. McDonald CJ: Protocol-based computer reminders, the quality of care and the non-perfectibility of man. N Engl J Med 1976; 295: 1351-1355
94. McDonald CJ, Wilson GA, McCabe GP: Physician response to computer reminders. JAMA 1980; 244: 1579-1581
95. McDonald CJ, Hui SL, Smith DM et al: Reminders to physicians from an introspective computer medical record, a two-year randomized trial. Ann Intern Med 1984; 100: 130-138
96. McDonald CJ, Hui SL, Tierney WM: Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics. MD Comput 1992; 9: 304-312
97. McPhee SJ, Bird JA, Jenkins CNH et al: Promoting cancer screening: a randomized controlled trial of three interventions. Arch Intern Med 1989; 149: 1866-1872
98. McPhee SJ, Bird JA, Fordham D et al: Promoting cancer prevention activities by primary care physicians: results of a randomized controlled trial. JAMA 1991; 266: 538-544
99. Morgan M, Studney DR, Barnett GO et al: Computerized concurrent review of prenatal care. QRB 1978; 4: 33-36
100. Nattinger AB, Panzer RJ, Janus J: Improving the utilization of screening mammography in primary care settings. Arch Intern Med 1989; 149: 2087-2092
101. Norton PG, Dempsey LJ: Self-audit: its effect on quality of care. J Fam Pract 1985; 21: 289-291
102. Palmer RH, Louis TA, Hsu LN et al: A randomized controlled trial of quality assurance in sixteen ambulatory care practices. Med Care 1985; 23: 751-770
103. Perera DR, LoGerfo JP, Shulenberger E et al: Teaching sigmoidoscopy to primary care physicians: a controlled study of continuing medical education. J Fam Pract 1983; 16: 785-788
104. Pollack MM, Getson PR: Pediatric critical care cost containment: combined actuarial and clinical program. Crit Care Med 1991; 19: 12-20
105. Pozen MW, D'Agostino RB, Mitchell JB et al: The usefulness of a predictive instrument to reduce inappropriate admissions to the coronary care unit. Ann Intern Med 1980; 92: 238-242
106. Pozen MW, D'Agostino RB, Selker HP et al: A predictive instrument to improve coronary-care-unit admission practices in acute ischemic heart disease: a prospective multi-center clinical trial. N Engl J Med 1984; 310: 1273-1278
107. Restuccia JD: The effect of concurrent feedback in reducing inappropriate hospital utilization. Med Care 1982; 20: 46-62
108. Rodman JH, Jelliffe RW, Kolb E et al: Clinical studies with computer-assisted initial lidocaine therapy. Arch Intern Med 1984; 144: 703-709
109. Rogers JL, Haring OM: The impact of a computerized medical record summary system on incidence and length of hospitalization. Med Care 1979; 17: 618-630
110. Rogers JL, Haring OM, Wortman PM et al: Medical information systems: assessing impact in areas of hypertension, obesity and renal disease. Med Care 1982; 20: 63-74
111. Rosser WW, McDowell I, Newell C: Use of reminders for preventive procedures in family medicine. CMAJ 1991; 145: 807-814
112. Sanazaro PJ, Worth RM: Concurrent quality assurance in hospital care. N Engl J Med 1978; 298: 1171-1177
113. Sibley JC, Sackett DL, Neufeld V et al: A randomized controlled trial of continuing medical education. N Engl J Med 1982; 306: 511-515
114. Steele MA, Bess DT, Franse VL et al: Cost effectiveness of two interventions for reducing outpatient prescribing costs. DICP 1989; 23: 497-500
115. Stross JK, Bole GG: Evaluation of a continuing education program in rheumatoid arthritis. Arthritis Rheum 1980; 23: 846-849
116. Tape TG, Campbell JR: Computerized medical records and preventive health care: success depends on many factors. Am J Med 1993; 94: 619-625
117. Tape TG, Givner N, Wigton RS et al: Process in ambulatory care: a controlled clinical trial of computerized records. In Greenes R (ed): Proceedings of the 12th Annual Symposium on Computer Applications in Medical Care, SCAMC Inc, Washington, 1988: 749-752
118. Tierney WM, Hui SL, McDonald CJ: Delayed feedback of physician performance versus immediate reminders to perform preventive care. Med Care 1986; 24: 659-666
119. Tierney WM, McDonald CJ, Hui SL et al: Computer predictions of abnormal test results effects on outpatient testing. JAMA 1988; 259: 1194-1198
120. Tierney WM, Miller ME, McDonald CJ: The effect on test ordering of informing physicians of the charges for outpatient diagnostic tests. N Engl J Med 1990; 322: 1499-1504
121. Turner RC, Waivers LE, O'Brien K: The effect of patient-carried reminder cards on the performance of health maintenance measures. Arch Intern Med 1990; 150: 645-647
122. Weingarten MA, Bazel D, Shannon HS: Computerized protocol for preventive medicine: a controlled self-audit in family practice. Fam Pract 198; 6: 120-124
123. Wexler JR, Swender PT, Tunnessen WW et al: Impact of a system of computer-aided diagnosis. Am J Dis Child 1975; 129: 203-205
124. White RH, Hong R, Venook AP et al: Initiation of warfarin therapy. J Gen Intern Med 1987; 2: 141-148
125. White PT, Pharoah CA, Anderson HR et al: Randomized controlled trial of small group education on the outcome of chronic asthma in general practice. J R Coll Gen Pract 1989; 39: 182-186
126. Wigton RS, Zimmer JL, Wigton JH et al: Chart reminders in the diagnosis of anemia. JAMA 1981; 245: 1745-1747
127. Winickoff RN, Coltin KL, Morgan MM et al: Improving physician performance through peer comparison feedback. Med Care 1984; 22: 527-534
128. Wones RG: Failure of low-cost audits with feedback to reduce laboratory test utilization. Med Care 1987; 25: 78-82
129. Hong SW, Ching TY, Fung JPM et al: The employment of ward opinion leaders for continuing education in the hospital. Med Teach 1990; 12: 209-217
130. Raisch DW, Bootman JL, Larson LN et al: Improving antiulcer agent prescribing in a health maintenance organization. Am J Hosp Pharm 1990; 47: 1766-1773
131. Freemantle N, Grilli R, Grimshaw J et al: Implementing findings of medical research: the Cochrane Collaboration on Effective Professional Practice. Q Health Care 1995; 4: 45-47

[Table of Contents]

---

Disclaimer