Canadian Medical Association JournalHome

Table of Contents
Free eCMAJ TOC

Back issues
Supplements
Selected series

eLetters
About this journal
Info for authors

PubMed

Evidence-based medicine: a commentary on common criticisms

Sharon E. Straus, Finlay A. McAlister
CMAJ 2000;163(7):837-41[PDF]


Abstract

Discussions about evidence-based medicine engender both negative and positive reactions from clinicians and academics. Ways to achieve evidence-based practice are reviewed here and the most common criticisms described. The latter can be classified as "limitations universal to the practice of medicine," "limitations unique to evidence-based medicine" and "misperceptions of evidence-based medicine." Potential solutions to the true limitations of evidence-based medicine are discussed and areas for future work highlighted.


Evidence-based medicine has been defined1 as "the process of systematically finding, appraising, and using contemporaneous research findings as the basis for clinical decisions." The reader will immediately recognize that this is not a new process — clinicians have always striven to combine their clinical expertise and their patients' values with the best available evidence. However, interest in evidence-based medicine has grown exponentially since the coining of the term in the early 1990s1,2 (from 1 MEDLINE citation in 1992 to 2957 in February 2000) and has led to calls to increase the teaching of evidence-based medicine at the undergraduate and postgraduate levels.3

Evidence-based medicine is a multistep process (Table 1).4 Clinicians can incorporate evidence into their practices in 3 ways. First is the "doing" mode, in which at least the first 4 steps in Table 1 are carried out before an intervention is offered. Second is the "using" mode, in which searches are restricted to evidence sources that have already undergone critical appraisal by others, such as evidence-based guidelines or evidence summaries (thus skipping step 3 in Table 1). Third is the "replicating" mode, in which the decisions of respected opinion leaders are followed (abandoning at least steps 2 and 3). Of course, even clinicians trained to the "doing" level move back and forth between these modes, typically depending on whether they are dealing with clinical problems they encounter frequently or only rarely.

Discussions about evidence-based medicine engender both negative and positive reactions from clinicians and academics, and this paper describes our efforts to categorize and respond to the most common criticisms.


Literature search

Criticisms of evidence-based medicine were systematically sought through an electronic literature search, from published surveys of front-line clinicians,56,7,8,9,10 and from the written records of questions posed during seminars held around the world from 1994 to 1999 by the director of the NHS Research and Development Centre for Evidence-Based Medicine in Oxford, United Kingdom (http://cebm.jr2.ox.ac.uk) (Dr. David Sackett, personal communication, 1999). MEDLINE was searched (without language restrictions) for articles published from 1966 to 1999 using the following search strategy: "evidence-based medicine" [MH] OR ("evidence-based" [TW] AND "medicine" [TW]) OR ("evidence" [TW] and "based" [TW] and "medicine" [TW]) AND "limitations" [MH] OR "criticisms" [MH] OR "limitations" [TW] OR "criticisms" [TW]. The titles and abstracts of the 95 articles identified (and the full text of the 47 felt to be potentially relevant) were reviewed by both of us for potential criticisms. Discrepancies (3 cases) were resolved by consensus. The reference lists of the retrieved articles were searched and experts in the field contacted in order to identify other relevant articles.

Criticisms expressed more than once (i.e., in 1 or more articles or at 1 or more seminars) were identified by content or qualitative analysis and included in this paper. The classification of criticisms was developed by both of us after a review of the criticisms that had been identified, with input from several members of the Evidence-Based Medicine Working Group (see Acknowledgements).


Limitations

Our classification of the commonly cited limitations of evidence-based medicine appears in Table 2. The first 3 limitations outlined here are not unique to evidence-based medicine but are universally encountered in the practice of medicine.

Shortage of coherent, consistent scientific evidence

Clinicians frequently encounter situations in which there is no relevant evidence from either basic or applied research.11 The exponential growth in clinical research, coupled with international efforts to identify, sort and rationalize this evidence systematically, will eventually close many of these gaps. However, until that time, clinical experience and reasoning (based on principles derived from basic scientific research) "must be applied to traverse the many grey zones of practice."11

Even when evidence exists, difficulties arise when it is inconclusive, inconsistent with previous studies, irrelevant to clinical realities or of poor quality.12 Indeed, in few research studies are the results reported in the context of the totality of available evidence.13 Although systematic reviews are a potential solution to this problem, inadequate attention to their methodology may lead to surprising variation in results and recommendations.14 Thus, steps must be taken to improve and standardize the methodology and reporting of systematic reviews.15,16

In our view, these problems, far from constituting a limitation of evidence-based medicine, highlight the importance of training clinicians to appraise research critically, to recognize the indeterminacy represented by confidence intervals and to apply the evidence, taking into account their patients' unique risks and values.

Difficulties in applying evidence to the care of individual patients

The universal occurrence of biological variation hampers attempts to extrapolate evidence, whether from basic or applied research, to individual patients. Thus, we disagree with the criticism that this problem is unique to evidence-based medicine.17

To address this concern, researchers increasingly use particular study architectures (such as "N-of-1" and large, simple trials) and carry out judicious subgroup analyses that are intended to improve our ability to extrapolate research results to individual patients in the "real world."18,19,20 Furthermore, novel formats that enable clinicians to describe evidence to each other and to individual patients have been developed. For example, the number needed to treat and the number needed to harm have gained acceptance as useful means to make the evidence relevant to the individual patient.21,22 Patient values can be incorporated into these expressions by means of formal decision analysis or bedside simplifications such as the likelihood of being helped or harmed.23

Barriers to the practice of high-quality medicine

The gap between the demand for health care and the resources available to meet that demand is growing and results in clinicians having to care for more patients in less time.24 This pressure impairs the ability of clinicians to apply any evidence, whether from basic or applied science, to their patients.

Per capita health expenditures have more than doubled over the past 2 decades, and over one-third of this rise is owing to the increased intensity of services.25 Thus, it is not surprising that purchasers have increasingly attempted to control escalating health care costs by setting priorities and rationing services (explicitly as in the case of efficacious but expensive drugs like sildenafil in the United Kingdom26 or implicitly as in proposals to provide antihypertensive drugs only to patients deemed to be at high risk27).

Some have criticized evidence-based medicine for this curtailing of clinical freedom.28 However, this process was well underway before the elucidation of evidence-based medicine. Indeed, increased attention to the principles of evidence-based medicine among policy-makers and purchasers should lead to the preservation of funding for proven efficacious therapies and the elimination only of interventions that have been shown to be harmful or ineffective.

The need to develop new skills

Unquestionably the practice of evidence-based medicine requires the acquisition and development of new skills (in literature searching and critical appraisal). Their mastery and application are formidable tasks and should not be underestimated. However, the assertion of some critics that clinicians are not interested in learning such skills is contradicted by surveys of practising clinicians.5,6,7,8,9,10

Evidence-based medicine skills can be acquired at any stage in clinical training. Incorporating their acquisition into the routine of grand rounds, postgraduate and undergraduate seminars, and "morning report" integrates them with the other skills being developed in these settings.29 Members of clinical teams at various stages of training can collaborate by sharing the searching and appraising tasks. The different skills required for practising in the "using" and "doing" modes can be learned in sequence, thus avoiding learner overload. Indeed, for many clinicians the most appropriate means to achieving evidence-based practice may be through the "using" mode outlined earlier in this article.

Limited time and resources

Critics of evidence-based medicine have correctly pointed out that its practice may require time and resources unavailable to the busy clinician.30

Important developments to help overcome this barrier include the systematic reviews generated by the Cochrane Collaboration, the growing numbers of evidence-based journals (such as ACP Journal Club) containing abstracts of quality- and relevance-filtered studies, and the creation of "best evidence" sections in a number of established journals. Moreover, electronic searching is increasingly being made available at the point of care, cutting time of access to the evidence to a few seconds.31 The generation of databases of critically appraised topics (1-page summaries of evidence relevant to common clinical questions),32 which can be quickly accessed at the point of care,31 represent another time- and energy-saving solution for busy clinicians, as is the division of labour between members of the clinical team noted previously. Finally, although we can generate several questions for each patient we see (and become paralyzed by trying to address them all), we can pare them down to just one by balancing the question that would be most important to our patient's well-being against that which may be answered most easily, that which is most interesting to us and that which is most likely to be raised by subsequent patients.4

Paucity of evidence that evidence-based medicine "works"

Although agreeing that evidence-based medicine makes good sense in theory, its critics have quite appropriately demanded evidence for whether it improves patient out-comes.33

No such evidence is available from randomized trials because no investigative team has yet overcome the problems of sample size, contamination and blinding that such a trial raises. Moreover, it is questionable whether withholding access to evidence from the control arm in such a trial would be ethical. However, outcomes researchers consistently document that patients who receive proven efficacious therapies have better outcomes than those who do not.34,35,36

Given this evidence, the focus has shifted from whether to teach evidence-based medicine to how to do so, and recent randomized trials have compared alternative strategies for enhancing evidence-based practice. These trials have both discredited traditional approaches such as didactic lectures and validated newer approaches such as academic detailing (one-on-one educational sessions with a content expert) and seeking advice from local opinion leaders.37


Misperceptions

Many criticisms of evidence-based medicine stem from misperceptions or misrepresentations and may be answered by careful consideration of the definition of evidence-based medicine and the 5 steps outlined in Table 1.4 We include these misperceptions in Table 2 in order to clarify that they represent only pseudolimitations of evidence-based medicine.

For example, criticisms that evidence-based medicine denigrates clinical expertise,17,38 ignores patients' values39 or promotes "cookbook medicine"38,40 arise because of a failure to appreciate step 4 in Table 1.41 Moreover, because evidence-based medicine is cost-indifferent and directed toward maximizing the quality of life of individual patients, it may (and often does) result in policies that will increase, rather than decrease, costs (consider the provision of statin drugs for normocholesterolemic patients following myocardial infarction).42,43 The most commonly cited pseudolimitation is that evidence-based medicine is an ivory-tower concept;44 however, surveys and audits of front-line clinicians clearly refute this claim.5,6,7,8,9,10,45,46,47,48,49,50 Furthermore, a common misperception is that evidence-based medicine is limited to doing, as opposed to using, clinical research.17 Although a minority of practitioners of evidence-based medicine also do research, its practice is a method for providing care for patients, not a method for performing research.

The final misperception is that only randomized trials or systematic reviews constitute the "evidence" in evidence-based medicine.44,51 Even the most vehement protagonist of evidence-based medicine would acknowledge that several sources of evidence may inform clinical decision-making. However, the practice of evidence-based medicine stresses finding the best available evidence to answer a question, and hierarchies of evidence have been developed to help describe the quality of evidence that may be found to answer various questions. Thus, randomized clinical trials are usually considered the "gold standard" for establishing the effects of an intervention, but they are not the best sources for answering questions about diagnosis, prognosis or harm. Although this hierarchy has been criticized for devaluing the basic sciences,51 we would submit that numerous studies over the past 4 decades have demonstrated the potential fallibility of extrapolating directly from the bench to the bedside, without the intervening step of proving the assumptions to be valid in human subjects.52,53,54


Conclusion

Evidence-based medicine, like other models of care,55 has limitations, and further innovation and study are required to resolve the issues raised in this paper. In particular, efforts need to be directed toward improving clinicians' access to evidence at the point of care; developing better methods of describing evidence to patients in order to facilitate shared decision-making; and conducting studies to test whether and how evidence-based medicine affects processes of care and patient outcomes.

Competing interests: None declared for Dr. Straus. Dr. McAlister has received speaker fees from Merck Frosst Canada & Co.

Contributors: Both authors took part in the conception, design and analysis of the study and wrote the first draft of this paper. Dr. McAlister wrote subsequent drafts, and both authors revised them for their intellectual content.

Acknowledgements: We thank Drs. David Sackett, Ian Chalmers, Scott Richardson, William Rosenberg, Brian Haynes and Gordon H. Guyatt for their helpful comments on earlier versions of this manuscript.

Financial support was received from the Alberta Heritage Foundation for Medical Research, Edmonton, Alta.


Dr. Straus is with the Division of General Internal Medicine, Mount Sinai Hospital, Toronto, Ont., and Dr. McAlister is with the Division of General Internal Medicine, University of Alberta Hospital, and is Population Health Investigator, Alberta Heritage Foundation for Medical Research, Edmonton, Alta.

This article has been peer reviewed.

Correspondence to: Dr. F. McAlister, Division of General Internal Medicine, 2E3.24 WMC, University of Alberta Hospital, 8440 112 St., Edmonton AB T6G 2R7; fax 780 407-2680; Finlay.McAlister@ualberta.ca


References

    1.   Evidence-Based Medicine Working Group. A new approach to teaching the practice of medicine. JAMA 1992;268:2420-5.
    2.   Guyatt GH. Evidence-based medicine. ACP J Club 1991;114:A-16.
    3.   Bordley DR, Fagan M, Theige D. Evidence-based medicine: a powerful educational tool for clerkship education. Am J Med 1997;102:427-32.
    4.   Sackett DL, Richardson WS, Rosenberg WMC, Haynes RB. Evidence-based medicine: how to practice and teach EBM. London (UK): Churchill Livingstone; 1997.
    5.   McColl A, Smith H, White P, Field J. General practitioners' perceptions of the route to evidence based medicine: a questionnaire survey. BMJ 1998;316:361-5.
    6.   Tunis SR, Hayward RS, Wilson MC, Rubin HR, Bass EB, Johnston M, et al. Internists' attitudes about clinical practice guidelines. Ann Intern Med 1994;120:956-63.
    7.   McAlister FA, Graham I, Karr GW, Laupacis A. Evidence-based medicine and the practising clinician: a survey of Canadian general internists. J Gen Intern Med 1999;14:236-42.
    8.   Hagdrup N, Falshaw M, Gray RW, Carter Y. All members of primary care team are aware of importance of evidence based medicine. BMJ 1998;317:282.
    9.   Ghali WA, Saitz R, Eskew AH, Lemaire JB, Gupta M, Hershman WY. Evidence-based medicine: behaviors, skills, and attitudes of medical students. Ann R Coll Physicians Surg Can 1998;31:177-82.
    10.   Olatunbosun OA, Edouard D. Pierson RA. Physicians' attitudes toward evidence based obstetric practice: a questionnaire survey. BMJ 1998;316:365-6.
    11.   Naylor CD. Grey zones of clinical practice: some limits to evidence-based medicine. Lancet 1995;345:840-2.
    12.   Miettinen OS. Evidence in medicine: invited commentary. CMAJ 1998;158(2):215-21.
    13.   Clarke M, Chalmers I. Discussion sections in reports of controlled trials published in general medical journals. Islands in search of continents? JAMA 1998;280:280-2.
    14.   Guyatt GH, Sinclair J, Cook DJ, Glasziou P, the Evidence-Based Medicine Working Group and the Cochrane Applicability Methods Working Group. Users' guides to the medical literature: XVI. How to use a treatment recommendation. JAMA 1999;281:1836-43.
    15.   Chalmers I, Hetherington J, Elbourne D, Keirse MJN, Enkin M. Materials and methods used in synthesizing evidence to evaluate the effects of care during pregnancy and childbirth: provisions for updating and amending overviews (meta-analyses) in the light of new data and criticisms. In: Chalmers I, Enkin M, Keirse MJN, editors. Effective care in pregnancy and childbirth. Oxford (UK): Oxford University Press; 1989.
    16.   Moher D, Olkin I. Meta-analysis of randomized controlled trials. A concern for standards. JAMA 1995;274:1962-4.
    17.   Charlton BG. Restoring the balance: evidence-based medicine put in its place. J Eval Clin Pract 1997;3:87-98.
    18.   Yusuf S, Held P, Teo KK, Toretsky ER. Selection of patients for randomized controlled trials: implications of wide or narrow eligibility criteria. Stat Med 1990;9:73-86.
    19.   Dans AL, Dans LF, Guyatt GH, Richardson S, and the Evidence-Based Medicine Working Group. Users' guides to the medical literature: XIV. How to decide on the applicability of clinical trial results to your patient. JAMA 1998;279:545-9.
    20.   Guyatt GH, Sackett DL, Cook DJ, and the Evidence-Based Medicine Working Group. Users' guides to the medical literature. II. How to use an article about therapy or prevention. B. What were the results and will they help me in caring for my patients? JAMA 1994;271:59-63.
    21.   Laupacis A, Sackett DL, Roberts DS. An assessment of the clinically useful measures of the consequences of treatment. N Engl J Med 1988;318:1728-33.
    22.   Cook RJ, Sackett DL. The number needed to treat: a clinically useful measure of treatment effect. BMJ 1995;310:452-4.
    23.   McAlister FA, Straus SE, Guyatt GH, Haynes RB, and the Evidence-Based Medicine Working Group. Users' guides to the medical literature: XX. Integrating research evidence with the care of the individual patient. JAMA. 2000;283:2829-36.
    24.   Improving communication between doctors and patients. Summary and recommendations of a report of a working party of the Royal College of Physicians. J R Coll Physicians Lond 1997;31:258-9.
    25.   Eddy DM. Health system reform: Will controlling costs require rationing services? JAMA 1994;272:324-8.
    26.   Chisholm J. Viagra: botched test case for rationing. BMJ 1999;318:273-4.
    27.   McAlister FA, Laupacis A. Towards a better yardstick: the choice of treatment thresholds in hypertension. Can J Cardiol 1998;14:47-51.
    28.   Maynard A. Evidence-based medicine: an incomplete method for informing treatment choices. Lancet 1997:349:126-28.
    29.   Reilly B, Lemon M. Evidence-based morning report: a popular new format in a large teaching hospital. Am J Med 1997;103:419-26.
    30.   Jacobson LD, Edwards AGK, Granier SK, Butler CC. Evidence-based medicine and general practice. Br J Gen Pract 1997;47:449-52.
    31.   Sackett DL, Straus SE. Finding and applying evidence during clinical rounds: the "evidence cart." JAMA 1998;280:1336-8.
    32.   Sauve S, Lee HN, Meade MO, Lang JD, Farkouh M, Cook DJ, et al, and the General Internal Medicine Fellowship Programme of McMaster University. The critically-appraised topic (CAT): a resident-initiated tactic for applying users' guides at the bedside. Ann R Coll Physicians Surg Can 1995;28:396-8.
    33.   Miles A, Bentley P, Polychronis A, Grey J. Evidence-based medicine: Why all the fuss? This is why. J Eval Clin Pract 1997;3:83-5.
    34.   Krumholz HM, Radford MJ, Ellerbeck EF, Hennen J, Meehan TP, Petrillo M, et al. Aspirin for secondary prevention after acute myocardial infarction in the elderly: prescribed use and outcomes. Ann Intern Med 1996;124:292-8.
    35.   Krumholz HM, Radford MJ, Wang Y, Chen J, Heiat A, Marciniak TA. National use and effectiveness of beta-blockers for the treatment of elderly patients after acute myocardial infarction. National Cooperative Cardiovascular Project. JAMA 1998;280:623-9.
    36.   Wong JH, Findlay JM, Suarez-Almazor ME. Regional performance of carotid endarterectomy: appropriateness, outcomes and risk factors for complications. Stroke 1997;28:891-8.
    37.   Thomson MA, Oxman AD, Davis DA, Haynes RB, Freemantle N, Harvey EL. Audit and feedback to improve health professional practice and health care outcomes. In: The Cochrane Library; Issue 3, 1998. Oxford (UK): Update Software.
    38.   Horwitz RI. The dark side of evidence-based medicine. Cleve Clin J Med 1996;63:320-3.
    39.   Cohn JN. Evidence-based medicine: What is the evidence? J Cardiac Failure 1996;2:159-61.
    40.   Charlton BG, Miles A. The rise and fall of EBM. Q J Med 1998;12:371-4.
    41.   Sackett DL, Rosenberg WMC, Gray JAM, Haynes RB, Richardson WS. Evidence-based medicine: what it is and what it isn't. BMJ 1996;312:71-2.
    42.   Grahame-Smith D. Evidence based medicine: socratic dissent. BMJ 1995;310:1126-7.
    43.   Sacks FM, Pfeffer MA, Moye LA, Rouleau JL, Rutherford JD, Cole TG, et al. The effects of pravastatin on coronary events after myocardial infarction in patients with average cholesterol levels. Cholesterol and Recurrent Events Trial investigators. N Engl J Med 1996;335:1001-9.
    44.   Hampton JR. Evidence-based medicine, practice variations and clinical freedom. J Eval Clin Pract 1997;3:123-31.
    45.   Ellis J, Mulligan I, Rowe, J, Sackett DL. Inpatient general medicine is evidence based. Lancet 1995;346:407-10.
    46.   Geddes JR, Game D, Jenkins NE, Peterson LA, Pottinger GR, Sackett DL. What proportion of primary psychiatric interventions are based on randomised evidence? Qual Health Care 1996;5:215-7.
    47.   Gill P, Dowell AC, Neal RP, Smith N Heywood P, Wilson AK. Evidence based general practice: a retrospective study of interventions in our training practice. BMJ 1996;312:819-21.
    48.   Kenny SE, Shankar KR, Rentala R, Lamont GL, Lloyd DA. Evidence-based surgery: interventions in a regional paediatric surgical unit. Arch Dis Child 1997;76:50-3.
    49.   Baraldini V, Spitz L, Pierro A. Evidence-based operation in paediatric surgery. Pediatr Surg Int 1998;13:331-5.
    50.   Howes N, Chagla L, Thorpe M, McCullough P. Surgical practice is evidence based. Br J Surg 1997;84:1220-3.
    51.   Swales JD. Evidence-based medicine and hypertension. J Hypertens 1999;17:1511-6.
    52.   Cobb LA, Thomas GI, Dillard DH, Merendino KA, Bruce RA. An evaluation of internal mammary artery ligation by a double-blind technic. N Engl J Med 1959;260:1115-8.
    53.   Failure of extracranial-intracranial arterial bypass to reduce the risk of ischemic stroke. The EC/IC Bypass Study Group. N Engl J Med 1985;313:1191-1200.
    54.   Echt DS, Liebson PR, Mitchell LB, Peters RW, Obias-Manno D, Barker AH, et al. Mortality and morbidity in patients receiving encainide, flecainide, or placebo. The Cardiac Arrhythmia Suppression Trial. N Engl J Med 1991;324:781-8.
    55.   Shaughnessy AF, Slawson DC, Becker L. Clinical jazz: harmonizing clinical experience and evidence-based medicine. J Fam Pract 1998;47:425-8.

 

 

Copyright 2000 Canadian Medical Association or its licensors