Why randomized controlled trials fail but needn't: a new series is launched

David L. Sackett, John Hoey

CMAJ 2000;162(9):1301-2

This growth has major implications for patients, their clinicians and the agencies responsible for funding RCTs. Ever larger numbers of clinicians are being asked to invite ever larger numbers of patients to enter RCTs, and both groups want to be certain that patient care will not be compromised and that participation will lead to valid, credible, important advances in the treatment or prevention of disease. Moreover, the time lag between the publication of individual trials and their systematic review by groups such as the Cochrane Collaboration means that patients and clinicians will often have to make sense out of growing numbers of individual reports. Finally, as government after government fails to provide their nonprofit independent granting agencies with sufficient resources to meet society's growing demand for high-quality evidence,¹ the drug industry, with its inescapable competition between health and profit, pays and calls the tune for more and more RCTs.²

Some RCTs are flawed. Often these flaws are detectable only after the trial is complete, but sometimes they are apparent at the outset. How can the alert clinician and patient avoid being misled by faulty RCT results and, more important, avoid participating in faulty RCTs at their inception? Beginning in this issue (page 1311) CMAJ will publish an occasional series of essays on why RCTs fail but needn't. Their intentions are 3. First, they intend to help clinicians advise the patient who is eligible for, or has been invited to join, an RCT by showing both of them how to determine whether the proposed trial is likely to generate results that are valid, clinically useful and of sufficient value to the patient that they ought to consider joining it. Second, these essays intend to help clinicians and patients decide whether the report of an RCT is true enough and clinically sensible enough for its results to be offered by clinicians and requested by patients. The series' third objective, narrower but deeper, is to help clinicians at any stage of their training and experience who are contemplating becoming co- or principal investigators in an RCT and want to evade the avoidable errors.

There are plenty of excellent textbooks on how to conduct RCTs the right way.^3,4,5,6,7 The CMAJ series is based on the premise that it is at least as instructive, and certainly more stimulating and fun, to use RCTs that were done the wrong way as the starting point for learning. The series' author (D.L.S.), involved to varying degrees in over 200 RCTs over the last 40 years, claims pre-eminence in knowing ways to fail in the design, conduct, analysis and interpretation of RCTs.

Each essay begins with a vignette, taken from the author's career at the interface between clinical medicine and clinical research methods, and shows how an RCT (almost) failed. After reporting the effect of this failure on the RCT's conclusions, the essay tracks the failure back to its root cause. Next, some specific principles are described that, had they been acted upon, would have prevented the failure. Wherever possible these principles will be based on empirical methodological research, such as that being gathered and systematically reviewed by the Cochrane Empirical Methodological Studies Working Group. Finally, these principles are converted into precise preventive strategies to avoid similar failures in future RCTs. Throughout, the emphasis will be on clarity for front-line clinicians; arithmetic will be minimized and statistical formulas avoided like the plague.

The preventive strategies that appear in this series make no claim of exclusivity; sometimes there will be other ways to rescue (at least in theory) the RCTs in question. The strategies presented, however, have actually been applied by the author and thus have worked at least once. Occasionally these proposals will disagree with established views on how RCTs should be designed and executed, and often they will challenge both the assumptions and the behaviours of contemporary trialists, ethics committees and trial monitors. Readers who disagree, have better ideas or simply feel peevish are encouraged to take it out on the author in the Letters column.

Competing interests: Dr. Sackett has been wined, dined, supported, transported, and paid to speak by countless pharmaceutical firms for over 40 years, beginning with 2 research fellowships and interest-free loans that allowed him to stay to finish medical school. Dozens of his randomized trials have been supported in part, but never in whole, by pharmaceutical firms, who never received or analysed primary data and never had veto power over any reports, presentations or publications of the results. He has twice worked as a paid consultant to advise pharmaceutical firms whether their products caused lethal side-effects (on both occasions he told them Yes). He has testified as an unpaid expert witness for a stroke victim who successfully sued a manufacturer of oral contraceptives, and he has been paid by a second to develop "levels of evidence" for determining the causation of adverse drug reactions. His wife inherited and sold stock in a pharmaceutical company. While head of a division of medicine he enforced the banning of drug-detail personnel from clinical teaching units (despite the threat of withdrawal of drug-industry funding for resident research projects). He received the Pharmaceutical Manufacturers' Association of Canada Medal of Honour (and cash) for "contributions to medical science in Canada" for the decade 1984­1994. No competing interests declared for Dr. Hoey.

Send a letter to the editor Envoyez une lettre à la rédaction

This commentary was posted early on eCMAJ (Apr. 13, 2000).

1. Sackett DL. Time to put the Canadian Institute for Health Research on trial [editorial]. CMAJ 1999;161(11):1414-5.
2. Hailey D: Scientific harassment by pharmaceutical companies: time to stop. CMAJ 2000;162(2):212-3.
3. Schwartz D, Flamant R, Lellouch J. Clinical trials [translated by Healy MJR]. Toronto: Academic Press; 1980.
4. Friedman LM, Furberg CD, DeMets DL. Fundamentals of clinical trials. London: John Wright; 1981.
5. Shapiro SH, Louis TA, editors. Clinical trials: issues and answers. New York: Marcel Dekker; 1983.
6. Pocock SJ. Clinical trials: a practical approach. Toronto: John Wiley & Sons; 1983.
7. Jadad A. Randomised controlled trials. London: BMJ Books; 1998.

© 2000 Canadian Medical Association or its licensors