Canadian Medical Association Journal 1996; 155: 1697-1702
Paper reprints may be obtained from: Dr. Patricia Huston, CMAJ, 1867 Alta Vista Dr., Ottawa ON K1G 3Y6; fax 613 523-0937; pubs@cma.ca
© 1996 Canadian Medical Association
Health services research has its roots in observational research showing regional variations in practice patterns. These variations provide evidence for uneven diffusion of medical concepts and technologies into practice and may reflect legitimate uncertainty about what is the best standard of care. Today, experimental and quasi-experimental methods are also used in health services research. For example, randomized designs may be used to determine the best way of ensuring that a new treatment is adopted effectively into routine practice. However, observational studies of practice patterns and patient outcomes still predominate in the health services research literature. These observational studies may be descriptive, documenting the existence of variations in practice patterns or patient outcomes, or they may be more analytic, seeking to explain observed variations.
The costs of assessing practice patterns and health outcomes for large populations would be enormous if every study developed its own mechanisms for collecting data. Thus, health services researchers often rely on data derived from secondary sources. We focus here on health services research using secondary data sources because it is common and has specific challenges in its reporting. For the purposes of this paper, we defined these secondary sources in two ways: (1) databases that were designed for ongoing epidemiologic surveillance of medical care, not for addressing a specific hypothesis; and (2) databases that were designed for administrative purposes but are now also being used to answer research questions (Table 1).
Although much has been written on how to conduct health services research, there is a dearth of material on how best to report it. This article identifies some of the special issues arising with studies that rely on secondary data sources and offers some general guidelines to authors and peer reviewers of papers reporting on this genre of health services research. It will also inform readers on what to look for when assessing reports of this nature.
Because observational studies using secondary data sources generally carry no risk for physical harm to participants, individual consent to analyse anonymous data is not usually required.[2] However, if investigators use a secondary data source to identify actual clinical records, then hospital review and permission are needed before the records can be audited. Requirements are more stringent still if patients are to be contacted based on identifiers and clinical information obtained from a secondary data source. In such situations the patient's personal physician should be the intermediary for any initial overture to the patient. Informed consent is required for any study manoeuvres involving the patients directly.
For studies that will use only secondary data sources, approval from a research ethics board is usually not required.2 However, if there is any doubt, the research ethics board should be consulted. It is critical in any study using secondary data sources that the confidentiality of the information be respected. Thus, use of personal identifiers should be minimized, anonymous records should be used whenever possible, any identifiers that are necessary should be removed as soon as possible from study records and data files, and strict control of the information must be maintained. Procedures to help maintain confidentiality include ensuring that all files are password protected, securing the area where the data are held and analysed, and entering the data only into stand-alone computers that are not connected to telephone lines.
The structured abstract should be prepared in the same manner as those for original research articles.[3]
As noted, designs used in health services research may vary. A randomized controlled trial may be performed to test manoeuvres aimed at changing a particular practice pattern. Quasi-experimental, cohort and casecontrol designs are also used, often with supplementation of the secondary data with additional primary data. However, most health services researchers currently use cross-sectional, survey, beforeafter or time-series designs.
The structured abstract should briefly state not only the study design but also how secondary data sources were used. It is often useful to provide a generic description in the design section (e.g., "Retrospective cohort study, using hospital discharge abstracts") with further details in the section on outcome measures (e.g., "Readmission to hospital within 6 months after discharge, based on probabilistic matching of the index and subsequent admissions").
Introduction
In the introduction the problem under investigation needs to be clearly identified. A concise literature review should be included to provide the context for the study and should justify how this study was designed to add to current knowledge. Finally, the specific research question or objective needs to be clearly stated.
The research question
Descriptive studies usually examine variations in the rate of health care services in a specified geographic area, such as rates of hysterectomy by indication in different regions of a province.[4] Alternatively, they can look at changes in rates over time, such as the dramatic 43% relative reduction in the rate of transurethral resection of the prostate noted in Ontario for the period 199091 to 199495.[5] Usually the null hypothesis is that the difference in rates among different regions (or over time) is no greater than expected by chance alone. This presumes that some level of variation is expected, and what is expected needs to be specified beforehand.[6]
Studies of geographic variation may also use primary data to clarify some aspect of the observed variation. For example, in a study by van Walraven and associates,[7] data from a secondary source indicating areas with high and low rates of hip and knee replacement were coupled with primary data from an audit of randomly selected charts in the high- and low-service areas to determine whether case-selection criteria differed.
Analytic studies compare patterns of care and outcomes across specific groups of patients or providers, with a view to understanding why variations occur. An example of this is the assessment of factors associated with complications immediately following abortion.[8] The research question needs to be very focused, as it tests a specific hypothesis about the degree and sources of variability in patient care or outcomes.
For both descriptive and analytic studies, care must be taken to assess whether the research question has been driven unduly by data availability. The question must not be chosen and tailored to fit the data; instead, the question must be important in its own right. One indicator that the data have driven the study question is if the question looks feasible in relation to the data source but somehow misses the relevant clinical or health systems issues. Studies draw on large datasets collected without prespecified hypotheses; therefore, there is always a risk of "data dredging," with multiple passes through the data and spurious statistical significance obtained by chance alone. The very large samples, particularly with administrative data, make it possible to find statistical significance when there is little clinical or policy significance to the findings. (By clinical significance we mean effects that might be judged important by clinicians or patients; by policy significance we mean findings that might be judged worthy of action by policy-makers or administrators.) Well-defined hypotheses, with parsimony in analysis and scepticism about apparently significant findings, are crucial if meaningful results are to be obtained from secondary data sources.
Methods
General considerations
Close attention needs to be given to the methods section. The report must address the assessment of sources of both random and systematic errors.
To begin, the setting and the population under study are described. Inclusion and exclusion criteria need to be identified so that the representativeness or completeness of the sampling frame can be assessed. If the total target population was not included in the study, the sampling technique needs to be described to reveal any possible sampling bias.
A checklist for descriptive and analytic studies using secondary data sources is outlined in Table 2. If an experimental design is used, the CONSORT (Consolidated Standards of Reporting Trials) checklist for randomized trials should be consulted.[9] If studies combine secondary data analysis with primary data collection or conduct criteria-based audits of appropriateness of care across different jurisdictions, institutions or providers,[7,10,11] critical appraisal guides for utilization reviews can be consulted as a source for basic reporting requirements.[12]
Description of secondary data sources
A detailed description of the secondary data source(s) is critical. If the use or manipulation of the data is such that consent is indicated, there should be a description of how it was sought and obtained, and how confidentiality was maintained throughout the study.
The specific information obtained from the data source needs to be identified. For example, in Canada, every hospital admission is recorded in computerized discharge abstracts; health records technicians review hospital charts and, on the basis of standard criteria, enter codes for specific procedures, diagnoses and complications. If a study uses hospital discharge abstracts, the report should describe the diagnosis or procedure codes that defined the study sample. The level of detail should be sufficient to allow for possible replication of the study.
The appropriateness of the data source should be defended by showing that it truly captures the process-of-care indicator or patient outcome of interest. Ideal outcome measures such as relief of symptoms, improved functional status and level of satisfaction with care are not generally available. Instead, hospital admissions, acute care episodes and procedures are considered proxies for the burdens of illness.
The report needs to identify the validity and reliability of all relevant aspects of the secondary data sources used, including all procedures, diagnoses or outcomes of interest. Researchers may include a component in the study design in order to validate the secondary data source used, or they may cite relevant published work. Methods for validating the quality of data sources vary.[13] For hospital records, a set of charts may be reviewed independently by two or more health records technologists and the extent of interobserver agreement determined. Alternatively, clinical experts may review the original records to determine whether the information agrees with what is entered into the secondary data source. Cross-validation is also possible, whereby two or more secondary data sources addressing similar issues are compared with each other for consistency. Whatever the method, some reassurance about the quality of the data analysed is indicated.
Any linkages of patient data across two or more secondary data sources (e.g., hospital admission rates and vital statistics to capture out-of-hospital deaths), or longitudinally within the same data source, must be described. Such linkages may be on a deterministic basis, whereby records are matched exactly using unique identifiers, or on a probabilistic basis, whereby similarities in non-unique identifiers (e.g., sex, date of birth and postal code of residence) are used to determine with extremely high probability that two records are for the same person.[14] Whatever the method of linkage, the threats to privacy and confidentiality increase when individuals are identified and tracked through multiple secondary data sources. Thus, all the ethical precautions outlined earlier are particularly important when developing and using linked datasets.
Analytical and statistical issues
Whereas in most clinical studies the patient is the unit of analysis, in health services research the unit of analysis must be very clearly defined, because it may be the region, the hospital or sets of hospitals, the physician or the individual patient. Ideally, there should be some characterization of the unit of analysis, including descriptions of how the relevant services or systems differ across the units of analysis, so that readers will appreciate the range of possible causes for the findings.
Whether the study is broadly descriptive or involves detailed comparative analysis, the report should outline a series of sensitivity analyses. Such analyses assess how sensitive the results of the primary analysis are to changes in definitions or interpretations of certain independent variables. If the results change dramatically in the face of plausible alternative definitions of one or more variables, then the conclusions cannot be considered robust. These analyses are important because of the inherent limitations of secondary data. Specifically, sensitivity analyses should explore issues such as the influence of outliers, different ways of defining procedures and populations, and different methods of linking or analysing data.
Descriptive studies
Descriptive studies assessing variations in practice patterns generally involve the straightforward determination of outcomes such as mean lengths of stay by institution or rates of service per unit population. Adjustments for differences in patient groups or populations under analysis tend to be rudimentary (e.g., straightforward age and sex adjustments).
Any population-based descriptive study requires clarity about the denominators. This can be achieved by calculating rates on the basis of place of residence so that the population of interest can be defined geographically. Further analysis will then be necessary to determine whether differences in service intensity arise from a particular set of hospitals or providers. If overall variation is significant, statistical tests can be used to determine whether a given county or hospital differs significantly from the remainder of the group under analysis. However, when such individual tests are performed for a number of counties or hospitals, the risk of type I error increases because of multiple comparisons; this must be factored into the analysis or interpretation.
Analytic studies
In studies that involve more detailed attempts to understand sources of variations in processes of care, two problems recur: incomplete exploration of potential explanatory factors, and inadequate statistical power.
Significant geographic variations in rates of a specific procedure or service can arise for many reasons, including differences in disease incidence, relative availability of the procedure of interest and alternative or competing treatments, patient education and expectations, referral patterns of other physicians, and practice styles of the professionals who provide the service of interest. The study report should be explicit about whether and how these explanatory factors were explored.
Although very large numbers of patients are usually included in secondary data sources, statistical power may be limited if the unit of analysis is a small number of counties or hospitals. If this is the case, a sensitivity analysis can be undertaken at the level of individual patients, with counties or hospitals as ecological covariates.[15] Considerations of statistical power are also important when primary data collection is used to address potential sources of variation not captured by secondary data sources.[16,17] In these types of studies, the analysis goes from thousands of subjects distributed across 40 or 50 regions in a secondary dataset to what may be only 200 or 300 patients whose charts are audited. If, for example, one hypothesizes that the functional status of patients differs with the rate of surgery, one has to be sure that there will be enough patients in each cell to conduct a proper analysis. Thus, explicit justification of the sample size is necessary for analyses that rely on primary data collection.
For detailed comparisons of patients' outcomes, two special issues arise.[18] First, are the outcome measures appropriate? As we have already cautioned, the most appropriate outcomes may be missing, and inferences must instead be drawn on the basis of proxy measures. Second, were the comparison groups adequately characterized by factors that might affect the outcomes of interest? Whereas clinical epidemiologists doing a prognostic study may be concerned with determining the patient characteristics that account for different outcomes, health services researchers will attempt to control for those characteristics and find variations attributable to some aspect of the health system.
Patient outcome studies usually rely on multivariate statistical methods to adjust for factors that might affect outcome and that were nonrandomly distributed among comparison groups. Both the crude and adjusted results should be presented when multivariate methods are used. A subgroup analysis including only patients at low risk of the outcome of interest is a simple way to reduce confounding arising from subtle and unmeasured differences among comparison groups19 and should be done routinely. This is based on the assumption that confounding factors, such as incompletely measured comorbid conditions, are less likely in low-risk groups.
Results
To optimize readability, the results should follow the same sequence of items noted in the methods section. To deal with concerns about possible systematic bias, the results section should include details on missing data or excluded subjects. Tables and figures can be used liberally, with two caveats. First, tables and figures should present only data that directly address the research question. Second, data presented in tables and figures should only be highlighted in the text, to avoid unnecessary repetition. For example, in the tables all study participants should be accounted for and both the number and percentage of participants should be identified, whereas in the text the most common groups can be identified by their percentages alone.
Discussion
The most common problem with discussion sections is that they are too long; usually because the results are reiterated. A word count of 2500 for the entire text (excluding the abstract and references) should be kept in mind when writing or reviewing this section.
In the discussion section the main findings should be summarized in the first one or two paragraphs. Interpretation of the results, or a response to the inevitable question "So what?," is then indicated. Although broad conclusions can sometimes be drawn from narrowly focused studies, care should be taken to ensure that unwarranted inferences are not made about the findings. For example, a descriptive study showing significant variation in practice patterns does not permit a definitive conclusion about the cause of the variations. Significantly longer lengths of stay at one hospital compared with another may reflect less on the institution and more on patient characteristics, long-distance referral networks or lack of adequate home care services in the surrounding community. Alternative interpretations of the data should always be considered and outlined. Limitations of the study, including those inherent in secondary data sources, must be reviewed (Table 3). Finally, suggestions for future research in the area should be provided.
For now, health services researchers will continue to draw on a variety of secondary data sources and will rely heavily on observational methods. The guidelines presented here are designed to help ensure that such studies are reported in a fashion that makes transparent the strengths and weaknesses of the data sources and analytical methods. Better reporting, in turn, will help ensure that this genre of health services research contributes constructively to the goal of improving health care.