Canadian Medical Association Journal 1996; 154: 1695-1698
© 1996 Canadian Medical Association
In the medical literature, surveys generally examine health status (i.e., prevalence studies), identify risk factors or chronicle activities, attitudes and health outcomes. Repeated surveys can reveal trends. Although information on how to conduct a survey is readily available,[1-5] information on how to report a survey is not. Because the type and scope of surveys vary widely, the objective of this article is to offer some general recommendations on reporting survey research. These recommendations form the basis for a checklist for authors and peer reviewers of survey articles (Table 1). They are also used by CMAJ editors to help determine the acceptability of a manuscript.
[Table of Contents]
Structured abstract
The structured abstract for a survey is the same as that for any original research article except that it does not include the heading "Intervention." Specific guidelines for preparing structured abstracts are included in the information for authors published in the Jan. 1 and July 1 issues of CMAJ. It is important that all the other headings are used and that the information given under each is presented in a logical and consistent fashion. The wording of the objective deserves careful consideration, because from this flows the logic of the abstract (and of the survey itself). For example, the description of outcome measures should address the key issues noted in the objective, and the results should address the outcome measures. If the research objective is concerned with a number of questions or variables (for example, the age group, sex and prescribing patterns of physicians who have a chemical dependence) then the outcome measures, results and conclusions should address these items in the same sequence throughout.
[Table of Contents]
Introduction
The introduction in a survey article varies little from that in other types of scientific studies. It identifies the problem or research issue, explains why it is important and provides a critical review of the relevant literature. Often, there is not enough critical appraisal of previous studies -- yet this is valuable information that should reveal the rationale for the current survey.
Authors need to address the question of how their survey promises to add to current knowledge. It is also useful to explain why a survey was the most appropriate research method. Finally, the specific question or questions addressed by the survey need to be noted. A clearly delineated research question at the outset is very important: if the research question is too general it often leads to multiple analyses or "data dredging."
[Table of Contents]
Methods
The fundamental rule in reporting scientific research is to describe the methods in sufficient detail to enable other investigators to replicate the study. In survey research three main topics are essential: the study population and survey sample, the survey instrument and the statistical analysis. It is useful to identify these under separate subheadings.
It is important to specify both the sampling strategy and to explain how the sample size was determined. Was the sampling stratified, randomized, sequential or based on quotas or clusters? Inclusion and exclusion criteria (such as language or age requirements) should be noted and justified. For example, if women aged 20 to 50 years were surveyed on their attitudes toward abortion, the reasons for not surveying men or teenaged and postmenopausal women need to be given.
Ideally, the sample size is determined beforehand on the basis of the type of information sought and the statistical analysis to be performed. If the sample size was limited by logistical factors, this should be acknowledged. Using a small sample poses a problem for significance testing, as it increases the chance of a type II error (i.e., the sample is too small to show a significant difference when there is one). As well as having insufficient power for significance tests, results derived from small samples lack precision. Prevalence estimates are directly related to sample size, and smaller samples widen the confidence interval around any prevalence estimate.
The authors should describe how the survey was conducted. How were people approached, and what information was given to them before they agreed to participate? Was the survey conducted through face-to-face interviews, by telephone or by mailed questionnaire? Was any payment offered?
An adequate response rate is critical to the acceptability of a survey article. The process used to maximize the response rate needs to be identified, as well as the rationale for this process. For example, in mailed surveys the Dillman technique,[6] which involves sending up to three mailings to nonresponders, is often used.
Except in unusual circumstances, surveys are not considered for publication in CMAJ if the response rate is less than 60% of eligible participants. At response rates of less than 60% it is very difficult to interpret the results.
A lower response rate might be justified in two situations. The first is when the authors can allay concerns about selection bias by clearly demonstrating that their sample is representative of the general study population. The second situation is one in which the responses yield results that are counterintuitive. In a survey investigating physicians' knowledge about the treatment of hirsutism, for example, physicians who agree to fill out the questionnaire are likely to be a little more confident about their knowledge in this area than physicians who do not respond. If, despite this possible source of bias, the results show that most physicians know very little about the treatment of hirsutism, then the results are probably valid - although they may underestimate the extent of the problem.
The description of questionnaire development covers three main areas. First, what process was used to create the questionnaire? Was a literature search done to identify key areas? Were previous questionnaires adapted or combined?
Second, how were the reliability and validity of the questionnaire assessed? This critical issue is sometimes forgotten by authors who, in their enthusiasm for the research, assume that the reliability and validity of the survey instrument are self-evident. All questionnaires should undergo formal reliability testing before they are used in a survey. This includes assessing the reproducibility of the test results or conducting consistency corre- lations.[7,8] In addition, questionnaires should undergo at least preliminary validity testing. Comprehensive criterion or construct-validity testing is time-consuming, expensive and may not be feasible,[7] but assessment of content or face validity is recommended. Systematic approaches for establishing content validity have been well described.[8]
Third, how was the questionnaire pretested? Pilot testing is necessary to ensure that the format of the questionnaire does not prevent it from eliciting the desired information. Failure to conduct reliability, validity and pilot testing of a survey questionnaire may preclude the publication of results because of uncertainty as to whether the questionnaire truly was able to ascertain what it was supposed to.
Authors greatly assist editors and reviewers when they enclose a copy of the questionnaire with the submitted article. Although the questionnaire is usually not reproduced in the journal, authors should note in the text that it is available to interested readers upon request.
Descriptive statistics (e.g., mean, standard deviation and confidence interval) are always valuable, as they help to characterize the study population and the survey sample. Inferential statistics are used for hypothesis testing. This may involve comparisons and the identification of possible risk factors. Parametric tests (such as Student's t-test) assume a normal distribution of the survey sample; this generally necessitates a random sampling technique. Nonparametric tests (such as the chi-squared test) do not require a normal distribution.
All statistical analyses carried out need to be described. The appropriateness of a statistical procedure should be determined carefully beforehand. When trying to identify risk factors, for example, one might be tempted to do multiple chi-squared tests because of their simplicity. However, a multivariate analysis, which controls for possible confounding variables, may be more appropriate. If multiple hypothesis testing is done, techniques to compensate for this (such as a Bonferroni adjustment) should be included.[11]
Any transformation of the data obtained in the questionnaire into a format more amenable to statistical analysis should be identified and explained. For example, if a Likert scale[12] of "strongly disagree, disagree, neutral, agree, strongly agree" was used in the questionnaire but these categories were collapsed into "agree" and "disagree" for the purpose of analysis, the authors must explain the rationale for doing so and specify how the "neutral" responses were handled.
[Table of Contents]
Results
The results section usually begins with a description of the survey sample. A table comparing the survey sample with the target population using descriptive statistics may help readers assess the representativeness of the sample and the relevance of this group to their own situation (as in a sample showing characteristics of patients similar to those in an urban family-medicine practice).
The data should be described in the sequence that was established in the structured abstract. This vastly increases the readability of the report.
Tables provide a convenient way to display the bulk of the data, and it is neither necessary nor desirable to repeat in the text all the results shown in a table. The text of the results section need only highlight the data in the tables.
First, the tables should present data that address the main objectives of the study. Although this rule of thumb may seem obvious, researchers often gather more data than are necessary and are tempted to present them all.
Second, all respondents must be accounted for. It is CMAJ style to display the number of people who responded to each question, followed in parentheses by the percentage of the sample that this number represents. It is well acknowledged that not all respondents answer every question. Nonrespondents should be accounted for in a separate row or column. When a large number of respondents do not answer a particular question this should be emphasized in the txWhen responses from an identified subgroup are used, the number of respondents in this subgroup should be clearly identified as the relevant denominator. When appropriate, measures of significance testing should be displayed.
[Table of Contents]
Discussion
The discussion section is usually short. The first paragraph or two should highlight the main findings and directly address the objectives of the study.
Authors should offer an explanation of the findings and note (and possibly refute) other plausible interpretations of the data. The significance of the findings needs to be identified. This may require a brief discussion of how the survey results compare with those of similar studies.
No discussion is complete without a careful consideration of the limitations of the study. Sources of possible bias and other threats to validity may be noted. Bias may enter from the sample selection, the questionnaire itself or the statistical analysis. If one or more questions had a low response rate this should be noted and identified as detracting from the strength of the result.
The discussion section usually ends with a commentary on directions for future research that considers the whole research area in general and the implications of the study findings in particular.
Authors must be very careful not to extrapolate beyond their data. The most common extrapolation is the assumption that whatever is reported by the respondents is true. The likelihood of both underestimation and overestimation needs to be considered. Any summary of the results should not state "this is" but rather "this was reported to be."
Another common extrapolation is to call for more educational programs when attitudes or patterns of practice are found to be less than ideal. This assumes that education will be effective in changing behaviour; this may not be the case. Unless the survey specifically includes questions on past educational programs and their effectiveness with respect to the study topic there is usually no evidence from the study on which to base this assertion.
[Table of Contents]
Conclusion
Surveys can yield relevant and timely information that may not be attainable by other methods. With careful planning, problems such as uncertain reliability and validity, low response rates and inappropriate statistical analyses can be avoided. When relevant, well conducted and reported clearly, surveys make a substantial contribution to the medical literature.
[Table of Contents]
References