Critical Appraisal of the Health Research Literature: Prevalence or Incidence of a Health Problem

Patricia L Loney, Larry W Chambers, Kathryn J Bennett, Jacqueline G Roberts and Paul W Stratford

For our specific critique example, two authors appraised the selected articles on the prevalence of dementia, using consensus, based on the guidelines outlined in Table 1. They used a methodological scoring system²⁰ previously used to rate the quality of studies on the prevalence of back pain (Table 2). To determine the relative weighting of each item for the scoring system, three authors were asked to recommend weights for each item and all agreed that each item should be weighted equally. Thus each item was assigned a score of 1 point, making 8 the maximum score possible.

Critical Appraisal of Studies

A. ARE THE STUDY METHODS VALID?

1. Study Design and Sampling Method: Are the study design and sampling method appropriate for the research question?

A survey (observational study) is the appropriate study design to determine the prevalence of a particular health problem. If the whole population of interest is not surveyed, then the best sampling technique is random (probability) sampling of persons from a defined subset of the population. Stratification (sampling purposely from subgroups) may be required to appropriately represent subgroups such as the very old. Stratified random sampling, with eligibility criteria, will ensure that the sample is representative of the population to whom the researchers wish to generalize the results.

For larger surveys, cluster sampling is sometimes used. In cluster sampling, groups of individuals (e.g. families or people living in defined geographical areas) are selected as the survey units.³ If the population is small, some studies survey the whole population and do not generalize the results to other populations.

A study to determine the incidence of a disease must have a prospective or longitudinal design, and should include persons known not to have the disease, who are then observed over a suitable time period.^1,3

As an example, one of the studies reviewed in our critical appraisal of studies on the prevalence of dementia in Canada was the Canadian Study of Health and Aging (CSHA).¹⁵ This was a survey that covered five regions across Canada through a random sample of 10,263 persons in community and institutional settings to determine the prevalence of dementia and Alzheimer's disease. The CSHA used random sampling, stratified for age, sex, region and place of residence (community or institution).

2. Sampling Frame: Is the sampling frame appropriate?

The type of sampling frame (list for study recruitment) from which subjects are selected is important. Census data provide one of the few data sets from which one can draw a sample that is thought to have minimal bias since certain groups of persons are thought not to be excluded as they might be in an electoral list or telephone list. For example, electoral lists may underrepresent the elderly or people who are cognitively impaired. In relation to our critical appraisal of dementia prevalence studies, a sample of "convenience" could be very biased in that persons with dementia were hard to reach, thus reducing the prevalence of dementia in the sample. Studies of whole, narrowly defined communities are usually done as door-to-door surveys, but this limits the generalizability of the findings outside that community.

In the Canadian Study of Health and Aging,¹⁵ the following databases were used for sample selection: provincial health insurance plans, enumeration composite records, election records and municipal records. The study included both institutionalized persons and community dwellers. In the one province (Ontario) where election records were used, the sample may have been biased if the extreme elderly were missed.

Bachman et al. (l992)¹⁰ used the Framingham cohort in the United States, which limits the generalizability of these results to a particular group of subjects. Some electoral or physician utilization lists might not represent all elderly persons (i.e. both those in the community and those in institutions, and healthy and unhealthy seniors), and thus might underrepresent persons with dementia.

3. Sample Size: Is the sample size adequate?

A large sample size produces narrow confidence limits, which is doubly important if the prevalence or incidence of a given condition is low. Small sample sizes produce large confidence intervals, making the findings less precise. It is critical to be as confident as possible that any changes in health care policy are based on results that did not occur by chance due to probability sampling inadequacy. In fact, the sample size required to estimate a proportion (prevalence of a disease) with a specified degree of precision (i.e. 95% confidence) can be calculated.²¹

Using a conservative sample size estimate of proportions for our review of dementia prevalence studies (assumptions based on CSHA study:¹⁵ prevalence = 8%, error rate <3%, 95% confidence level), the calculated sample size needed would be 314.²¹ In their study of dementia prevalence, Rockwood and Stadnyk (1994)¹⁸ indicated that the sample should be at least 300 subjects. Thus, a sample size of 300 was considered adequate for the purposes of our review. If prevalence rates were needed for subgroups, then the suggested sample size would be required for each subgroup.

The sample sizes used in the dementia prevalence studies we critiqued varied from 500¹³ to 10,263.¹⁵ The largest sample, from the CSHA, produced smaller error rates and smaller confidence intervals, which is important when making a health care decision or policy.

4. Appropriate Measurement: Are objective, suitable and standard criteria used for measurement of the health outcome?

Often crude outcome measures are used in population health research due to the expense of complicated diagnostic tests. However, these measures may not be capable of including or excluding appropriate levels or stages of the health problem. It is important that published studies describe the measurements used well enough that the different outcome measures can be compared. If a worldwide standard measure of a particular health outcome exists, any studies not using it should indicate how their measure is related to the more common accepted measure.

The outcome measure must be reliable (reproducible) and valid with high sensitivity and specificity. Since health problems can be defined in many ways, the measurement of the problem must be the best possible one used by health care providers. If a disease is rare, there are often two phases to a prevalence study: subjects are first screened quickly for the condition using an inexpensive, broad screening test with good sensitivity and specificity, and then more complicated and detailed clinical assessments are made in the second phase. The screening test should not miss true positives-people who truly have the disease-and it should also have a low false negative rate, meaning it does not incorrectly label subjects with the disease as being disease-free.

For example, dementia is sometimes classified in research studies according to different systems from the United States (DSM), continental Europe (International Classification of Diseases) and the United Kingdom (CAMDEX: Cambridge Mental Disorders of the Ederly Examination).²² Research has indicated that these commonly used criteria can differ by a factor of 10 in the number of subjects classified as having dementia.²² In the CSHA,¹⁵ a variety of measures were used by independent assessors who were unaware of the initial screening test results. The community screening measure was the Modified Mini-Mental State Examination (3MS),⁷ which was given by trained interviewers in the subject's home. Subjects who screened positive (score <78) and a randomly selected group of subjects who screened negative were given clinical examinations by a nurse, a psychometrician (blind to 3MS testing), a neuropsychologist and a physician. These health professionals were trained and given guidance about how to assess for dementia. In addition, biological tests were carried out. Other assessments included the DSM III-R criteria⁸ and CAMDEX.²³ In the American Framingham study,¹⁰ presence of dementia was determined through the MMSE,²⁴ the CES-D²⁵ (Center for Epidemiologic Studies Depression Scale) and general examinations by an independent neurologist.

Many health problems are not easily diagnosed or defined, and some, such as dementia, include stages where mild cases are not always easily distinguished.

5. Unbiased Measurement: Is the health outcome measured in an unbiased fashion?

Considerable judgment by assessors or interviewers is required to determine the presence of some health outcomes under scrutiny; thus, it is best that trained assessors are independent and not aware (i.e. blinded) of the subjects' clinical status or, sometimes, even the purpose of the study. It is important that the subjects under assessment include those thought to be negatives as well as positives.

If more than one rater is used, interobserver and/or intraobserver reliability of clinical assessments must be high and should be noted in the articles published. The interviewers or assessors must all be using the same criteria, including specifics related to each health problem, such as its duration. This is especially pertinent when diagnosing an illness such as Alzheimer's disease, since investigators must evaluate clinical signs and symptoms in the subjects in addition to caregivers' views of these. Sometimes, as for Alzheimer's disease, multiple measurements or assessments are conducted to rule out other health conditions. Thus the numerators (health problems) of the rates must all be defined or diagnosed in the same way.

6. Response Rate: Is the response rate adequate? Are the refusers described?

The greater the number of selected subjects who are not available for measurement, the less valid the estimate. A response rate in population surveys of two thirds to three quarters has been suggested to be generalizable to the population samples.²⁶ Therefore, we chose a response rate of 70% as acceptable in our review. In the case of dementia, a significant proportion of those persons not responding to a survey might be suffering from dementia, which could lead to an underestimate of its prevalence.¹⁹

Since a large number of dropouts, refusals or "not founds" among the subjects selected may jeopardize a study's validity, the authors should describe the reasons for non-response and compare persons in the study with those not in the study as to their sociodemographic characteristics. If the reasons for non-response seem unrelated to the health outcome measured and the characteristics of those individuals not in the sample are comparable to those in the study, researchers may be able to justify a more modest response rate.

Response rates may be improved if the assessment or measurement is easily accessible, conveniently timed for the subjects, acceptable in length and suitable in content. Home visits may be more acceptable for many elderly persons.

In our review, prevalence rates of dementia differed as did study response rates. The CSHA¹⁵ accounted for all subjects, giving reasons for non-response. The compliance rate for the initial screening (phase 1) was 72%, and 73% of these respondents were compliant for clinical examination during the prevalence study. The CSHA authors considered these rates slightly low and thought that dementia prevalence might be underrepresented in the sample since 27% refused the clinical exam and their reasons for refusal might have included the presence of dementia.

To determine incidence ideally, all study subjects should be followed and measured to prevent bias. Usually patients are available for follow-up and if randomly selected subjects are not found or studied, one is never sure if there is a consistent bias known to influence incidence. If persons die during the period of the study, the cause of death must be ascertained. It is necessary to follow subjects over a clinically sensible period of time, depending on the illness under study and the age of the population. For dementia, if the follow-up period is too long, cases may be missed due to death, especially in the older subgroups.

B. WHAT IS THE INTERPRETATION OF THE RESULTS?

7. Results: Are the estimates of prevalence or incidence given with confidence intervals and in detail by subgroup, if appropriate?

The quantitative results from studies of prevalence or incidence are proportions or rates over a fixed period of time. The prevalence rates found in studies reviewed provide only estimates of the true prevalence of a problem in the larger population. Confidence intervals then indicate the level of confidence one can have in the estimates and their range. Since some subgroups are very small, usually 95% confidence intervals are given.

The CSHA authors¹⁵ provided confidence intervals and described prevalence rates in detail by age group, sex, setting (community or institution) and region of Canada. Their estimates of the prevalence of dementia ranged from 2.4% among persons aged 65-74 years, to 34.5% among those aged 85 and over.

C. WHAT IS THE APPLICABILITY OF THE RESULTS?

8. Study Subjects: Are the study subjects and the setting described in detail and similar to those of interest to you?

Certain diseases are known to vary in prevalence or incidence across different geographic regions and population sectors. For example, persons over 85 years of age and those residing in institutions are expected to have higher prevalence rates of dementia. For some health problems, rates for women may differ from those for men.

Sociodemographic variables, such as educational status, may vary between countries. Therefore, the study sample needs to be described in enough detail that other researchers can determine if it is comparable to the population of interest to them.

In the CSHA article,¹⁵ study subjects are described in detail by age, sex and region of residence in Canada. Institutionalized subjects are also included in the sample.

If the study being appraised estimates the prevalence of a sign or symptom in an experimental group, such as a control group in a randomized controlled trial, the sociodemographic characteristics of the subjects must be reported in order to understand the applicability of the results. Similarly, providing a comparison of study participants with those who refused or were ineligible can help others determine for whom the study group is representative.

Conclusion

Table 3 summarizes the findings from our critical appraisal of studies on the prevalence of dementia according to the guidelines we developed. Prevalence rates from these nine studies varied from 2% to 9%, and the scores we assigned varied from 3 to 8, the maximum possible.

Some of the studies seemed inappropriate for determining the prevalence of dementia in Canada due to the lack of relevance of the subjects studied, while others had methodological weaknesses. For example, if the assessors did not have negatively screened subjects included in their sample, the results might be biased. In addition, some studies had low response rates, raters were not always blinded and the sampling frames were not always the best.

The study by the CHSA Working Group estimated the prevalence of dementia in Canada to be at least 8%. The sample was randomly selected, subjects in institutions were included and outcome measures were appropriate. The results included subgroup analysis, the sample size was large and the setting and subjects were applicable to our community situation. We found the weaknesses of the study to be a slightly low response rate and use of electoral lists in one province, suggesting an underreporting of cases of dementia. Our final rating of this study was a favourable score of 7.