Volume 22, No. 2
2001

[Table of Contents]

Completeness and Accuracy of the Birth Registry Data on Congenital Anomalies in Alberta, Canada

Fu-Lin Wang, Stephan Gabos, Barbara Sibbald and R Brian Lowry

Abstract

Vital statistics and other administrative data are becoming an increasingly important source for epidemiologic research and surveillance. This study, the first in Canada, evaluated the usefulness of birth registry data on congenital anomalies in Alberta. We compared the number of birth defects recorded in the birth registry with the number collected through the Alberta Congenital Anomalies Surveillance System (ACASS) between 1985 and 1996. In addition, records of 3,881 (99.9%) babies with a birth defect(s) from the ACASS during 1994-1996 were matched to the birth registry by deterministic linkage. Of these, 2,969 babies had single anomalies that were used for validity analysis. The anomalies were grouped by those within International Classification of Disease (ICD) ICD-9 Section XIV (ICD-9=740.0-759.9) and those outside Section XIV. For those within Section XIV, 24 summary diagnostic categories were examined. As shown, the total case count from the birth registry was on average about 3% lower than that from the ACASS between 1985 and 1996. The validity of diagnostic categories is high for the 24 categories examined, with an overall agreement of between 80% and 100%. The sensitivity, positive predictive value, and kappa are also high for all these anomalies combined during 1994 and 1996, showing 95.7%, 99.8, and 0.81 respectively.

Key words: birth registry; congenital anomalies; reliability; validity

Introduction

Secondary data, such as vital statistics birth/death registries, physician claims, and hospital records, have become increasingly important sources for epidemiological research and surveillance.^1-5 The birth registry, for instance, may be an important source of information for studies of congenital anomalies and other health events.^2,6 The province of Alberta birth registry has been readily available in a standard format for years and records some 40,000 births each year. Each record also contains infant health and reproductive health-related information such as birth weight, mother's age at delivery, and the presence of congenital anomalies. The major variables available in the registry are listed in Appendix 1. To date, no study has evaluated the usefulness of birth registry data on the study of congenital anomalies in Canada.

Only a few studies have evaluated the adequacy and completeness of the congenital anomalies data. Knox and his colleagues evaluated a national congenital anomaly surveillance system in the United Kingdom (UK)⁷ and concluded that the system was inadequate when a case is ascertained from a single source, mainly the birth notice. Another study examined the completeness and accuracy of diagnosis by comparing data from several birth defect registries in the UK and found a high degree of validity in diagnosis.² A Canadian study discussed the limitations of the Canadian Congenital Anomalies Surveillance System (CCASS), focusing on data collected from hospital stays.⁸

The Alberta Congenital Anomalies Surveillance System (ACASS), one of the earliest congenital anomalies registries in Canada, routinely collects data on congenital anomalies in the province of Alberta from multiple sources,^9-10 making it a valuable reference for evaluation studies on congenital anomalies in the province. Using data from the ACASS as a "gold standard", this study evaluates the completeness and accuracy of congenital anomalies data available in the Alberta birth registry. The total number of cases in the birth registry and the ACASS between 1985 and 1996 were compared for completeness. Like previous studies,^2,11-12 the overall percent agreement, estimated kappa, and sensitivity were used for validity assessment. Regional variations in the overall quality of congenital anomalies data in the province were also compared.

Materials and Methods

Description of Data Sources

Data on congenital anomalies were made computer accessible from the ACASS (1980-1996) and from the birth registry (1985-1996). The ACASS started in 1966 and has the best information on congenital anomalies available in the province. Like surveillance systems in other countries, the ACASS collects data on selected malformations for infants up to one year of age.⁹ Cases are obtained from a variety of independent sources, such as the Congenital Anomalies Reporting Form, the Medical Certificate of Stillbirth, the Notice of Birth form, the Medical Certificate of Death, acute care hospitals, and some agencies, outpatient specialty clinics and laboratories. For suspected cases, the ACASS verifies the diagnosis with the relevant physicians/laboratories. Quality measures are taken to ensure the accuracy of diagnosis. The methodology in case ascertainment and evaluation has been detailed elsewhere.^9-10 The British Paediatric Association Classification of Diseases (BPA) code, an adaptation of the International Classification of Disease 9^th version (ICD-9), is used in the ACASS for congenital anomalies.

After receiving data on congenital anomalies (in hard copy) from the ACASS, the birth registry staff will, whenever possible, enter the data into the birth registry database. Data on congenital anomalies from the Alberta birth registry are expected to be of a reasonably good quality. The anomalies in the birth registry are also coded according to the BPA scheme. After receiving data from the birth registry, Alberta Health and Wellness determines the geographic location of the mother's residence according to Regional Health Authority (RHA). Data are grouped into five geographic regions for analysis: Southern (RHA 1-3, 5), Central (RHA 6-9), Northern (RHA 11-17), Calgary (RHA 4), and Edmonton (RHA 10).

Case Definition and Assessment of Completeness of Registry

In this study, a case refers to an individual live or stillborn infant who is coded as having one or more anomaly(ies). The diagnosis may be made at birth (in the birth registry) or during infancy (in the ACASS). An anomaly includes structural defects, chromosomal and monogenic syndromes, inborn errors of metabolism, and other related inborn disorders. An individual can have more than one birth defect. Because only the first defect, the major anomaly of a newborn, is available in the birth registry, the number of cases rather than the number of defects is used in this study.

Similar to the ACASS and other studies,¹³ the birth case prevalence for this study refers to the number of individual live and stillborn infants with at least one birth defect per 1,000 total births.

It is estimated as the following:

Prevalence =

where:

B_CASE = still births (>= 20 weeks) and live births with
a birth defect; and

B_TOTAL = total still births (>= 20 weeks) and live births

The completeness of the registry of congenital anomalies in the birth registry is evaluated by monitoring:

1. the difference in the total number of cases (the birth registry minus the ACASS), and

2. the prevalence ratio (the birth registry divided by the ACASS). The implication of the difference and ratio is:

• difference = 0 or ratio = 1:
  Equal reporting in the birth registry
• difference > 0 or ratio > 1:
  Overreporting in the birth registry
• difference < 0 or ratio < 1:
  Underreporting in the birth registry

Data Linkage and Validity Analysis

Validity is the extent to which the study measures what it is intended to measure. In Stone's work,² it was said "Validity has been defined as 'an expression of whether a response or measure actually represents what it purports to; essentially a measure of truth within the terms of reference'.¹⁴ Since the truth is seldom at hand, a more pragmatic definition is 'the extent to which the results of a method agree with an independent external criterion.'"¹⁵ In this study, validity refers to the agreement in each specific diagnostic category/section of congenital anomalies between the birth registry and the ACASS. The diagnosis from the ACASS was used as a "gold standard" for comparisons.

Data from the ACASS during 1994 to 1996 (incomplete at the time of study) were linked to the birth registry file of the same period by:

• BRN (birth registry number), and
• DOB (date of birth of the baby).

During the three-year period, 3,886 babies with congenital anomalies were recorded in the ACASS database. Of these, 3,881 (99.9%) were matched to the birth registry. Five cases that did not match were excluded from further analysis. For those matched cases, 2,969 (76.5%) had single anomalies. These single anomalies were used for validity analysis. The anomalies were grouped by those within the adaptation of ICD-9 Section XIV (ICD-9 = 740.0-759.9) and those outside Section XIV. The latter include over 20 diagnostic categories, such as umbilical hernia (ICD-9 = 553.10), inguinal hernia (ICD-9 = 550.90), anomalies of jaw size (ICD-9 = 524.00), hereditary hemolytic anemia (ICD-9 = 282.00), cystic fibrosis (ICD-9 = 277.00), disorders of carbohydrate transport and metabolism (ICD-9 = 271.00), etc. Twenty-four summary diagnostic categories were examined for those within Section XIV.

The diagnostic categories recorded in the birth registry were compared with those in the ACASS. The overall percent agreement, kappa estimate, sensitivity and positive predicative value were estimated according to Fleiss¹⁶ and Sorensen et al.⁴ The definition and calculation of each measure are illustrated in Appendix 2. The criteria used to judge the level of percent agreement are greater than 80% for excellent, 61-80% for good, 41-60% for moderate, and less than 40% for poor.¹² The clinical significance of the estimated kappa, which excludes the chance-induced agreement (when positive), was interpreted according to the "benchmarks" suggested by Landis and Koch¹⁷ as the following:

Kappa statistics	Strength of agreement
< 0.00-0.00	Poor
< 0.00-0.20	Slight
< 0.21-0.40	Fair
< 0.41-0.60	Moderate
< 0.61-0.80	Substantial
< 0.81-1.00	Almost perfect

A chi-square test was applied as appropriate for regional variations and for trend analysis.¹⁸

Results

Completeness of the Registry

Table 1 summarizes the annual birth case prevalence of congenital anomalies (ICD-9 XIV) from the birth registry and ACASS in Alberta from 1985-1996. The difference in the total number of cases and the prevalence ratio are presented for comparison. As shown, the number of cases from the birth registry is on average about 3% less than that from the ACASS during the 12-year study period. The number of non-captured cases by the birth registry varies by year, from nine in 1986 to 100 in 1991, with a total of 544 under-reported during the 12 years. It is interesting to note that the difference is larger in recent years, especially for 1995, and that in 1993 the birth registry had one case of overreporting.

Accuracy of the Information - Validity of Diagnosis

Table 2 presents the data on single anomalies from the two sources for years 1994, 1995, and 1996. Of a total of 2,969 cases recorded in the ACASS, 2,662 (89.7%) are within the adaptation of ICD-9 Section XIV (7400-7599), and 307 cases are outside ICD-9 XIV. Overall, 93.4% of diagnoses during the three-year period from the two sources are in the same anomaly section. The percentage is relatively higher for anomalies within ICD-9 Section XIV (94.3%) and lower for anomalies outside this section (89.3%). From 1994 to 1996 the agreement was decreasing for anomalies within ICD-9 Section XIV (p< 0.001). It was higher in 1994 (96.5%), dropped slightly in 1995 (95.3%), and was lower in 1996 (90.3%). For those outside ICD-9 Section XIV, the agreement was 90.6%, 88.3%, and 88.9% for the years 1994, 1995, and 1996 respectively.

Data Quality across Geographic Regions

Health surveillance and studies of congenital anomalies often require comparisons of regional differences. Understanding the accuracy of diagnosis and completeness of data across regions is crucial for appropriate interpretation.

Table 5 presents the overall agreement of diagnoses on congenital anomalies between the birth registry and the ACASS, 1994-1996. As shown, there is no appreciable difference in the agreement across the five geographic regions for all congenital anomalies combined, and anomalies within or outside the ICD-9 Section XIV. The agreement is slightly higher for the anomalies within ICD-9 Section XIV regardless of the region.

Congenital Anomalies Not Captured by the Birth Registry

Table 7 shows the number of babies with a single birth defect who were not captured by the birth registry during the 3-year period. Overall, 135 babies (4.6%) with a single anomaly were not captured by the birth registry. The proportion of these non-captured anomalies increased, from 2.9% (33/1152) in 1994 to 3.2% (30/950) in 1995, and 8.3% (72/869) in 1996 (p < 0.05). When looking at the anomalies subgroups, the proportion of the not-captured cases was higher for those outside the adaptation of ICD-9 Section XIV. The majority of the anomalies outside ICD-9-XIV are of less clinical significance, such as umbilical hernia (ICD-9 = 553.10), inguinal hernia (ICD-9 = 550.90), anomalies of jaw size (ICD-9 = 524.00), etc. However, some important genetic diseases, such as hereditary hemolytic anemia (ICD-9 = 282.00), cystic fibrosis (ICD-9 = 277.00), and disorders of carbohydrate transport and metabolism (ICD-9=27100) were missing from the birth registry.

Table 8 summarizes the number and proportion of single anomalies non-captured by the birth registry for the five geographic regions from 1994-1996. As shown, the proportion is fairly close across the five geographic regions, although it appears slightly higher in Edmonton.

Discussion

Using administrative data sources for surveillance and research is becoming increasingly important in the health setting.⁴ Understanding the completeness and accuracy of this information will assist data analysis and interpretation. This study, using the birth registry as an example, examined the completeness and accuracy of such data on congenital anomalies in Alberta, Canada.

It was found that total case counts recorded in the birth registry were on average 3% lower than those recorded in the ACASS between 1985 and 1996. The difference is fairly close for most of the years under study but is larger in 1991 (6% lower), 1994 (5% lower) and 1995 (8% lower). This difference should be taken into account in interpretation when the birth registry data are used. It is important to emphasize that the data on congenital anomalies from the two sources are expected to be the same. The observed differences can probably be attributed to the "lag period" between data collection, verification, and entry into the system, and perhaps to other sources (i.e., transcription error in updating, data and data entry errors). Staff shortages and staff change over time are likely reasons for the difference. There is also a 6-10 month delay, depending on the year, in accessing the birth registry data that may also account for some of the differences. The larger differences in the most recent two years may, in part, be attributed to the improved case ascertainment from the ACASS and a decrease in the length of hospital stay for obstetric cases.

The validity of diagnostic categories (single anomalies from the birth registry during 1994-1996) is excellent and clinically almost perfect for the 24 categories examined as well as for the overall anomalies. There is no evidence suggesting regional variations in data quality with respect to both data validity and completeness. These findings suggest the data on congenital anomalies from the birth registry in Alberta are fairly useful.

It must be emphasized that the usefulness of vital statistics for surveillance of a particular health event depends on the characteristics of that health event, as well as on the procedures used to collect, code and summarize relevant information. In general, vital statistics will be more useful for conditions that can be ascertained easily at the time of birth or death. Likewise, mortality rates derived from death certificate data will more closely approximate true incidence for conditions with a short clinical course that are easy to diagnose, are easily identified as initiating a chain of events leading to death, and are usually fatal.

The majority of birth defects can be easily identified at birth. The Physician Notice of Livebirth or Stillbirth used in a birth registry may serve as an important source for case identification. If the data on congenital anomalies in the Alberta birth registry have reached such a high quality, it is because they are all obtained from the ACASS, demonstrating that a close tie between the birth registry and the ACASS is required to maintain this level. Without the use of Physician Notice of Livebirth or Stillbirth in the birth registry, some cases may be not initially identified and may miss attention from the ACASS. However, without a well-established mechanism of case identification, verification, confirmation, coding, and entry into a data system from the ACASS, which then sends the data to the birth registry, the case registration in the province may not be as complete and accurate. The rich information and the high quality of birth defect data collected by the Alberta birth registry (Appendix 1), and perhaps of other provinces with similar mechanisms of case ascertainment, would suggest its potential value for surveillance and research of congenital anomalies. This study also found that some factors such as the lag period, errors in data entry and transcription - common in administration of all birth registries - appear to have little impact on the completeness and accuracy of case registration in a birth registry. Of note, although the birth and death certificates are filed shortly after the event occurs, the process of producing final vital statistics at a provincial level from these data sources can take at least eight months or longer.

This exercise has demonstrated an approach to evaluating an administrative database, and may help future evaluation studies of secondary data. In fact, a similar approach is being used in another study comparing congenital anomalies data between physician claims, hospital morbidity and the ACASS.

It is important to keep in mind, however, that this study also has important limitations. Although comparing data from several sources has been used and recommended by others,^1-4 the data from the reference source are assumed to be valid. Data errors in the reference group may lead to some variations in the level of the validity measures, though this error is likely very small for the ACASS data. There are hundreds of specific anomalies and the present study evaluated only 24 diagnostic categories, so the potential differences among specific anomalies within each diagnostic category are not revealed by the present study. Many fetal anomalies, especially those of less than 20 weeks' gestation, were not captured by the birth registry/ACASS in the past, and about 20% of cases with multiple anomalies were not included in this evaluation.

Nonetheless, findings from this study lead to the following conclusions:

• The data on congenital anomalies from the birth registry represent about 97% of total infants with a single birth defect during 1985-1996. For infants with multiple anomalies, the "minor" defects were not available in the birth registry during this period. Thus, the total number of defects from the birth registry was likely underreported by about 20%.
• Most of the diagnostic categories of single congenital anomalies (within ICD-9 Section XIV) from the birth registry agree with those from the ACASS during 1994-1996, suggesting accuracy of diagnostic information for the 24 categories examined. Overall, 99.8% of single anomalies in the birth registry during 1994 to 1996 were likely "true" cases. During the same period, the probability of correctly classifying the single anomalies within ICD-9 Section XIV was 95.7%. Clinically, the validity of the diagnosis for single anomalies within ICD-9 Section XIV was almost perfect during the three-year period.
• No significant regional variations were found with respect to the completeness of information, sensitivity, predictability, kappa, and overall agreement of diagnosis on single anomalies within ICD-9 XIV as a whole during 1994-1996.
• The ACASS has played an essential role in surveillance of congenital anomalies in the province. Further collaboration between Alberta Health and Wellness, the ACASS, the birth registry and other agencies is required for the surveillance, prevention and control of congenital anomalies in the province.

It is recommended that data on congenital anomalies from the birth registry, in combination with data from the ACASS and other sources, such as Alberta Health Care Insurance Plan (AHCIP), physician claims, and hospital morbidity, be used for monitoring/surveillance purposes such as:

• monitoring long-term trends of birth prevalence of selected congenital anomalies;
• identifying differences in infant health status within ethnic (Aboriginal vs. Non-Aboriginal) or other subgroups of the population;
• assessing differences in congenital anomalies, stillbirths, low birth weight by geographic area (spatial pattern) or by maternal age or other factors;
• monitoring congenital anomalies that are generally considered preventable (such as neural tube defect);
• generating hypotheses regarding possible causes or correlations of congenital anomalies or other infant health indicators;
• conducting health-planning activities related to infant health; and
• monitoring progress toward achieving improved health status of the infant or child population.

Acknowledgements

The authors wish to thank Mr. Larry Svenson of Health Surveillance and Mr. Barry Chatwin of Information Technology for providing data necessary for linkage as well invaluable advice for the linkage process. Special thanks to Mrs. Susan Shaw and Dr. Don Schopflocher for their support and editorial changes.

References

1. Roos LL, Nicol JP, Cageorge SM. Using administrative data for longitudinal research: comparisons with primary data collection. J Chronic Dis 1987; 40:41-49.

2. Stone DH. A method for validation of data in a register. Public Health 1986;100: 316-24.

3. Tennis P, Bombardier C, Malcolm E, Downey W. Validity of rheumatoid arthritis diagnosis listed in the Saskatchewan hospital separations database. J Clin Epidemiol 1993;46:675-83.

4. Sorensen H, Sabroe S, Olsen J. A framework for evaluation of secondary data sources for epidemiological research. Int J Epidemiol 1996;25:435-442.

5. Wen SW, Liu S, Marcoux S, Fowler D. Uses and limitations of routine hospital admission/separation records for perinatal surveillance. Chronic Dis Can 1997;18:113-119.

6. Wang FL, Gabos S, Yan J, Li FX. Parental birth cohort, birth weight and congenital anomalies. Am J Epidemiol 1997;145:S3.

7. Knox EG, Armstrong EH, Lancashire R. The quality of notification of congenital malformations. J Epidmiol Community Health 1984:38:296-305.

8. Rouleau J, Arbuckle TE, Johnson KC, Sherman GJ. Description and limitations of the Canadian Congenital Anomalies Surveillance System (CCASS). Chron Dis Can 1995; 16:37-42.

9. Lowry RB, Thunem NY, Anderson-Redick S. Alberta Congenital Anomalies Surveillance System. CMAJ 1989;141:1158-59.

10. Lowry RB, Anderson-Redick S. Report from the Alberta Congenital Anomalies Surveillance System. Bulletin of the Hereditary Diseases Program of Alberta 1992 (ISSN 0844-1316);11:13-16.

11. Wang FL, Semchuk KM, Love EJ. An assessment of usefulness of demographic data provided by surrogate respondents in a case-control study of Parkinson's disease. J Clin Epidemiol 1992;45:1219-1227.

12. Wang FL, Semchuk KM, Love EJ. Reliability of environmental and occupational exposure data provided by surrogate respondents in a case-control study of Parkinson's disease. J Clin Epidemiol 1994;4:797-807.

13. Johnson KC, Rouleau J. Temporal trends in Canadian birth defects birth prevalences, 1979-1993. Can J Public Health 1997;88:169-176.

14. Schaefer M. World Health Organization Scientific Group on the Evaluation of Environmental Health Programs, Geneva: World Health Organization 1972: DIS/WP/72.2.

15. Bennett AE, Ritchie K. Questionnaires in Medicine. Oxford University Press: London (for the Nuffield Provincial Hospitals Trust) 1975. p. 29.

16. Fleiss JL Statistical Methods for Rates and Proportions 2^nd ed. New York: Wiley 1981; 44-46, 212-225.

17. Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-174.

18. Altman DG. 2 H K tables - Comparison of several proportions. In DG Altman ed. Practical Statistics for Medical Research. London/Glasgow/Weinheim/New York/Tokyo/Melbourne/Madras: Chapman & Hall, 1995. 

APPENDIX 2 Terms and Definitions
<>Sensitivity is a measure of the probability of correctly diagnosing/classifying a case or event (Syn: true positive rate). <>Specificity is a measure of the probability of correctly identifying/classifying a non-case or non-event (Syn: true negative rate). <>Positive Predictive Value: In screening and diagnostic tests, the probability that a person/event with a positive test is a true positive. <>Overall agreement is a measure of the probability of correctly diagnosing/classifying a case or event plus a non-case or non-event. <>Kappa is a measure of the degree of non-random agreement between observers or measurements of the same categorical variable.
These measures have been widely used in screening tests and interclass or intraclass correlations. The calculations are illustrated by the following two by two table:
Table layout for sensitivity, specificity, predictive value, and agreement analysis
		Cases or events from the standard		Total
		Yes	No
Cases or events being classified from the study	Yes	a  (90)	b  (15)	a  +  b
	No	c  (10)	d  (25)	c  +  d
Total		a + c	b  +  d	a + b + c + d = N
<>Sensitivity = a / (a + c) * 100  =  90 / (90 + 10) * 100 = 90.0% <>Positive Predicative Value (PV) =  a / (a + b) * 100 = 90 / (90 + 15) * 100 = 85.7% <>Specificity  = d / (b + d) * 100 = 25 / (15 + 25 ) * 100 = 40.0%
<>Overall Agreement = (a + d) / N * 100 = (90 + 25) / 140 * 100 = 82.1% <>Kappa = P0 - Pe / 1 - Pe =  [2(ad - bc)] / {[(a + b)(b + d)] + [(a + c)(c + d)]} <>Kappa = 2(90*25 - 15*10) / (105*40 + 100*35) = 4200 / 7700 = 0.54 <>where P0 is the observed agreement and Pe is the expected agreement.

Author References

Fu-Lin Wang, Epidemiologist, Health Surveillance, Alberta Health and Wellness, Edmonton, Alberta

Stephan Gabos, Director, Health Surveillance, Alberta Health and Wellness, Edmonton, Alberta

Barbara Sibbald, Manager, Alberta Congenital Anomalies Surveillance System, Department of Medical Genetics, Alberta Children's Hospital, Calgary, Alberta

R Brian Lowry, Director, Alberta Congenital Anomalies Surveillance System, Department of Medical Genetics, Alberta Children's Hospital, Calgary, Alberta

Correspondence: Dr Fu-Lin Wang, Health Surveillance, Alberta Health and Wellness, 24th floor, TELUS Plaza North Tower 10025 Jasper Ave., PO Box, 1360 STN Main, Edmonton, Alberta T5J 2N3; Fax: (780) 427-1470; E-mail: fu-lin.wang@health.gov.ab.ca

[Previous] [Table of Contents] [Next]