Chronic Diseases and Injuries in Canada

Volume 32, no. 1, December 2011

2008 Niday Perinatal Database quality audit: report of a quality assurance project

S. Dunn, PhD (1,2); J. Bottomley, MHA (3); A. Ali, MSc (4); M. Walker, MD (1,5,6,7)

This article has been peer reviewed.

Abstract

Introduction: This quality assurance project was designed to determine the reliability, completeness and comprehensiveness of the data entered into Niday Perinatal Database.

Methods: Quality of the data was measured by comparing data re-abstracted from the patient record to the original data entered into the Niday Perinatal Database. A representative sample of hospitals in Ontario was selected and a random sample of 100 linked mother and newborn charts were audited for each site. A subset of 33 variables (representing 96 data fields) from the Niday dataset was chosen for re-abstraction.

Results: Of the data fields for which Cohen’s kappa statistic or intraclass correlation coefficient (ICCC) was calculated, 44% showed substantial or almost perfect agreement (beyond chance). However, about 17% showed less than 95% agreement and a kappa or ICC value of less than 60% indicating only slight, fair or moderate agreement (beyond chance).

Discussion: Recommendations to improve the quality of these data fields are presented.

Keywords: audit, data quality, quality assurance, reliability

Background

The Ministry of Health and Long-Term Care (MOHLTCC) in Ontario recognized that producing and sustaining quality surveillance data is the foundation of an effective and efficient health system.¹ Surveillance is defined as the ongoing systematic collection, analysis and interpretation of health data essential to the planning, implementation and evaluation of public health practices, integrated with the timely dissemination of these data to key stakeholders.²  A surveillance system can function as both measurement tool and stimulus for action³  by providing early warning of health problems and evidence for policy and program development, risk assessment, trend analysis and the evaluation of prevention and control strategies.⁴  However, the usefulness of a surveillance system is limited by the quality of the data it collects and analyzes.

In Ontario, the Niday Perinatal Database (the “Niday”) is the source of data to assess outcomes, risk factors and interventions related to perinatal care. It was created in 1997 under the direction of the Perinatal Partnership Program of Eastern and Southeastern Ontario (PPPESOO) to provide perinatal data to PPPESO partners. This Internet-based system has evolved significantly since its inception and has become a unique co-operative venture with over 100 health care organizations across the province contributing real-time perinatal data. It enhances the ability of health care providers in different parts of the province and within different service sectors to work together to improve perinatal health. At the time of the audit, 96% of Ontario births were captured in the Niday, and there were 90 defined patient elements covering the full spectrum of perinatal care (Table 1). In 2001, the province adopted the variables in the Niday as the minimum dataset.

Table 1. List of variables in the Niday Perinatal Database in 2008 (n = 90) including variables chosen for re-abstraction as part of the 2008 quality audit

This is the only database in Ontario that provides immediate access to real-time population-based perinatal data for an entire region. The Better Outcomes Registry and Network (BORNN Ontario) Steering Committee now manages the project. The involvement of most hospitals in the province also permits inter-hospital/health unit comparisons necessary for benchmarking and performance improvement based on learning from others’ successes. As the system evolves, BORN is committed to ensuring high quality data, with powerful and efficient reporting tools.⁵

In light of the fact that approximately 40% of all live births in Canada occur in Ontario (37.1% in 2008/2009),⁶ this database provides rich perinatal information for a large proportion of the births in Canada. Although it is well recognized that the foundation of an effective and efficient health system requires the production of quality data,¹ it was unclear whether the Niday, as configured, was a reliable source of information. The goal of this quality assurance project was to assess objectively the reliability, completeness and comprehensiveness of the data in the Niday Perinatal Database.

Methods

The Data Quality Management Framework,⁷ developed by the MOHLTC Health Results Team for Information Management, was used to guide this project. According to the Tri-council policy, and given the fact this was a quality assurance project, Research Ethics Board approval was not required.⁸ Hospital participation in this project was voluntary, and every effort was made to ensure the confidentiality of patient information and privacy of participating hospitals.

Data re-abstraction

In order to determine the reliability and completeness of the data, re-abstraction of information from patient records was carried out to assess agreement between selected variables in the perinatal database and the mother and infant charts. Written consent was requested from and given by each site participating in the re-abstraction phase of the project. Information was handled confidentially, and each auditor signed a Pledge of Confidentiality Form. The auditors re-entered data from the patient records that had already been collected and entered by the hospital data entry person into the Niday. The laptops used for data entry were supplied to the auditors and returned following the re-abstraction process. The electronic data were then securely transferred to the statistician for analysis and deleted from the laptops. Data were aggregated for analysis, and findings were anonymized.

Setting and sample size (hospitals)

Purposive sampling was used to recruit 14 hospitals for the audit representing five regions of the province: East/Southeast, Greater Toronto Area (GTAA), Central West, South West, and North. The sample captured both obstetrical and newborn care practices and included all levels of care: level 1, or low-risk pregnancies (4 of 51 hospitals in Ontario); level 2, or women/babies with health problems (8 of 37 hospitals in Ontario), level 3, or specialized care (2 of 7 hospitals in Ontario). A combination of both paper and electronic documentation systems and a variety of data entry processes were used by the sample hospitals.

Sample size

A computer-generated random sample of 100 maternal chart numbers (and linked baby records) was identified for each participating site from existing records that had already been entered into the Niday in 2008 (total of 200 charts per site). The total sample size for this project was 1395 linked mother-baby dyads; in three cases the patient charts could not be located at the time of the re-abstraction, and in two cases the chart numbers were not for a perinatal client.

Variables for re-abstraction

A subset of variables (33/90; 36.7%) from the Niday perinatal dataset was chosen for re-abstraction. Selection was based on the following criteria: a) a mandatory variable; b) a non-mandatory variable with less than 10% missing data based on verification reports; and c) a variable that addressed a practice issue of interest (e.g. use of antenatal steroids, indication for Caesarean section, episiotomy, lacerations, fetal surveillance, forceps/vacuum, indications for induction, method of induction, maternal pain relief, smoking). This resulted in 96 data fields available for re-abstraction because some of the variables consisted of multiple data fields (e.g. indications for induction included 17 data fields; maternal pain management included 11 data fields). Table 1 lists the variables selected for re-abstraction and those excluded (with rationale).

Auditors

Due to the wide geographic distribution of the participating hospitals, and the travel and time involved to complete an audit in 14 sites across the province, six auditors with a health care background were hired and trained to expedite the process. Two auditors entered data at five sites each and each of the remaining four auditors re-abstracted data at one site each. Figure 1 shows a flow sheet of the data collection process.

Figure 1 - The data collection process

Figure 1, Text Equivalent

Each of the auditors was told about the project and trained in the re-abstraction process, including where to find the information in the patient record and how to use the SPSS (version 15.0) spreadsheet for data collection to ensure consistent re-abstraction. Each received a handout containing the definition of terms for each of the variables in the Niday, contact information for the project coordinator, a list of their designated hospital(s) and an SPSS spreadsheet with pre-entered sample data (maternal chart number, baby chart number, baby date of birth) for each of their designated sites. For practice, the auditors entered data into the SPSS spreadsheet based on the same two charts; inter-rater reliability was evaluated based on these cases.

Data collection procedure

Following the random chart selection process, a list of the patient records from each of the participating hospitals was prepared. The identifying variables used were the mother’s chart number and the matching baby’s chart number. For added precision, the date of birth was printed out for each baby. This enabled auditors to verify that the record entered was the correct one. In each of the 14 participating hospitals, a key contact person was identified and informed about the project by the project manager. The key contact was asked to assist (or designate someone who could assist) the auditors to obtain entry to the site, access the patient charts from health records and problem-solve any site-related issues. Prior to data collection, the key contact person (or designate) met with the auditor to show the patient documentation systems and where to find the key information.

Primary data abstraction took place from April to July 2008. Data collection for one site had to be repeated in October 2008 as the original file for this site was overwritten and the data were lost.

The charts (paper or electronic records) were obtained from the Health Records Departments of each of the participating hospitals. The auditors reviewed and re-abstracted the data using the standardized data entry procedures. The data were collected using an SPSS version 15 data file template. A spreadsheet was created that included the data fields under review and pull-down menus matching those found on the current Niday entry screen. For ease of data entry, the variables were placed in the same order as they appeared in the majority of hospital records. Data were entered into two portable laptop computers. Re-abstraction took two to four days per site, due to standard delays when accessing patient records and the time it takes to work through the information in each patient record. The project manager was available by pager, phone or email during the re-abstraction process to address any questions that arose.

Patient records

Although hospital patient documentation systems are not standardized throughout the province, the chart reviews were conducted as consistently as possible. Auditors were trained to obtain information from the same sources used for the original data entry. The postal code, mother’s age and maternal transfer from another hospital were obtained from the admission record; the rest of the variables were obtained from the labour record, the delivery record, the antenatal record, the discharge summary, lab results, nurses’ notes, doctors’ orders, medication records and the postpartum screening record. Terminology and the organization of the patient chart varied somewhat from site to site, but the overall layout of the information was similar. In one region, a standardized documentation system was used by all of the participating hospitals except one. All of the records were in either English or English/French.

Analysis

Descriptive statistics (frequencies, means and percentages) were calculated using SPSS version 15 to describe the characteristics of the study sample groups. The reliability of the data was assessed by comparing the re-abstracted data from the patient record to the original data entered into the Niday Perinatal Database. Cross-tabulations were generated to explore non-agreements and missing data in an attempt to identify potential reasons for the variation between the auditor and the original data entered for each field.

Although sensitivity and specificity can be used to measure the accuracy of data gathered from an external source compared to a primary source of information, this approach requires that one of the data sources is identified as the gold standard.⁹  Many factors can affect the transfer of information from a patient record, such as observer variation, poor documentation, illegible charts, data loss, unavailability and timeliness of chart completion.¹⁰ This makes it impossible to identify a gold standard from either the original data entered into Niday or the re-abstracted data entered by the auditors. When neither data source can be designated as the gold standard, high agreement between the two suggests high reliability. In other words, when two similar datasets are compared and a high proportion of the data are the same, then it can most likely be interpreted that they are both correct. This is an indicator of having high quality data.

Therefore, for the purposes of this audit, we used percent agreement, Cohen’s kappa statistic (κ) and intraclass correlation coefficient (ICCC) between the variables11 to compare the data newly re-abstracted from patient records with data previously entered into the Niday by the participating hospitals. Percent agreement was calculated for all variables. For kappa and ICC, categorical/nominal variables (n = 87), and continuous variables (n = 3) were considered separately.

Categorical variables

The analysis for all the categorical/nominal variables (except for postal code) was by two-way cross tabulations of each variable and comparison of the entries, as explained above. Since postal codes are string variables, cross tabulation was not feasible so an equivalent equal/not equal statement on the SPSS program was used to calculate the percent agreement.

We used Cohen’s kappa statistic to examine the proportion of responses in agreement in relation to the proportion of responses that would be expected by chance, given symmetrical marginal distributions.^12-14 Cohen’s kappa statistic represents the proportion of agreements after chance agreement has been excluded. Kappa values range from 0 (no agreement) to 1 (total agreement). According to Landis and Koch, a kappa value of 0.90 (or 90%) indicates almost perfect agreement while a kappa value of 0.55 (or 55%) reflects only moderate agreement.¹⁵

Continuous Variables

For continuous variables, agreement was assessed using an equal/not equal statement on the SPSS program and by calculating the ICC. ICC is a more appropriate measure of reliability for continuous data than Pearson’s product moment correlation coefficient or Spearman’s rank-order correlation coefficient since these measure association rather than agreement.12-14 ICC values range between 0 (no agreement) and 1 (total agreement), “with values approaching 1 representing good reliability.”^{16, pg. 357}  According to Portney and Watkins,¹⁷ an ICC of over 0.9 (or 90%) indicates excellent agreement, while an ICC of 0.35 (or 35%) indicates poor agreement between variables. The notes to Table 2 shows more detailed interpretation of kappa and ICC values.

Results

This quality assurance project evaluated the reliability, completeness and comprehensiveness of the Niday Perinatal Database and found that the database met expectations either fully or partially.

Reliability

A total of 33 out of 90 variables (96 data fields) in the Niday were re-abstracted from patient records to determine the degree of agreement with data already entered in the database. Of the 89 data fields for which kappa or ICC was calculated, almost one-half (n = 39; 43.8%) showed substantial or almost perfect agreement (beyond chance), suggesting that these variables may be used with confidence. Just over one-third of the data fields (n = 34; 38.2%) were found to have kappa values below the moderate level (60% beyond chance) despite having excellent agreement rates. However, a prevalence effect due to asymmetrical imbalances of marginal totals was the likely cause of the low kappa value in this group.¹⁸  The remaining data fields (n = 15; 16.9%) showed both percent agreement of less than 95% and a kappa or ICC value less than 60% indicating only slight, fair, poor or moderate agreement (beyond chance). This suggests these data fields may be problematic and require further investigation. Table 2 summarizes the percent agreements, Cohen’s kappa or ICC for each data field.

Completeness

Approximately 34% of the variables in the Niday were missing more than 10% of data based on verification reports generated prior to the start of the audit. Only variables that were mandatory or had low rates of missing data (< 10%) just prior to the audit were selected for re-abstraction (Table 1).

Missing (not entered) data were also evaluated as part of the re-abstraction and were found to be associated with the following variables: antenatal steroids, forceps/vacuum, episiotomy, laceration and smoking. The missing data were limited to only three sites (F, J and K; see Figure 1). The primary reason for missing data at these sites was due the auditors or original hospital data entry personnel deciding to leave a cell empty rather than selecting “none” or “unknown.” At site F the auditor left the field empty while the hospital data entry person entered “none” or “unknown,” while the reverse took place at sites J and K. Missing data was not a significant issue and these data points were not excluded from the assessment of agreement. This was not a surprising finding, given the fact that these variables were selected for abstraction in the first place because of high completion rates.

Comprehensiveness

At the time of the audit over 96% of births in the province (involving 95 delivering hospitals and including midwifery hospital births and some home births) were captured in the Niday. There were 90 defined patient elements with 23 mandatory fields (at the start of the audit).

Discussion

Although neither of the datasets used during the audit can be declared as a gold standard, the moderate-to-high levels of agreement (beyond chance) between the two sources suggest that the variables are comparable across two methods of data collection.¹⁹  The worst case scenario in interpreting these findings would be that all the differences are due to having wrong data in the Niday. When there is a level of disagreement between the two data sources for some data fields, part of this difference may be as a result of wrong data in the Niday, wrong data entered during the audit, or wrong data in both datasets.

Although the reasons for non-agreements could not always be discerned, a variety of potential factors were identified during detailed exploration of the data. Results from the audit indicated disagreement between the two data sources occurred across multiple sites, and included both hospital and auditor data entry issues. These issues have been clustered into four themes (data entry choice, clarity of information, inaccurate documentation and human error).
The first issue related to choices available for data entry has to do with the designation given to some variables. At the time of the audit, all data fields in the Niday were designated as either mandatory or non-mandatory. In reviewing non-agreements, it was evident that in some cases the auditor found information in the patient record that the original hospital data entry person did not record. Although, both groups were tasked with finding and entering as much information as possible, in reality it is possible that discretionary completion of some of the non-mandatory data fields at some sites contributed to the non-agreements. This example illustrates the importance of ensuring that all data fields are mandatory and that only essential, meaningful data are collected.

The second issue related to this theme was about pick-list choices and the availability of information in the patient health record. If the information is not documented in the patient record in such a way as to match the pick-list choices, data quality can be affected. For example, in the case of smoking during pregnancy, documentation may indicate that a women smoked, but not provide the detail required to determine the duration of smoking through pregnancy (e.g. above or below 20 weeks as required for Niday at the time of the audit). In some cases where non-agreement occurred, it was because some people entered “unknown” while others left the field empty when the required data was not available in the patient health record. This example illustrates the importance of aligning documentation tools with data entry processes to enhance data quality.

The second theme has to do with clarity of information available for each data field. Confusion over the wording, use of double negatives and different interpretations of the definitions for some variables may have contributed to non-agreements (e.g. interpreting what qualifies as an induction or augmentation of labour). This example illustrates the importance of ensuring the definitions for each variable are precise and applicable to practice.

The third theme was related to inadequate, illegible or inaccurate documentation. Data entry is dependent on the accuracy of the information recorded in the patient health record. Even though specific documents were identified to be the source of information for data entry for both the primary and audited datasets, some of the information entered was difficult to find, or inconsistent, contributing to non-agreement. For example, gestational age and birth weight both require double entry of the data. Double entry of these variables may provide verification that the original number entered is correct, which enhances reliability of the variable, but it does not ensure validity of the information. This is evidenced by the discrepancies between the original data entered and the auditors’ data for these variables.

Finally, even though every attempt was made to ensure a consistent process for data entry, it is always possible that human error contributed to non-agreements between the two datasets. Results of this audit have provided information about potential issues related to data entry for some variables in the database. A number of variables were more problematic. Further exploration of the issues is required in order to develop strategies to improve the data quality for these variables in the Niday.

Interestingly, eight of the data fields identified in this audit as less reliable were also found to be problematic during a previous audit of the Niday (Table 2).²⁰  This is significant in that some of these variables have been identified as priority items highly relevant for the perinatal reports being developed by BORN Ontario.

Table 2. Comparison of abstracted data from patient records (N = 1395) and data entered in Niday Perinatal Database using percent agreement, Cohen’s kappa and intraclass correlation coefficient (ICC)

No.	Variable Name	Data Field Label	Coding	Not matched n/1395 (%)	Percent agreement (%)	Cohen’s kappa [κ] (%)	ICC (%)
Abbreviations: FF02, free flow oxygen; FS, fetal surveillance; ICC, intraclass correlation coefficient; IUGR, intrauterine growth restriction; LGA, large for gestational age; NST, non-stress test; PPV, positive pressure ventilation; PROM, premature rupture of membranes; SGA, small for gestational age. Notes: Cohen’s kappa statistic (κ) degrees of agreement after chance agreement has been excluded15: Poor < 0; Slight = 0–0.20; Fair = 0.21–0.40; Moderate = 0.41–0.60; Substantial = 0.61–0.80; Almost perfect = 0.81­–1.00. Intraclass correlation coefficient (ICC) degrees of agreement17: Poor < 0.50; Moderate = 0.50–0.75; Good ≥ 0.75-0.90; Excellent > 0.90. a N/A - not applicable as equal/not equal was used, hence no cross tabulation to generate the kappa statistic. b Data fields with < 95% agreement and kappa or ICC values < 60% indicating only slight, fair, poor or moderate agreement (beyond chance). c Data fields also found to be problematic during a previous audit of the Niday Perinatal Database.20
Mandatory data fields
1.	SITE	Site name			Pre-entered
2.	Maternal chart number	Maternal chart no.			Pre-entered
3.	Baby chart number	Baby chart no.			Pre-entered
4.	Baby birth date	Baby birth date - DMY			Pre-entered
5.	Number of previous preterm babies	No previous preterm babies	Number (0–15) Unknown	64 (4.6)	95.4	54.5
6.	Number of previous term babies	No previous term babies	Number (0–15) Unknown	79 (5.7)	94.3	91.2
7.	Previous Caesarean section	Previous C/S	Yes No Unknown	50 (3.6)	96.4	81.8
8.	Maternal transfer from	Maternal transfer from	Pick from site list Planned home birth Out of region No transfer	35 (2.5)	97.5	25.0
9.	Multiple gestation	Multiple gestation	Singleton Twin Triplet Quadruplet Quintuplet Sextuplet Septuplet	1 (0.1)	99.9	98.8
10.	Labour type	Labour type	Spontaneous Induced No labour	135 (9.7)	90.3	81.8
11.	Delivery type	Delivery type	Vaginal Caesarean section Unknown	4 (0.3)	99.7	97.3
12.	Mother’s birth date	Mothers birth date - DMY	Date of birth (D/M/Y)	128 (9.2)	90.8	N/Aa	N/Aa
13.	Birth weight	Birth weight b,c	Birth weight (grams)	114 (8.2)	91.8		35.1
14.	Gestational age at birth	Gestational age at birthb	Gestational age (weeks) Unknown	119 (8.5)	91.5		32.0
15.	Baby’s sex	Baby gender	Male Female Ambiguous Unknown	29 (2.1)	97.9	96.0
16.	APGAR - 1	APGAR1	Number (0–10) Unknown	58 (4.2)	95.8	92.5
17.	APGAR - 5	APGAR5	Number (0–10) Unknown	51 (3.7)	96.3	87.7
18.	Newborn resuscitation	Noneb	Not checked Checked	352 (25.2)	74.8	46.7
19.		Drugs		12 (0.9)	99.1	64.3
20.		FF02		118 (8.5)	91.5	70.2
21.		Intubation		10 (0.7)	99.3	63.9
22.		PPV		54 (3.9)	96.1	63.4
23.		Chest Compression		5 (0.4)	99.6	28.4
24.		Unknown b,c		86 (6.2)	93.8	3.0
25.	Neonatal transfer to	Neonatal transfer hospital	Pick from site list No transfer (if birth hospital) Out of region	11 (0.8)	99.2	50.0
26.	Neonatal death / stillbirth	Neonatal death / stillbirth	Not applicable Stillbirth ≥ 20 weeks Neonatal death < 7 days Neonatal death > 7–28 days	2 (0.1)	99.9	50.0
Non-mandatory data fields
27.	Maternal postal code	Maternal postal code	Full postal code	97 (7.0)	93.0	N/Aa	N/Aa
28.	Antenatal steroids	Antenatal steroidsb,c	None 1 dose < 24 hr 2 doses: last dose < 24 hours 2 doses: last dose ≥ 24 hours Unknown	354 (25.4)	74.6	7.5
29.	Fetal surveillance	FS - Admission stripb,c	Not checked Checked	424 (30.4)	69.6	39.2
30.		FS - Auscultationb,c		263 (18.9)	81.1	60.0
31.		FS – Intrapartum electronic fetal monitoring (external)b,c		265 (19.0)	81.0	53.2
32.		FS – Intrapartum electronic fetal monitoring (internal) b,c		125 (9.0)	91.0	45.0
33.		FS – No Monitoring		29 (2.1)	97.9	11.4
34.		FS - Unknown		36 (2.6)	97.4	13.5
35.	If induced - indication for induction	None	Not checked Checked	10 (0.7)	99.3	12.5
36.		Diabetes		9 (0.6)	99.4	74.0
37.		Elective		31 (2.2)	97.8	26.8
38.		IUGR/SGA		14 (1.0)	99.0	64.5
39.		LGA		8 (0.6)	99.4	55.3
40.		Maternal obstetrical conditions		32 (2.3)	97.7	14.6
41.		Multiple gestation		4 (0.3)	99.7	66.5
42.		Non-reactive NST		5 (0.4)	99.6	28.4
43.		Oligohydramnios		7 (0.5)	99.5	79.8
44.		Poor biophysical score		5 (0.4)	99.6	28.4
45.		Post dates		64 (4.6)	95.4	73.8
46.		Pre-eclampsia		25 (1.8)	98.2	43.6
47.		Pre-existing maternal medical conditions		6 (0.4)	99.6	24.8
48.		PROM		42 (3.0)	97.0	52.8
49.		Other maternal		51 (3.7)	96.3	32.1
50.		Other fetal		24 (1.7)	98.3	32.5
51.		Other		16 (1.1)	98.9	24.5
52.	If induced - method of induction	None	Not checked Checked	2 (0.1)	99.9	85.0
53.		Amniotomyb		125 (9.0)	91.0	51.2
54.		Cervidil		53 (3.8)	96.2	70.0
55.		Cytotec/ Misoprostol		15 (1.1)	98.9	20.5
56.		Mechanical		10 (0.7)	99.3	63.9
57.		Oxytocin		129 (9.2)	90.8	66.1
58.		Other		26 (1.9)	98.1	18.0
59.		Other - Prostaglandin		31 (2.2)	97.8	38.3
60.	If Caesarian section - indication for Caesarian section	None	Not checked Checked	2 (0.1)	99.9	85.0
61.		Breech		21 (1.5)	98.5	82.4
62.		Cord prolapse		1 (0.1)	99.9	80.0
63.		Diabetes		7 (0.5)	99.5	49.0
64.		Failed forceps/vacuum		3 (0.2)	99.8	72.6
65.		Fetal anomaly		0	100	100
66.		IUGR/SGA		5 (0.4)	99.6	54.4
67.		LGA		4 (0.3)	99.7	33.3
68.		Maternal request		26 (1.9)	98.1	17.9
69.		Multiple gestation		12 (0.9)	99.1	64.3
70.		Non-progressive labour/ descent/ dystocia		34 (2.4)	97.6	76.6
71.		Non-reassuring fetal status		31 (2.2)	97.8	72.3
72.		Placenta previa		1 (0.1)	99.9	90.9
73.		Placental abruption		4 (0.3)	99.7	60.0
74.		Preeclampsia		8 (0.6)	99.4	42.6
75.		Prematurity		8 (0.6)	99.4	19.8
76.		Previous Caesarean		22 (1.6)	98.4	89.7
77.		PROM		4 (0.3)	99.7	60.0
78.		Other fetal health problem		14 (1.0)	99.0	50.0
79.		Other maternal health problem		17 (1.2)	98.8	31.4
80.	Forceps vacuum	Forceps/vacuumb	None Forceps Vacuum Forceps and vacuum Unknown	189 (13.5)	86.5	55.5
81.	Episiotomy	Episiotomyb	None Mediolateral Midline 3rd degree extension 4th degree extension Unknown	241 (17.3)	82.7	46.9
82.	Laceration	Laceration	None 1st degree 2nd degree 3rd degree 4th degree Cervical tear Other Unknown	347 (24.9)	75.1	63.0
83.	Maternal pain relief	None	Not checked Checked	69 (4.9)	95.1	52.4
84.		Epidural		101 (7.2)	92.8	85.5
85.		General		8 (0.6)	99.4	73.1
86.		Localb		111 (8.0)	92.0	45.8
87.		Narcotics		97 (7.0)	93.0	82.4
88.		Nitrous Oxide		94 (6.7)	93.3	71.9
89.		Non-pharmacologicalb		319 (22.9)	77.1	49.5
90.		Pudendal		1 (0.1)	99.1	92.3
91.		Spinal epidural combination		21 (1.5)	98.5	50.4
92.		Spinal		51 (3.7)	96.3	85.3
93.		Unknown		15 (1.1)	98.9	46.0
94.	Time of birth	Time of birth	Time of birth (24 hour format) None	127 (9.1)	90.9	N/Aa	N/Aa
95.	Delivered by	Delivered by	Obstetrician Family physician Midwife at hospital Midwife at home Nurse practitioner Specified midwife group Other Unknown	159 (11.4)	88.6	71.8
96.	Smoking status	Smoking b,c	No smoking ≤ 20 weeks > 20 weeks ≤ 20 and > 20 weeks Unknown	294 (21.1)	78.9	50.7

 

A previous validation study that explored record linkage of births and infant deaths in Canada examined gestational age and birth weight and indicated good overall agreement.^{21, 22}  Gestational age was also found to have a relatively high degree of agreement between the Discharge Abstract Database (DADD) of the Canadian Institute for Health Information (CIHII) and the Nova Scotia Atlee Perinatal Database (NSAPDD).²³  This finding is in contrast to our study, where gestational age and birth weight achieved ICC values of between 30% and 40%, indicating poor agreement.

Caesarean delivery was found to be coded accurately in the DAD, and information on first to fourth degree perinatal lacerations and induction of labour was also reasonably accurate in this study.²³  Results of our audit were consistent with respect to delivery type and lacerations, with substantial or almost perfect agreement (beyond chance) achieved between the re-abstracted data and the information previously entered into the Niday. However, induction method (amniotomy) was less reliable with only 51.2% agreement (beyond chance) noted between the two datasets.

Ensuring completeness and reliability of the data entered into the Niday is a challenge. Data are entered manually via a secure Internet website or uploaded directly into the database from electronic documentation systems. Regional coordinators send reminders to hospital staff to facilitate the process of data entry and to troubleshoot problems when needed. Verification reports are generated quarterly by a data analyst to identify inconsistencies in numbers and types of births and find errors in the data. A training program has been developed so that all users have a thorough understanding of the system. Sustainability of this database depends on achieving broad support at all levels and valuing the system as a key attribute of the patient safety movement. Based on the results of this audit, and through consultation with experts in the field, a number of recommendations have been put forward to improve data quality (Table 3).

Table 3. Recommendations to improve quality of data

This audit is in line with the MOHLTC quality assurance initiatives, and it is a logical step to improving data quality and perinatal care practices. The Niday Perinatal Database is a comprehensive, multifaceted system providing data to perinatal care providers, decision makers, educators and researchers in Ontario. Since the audit, the Niday has expanded to capture data for 100% of births in the province. Many upgrades and improvements to the system have already been completed. Further exploration of quality issues is ongoing as part of the initiative to integrate the database with four other perinatal/newborn databases (Fetal Alert Network, Maternal Multiple Marker Screening, Newborn Screening, and the Ontario Midwifery Program (OMPP) Database. Recent Ministry funding and a newly established administrative body (BORN Ontario) have been established to carry these recommendations forward.

Limitations

There are two potential limitations to this audit: completeness and clarity of the patient health record and sampling method. Of the hospitals entering data into the Niday at the time of the audit, 14% were recruited to participate in the re-abstraction process. This sample pool was sufficient to identify a number of issues. Although, the patient charts were selected randomly, the hospitals were selected through purposive sampling; therefore, the results of these analyses may not be generalizable to all hospitals in the province. Data entry personnel for both the original data entry to the Niday database and the re-abstraction process were asked to collect as much information as possible from the patient chart and to be vigilant in entering the data. However, reliability of the data entered into the Niday database is dependent on completeness and clarity of the information documented. Deficits in either regard can influence the reliability of the data entered and influence the results of an audit.

Conclusions

There were 90 defined patient elements within the Niday Perinatal Database at the start of the audit. Approximately one-third of the variables were re-abstracted from the patient record to determine agreement with the data already entered in the Niday Database. Approximately 17% of the data fields audited showed both percent agreement of less than 95% and a kappa or ICC value of less than 60%, indicating only slight, fair, poor or moderate agreement (beyond chance) between the data originally entered into the Niday database and the data re-entered during the audit. This suggests these data fields may be less than reliable and require further investigation to ensure quality.

Acknowledgements

This project is the result of the effort of many individuals and organizations in Ontario. The Niday Quality Audit was conducted under the auspices of the Ontario Perinatal Surveillance System (OPSS). We thank Monica Prince (Prince Computing) who conducted the data analyses and provided input into the final report. We would also like to thank Dr. Ann Sprague for her assistance reviewing the final report, Deshayne Fell for her help in reviewing the manuscript, the auditors for their tireless efforts collecting data at the participating sites across the province and the countless practitioners, data entry personnel and decision makers who provided assistance to make the project possible.

References

1. Brown A. Update: building on a foundation to sustain quality data. Toronto (ON): Ministry of Health and Long-Term Care; 2007 Jan. 
2. Last JM. A dictionary of epidemiology. 4th ed. New York (NY): Oxford University Press; 2001.
3. Health Canada. The Canadian Perinatal Health Report. Ottawa (ON): Minister of Public Works and Government Services Canada; 2000.
4. Choi BC. Perspectives on epidemiologic surveillance in the 21st century. Chronic Dis Can. 1998;19(4):145-51.
5. BORN Ontario [Internet]. Ottawa (ON): BORN [updated 2011] Mission and Vision [cited 2011 Apr 26]. Available from: http://www.bornontario.ca/about-born/vision-and-mission
6. Statistics Canada. Births, estimates, by province and territory [Internet]. Ottawa (ON): Statistics Canada; [modified 2010 Oct 26; cited 2011 Apr 26]. Available from: http://www40.statcan.gc.ca/l01/cst01/demo04a-eng.htm
7. Freedman G; Health Results Team for Information Management. Building a data quality management framework for Ontario. Toronto (ON): Ministry of Health and Long-term Care; 2006.
8. Canadian Institutes of Health Research; Natural Sciences and Engineering Research Council of Canada; Social Sciences and Humanities Research Council of Canada. Tri-council policy statement: ethical conduct for research involving humans. Ottawa (ON): 2010.
9. Iron K, Manuel DG. Quality assessment of administrative data (QuAAD): an opportunity for enhancing Ontario's health data. ICES Investigative Report. Toronto (ON): Institute for Clinical Evaluative Sciences; 2007 Jul.
10. Hierholzer WJ. Health care data, the epidemiologist's sand: comments on the quantity and quality of data. Am J Med. 1991;91(3B):21S-26S.
11. Juurlink D, Preyra C, Croxford R, Chong A, Austin P, Tu J, et al. Canadian Institute for Health Information Discharge Abstract Database: a validation study. Toronto (ON): Institute for Clinical Evaluative Sciences; 2006 Jun.
12. Streiner DL, Norman GR. Health measurement scales: a practical guide to their development and use. 4th ed. Oxford (UK): Oxford University Press; 2008.
13. Norman GR, Streiner DL. Biostatistics: The bare essentials. 2nd ed. Hamilton (ON): BC Decker; 2000.
14. Bartko JJ. Measurement and reliability: statistical thinking considerations. Schizophr Bull. 1991;17(3):483-9.
15. Landis RJ, Koch GG. The measurement of observer agreement for categorical data. Biometrics. 1977;33(1):159-74.
16. Bedard M, Martin NJ, Krueger P, Brazil K. Assessing reproducibility of data obtained with instruments based on continuous measures. Exp Aging Res. 2000; 26(4):353-65.
17. Portney LG, Watkins MP. Foundations of clinical research: applications to practice. 2nd ed. New Jersey:Prentice Hall; 2000.
18. Sim J, Wright CC. The kappa statistic in reliability studies: use, interpretation, and sample size requirements. Phys Ther. 2005;85(3):257-68.
19. Bader MD, Ailshire JA, Morenoff JD, House JS. Measurement of the local food environment: a comparison of existing data sources. Am J Epidemiol. 2010;171(5):609-17.
20. Ali AH. An evaluation of perinatal surveillance system in Eastern and Southeastern Ontario. Ottawa (ON): University of Ottawa; 2003.
21. Fair M, Cyr M, Allen AC, Wen SW, Guyon G, MacDonald RC, et al. Validation study for a record linkage of births and infant deaths in Canada. Ottawa (ON): Statistics Canada; 1999 [Statistics Canada, Catalogue No.: 840013XIE].
22. Fair M, Cyr M, Allen AC, Wen SW, Guyon G, MacDonald RC. An assessment of the validity of a computer system for probabilistic record linkage of birth and infant death records in Canada. The Fetal and Infant Health Study Group. Chronic Dis Can. 2000;21(1):8-13.
23. Joseph KS, Fahey J. Validation of perinatal data in the Discharge Abstract Database of the Canadian Institute for Health Information. Chronic Dis Can. 2009;9(3):96-100.