Early Hearing and Communication Development

 

Chapter IX: Program Evaluation

Program Evaluation and Quality Improvement

Authors: Dr. David Brown and Dr. Martyn Hyde

Program evaluation and quality improvement (PEQI) are essential components of any high quality program for early hearing and communication development (EHCD). Both components must exist as a sub-program of the overall EHCD program. The PEQI sub-program should address structure, process and outcome elements of the entire EHCD program. The PEQI sub-program requires explicit and precise a priori definition of quantitative and realistic objectives for the EHCD program as a whole, as well as for each and every one of its major, programmatic components. Additionally, PEQI programs should include clear and demonstrably effective mechanisms for EHCD program adjustment in response to observed deficiencies.

Program evaluation is a formal method by which the EHCD program directorate can determine whether the overall goals and specific objectives of the program are actually being achieved. To be worthwhile, such an evaluation must lead ultimately to concrete changes in each and every component of the program that is thought to contribute to any observed shortfall in achievement of objectives. Quality improvement is a related conceptual approach that incorporates ongoing and continuous re-examination of the program components in order to determine proactively whether the program is functioning as effectively and efficiently as possible, and to determine when and where problems or concerns arise, so that the components can be maintained, repaired and, wherever feasible, enhanced (see, for example, Donabedian¹).

In accordance with a widely-accepted health services evaluation conceptual framework, a good PEQI program can be formulated in terms of the structural, process and outcome components that define the overall EHCD program.² The structural elements of the EHCD program include but are not limited to:

• administration
• personnel training, performance management and continuing education
• information management, including tracking and reporting
• instrumentation procurement, calibration, maintenance and manufacturer liaison
• clinical protocol development, dissemination and updating
• family consenting, infant risk assessment; contact, screening and follow-up compliance procedures
• audiologic assessment procedures
• communication development options provision
• family support provision, surveillance procedures, referral procedures, measurement procedures for outcomes and proxy outcomes
• quality management
• public and professional education
• ongoing technology
• evidence assessment

Program Evaluation

Program evaluation requires that the program and its component sub-programs have associated outcome measures that are not only explicitly definable but also quantifiable. The next step is to develop realistic indicators and benchmarks for each of the outcome measures; the program can then use these to determine if it is meeting its intended performance targets. The benchmarks are levels to which programs should aspire, in order to know that what is being achieved satisfies defined standards of care. They are important in EHCD programs to prove to health professionals and decision makers that EHCD is both beneficial and cost-effective.³ It should be recognized that there is clear evidence of program improvement over time and that benchmarks might be difficult to achieve, particularly at the outset. However, asymptotic performance benchmarks should be achievable in the second or third year of implementation. Program administrators should encourage and assist individual sites in the pursuit of performance benchmarks that are reasonably uniform and consistent with the program's overall benchmarks. Failure to achieve benchmark performance should be clearly evidentiated and improvements aggressively sought and documented.

Outcome Measures

The outcome measures that a high quality EHCD program should routinely quantify and document are listed below. Measurements may include both true outcomes and process events that may serve as proxies for true outcomes. The latter type of measurement is common in situations wherein true outcomes are difficult, expensive or impossible to obtain. An example is use of a habilitative service event as a proxy for the desired effect of that service on language development.

• The number and proportion of the overall target population successfully screened by 1 month of age, or within one month of discharge from the birth hospital admission.
• The numbers and proportions of infants with a refer result from screening overall and broken down by screening site, screening personnel and risk status.
• The numbers and proportions of screening referrals for whom audiologic assessment is initiated by 3 months of age and within two months of the initial screening result.
• The numbers and proportions of screening referrals with completed audiologic assessments by 4 months of age and within one month of assessment initiation.
• The numbers and proportions of the target birth, screened and referred cohorts of infants who have confirmed permanent childhood hearing impairment (PCHI) at 6 months of age, also broken down by ear and severity categories, risk status and assessment site/personnel.
• The numbers and proportions of infants with PCHI who have been recommended for hearing aids by 6, 9 and 12 months of age.
• The numbers and proportions of infants with PCHI who have received medical intervention for otitis media (OM) by 6 months of age.
• Documentation of reasons for non-fitting of hearing aids by 6, 9 and 12 months of age, broken down by risk status.
• The number of infants with confirmed PCHI at 12 and 24 months of age who passed newborn screening and were detected by surveillance and referral-in routes.
• Family satisfaction with EHCD program processes.
• Communication development outcomes, broken down by type and degree of PCHI.

Performance Benchmarks

Using the outcome measures above, programs can determine if benchmarks have been achieved. The following are examples of feasible benchmarks. (A more comprehensive set of benchmarks can be found in JCIH.⁴)

• Within six months of program initiation, hospitals or birthing centres screen a minimum of 95% of infants during their birth admission or before 1 month of age.
• The referral rate for audiologic and medical evaluation following the screening process should be 4% or less within one year of program initiation.
• The agency within the EHCD program with defined responsibility for follow-up documents efforts to obtain follow-up on a minimum of 95% of infants who do not pass the hearing screening.
• Infants referred from universal newborn hearing screening (UNHS) begin audiologic and medical evaluations before 3 months of age, or three months after discharge for neonatal intensive care unit (NICU) infants.
• Infants with hearing impairments are enrolled in a family-centred early intervention program before 6 months of age.

Long-Term Outcomes

Measures that may appear to be proxies for long-term outcomes may constitute true outcomes at the sub-program level. For example, the UNHS component of an EHCD program exists solely in order to deliver all correctly identified infants with PCHI to appropriate follow-up services, with minimal delivery of false positives.

In the larger sense, however, EHCD programs exist not to screen or to diagnose hearing impairment but to deliver effective interventions and communication development services. Ultimately, therefore, the true measures of a successful EHCD program lie in outcomes such as family satisfaction with services, improved hearing ability at an early age, and improved communication development over the long term. There is an acute need to develop and apply valid and accurate measures that reflect these diverse outcome domains. Because randomized clinical trials of EHCD programs are clearly unfeasible, the ultimate validation of the benefits of EHCD will lie in comparisons of a variety of outcomes with their historical controls. This approach has been strongly advocated in several critical evaluations of UNHS/ EHCD initiatives, including formal, evidence-based systematic reviews.

Economic Evaluations

Aside from the outcome measures listed above, which provide information service processes and associated outcomes, funding agencies routinely require both budgetary and more comprehensive economic evaluations of EHCD programs. These are relevant for accountability and sustainability issues, especially in relation to cost containment and to opportunity costs of alternative health care programs. The functionality of the EHCD program is contingent upon its continued funding. Therefore, economic data need to include, but are not limited to, the capital and operating direct costs of all program structures and processes, including infrastructural elements such as information systems. A common problem in such analyses is to differentiate the true costs of the EHCD program from other costs that are associated with the health care environment within which the EHCD program operates. These analyses must also take into account the actual costs that would ultimately be sustained in the absence of the EHCD program, and this is a difficult area for which quantitative data are frequently lacking. An example of the difficulty is the weighing of direct EHCD costs against the actual, historical costs of the ad hoc and frequently inadequate assessment and intervention services that are the norm in the absence of EHCD programs.

Basic measures that are commonly used for EHCD direct cost evaluation are the overall cost per infant screened and the cost per infant identified with a PCHI.⁵ Such analyses are fundamental and are a useful facet of PEQI, especially for comparative cost-effectiveness analysis of various process options. Cost-effectiveness analysis (CEA) will be discussed later in this chapter. For example, cost per infant identified may be very sensitive to the referral rates to audiologic assessment that are achieved by specific screening strategies, and this may be more meaningful than simple use of process proxies such as the raw referral rates. However, a much more comprehensive analysis is required to approach true cost-benefit issues, because cost-benefit analysis (CBA) requires attribution of quantitative, monetary costs to long-term outcomes such as educational achievement, earning capacity and quality of life. This is an area that is at a rudimentary stage and requires much further investigation.

False-Positive Referrals

Some of the "costs" associated with program performance are extremely difficult to quantify. An important example are the costs associated with false-positive screening referrals. Much concern has been expressed in the literature about the impact of false positives on both parental anxiety and child bonding, as well as on more easily determined costs such as those of unnecessary audiologic assessments, assessment errors and inappropriate interventions. At present, a major focus of program quality is based upon minimization of the screening false-positive rate, as a proxy cost measure. A related key focus is upon audiologic assessment error minimization.

Key References

1. Donabedian A. Explorations in quality assessment and monitoring. Volume 1: The definition of quality and approaches to its assessment. Ann Arbor, MI: Health Administration Press; 1980.
2. Hyde ML, Riko K. Design and evaluation issues in universal newborn hearing screening programs. J Speech Lang Pathol Audiol. 2000;24(3):102-18.
3. Finitzo T, Crumley W. A model universal newborn hearing screening program for hospitals and birthing facilities. J Speech Lang Pathol Audiol. 2000;24(3):130-8.
4. Joint Committee on Infant Hearing. Year 2000 Position Statement: Principles and Guidelines for Early Hearing Detection and Intervention Programs. Am J Audiol. 2000;9(1):9-29.
5. Keren R, Helfand M, Homer C, et al. Projected cost-effectiveness of statewide universal newborn hearing screening. Pediatrics. 2002 Nov;110(5):855-64.

Cost-Effectiveness Analysis

Authors: Dr. Martyn Hyde and Dr. Brian Westerberg

The decision to implement a systematic program for early hearing and communication development (EHCD) must take account of the net costs to the individual family and to society as a whole. This is especially relevant in the context of a limited health care cost envelope, within which expenditure on one type of service implies restriction of some other health services. Cost analysis increases in importance if the proposed program does not clearly reflect an ethical imperative or a core societal value, for example, that "all children and families have a basic right to early, effective communication." Debates about such value statements in relation to EHCD are as yet at an immature stage.

Cost-effectiveness analysis (CEA) seeks to comparatively evaluate different service models, usually in terms of cost per unit valued outcomes, such as cases successfully identified, diagnosed or achieving specific long-term criteria of success. Cost-benefit analysis (CBA) can be viewed as an extension of CEA, within which all activities, outcomes and side effects of a program are expressed in a common, monetary unit.

When the net costs of a program are assessed comprehensively and are negative (i.e., a net monetary gain), then the cost issue is moot, except for the politically common issue of short-term cost of long-term benefit. When there is significant, positive net cost, society must consider value judgments, typically relating the proposed program to other, comparable activities already occurring (such as other newborn mass screening activities). Society must also consider the costs associated with the status quo (in the absence of the proposed program) and how they may change, especially in relation to the quality of life of affected individuals and families. It is often difficult and contentious to assign monetary values to such domains, although several sophisticated methods are available (such as utility analysis).

There have been several published cost-related analyses in connection with EHCD programs. The approaches and models used vary in sophistication and completeness, and the field is characterized by variability of methods, assumptions and results, as well as limitations of data underlying rational choice of key parameters. By far the most sophisticated analysis to date is a report by Keren et al.,¹ which includes an extensive bibliography of earlier publications in the area. Keren et al.¹ attempted to identify the best data available for the many key parameters in a comprehensive model. In addition to the usual program capital and operating costs, they estimated long-term savings from reduced, special educational costs, as well as improved work force productivity. The key methods, findings and limitations of the study are presented here.

The target disorder for the CEA was defined as a bilateral hearing impairment of >= 40 dBHL. This is a typical selection, but it is the most conservative criterion within reason. Many EHCD programs, especially in the United States, have much more liberal criteria, typically including lesser degrees of hearing impairment and unilateral impairments. A change in criterion definition has a strong effect on the prevalence of the disorder, and increases the incremental yield of both targeted and universal systematic screening. This limits the direct applicability of the analysis results to many current screening programs, although the methodology remains generally appropriate.

Many probabilities used in the model were available through data from current hearing screening programs. Reasonable estimates were available for the prevalence of hearing impairment in low- and high-risk infants, the proportions who complete testing at each stage of the protocol, and the proportions of infants with hearing impairments detected prior to 6 months of age and who had intervention prior to 12 months of age. Some probabilities were estimated - for example, sensitivity and specificity of screening tests. While there are excellent data available on specificity, current data on sensitivity are limited, so actual program sensitivity may diverge considerably from the assumed values of 0.95. The assumption of identical sensitivities for otoacoustic emissions (OAE) and auditory brainstem response (ABR) is also questionable, given that the ABR is sensitive to auditory neuropathy (AN), whereas the OAE are not, and that AN may be present in up to 10% of all infants with bilateral permanent childhood hearing impairment (PCHI).²

Screening costs included capital costs for equipment, as well as labour costs. Long-term societal costs accounted for lost productivity, special education, vocational rehabilitation, medical costs and assistive devices.

With the nominal values of all base parameters in the model, three situations of no screening, targeted screening for high-risk infants and universal screening detected 30, 66 and 99 of 128 infants with PCHI, respectively. Costs per infant diagnosed were $2,300, $10,100 and $21,400, respectively. The incremental (marginal) cost per additional case diagnosed was $16,400 for targeted screening, and $44,300 for universal screening (all values in $U.S.). These values were considered comparable to those that apply in detection of hypothyroidism and phenylketonuria.

When lifetime costs were considered, both high-risk and universal screening programs were found to result in overall cost reduction, assuming that intervention prior to 12 months of age improved speech and language outcomes. For universal screening, a majority of cases detected were required to achieve normal language outcomes, in order for there to be a net cost reduction.

An interesting feature of the model is that the numbers of hearing impaired children with normal language outcomes were 53, 59 and 65 for the three situations noted; clearly, the assumptions used were such as to yield these very modest differences in this important outcome.

Keren et al.¹ used sensitivity analysis to examine the extent to which their result varied for different values of key parameters in the model. This is an essential component of any such analysis, if it is to be relevant, valid and generalizable. The incremental cost per case diagnosed by universal screening was found to be very sensitive to the assumed proportions of successful long-term language outcomes and to the impact on lifetime productivity. It was moderately sensitive to the assumed success rate for targeted screening, and to the losses to follow-up in a universal program.

The Keren et al. model¹ highlights a need for better data on long-term impact of EHCD programs on language levels and on lifetime productivity. It also underscores the importance of high rates of follow-up diagnostic evaluation in infants who do not pass screening; this is a common area of deficiency in EHCD program reports to date. It is important to note that, for example, a 20% loss to follow-up of screening referrals translates to a 20% reduction in the effective sensitivity of the screening process, whatever the estimated, intrinsic sensitivity of the screen.

A final caveat is that overall performance of a screening program is a function of many parameters of program operation, as well as contextual variables relating to the health care system in which the program is embedded. Moreover, simple sensitivity analysis involving adjustment of one or even two parameters may not reveal the true effects of program improvement in several aspects simultaneously. Because of these and other limitations, the Keren et al. report¹ may be considered as illustrating an appropriate methodological approach to CEA in relation to early detection of hearing impairment. However, further work in this area is required to ensure its generalizability to specific program structures within specific health care systems.

Conclusions

• Universal newborn hearing screening (UNHS) must be accompanied by appropriate, accessible services for confirmation, audiologic and etiologic diagnoses, and effective hearing and communication development (HCD) options for all children referred through screening programs.
• Early hearing and communication development (EHCD) programs should reflect demographic and cultural factors as well as existing systems, infrastructure and well-developed collaborative linkages with other health care, social support and educational systems.
• A well-designed program will include ongoing evaluation and continuous quality improvement components as well as cost-effectiveness analyses (CEA) and costbenefit analyses (CBA).

Key References

1. Keren R, Helfand M, Homer C, et al. Projected cost-effectiveness of statewide universal newborn hearing screening. Pediatrics. 2002 Nov;110(5):855-64.
2. Sininger Y. Auditory neuropathy in infants and children: implications for early hearing detection and intervention programs. Audiology Today. 2002 Oct:16-21.

Previous Page | Table of Contents | Next Page