PROGRAM EVALUATION AND RESEARCH

This section of the website will provide you with information about the national evaluation of Aboriginal Head Start, and about research endeavours in AHS. In the evaluation section you will find information that explains what the national evaluation is all about, and what stage in the process we are at.

There has not been research conducted on AHS, and we hope to build this section of the website as research on the program is stimulated.

• About National Program Evaluation

About National Program Evaluation

• About Evaluation
• About AHS Program Evaluation
• Why a National Evaluation?
• Where are we in the National Evaluation?

Evaluation assists in describing program operations and in guiding AHS program development as well as providing information to inform public accountability. Read on to learn more about the nature of AHS evaluation and some of the challenges and creative solutions to a rigorous, culturally-sensitive approach.

About Evaluation

Evaluation is a field of science that strives to describe or measure the extent to which a program is meeting its goals. The evaluation process examines the design and implementation of a program as well as the changes resulting from program participation. Predetermined indicators of program performance generally provide the basis for data collection; additional indicators generated in the process of the evaluation are captured as well. Information gathered in an evaluation contributes to informed decision-making affecting program direction, determining the worth and effectiveness of a program, and identifying the strengths of the program. Program evaluation is a complex and continuous process of gathering and analyzing information about the status of a program and of communicating results to the public.

About AHS Program Evaluation

AHS Evaluation provides an opportunity to gather evidence on whether the program is doing what it set out to do, and to share the results with participating communities, the Government of Canada and with all Canadians. AHS evaluation takes place at the national, regional and local levels.

There are two main components to the AHS national evaluation: a process evaluation and an impact evaluation.

The National Process and Administration Evaluation collects demographic and descriptive data from each AHS site about participants, activities, needs and finances. This evaluation survey is conducted annually. Ongoing operations within each of the six program components are captured in this process. Data are highly useful in explaining and promoting the program within and outside government. Data are analysed to capture the sites' diversity in size, geographic location, culture and program delivery. Highlights of the process evaluation report are published annually on the AHS website and have a wide distribution in print.

The Impact Evaluation describes the changes in children, parents and communities as a result of their participation in the program. The evaluation examines the impacts of all six program components. A sample of AHS sites took part in this study which demonstrates the effects of the program based on information gathered by community evaluators through child assessment and observation and structured interviews with community members. The impact evaluation began in the fall of 2002 and will report on the findings in 2005.

The National Evaluation has been designed from the outset on principles of cultural sensitivity, community participation and capacity-building. Academics, evaluation consultants and community members are partners in the development of the methodology and tools and in the evaluation process itself. A holistic approach to the evaluation views the children, families and programs in the context of their community and extended society.

Regional evaluations focus on collecting data on program operations and use of resources that will assist program planning and improvements regionally. Regional evaluations sometimes include aspects of impact evaluation.

Local evaluations provide information to sites on programming and management operations that assist them in assessing program quality and effectiveness in areas that will improve program delivery and highlight program strengths. The experiences and observations of participating families and other community members give valuable input to local evaluations.

Why A National Evaluation?

AHS is required to complete an evaluation of the program through a commitment made to the Treasury Board of Canada, which provides all of the ongoing funds to AHS. As the federal department that administers AHS, Health Canada coordinates and implements a national evaluation of AHS. Gathering national data allows the program to compile statistics and evidence to support reports and claims that the program is operating at full capacity, could utilize more resources, and is having a positive impact on its participants. It also contributes to our ability to promote the program with hard evidence and to raise awareness about the strengths and challenges facing Aboriginal people living in urban and northern communities.

All AHS projects have a responsibility to participate in national, regional and local evaluation activities. By signing a Contribution Agreement with Health Canada to operate an AHS project, project sponsors are accountable to demonstrate to their communities and to the Government of Canada that they spend responsibly and effectively. This maintains community and government confidence in their ability to deliver AHS.

What Does the AHS National Office do With the Information Collected?

The information collected in national evaluation activities is used in a number of ways.

• It is used by Health Canada to demonstrate to the Treasury Board of Canada, participants, communities and other stakeholders that AHS is accountable for the ongoing funds to the program.
• It is used to produce AHS evaluation publications that foster information and knowledge exchange with key partners and promote the success of the program.
• It is used by the program to support arguments for program enhancement and expansion.
• It is used to guide program and policy development (e.g. data gathered about the lack of services for special needs children has guided the planning for the enhancement and expansion of the program to include a focus on this area).
• It is used to identify program, participant and community strengths and challenges that will lead to the overall improvement of AHS.

Where are we in the Evaluation Process?

The national evaluation of AHS is an ongoing process. In 1997, an Evaluation Framework was developed that outlines the plan and guidelines for the evaluation of the AHS Program. It provides an historical overview of the design of the evaluation. An Evaluation Subcommittee of the National Aboriginal Head Start Council is working together with Health Canada to guide and inform the evaluation of AHS.

The National Process and Administrative Survey is conducted each spring and data is analyzed over the summer and fall. The survey collects data on characteristics of the program and its participants, project administration and coordination, program components and the various activities associated with them, and program needs and finances. The first annual National Process and Administrative Evaluation was conducted in 1999. The survey results contribute to the development of the National Impact Evaluation and to AHS national program planning. Visit the publications section of this website to view or order copies of past evaluation reports.

The National Impact Evaluation is underway. There are a number of challenges involved in evaluating the impact of AHS. The challenges are common to evaluating child outcomes, Aboriginal programs in general, and challenges specific to AHS:

• ensuring that the evaluation approach and tools are culturally appropriate;
• determining the indicators of impact on children, parents and the communities;
• attributing impact to AHS participation;
• addressing all six program components;
• representative and valid pilot testing; and,
• the diverse nature of AHS (i.e. First Nations, Inuit and Métis participants in the far north, south, east and west).

Together with Health Canada, an expert Tool Development Team (TDT) was convened in October 2000 to address these issues. The TDT developed a package of ten tools to be piloted in five AHS sites in the winter and spring of 2001 and 2002. Eight of them are questionnaires that will be administered by trained community evaluators to AHS staff, parents, elders and traditional teachers, community health care professionals, kindergarten teachers, social workers, and other community professionals. The questionnaires are intended to capture interviewees' perceptions of change. In addition to the questionnaires are two child oriented tests: the Work Sampling System; and the Aboriginal Vocabulary Aquisition test.

The Work Sampling System (WSS) is a standardized comprehensive approach to observing and recording a child's progress in school in different areas of development. AHS teaching staff are trained in the WSS, and will chart children's performance in four areas: personal and social development; language and literacy; mathematical thinking; and, physical development at the beginning and end of each school year. The WSS was selected because it evaluates children individually rather than comparing them to one another. As well, it is a process for observing children and their work and applying those observations to the enhancement of their growth and development. Outcomes of WSS are highly correlated with other school readiness tests.

The Aboriginal Vocabulary Acquisition Test is being developed by the TDT with input from AHS sites. It involves one-on-one testing by an evaluator with the child. The test is intended to measure the acquisition of Aboriginal vocabulary and cultural knowledge. This test was designed to be applicable to the range of languages and cultures and demographic regions relevant to AHS. For example, the child will be shown a picture and asked to say the Aboriginal word for the object, animal or body part presented. Children will only be asked to identify things that are being taught in their particular site in the Aboriginal language taught in the site. The test is designed to test both expressive and receptive language (e.g.. "can you say the Cree name for this" while the evaluator shows the child a picture of a bear, tests expressive language, while "show me inukshuk" while the evaluator shows the child four pictures, one of which is an inukshuk, tests receptive language).

AHS is pilot testing National Impact Evaluation tools in five AHS sites across the country that are representative of the four directions, First Nations, Inuit and Métis sites, as well as English and French sites. The pilot sites are located in: Lorretville, Quebec; Igloolik, Nunavut; Toronto, Ontario; Dauphin, Manitoba; and, Prince George, British Columbia. Pilot testing will establish what needs to be adjusted or improved in the tools and the methodology. Results of the pilot testing will be available in the fall of 2002. A two year National Impact Evaluation began in the fall of 2003.

Also completed as a component of the Aboriginal Head Start (Urban/Northern) National Evaluation was Johnny National, Super Hero (html/pdf) — a story about AHS that could be true — by Tomson Highway, and illustrated by Leo Yerxa. It is a narrative report prepared in the form of a short story created by Tomson Highway, noted Cree playwright and novelist. Highway's story profiles Aboriginal Head Start (Urban/Northern) in a dimension that complements and enhances the more quantitative process evaluation reports. The fictional characters and text of the story are based on material collected from Mr. Highway's site visits, local evaluation reports, and letters from staff and parents describing their experiences of the program. The story celebrates and highlights the accomplishments of the program through storytelling, a traditional teaching and communication approach. A Reader's Guide was developed to accompany the story to present information about this innovative approach, and intends to stimulate discussion about the story and its characters.