POR Registration #: POR 071-13
PWGSC Contract # EP363: 090027/011/CY
Contract Award Date: 5/02/2014
Delivery date: 07/03/14

Winter 2014 Skills Initiative Advertising Campaign Evaluation (ACET)

Methodology Report

Prepared by:
EKOS RESEARCH ASSOCIATES INC.

Prepared for:
Employment and Social Development Canada

Ce rapport est aussi disponible en français sur demande

For more information on this report, please email: nc-por-rop-gd@hrsdc-rhdcc.gc.ca

EKOS RESEARCH ASSOCIATES

Ottawa Office
359 Kent Street, Suite 300
Ottawa, Ontario
K2P 0R6
Tel: (613) 235 7215
Fax: (613) 235 8498
E-mail: pobox@ekos.com

Toronto Office
1235 Bay Street, Suite 400
Toronto, Ontario
M5R 3K4
Tel: (416) 598 8002
E-mail: toronto@ekos.com

Winnipeg Office
7 Prominence Point
Winnipeg, Manitoba
R3Y 0A9
Tel: (204) 221-9923
E-mail: winnipeg@ekos.com

www.ekos.com

Executive Summary

Background

The Government of Canada’s Better Job advertising campaign has been in place for three years. It was designed to increase awareness among youth about key government programs and services available to help them obtain education and skills to get better jobs. The new 2014 Skills Initiative campaign aims to increase awareness among all Canadians of the Government of Canada’s programs and services designed to assist Canadians to connect with available jobs. It is also intended to drive target audiences to ActionPlan.gc.ca/Jobs or the 1-800 O-Canada number to find programs and services that will help them make informed choices with regard to education and career.

As the Skills Initiative campaign exceeds $1,000,000, public opinion research is required by Treasury Board using the Advertising Campaign Evaluation Tool. Our proposal to conduct a survey of Canadians using the ACET will help to evaluate the campaign by effectively gauging Canadians awareness and knowledge of this initiative. Information gathered from this process will provide Employment and Skills Development Canada (ESDC) with concrete data that will assist in developing/adjusting future media strategies.

Research Purpose

Specifically, the purpose of the quantitative post-testing is to evaluate the 2013-2014 Skills Initiative Campaign and to gauge Canadians’ awareness/knowledge of the campaign as seen on television and social media sites. Information obtained through this public opinion research can be extrapolated to the wider population of Canadians and allows Employment and Social Development Canada to determine the impact of its advertising campaign and provide direction on areas where the campaign can be adjusted to reach a broader Canadian public. The findings provide useful information to improve future campaigns which are better targeted, more informative and better geared to the target population of working age Canadians under 55 and Canadians in general.

Methodology

A bilingual, national telephone survey was conducted of 2,007 Canadians 18 years of age or older. The survey instrument consisted of the ACET questions used by the Government of Canada in all of its advertising evaluation research. The sample included randomly selected respondents from all provinces and territories. The survey sample relied on a Random Digit Dial (RDD) method of sampling households from a wider sample frame of listed and unlisted landlines in Canada and is therefore considered to be a probability sample that is representative of the wider universe (exclusive of households with only cell phones). The number of total completed in the landline sample is 1,746. A subset of 13 per cent of the sample (269 interviews), however, were completed with Canadians who own only a mobile telephone and not a landline (i.e. cell phone only or CPO individuals) to increase the representativeness of the sample^{Footnote 1}. This sample was drawn from Probit, our online-hybrid panel of Canadians. Probit is assembled using a random digit dial (RDD) process for sampling from a blended land-line cell-phone frame, which provides full coverage of Canadians with telephone access. The cell phone only sample drawn for the survey is therefore a probability sample that is representative of Canadians with cell phone but not landline access. The combined overall survey sample allows for results that can be extrapolated to the population of Canadians with telephone access.

Expenditure

The survey entailed the expenditure of $48,062.99 (including HST).

Political Neutrality Certification

This certification is to be submitted with the final report submitted to the Project Authority. I hereby certify as Senior Officer of EKOS Research Associates Inc. that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leaders.

Signed by: Susan Galley (Vice President)

Methodology

The survey instrument consisted of the ACET questions used by the Government of Canada in all of its advertising evaluation research. The length of the interview was 6.5 minutes. Respondents to the telephone survey were 18 years of age and older, and were randomly selected. The sample included all provinces and territories, and the survey was administered in English and French. The survey sample relied on a Random Digit Dial (RDD) method of sampling households from a wider sample frame of listed and unlisted landlines in Canada and is therefore considered to be a probability sample that is representative of the wider universe (exclusive of households with only cell phones). The number of total completed in the landline sample is 1,740, with an associated margin of error of up to plus or minus 2.3 per cent, at a.95% confidence interval (i.e., 19 times out of 20).

In addition, a subset of 13 per cent of the sample, were completed with Canadians who own only a mobile telephone and not a landline (i.e. cell phone only or CPO individuals) to increase the representativeness of the sample^{Footnote 2}. These CPO cases (n=266 of the 2,007) were confirmed as cell phone only Canadians at the start of the survey. The CPO sample was drawn from Probit, our online-hybrid panel of Canadians. Probit is assembled using a random digit dial (RDD) process for sampling from a blended land-line cell-phone frame, which provides full coverage of Canadians with telephone access. The distribution of the recruitment process is meant to mirror the actual population in Canada (as defined by Statistics Canada). As such, our more than 90,000 member panel can be considered representative of the general public in Canada (meaning that the incidence of a given target population within our panel very closely resembles the public at large) and margins of error can be applied. All households/individuals in the Probit panel are contacted by telephone and the nature of the panel is explained in greater detail (as are our privacy policies) and demographic information is collected. At this time the online/off-line as well as landline/cell phone status is ascertained in order to determine the method of completing surveys (i.e., online, telephone, or mail). This variable of ‘type of telephone service’ (cell phone only, landline only or both) collected at the time of screening is used to determine CPO sample. As with any RDD sample, Probit panel cases are considered to be a probability-based sample. The total CPO sample carries with it a margin of error of plus or minus 6.0 per cent, at a 95% confidence interval (i.e., 19 times out of 20).

Prior to beginning the full fieldwork, the study objectives and sampling of the survey, as well as the meaning and intent of specific items in the interview were thoroughly covered in training sessions with the full complement of interviewers who worked on the study. Test or practice interviews were conducted to familiarize interviewers with the questions, categories, flow and skip logic.

Survey supervisors continuously monitored interviewing during the data collection process, using a dual audio and visual monitoring system. A portion of interviews from each survey interviewer were monitored to ensure consistency of questionnaire administration and interviewing techniques.

The landline sample for this study was derived from Survey Sample Software. The last birthday method was used to select the respondent from within the sampled household. If this person was under 18, we asked to speak to another member of the household who was 18 years or older.

Eight call-backs (nine total calls) were made to each selected household in the original sample before retiring a case and substituting another household. Follow-up calls were made on subsequent days, at varying time periods to maximize the potential for reaching a given respondent and appointments were taken at the convenience of the respondent. All individuals were given the choice of conducting the interview in either official language. The survey was conducted over eight days starting after the campaign ended (February 24 to March 3, 2014). Call-backs were rotated over weekday evenings and weekends during that time.

The survey was registered with the National Survey Registration System. Survey data collection adhered to Government of Canada standard for public opinion research as well as all applicable industry standards as set out by the Market Research Intelligence Agency (MRIA), of which EKOS is a Gold Seal member. EKOS informed respondents of their rights under the Privacy Act and the Access to Information Act, and ensured that those rights were protected throughout the research process. This included: informing respondents of the purpose of the research; identifying both the sponsoring department and the research supplier; informing respondents that their participation in the study is voluntary, and that the information provided would be administered according to the requirements of the Privacy Act.

Once the survey data was collected, the database was reviewed for data quality. Coding was also completed (starting after the first 300 cases and completed two days after the completion of the survey collection period).

Survey results were weighted based on Statistics Canada data according to age, gender and region to ensure the sample was representative of the general public aged 18 years and older.

Data tables were created to isolate results for major subgroups to be used in the analysis (e.g., results for the target segment of parents with children, age segments, gender, education segments income segments, regions).

Response Rates^{Footnote 3}

The response rate for the telephone survey is calculated by dividing the cooperative call-backs (i.e., those who completed the study, those who we spoke to but were ineligible, and the quota filled) by the functional sample. The functional sample is the sample remaining after numbers not in service, business and fax numbers, duplicate numbers and numbers blocked by the phone company are removed.

Discussion of Non-Response Bias

A comparison of the unweighted sample with 2011 Census figures from Statistics Canada suggests that there are some sources of systematic sample bias in the survey, following patterns typically found in most general public surveys. These include an under representation of youth under 25 (six per cent versus 12 per cent in the population, and an over representation of older Canadians (41 per cent over 55 compared with 34 per cent in the population). There is a more educated sample in the survey than found in the population with 39 per cent reporting university degrees compared with 23 per cent in the population. The sample under represents Canadians with a college level of education (27 per cent compared with 36 per cent in the population as per 2011 Census figures). Households with children under the age of 18 are also under represented in the sample (28 per cent compared with 38 per cent in the population). There is also an under representation of Canadians born outside of Canada (17 per cent versus 25 per cent). As previously described, the sample was weighted by age, gender and region.

Appendix A Questionnaire

CPINTRO
CELL PHONE ONLY INTRO

Contact info:
Name:
Gender:
Age Group:
Region:

Hello, my name is _______________. May I speak with &&ANAME? I'm calling on behalf of EKOS Research Associates. I'd like to ask you a few questions on current issues of interest to Canadians. Your responses will be kept entirely confidential and anonymous. This survey is registered with the Marketing Research and Intelligence Association's (MRIA). Would you be willing to take part in this survey? We can do it now or at a time more convenient for you.

[IF ASKED] The survey will take about 6 minutes to complete.

NOTE: Reveal name of client department which commissioned survey at the end of the survey only.

• Yes now continue 1
• Yes call later / specify time and date 2
• Refused thank and terminate 9 ->THNK2

CPC
CELL PHONE ONLY

Have I reached you on a cellular phone or a traditional telephone line?

• Cellular phone Continue 1
• Traditional phone Thank and Terminate 2 ->THNK2
• Refused Thank and Terminate 9 ->THNK2

CPA
CELL PHONE ONLY

Are you in a safe place to answer a survey?

• Yes Continue 1
• No SCHEDULE CALLBACK DATE AND TIME 2 ->CPINTRO
• Refused Thank and Terminate 9 ->THNK2

CPB
CELL PHONE ONLY

Are you 18 years of age or older?

• Yes Continue 1
• No Thank and Terminate 2 ->THNK2
• Refused Thank and Terminate 9 ->THNK2

CPD
CELL PHONE ONLY

At home, do you have a traditional telephone line other than a cell phone?

• Yes Thank and Terminate 1 ->THNK2
• No Continue 2
• Refused Thank and Terminate 9 ->THNK2

CPE
CELL PHONE ONLY

In which province or territory do you live?

• Newfoundland 1
• Prince Edward Island 2
• Nova Scotia 3
• New Brunswick 4
• Quebec 5
• Ontario 6
• Manitoba 7
• Saskatchewan 8
• Alberta 9
• British Columbia 10
• Yukon 11
• Northwest Territories 12
• Nunavut 13

BDAYQ
RDD ONLY

Hello, my name is _______________. I'm calling on behalf of EKOS Research Associates. I'd like to ask you a few questions on current issues of interest to Canadians. Your responses will be kept entirely confidential and anonymous. This survey is registered with the Marketing Research and Intelligence Association's (MRIA).

May I please speak with the person in your household who is 18 years of age or older and who has had the most recent birthday? Would that be you?

IF THAT PERSON IS NOT AVAILABLE, ARRANGE CALLBACK

• Yes, continue 1
• No, ask to speak to 'eligible' person and start again 2
• REFUSED thank and terminate 9 ->THNK2

BDAYQ2
RDD ONLY

Would you be willing to take part in this survey? We can do it now or at a time more convenient for you.

[IF ASKED] The survey will take about 6 minutes to complete.

NOTE: Reveal name of client department which commissioned survey at the end of the survey only.

• Yes now continue 1
• Yes call later / specify time and date 2
• REFUSED thank and terminate 9 ->THNK2

SEX

Record gender of respondent

DO NOT ASK

• Male 1
• Female 2

PRIV

This call may be recorded for quality control or training purposes.

Q1A

Over the past few weeks or so, have you seen, heard or read any advertising about programs and services available to help Canadians improve their employment situation?

• Yes 1
• No 2
• DON'T KNOW 8
• REFUSED 9

Q1B
YES

What can you remember about this advertising? What words, sounds or images come to mind?

[RECORD VERBATIMS - ACCEPT MULTIPLE RESPONSES]

• INSERT RESPONSE 77
• DON'T KNOW/REFUSED 99 X

Q2A
<More precisely...>

Do you remember seeing a television ad over the past few weeks or so showing a woman talking about how she wanted to become a mechanic and decided to go online and apply for an apprenticeship grant. The ad ends with the woman saying that she now loves her life, her job and that she is really proud of herself.

• Yes 1
• No 2
• DON'T KNOW 8
• REFUSED 9
• VOLUNTEERED: Not applicable/do not watch TV 7

Q2B
USES INTERNET

And do you remember seeing this same ad as a video on the Internet over the past few weeks or so?

• Yes 1
• No 2
• DON'T KNOW 8
• REFUSED 9
• VOLUNTEERED: Not applicable/do not use the Internet 7

Q2C
USES INTERNET
<More precisely...>

Do you remember seeing an ad on the Internet over the past few weeks or so about programs and services available to help Canadians improve their employment situation?

• Yes 1
• No 2
• DON'T KNOW 8
• REFUSED 9
• VOLUNTEERED: Not applicable/do not use the Internet 7

Q3
RECALLED AD(S)

Thinking about the advertising that you saw, what do you think was the main point this advertising was trying to get across?

[RECORD VERBATIMS - ACCEPT ONE RESPONSE]

• INSERT RESPONSE 77
• DON'T KNOW / REFUSED 99

Q5A
RECALLED AD(S)

Who do you think produced those ads? In other words, who funded them?

[DO NOT READ LIST. ACCEPT ONE RESPONSE]

• Government of Canada/Federal Government 1
• Provincial Government 2
• Municipal Government 3
• Government (General) 4
• Taxpayers 5
• Other (specify) 77
• DON'T KNOW 88REFUSED 99

Q5B
THOSE WHO RECALL THE AD IN THE Q2 SERIES AND SAID GOV GENERAL (Q5A)

Which level of government?

READ LIST. ACCEPT ONE RESPONSE

• Federal government 1
• Provincial government 2
• Municipal government 3
• Other (specify) 77
• DON'T KNOW 88
• REFUSED 99

DEMIN

Finally, I'd like to ask you some questions for statistical purposes. I'd like to remind you that all your answers are completely confidential.

D1

Which of the following categories best describes your current employment status? Are you...?

[READ LIST, ACCEPT ONE RESPONSE]

• Working full-time (35 or more hours per week) 1
• Working part-time (less than 35 hours per week) 2
• Self-employed 3
• Student attending full time school (not working) 4
• Unemployed, but looking for work 5
• Not in the workforce (e.g. unemployed, but not looking for work, a full-time homemaker or parent) 6
• Retired 7
• Other 77
• REFUSED 99

D2

What is the highest level of formal education that you have completed to date?

[READ LIST. ACCEPT ONE RESPONSE]

• Elementary school or less 1
• Secondary school 2
• Some post-secondary 3
• College, vocational or trade school 4
• Undergraduate university program 5
• Graduate or professional university program 6
• REFUSED 9

D3

In what year were you born?

ANSWER THE FULL YEAR, I.E. 1977 as "1977"

• RECORD YEAR 1
• REFUSED 9

D4

Are there any children under the age of 18 currently living in your household?

• Yes 1
• No 2
• REFUSED 9

D5

Which of the following categories best describes your total household income?

That is, the total income of all persons in your household, before taxes?

READ LIST

• Under $20,000 1
• $20,000 to just under $40,000 2
• $40,000 to just under $60,000 3
• $60,000 to just under $80,000 4
• $80,000 to just under $100,000 5
• $100,000 to just under $120,000 6
• $120,000 to just under $150,000 7
• $150,000 and above 8
• REFUSED 99

D6

Were you born in Canada?

• Yes 1
• No 2
• REFUSED 9

D7A

In the last two weeks, have you...

Read a daily newspaper?

• Yes 1
• No 2
• DON'T KNOW 8
• REFUSED 9

D7B

In the last two weeks, have you...

Read a weekly newspaper?

• Yes 1
• No 2
• DON'T KNOW 8REFUSED 9

D7C

In the last two weeks, have you...

Watched television?

• Yes 1
• No 2
• DON'T KNOW 8
• REFUSED 9

D7D

In the last two weeks, have you...

Listened to the radio?

• Yes 1
• No 2
• DON'T KNOW 8
• REFUSED 9

D7E

In the last two weeks, have you...

Used public transit?

• Yes 1
• No 2
• DON'T KNOW 8
• REFUSED 9

D7F

In the last two weeks, have you...

Read a magazine?

• Yes 1
• No 2
• DON'T KNOW 8REFUSED 9

D7G
USES INTERNET

In the last two weeks, have you...

Used the Internet?

• Yes 1
• No 2
• DON'T KNOW 8
• REFUSED 9

D7J

In the last two weeks, have you...

Watched a movie at a movie theatre?

• Yes 1
• No 2
• DON'T KNOW 8
• REFUSED 9

E7H
USES INTERNET

In the last two weeks, have you...

Used Facebook?

• Yes 1
• No 2
• DON'T KNOW 8
• REFUSED 9

E7I
USES INTERNET

In the last two weeks, have you...

Watched videos on YouTube?

• Yes 1
• No 2
• DON'T KNOW 8
• REFUSED 9

QCLIENT

This survey was conducted on behalf of Employment and Skills Development Canada. In the coming months the report will be available from Library and Archives Canada.

CTHNK
CELL ONLY

In order to compensate you for your time, we require your full name and mailing address

INAME
CELL ONLY

May we have your name?

*INTERVIEWER: Confirm spelling of name.

• (Record name) 1
• No response 9

IDRESS
CELL ONLY

May we have your address?

*INTERVIEWER: Confirm spelling of address.

• (Record street address) 1
• No response 9

IDRESS2
CELL ONLY

• (City) 1
• No response 9

IDRESS3
CELL ONLY

• (POSTAL CODE) 1
• No response 9

THNK

That concludes the survey. We thank you very much for taking the time to participate, it is greatly appreciated.

End of Interview

• Completion 1 D

THNK2

We regret but your responses have shown that you are ineligible to participate in this survey. Thank you for your time.