POR Number: POR 034-14
Contract Number: G9292-151764/001/CY
Award Date: 2014-11-06
End date of fieldwork: February 22nd, 2015
Contracted cost of the qualitative research (including HST): $53,753.77
Prepared for Employment and Social Development Canada
nc-por-rop-gd@hrsdc-rhdcc.gc.ca
March 2015
Ce rapport est aussi disponible en français.
Phoenix SPI is a ‘Gold Seal Certified’ Corporate Member of the MRIA
Data Tables Available Under Separate Cover
Employment and Social Development Canada (ESDC) commissioned Phoenix Strategic Perspectives (Phoenix) to conduct quantitative research related to the Canada Apprentice Loan advertising campaign.
The Canada Apprentice Loan (CAL) is an ESDC program designed to help apprentices registered in any Red Seal certification with the direct costs of skills training (e.g., tuition and materials) by providing interest-free federal loans of up to $4,000 per period of technical training.
The goal of the CAL is to encourage greater employment in the Red Seal trades to align with job opportunities in sectors facing skilled trades mismatches and shortages, and to make it easier for Canadian apprentices to train for and find employment. Applications for the CAL have been accepted in provinces and territories as of January 1, 2015.
This campaign sought to increase awareness and take up of the CAL, as well as other Government of Canada apprenticeship programs and services, including the Apprenticeship Incentive Grant, Apprenticeship Completion Grant, Employment Insurance support for apprentices on training, and tax credits. The advertising campaign ran from December 22nd, 2014 to February 15th, 2015 with ads appearing on television, radio, newspaper, and web (video and banner ads).1 The total media-buy for the campaign was over $1 million.
According to the Government of Canada’s Communications Policy, federal government advertising valued at more than $1 million must be assessed post-campaign to determine its impact and value. The purpose of this research was to evaluate the “Canada Apprentice Loan” advertising campaign using the Government of Canada’s standardized Advertising Campaign Evaluation Tool, or ACET survey.
The specific objectives of this research were to measure:
The results will be used by the Government of Canada to assess the effectiveness of the Canada Apprentice Loan advertisements.
The total contract value of this research was $53,753.77, including HST.
A telephone survey was conducted with a nationally disproportionate sample of 2,210 Canadians, 18 years of age and older, which included an oversample of young adults, aged 18 to 25 years.
A dual-frame (landline and cell phone) sample was used to maximize coverage, with 55% of the sample being landlines and 45% cell phones. The frames overlapped, meaning that landline households were not screened out of the cell phone sample frame. The same random selection process was used for both the landline and cell phone sample frames to ensure the integrity of the probability sample.
The geographic distribution of the interviews was disproportionate in order to improve the accuracy of regional results. The regional stratification was as follows:
| Strata | Target Number of Interviews |
|---|---|
| Atlantic | 250 |
| Quebec | 500 |
| Ontario | 650 |
| Prairies/Northwest Territories/Nunavut | 510 |
| British Columbia/Yukon | 300 |
| Total | 2,210 |
Based on a sample of this size, the overall results are considered to be accurate to within ±2.2% (adjusted to take into account the stratified sample), 19 times out of 20 (95% confidence interval).
Probability sampling was undertaken using random digit dialling (RDD). Screening questions were used to ensure that all respondents were eligible (i.e., Canadian citizens at least 18 years of age at the time of the survey). To randomize participation within households from the landline sample, the ‘most recent birthday’ method was used to select a survey respondent. There was no unit selection undertaken with the cell phone sample.
The Government of Canada ACET questionnaire was used, and is appended to this report. This is the standard tool used to measure Government of Canada advertising campaigns. The survey was registered with the Marketing Research and Intelligence Association’s (MRIA) Research Verification System.
No pre-test was conducted because the ACET is a standard tool. In lieu of a pre-test, a total of ten interviews were digitally recorded (five in French and five in English) during the first day of fieldwork. These recordings were reviewed by Phoenix’s team and the Project Authority.
The data collection was carried out by Elemental Data Collection Inc., under subcontract to Phoenix, from February 11th to 22nd, 2015. Calling was conducted during the week and on weekends, abiding by the hours and procedures stipulated in the Government of Canada standards for public opinion research. With the landline sample, a minimum of eight call-backs were attempted to reach respondents before a record was retired. For the cell phone sample, no more than three call-backs were made. All survey respondents were informed that participation is voluntary and that information collected is protected under the authority of privacy legislation. Interviews were conducted in the respondent’s official language of choice and averaged seven minutes in length.
The overall response rate for this survey was 6% (10% for the landline sample and 5% for the cell sample). This is consistent with typical response rates for RDD surveys of the general public. The table below presents detailed information.
| Total Sample | Landline Sample | Cell Sample | |
|---|---|---|---|
| Total Numbers Attempted | 63,740 | 15,666 | 48,074 |
| Out-of-scope - Invalid | 16,336 | 3,582 | 12,754 |
| Unresolved (U) | 35,798 | 7,817 | 27,981 |
| No answer/Answering machine | 35,798 | 7,817 | 27,981 |
| In-scope - Non-responding (IS) | 8,660 | 3,048 | 5,612 |
| Language barrier | 372 | 146 | 226 |
| Incapable of completing (ill/deceased) | 80 | 58 | 22 |
| Callback (Respondent not available) | 985 | 466 | 519 |
| Refusal | 7,156 | 2,365 | 4,791 |
| Termination | 67 | 13 | 54 |
| In-scope - Responding units (R) | 2,946 | 1,219 | 1,727 |
| Completed Interview | 2,210 | 1,206 | 1,004 |
| NQ - Quota Full | 67 | 13 | 54 |
| NQ - Age | 669 | 0 | 669 |
| Response Rate | 6.21% | 10.09% | 4.89% |
Weighting adjustments were applied to the survey data to ensure that the data were representative of the 18+ population of Canada. Specifically, the data were weighted by region, gender and age according to the most recent Statistics Canada census of the population.
| Strata | Unweighted Sample Size | Weighted Sample Size |
|---|---|---|
| Atlantic | 245 | 157 |
| Quebec | 523 | 526 |
| Ontario | 679 | 845 |
| Prairies/Nunavut/Northwest Territories | 463 | 391 |
| British Columbia/Yukon | 300 | 292 |
| Total | 2,210 | 2,210 |
A non-response analysis was conducted to assess the potential for non-response bias. To undertake the analysis for this survey, the unweighted sample distribution by gender and age3 was compared to the actual population (based on 2011 Census figures from Statistics Canada).
Canadians were generally sampled in proportion to the population. However, consistent with most telephone surveys of the general public, age is a source of potential bias in the survey. As the table indicates, younger Canadians are under-represented and older Canadians are over-represented in the sample.
| Strata | % of unweighted sample | % of population |
|---|---|---|
| Male | 51% | 48% |
| Female | 49% | 52% |
| 18-34 | 23% | 27% |
| 35-54 | 33% | 38% |
| 55+ | 40% | 32% |
The survey data were weighted to address these variations, as well as the sample design for the survey, which was regionally disproportion. Weighting serves to reduce bias should it be present, but not to eliminate it completely.
To estimate the amount of bias introduced into the survey results, the unweighted and weighted results by age were compared for key survey variables. In this case, we selected unaided and aided awareness. Our analysis indicates that the age bias in the sample did not introduce any meaningful bias to the survey results.
I hereby certify as a Senior Officer of Phoenix Strategic Perspectives that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not contain any reference to electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leader.
Alethea Woods
President
Phoenix Strategic Perspectives Inc.
March 13th, 2015
Date
Hello, my name is [Insert: your name]. I’m calling on behalf of Phoenix SPI a public opinion research firm. I’d like to ask you a few questions on current issues of interest to Canadians. Before we begin, I would like to know if you prefer to do the survey in French or English.
Please note that your responses will be kept entirely confidential and anonymous and that this survey is registered with the Marketing Research and Intelligence Association (MRIA).
[IF ASKED] The survey will take about 7 minutes to complete.
CS1. Have I reached you on your cell phone?
CS2. Are you 18 years of age or older?
CS3. Are you in a place where you can safely talk on the phone and answer my questions?
CS4. When would it be more convenient for me to call back?
CS5. Would you be willing to take part in this survey?
C1. In addition to your cellular telephone, does your household currently have a traditional telephone or landline?
LS1. May I please speak with the person in your household who is 18 years of age or older and who has had the most recent birthday? Would that be you? [IF THAT PERSON IS NOT AVAILABLE ARRANGE CALLBACK]
LS2. Would you be willing to take part in this survey? We can do it now or at a time more convenient for you.
L1. In addition to your residential landline telephone, do you or someone else in your household also use one or more cell phone numbers?
Q1A:
Over the past few weeks or so, have you seen, heard or read any advertising about the skilled trades? By skilled trades, I mean bricklayers, plumbers, carpenters, or electricians or any other job or vocation that requires classroom and on-the-job training.
Q1B:
What can you remember about this advertising? What words, sounds or images come to mind? [ACCEPT MULTIPLE RESPONSES]
Q2A:
Do you recall seeing an ad on television over the past few weeks or so that showed electricians, welders and crane operators working on different sites, with the words "Building. Canada's. Future" appearing on the screen and talking about programs to help apprentices?
Q2B:
And do you remember seeing this same ad as a video on the Internet over the past few weeks or so?
[READ AND ROTATE Q2C&D&E]
Q2C. And do you remember seeing an Internet banner ad over the past few weeks or so that mentions $4,000 in interest-free loans to help Canadians complete their apprenticeships?
[ASK ALL RESPONDENTS EXCEPT THOSE FROM THE PROVINCE OF QUEBEC]
Q2D. And do you remember hearing an ad on the radio over the past few weeks that mentions the need for skilled tradesmen and women to keep Canada’s economy strong and talks about up to $4,000 in interest-free loans to help Canadians complete their apprenticeships?
[ASK ALL RESPONDENTS EXCEPT THOSE FROM THE PROVINCE OF QUEBEC]
Q2E. And do you remember seeing an ad in newspapers which introduces the idea of a $4,000 interest-free loan to help apprentices complete their training in Red Seal trades? The ad shows pictures of a crane operator, an electrician, and a steel-worker.
IF RESPONDENT RECALLS THE AD IN Q2A-E, CONTINUE; OTHERWISE SKIP TO D1.
Q3:
Thinking about the advertising that you saw, what do you think was the main point these ads were trying to get across? [OPEN-ENDED, ACCEPT ONE RESPONSE]
Q5A:
In your opinion, who produced these ads? In other words, who funded them? [DO NOT READ LIST, ACCEPT ONE RESPONSE]
[ASK Q5B IF Q5A = ‘Government (General)’ OTHERWISE GO TO D1]
Q5B:
Which level of government? [READ LIST, ACCEPT ONE RESPONSE]
Finally, I’d like to ask you some questions for statistical purposes. I'd like to remind you that all your answers are completely confidential.
D1:
Which of the following categories best describes your current employment status? Are you...? [READ LIST, ACCEPT ONE RESPONSE]
D2:
What is the highest level of formal education that you have completed to date? [READ LIST, ACCEPT ONE RESPONSE]
D3:
In what year were you born?
D4:
Are there any children under the age of 18 currently living in your household?
D5:
Which of the following categories best describes your total household income? That is, the total income of all persons in your household, before taxes? [READ LIST, ACCEPT ONE RESPONSE]
D6:
Were you born in Canada?
D7:
What is the language you first learned at home as a child and still understand? [DO NOT READ LIST, ACCEPT UP TO TWO RESPONSES]
D8 :
In the last two weeks, have you…
RANDOMIZE MEDIA CHANNELS
That concludes the survey. This survey was conducted on behalf of Employment and Social Development Canada. In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to participate, it is greatly appreciated.
INTERVIEWER RECORDS:
1 The television and web advertisements were run across the country; however, the radio and newspaper ads were not presented in the province of Quebec.
2 The response rates were calculated using MRIA’s Empirical formula: R=R/(U+IS+R).
3 Excluding the oversample of young adults.