Prepared for the Canada Revenue Agency (CRA)
Supplier name: EKOS Research Associates
Contract number: 46558-193758/001/CY
Contract Award Date: September 18, 2018
Delivery Date: May 28, 2019
Contract Value: $44,522 (HST included)
Registration Number: POR 050-18
For more information on this report, please contact the Canada Revenue Agency at:
cra-arc.media@cra-arc.gc.ca.
Ce rapport est aussi disponible en français
Assessment of the 2018-2019 CRA Benefits and Credits Advertising Campaign
Methodological Report
Prepared for the Canada Revenue Agency (CRA)
Supplier name: EKOS Research Associates
May 2019
This report summarizes the methodology of two Advertising Campaign Evaluation Tool (ACET) surveys.
Cette publication est aussi disponible en français sous le titre: Évaluation de la campagne publicitaire des prestations et crédits de l'ARC de 2018-2019.
This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from the CRA. For more information on this report, please contact the CRA at: cra-arc.media@cra-arc.gc.ca.
Catalogue Number: Rv4-137/2019E-PDF
International Standard Book Number (ISBN): 978-0-660-33167-6
Related Publication (Registration Number): POR 050-18
Catalogue Number Rv4-137/2019F-PDF (Methodological Report, French)
International Standard Book Number (ISBN): 978-0-660-33168-3
© Her Majesty the Queen in Right of Canada, as represented by the Minister of National Revenue, 2019
Table of Contents
Many Canadians depend on benefit payments and tax credits for a significant part of their household income. For lower income households, benefits can increase income by as much as 50 per cent.
In addition, vulnerable populations such as low-income households, Indigenous peoples, newcomers to Canada, and young adults face a number of barriers when it comes to doing their taxes and getting their benefits and credits. Some need more support to help complete their income taxes, while others lack awareness. These challenges make it more difficult to reach these populations to ensure they are aware of all of the benefits and credits that they are entitled to, but may not be receiving.
Considering these barriers, it is vital that the Canada Revenue Agency (CRA) connects with this audience and gives them the information and support that they need on the benefits and credits to which they are entitled when they file their taxes. This includes the Canada Child Benefit (CCB), the Working Income Tax Benefit (WITB)/Canada Workers Benefit (CWB), and the GST/HST credit. In addition, greater awareness is needed so these populations are aware that filing tax returns every year is necessary in order to be eligible for these benefits and credits.
Consequently, the CRA undertook an advertising campaign, including digital, television and out-of-home advertising, aimed at:
The CRA commissioned EKOS Research Associates to conduct research evaluating the effectiveness of this advertising campaign.
The evaluation involved two research components:
A pre-campaign evaluation using the standardized Advertising Campaign Evaluation Tool (ACET) pre-campaign online survey, to assess pre-campaign awareness of the subject matter, including:
A post-campaign evaluation using the standardized ACET post-campaign online survey testing post-campaign awareness of the subject matter, including:
The pre-campaign online survey was conducted between November 27-December 10, 2018 with 2,007 members of the general public (with quotas for low income individuals, youth, and Indigenous and First Nations people).
The post-campaign online survey was conducted between May 6-21, 2019 with 2,037 members of the general public (with quotas for low income individuals, youth, and Indigenous and First Nations people).
The findings from the research were used to assist in evaluating the effectiveness of the campaign.
The contract amount for this project was $44,522.00 (including HST).
Statement of Political Neutrality
I hereby certify as Senior Officer of EKOS Research Associates Inc. that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate or ratings of the performance of a political party or its leaders.
Signed Derek Jansen, Vice President EKOS Research
The target audience for the study was all Canadians over the age of 18. The methodology for this study involved the following two research components:
Online survey of 2,007 members of the general public (18 years and older), including quotas for low income individuals, youth, and Indigenous and First Nations people.
| Group | Sample Size | Quota |
|---|---|---|
| Low Income | 288 | 200 |
| Youth | 216 | 200 |
| Indigenous and First Nations | 112 | 100 |
| Other | 1,391 | 1,500 | TOTAL | 2,007 | 2,000 |
Online survey of 2,037 members of the general public (18 years and older), with quotas for low income individuals, youth, and Indigenous and First Nations people.
| Group | Sample Size | Quota |
|---|---|---|
| Low Income | 266 | 200 |
| Youth | 208 | 200 |
| Indigenous and First Nations | 127 | 100 |
| Other | 1,436 | 1,500 | TOTAL | 2,037 | 2,000 |
The sample for both research components was drawn from EKOS' in-house Probit panel. Probit is a research panel that has been designed by EKOS to provide statistically representative data. Our online panel offers complete coverage of the Canadian population (i.e., Internet, phone, cell phone), random recruitment (in other words, participants are recruited randomly, they do not opt themselves into our panel), and equal probability sampling. All respondents to our panel are recruited by telephone using random digit dialling and are confirmed by live interviewers.
Our panel is considered to be probability based as it is recruited from random digit dial (RDD) sample, both landline and cell phone. The overall panel size is roughly 90,000 Canadian households. Random stratified samples are drawn from the panel database for individual research assignments.
The ACET pre and post campaign survey instruments consisted of the Advertising Campaign Evaluation Tool questions used by the Government of Canada in all of its advertising evaluation research.
The average time it took respondents to complete the pre-campaign questionnaire was 6 minutes, and the average length of the post-campaign questionnaire was 10 minutes.
Pre-testing was conducted on November 26, 2018 (pre-campaign), and May 6, 2019 (post-campaign) to ensure each questionnaire was working well. The pre-tests included a series of questions allowing respondents to provide an assessment of the clarity, length, and flow of the questionnaire, and provide any final comments about the survey.
The pre-campaign survey was conducted between November 27-December 10, 2018. The post campaign survey was conducted between May 7-21, 2019. For each survey, respondents were 18 years of age and older, and were randomly selected. The sample included all provinces and territories, and the survey was administered in English and French.
Survey data collection adhered to Government of Canada standard for public opinion research. EKOS informed respondents of their rights under the Privacy Act and the Access to Information Act, and ensured that those rights were protected throughout the research process. This included informing respondents of the purpose of the research, identifying both the sponsoring department and the research supplier, informing respondents that their participation in the study is voluntary, and that the information provided would be administered according to the requirements of the Privacy Act.
Once the survey data was collected, each database was reviewed for data quality. Coding was also completed. Survey results were weighted based on Statistics Canada data according to age, gender, and region to ensure the sample was representative of the general public aged 18 years and older.
The response rate for the pre-campaign survey was 20.4%. This is calculated using the formula developed by the Marketing Research Intelligence Agency in conjunction with the Government of Canada. http://mria-arim.ca/about-mria/standards/response-rate-calculation-formula. A total of 12,195 email invitations were sent. A total of 2,007 valid cases were completed, and combined with another 479 respondents who were screened out of the survey due to their responses as the numerator in the calculation of the response rate.
For the post-campaign survey, the response rate was 14.7%, using the same formula. This includes 16,930 email invitations sent. The 2,037 valid cases completed were combined with another 445 respondents who were screened out of the survey due to their responses to form the numerator in the calculation.
INTRODUCTION
Thank you for taking the time to complete this survey dealing with current issues of interest to Canadians. Si vous préférez répondre au sondage en français, veuillez cliquer sur français [SWITCH TO FRENCH VERSION].
Your participation is voluntary and your responses will be kept entirely confidential and anonymous. The survey takes about 5 minutes to complete. This survey is being directed by EKOS Research, and is being administered according to the requirements of the Privacy Act. To view our privacy policy, click here. If you require any technical assistance, please contact online@ekos.com.
a) Does anyone in your household work for any of the following organizations?
SELECT ALL THAT APPLY
IF “NONE OF THESE ORGANIZATIONS” CONTINUE, OTHERWISE THANK AND TERMINATE.
b) Are you…
c) In what year were you born?
Year:
Prefer not to say
ADMISSIBLE RANGE 1900-2001. IF > 2001, THANK AND TERMINATE. ASK D IF QUESTION C IS LEFT BLANK.
d) In which of the following age categories do you belong? SELECT ONE ONLY.IF “LESS THAN 18 YEARS OLD” OR “BLANK”, THANK AND TERMINATE.
e) In which province or territory do you live? SELECT ONE ONLY. IF NO PROVINCE OR TERRITORY IS SELECTED, THANK AND TERMINATE.
CORE QUESTIONS
ASK ALL RESPONDENTS
Q1: Over the past three weeks, have you seen, read or heard any advertising from the Government of Canada?
Q2: Think about the most recent ad from the Government of Canada that comes to mind. Where have you seen, read or heard this ad? SELECT ALL THAT APPLY
Q3: What do you remember about this ad?
Q4: How did you know that it was an ad from the Government of Canada?
CAMPAIGN SPECIFIC QUESTIONS
ASK ALL RESPONDENTS
T1A: Over the past three weeks, have you seen, read or heard any Government of Canada advertising about Government of Canada benefits and tax credits?
T1B: Where have you seen, read or heard this Government of Canada ad about Government of Canada benefits and tax credits?
T1C: What do you remember about this ad?
ASK ALL RESPONDENTS
CAMPAIGN SPECIFIC ATTITUDINAL AND BEHAVIOURAL QUESTIONS MAY BE ADDED HERE
T1D. Are you aware of Government of Canada benefits and tax credits that are available if you qualify?
DEMOGRAPHIC QUESTIONS
These next questions will help us to analyze the survey results.
D1: Which of the following categories best describes your current employment status? Are you… SELECT ONE ONLY
D2: What is the highest level of formal education that you have completed? SELECT ONE ONLY
D3: Are there any children under the age of 18 currently living in your household?
D4: Which of the following categories best describes your total annual household income, including income from all household members, before taxes are deducted? SELECT ONE ONLY.
D5: Do you identify yourself as an Indigenous, or First Nations person?
D6: Where were you born?
ASK IF D6=BORN OUTSIDE CANADA. D7: In what year did you first move to Canada?
D8: What is the language you first learned at home as a child and still understand? SELECT UP TO TWO
That concludes the survey. This survey was conducted on behalf of Canada Revenue Agency (CRA). In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to answer this survey. Your help is greatly appreciated.
INTRODUCTION
Thank you for taking the time to complete this survey dealing with current issues of interest to Canadians. Si vous préférez répondre au sondage en français, veuillez cliquer sur français [SWITCH TO FRENCH VERSION].
The survey takes about 5 minutes to complete. Your participation is voluntary and your response will be kept entirely confidential and anonymous. This survey is being directed by EKOS Research, and is being administered according to the requirements of the Privacy Act. To view our privacy policy, click here. If you require any technical assistance, please contact online@ekos.com.
a) Does anyone in your household work for any of the following organizations?
IF “NONE OF THESE ORGANIZATIONS” CONTINUE, OTHERWISE THANK AND TERMINATE.
b) Are you…
c) In what year were you born?
Year:
Prefer not to say
ADMISSIBLE RANGE 1900-2001. IF > 2001, THANK AND TERMINATE. ASK D IF QUESTION C IS LEFT BLANK.
d) In which of the following age categories do you belong? SELECT ONE ONLY.IF “LESS THAN 18 YEARS OLD” OR “BLANK”, THANK AND TERMINATE.
e) In which province or territory do you live? SELECT ONE ONLY. IF NO PROVINCE OR TERRITORY IS SELECTED, THANK AND TERMINATE.
CORE QUESTIONS
ASK ALL RESPONDENTS
Q1: Over the past three weeks, have you seen, read or heard any advertising from the Government of Canada?
Q2: Think about the most recent ad from the Government of Canada that comes to mind. Where have you seen, read or heard this ad? SELECT ALL THAT APPLY
Q3: What do you remember about this ad?
Q4: How did you know that it was an ad from the Government of Canada?
CAMPAIGN SPECIFIC QUESTIONS
ASK ALL RESPONDENTS
T1A: Over the past three weeks, have you seen, read or heard any Government of Canada advertising about Government of Canada benefits and tax credits?
T1B: Where have you seen, read or heard this Government of Canada ad about Government of Canada benefits and tax credits?
T1C: What do you remember about this ad?
ASK ALL RESPONDENTS
CAMPAIGN SPECIFIC ATTITUDINAL AND BEHAVIOURAL QUESTIONS MAY BE ADDED HERE
T1D. Are you aware of Government of Canada benefits and tax credits that are available if you qualify?
T1E. Did you do anything as a result of seeing/hearing this advertising? [ONLY ASK IF YES TO T1A]
T1F. Which of the following actions did you take? [ACCEPT MULTIPLE RESPONSES] [ONLY ASK IF YES TO T1E]
Here are some ads that have recently been broadcast on various media.
[INSERT VIDEO, OOH AND RADIO ADS]
INSTRUCTIONS: SHOW TV AD, OOHAD, RADIO AD
T1H. Over the past two months, have you seen, read or heard any of these ads?
T1I. Where have you seen, read or heard any of these ads? SELECT ALL THAT APPLY
T1J. What do you think is the main point these ads are trying to get across?
| 1 Strongly Disagree | 2 | 3 | 4 | 5 Strongly Agree | |
| These ads catch my attention | o | o | o | o | o |
| These ads are relevant to me | o | o | o | o | o |
| These ads are difficult to follow | o | o | o | o | o |
| These ads do not favour one political party over another | o | o | o | o | o |
| These ads talk about an important topic | o | o | o | o | o |
| These ads provide new information | o | o | o | o | o |
| These ads clearly convey that you may qualify for Government of Canada benefits and credits. | o | o | o | o | o |
DEMOGRAPHIC QUESTIONS
These next questions will help us to analyze the survey results.
D1: Which of the following categories best describes your current employment status? Are you… SELECT ONE ONLY
D2: What is the highest level of formal education that you have completed? SELECT ONE ONLY
D3: Are there any children under the age of 18 currently living in your household?
D4: Which of the following categories best describes your total annual household income, including income from all household members, before taxes are deducted? SELECT ONE ONLY.
D5: Do you identify yourself as an Indigenous, or First Nations person?
D6: Where were you born?
ASK IF D6=BORN OUTSIDE CANADA. D7: In what year did you first move to Canada?
D8: What is the language you first learned at home as a child and still understand? SELECT UP TO TWO
That concludes the survey. This survey was conducted on behalf of Canada Revenue Agency (CRA). In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to answer this survey. Your help is greatly appreciated.