Prepared for the Canada Revenue Agency (CRA)
Supplier name: Ipsos Public Affairs
Contract Number: 46558-226728/001/CY
Contract value: $49,699.79
Award Date: February 2, 2021
Delivery Date: June 10, 2021
Registration number: POR 111-20
For more information, please contact: media.relations@cra-arc.gc.ca
Ce rapport est aussi disponible en français.
Prepared for Canada Revenue Agency (CRA)
Supplier name: Ipsos Public Affairs
June 2021
This public opinion research report presents the methodology of the CRA Benefits and Credits Campaign – Advertising Campaign Evaluation Tool (ACET) online survey conducted by Ipsos Public Affairs on behalf of the Canada Revenue Agency (CRA). The research study was conducted with a total sample of 4033 Canadians aged 18+: n=2000 in the pre-campaign wave between February 8th and 14th, 2021, and n=2033 in the post-campaign wave between May 7th and May 14th, 2021.
Cette publication est aussi disponible en français sous le titre : Évaluation de la campagne publicitaire des prestations et crédits de l'ARC de 2020/2021.
This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from the Canada Revenue Agency (CRA). For more information on this report, please contact the CRA at media.relations@cra-arc.gc.ca.
101 Colonel By Drive
Ottawa, Ontario K1A 0K2
Canada
Catalogue Number: Rv4-137/1-2021E-PDF
International Standard Book Number (ISBN): 978-0-660-39639-2
Related publications (registration number: POR 111-20):
Catalogue Number (Final Methodological Report, French): Rv4-137/1-2021F-PDF
ISBN (French): 978-0-660-39641-5
© Her Majesty the Queen in Right of Canada, as represented by the Canada Revenue Agency, 2021
This public opinion research report presents the methodology of the CRA Benefits and Credits Campaign – Advertising Campaign Evaluation Tool (ACET) online survey conducted by Ipsos Public Affairs on behalf of the Canada Revenue Agency (CRA).
The Canada Revenue Agency (CRA) administers taxes, benefits, and related programs for Canadians. There are a number of tax benefits, credits and programs that provide additional income or tax relief to individuals which can help make life more affordable. However, vulnerable populations such as low-income households, Indigenous peoples, newcomers to Canada, and youth face a number of barriers when doing their taxes and receiving their benefits and credits. These include a lack of awareness of programs, living in remote locations, language barriers, low financial literacy and/or mental or physical health issues.
The CRA is undertaking an advertising campaign to help increase awareness amongst vulnerable populations and working Canadians to ensure that they are aware of the benefits and credits that they are entitled to when they file their taxes.. The benefits and credits advertising campaign will feature the Canada child benefit (CCB), the Canada workers benefit (CWB), the GST/HST credit, and the Canada Training Credit (CTC). This campaign will also highlight the programs that support home buyers, such as the Home Buyers' Plan (HBP), the Home buyers' amount, the GST/HST new housing rebate and the First-Time Home Buyer Incentive. The campaign will also increase awareness of the availability of free tax clinics to help individuals with modest incomes and simple tax situations do their taxes.
The main objectives of this campaign are to:
The campaign launch date was Monday February 15, 2021. The campaign ran until April 30, 2021. The research consisted of online surveys conducted in English and French, with the pre-campaign evaluation taking place between February 8th and 14th, 2021 and the post-campaign evaluation taking place between May 7th to 14th, 2021.
Ipsos was contracted to conduct the data collection and tabulation, and to provide a methodology report. Ipsos programmed, hosted and provided sample management services, while CRA provided the online questionnaires. Ipsos was responsible for data collection and data storage in Canada, data processing and data weighting. The data collection from Canadians was handled in accordance with government-wide Public Opinion Research procedures.
The total contract value of this research was $49,699.79, including HST.
The project includes data collection for pre- and post-campaign evaluations that will assist the CRA in determining the effectiveness of the advertising campaign by measuring awareness of the subject matter with the audiences. Advertising campaign evaluation is mandatory for all advertising campaigns in which media buys exceed $1 million. The pre- and post-campaign ACETs are used for all major Government of Canada (GC) advertising campaigns in order to consistently evaluate these campaigns and comply with key requirements in the GC Communications Policy.
This research has two components:
In both the pre- and post-campaign survey, the identity of the CRA as the sponsor was not revealed at the outset of the questionnaire but was detailed at the conclusion.
The research study was conducted with a total sample of 4033 Canadians aged 18+: n=2000 in the pre-campaign wave between February 8th and 14th, 2021, and n=2033 in the post-campaign wave between May 7th and May 14th, 2021.
The survey questionnaire was administered online via the usage of an online panel, therefore the survey results are not projectable to the overall Canadian populations and a margin of error cannot be provided.
The findings from the research are being used to assist in evaluating the effectiveness of the campaign.
The contract amount for this project was $49,699.79 (including HST).
I hereby certify as Senior Officer of Ipsos that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate or ratings of the performance of a political party or its leaders.
Mike Colledge
President
Ipsos Public Affairs
The research consisted of a quantitative methodology using an online non-probability panel and Computer Assisted Web Interviewing (CAWI).
The total sample of respondents reflects the Canadian adult (18+) population's demographic characteristics by age, gender and region, with sufficient quotas of:
As required in all ACET studies, the surveys for the pre and post campaign evaluation were in the form of two separate GC templated questionnaires, one for each of the pre and post-campaign evaluations. The questionnaires were not altered in any way, including the wording, order and coding of questions. No questions were deleted. A few questions were added at the end of the survey, after the core questions had been asked.
Since the survey questionnaire was administered online via the usage of an online non-probability panel, the survey results are not projectable to the overall Canadian populations and a margin of error cannot be provided.
The total sample size for the pre-campaign survey was n=2000 and post-campaign survey was n=2033. The data was weighted by age, gender, region. Within each of these general population samples, the CRA set quotas for priority groups as noted above. The following table outlines the completes obtained by target sub-population.
| Priority subgroups | Pre-campaign wave # of completes (n=) |
Post-campaign wave # of completes (n=) |
|---|---|---|
| Indigenous peoples | 134 | 104 |
| Low-income individuals (household income under 40K) | 574 | 589 |
| Parents of children <18 years old | 628 | 510 |
| Secondary groups | Pre-campaign wave # of completes (n=) |
Post-campaign wave # of completes (n=) |
| Newcomers <5 years* | 50 | 52 |
| Parents of children <6 | 336 | 252 |
| Youth 18-24 | 170 | 208 |
| Purchasing first home in the next 3 years | 430 | 421 |
| Interested in changing careers /get training or courses | 858 | 800 |
Interviews were conducted in English and French, with respondents being able to complete the survey in either official language of their choice.
Respondents were invited to participate in the survey via email, with a unique URL link. This link could only be used once, with respondents being allowed to pause during completion and return to complete it at a later time.
For the pre- and post-campaign surveys, pre-tests were conducted to confirm survey length before fully deploying the questionnaire and to determine if respondents understood the questions. For the post-campaign survey, a question was also added to ensure there were no technical issues when listening to the audio in the TV ad. No issues were flagged during the pre-tests that required revision to the original questionnaire.
For this study, Ipsos partnered with a certified partner sample provider, Dynata, to obtain the required target samples. Dynata's panels are continually refreshed and recruited through various channels including 1) loyalty program sourcing across travel, entertainment and other sectors; 2) online banners, cable TV advertising, mailings, social media influencers, mobile app, etc.; 3) integrated channels including access to online communities, social media platforms, publishers and others (this last group has not opted into a panel) but each participant has a Dynata profile.
Dynata's panel includes members who have opted in and receive a form of incentive. Incentives are based on a point-based system cashed in for rewards (electronic gift certificates, high street vouchers, charity donations, and long-term loyalty rewards), as well as sweepstakes. The amount of incentives is based on the specific requirements of each survey, depending on the length and complexity of the survey, the subject matter of the study, and the time required to complete a minimum number of interviews.
The comprehensive background profiling data gathered when respondents join a panel allows for the targeting of respondents based on key criteria, such as region, age, gender, education and income level, intention to buy a home within 12 months, profession, and other characteristics.
We ensured panel members had not participated in any Government of Canada survey in the previous 30 days as a member of that panel or in a survey on similar subject matter by any mode.
Dynata uses a router as its sampling methodology. This routing technology uses weighted randomization to assign surveys to participants. Upon entry into the system, panelists are checked to ensure they have not exceeded survey participation limits. A list of potential survey matches is determined for each panelist based upon the information we know about them. Panelists may be asked additional screening questions within the system to ensure they meet the project criteria. Priority may be given to surveys that are behind schedule; however, this is kept to a minimum as survey randomization must remain in place as a key element for preventing bias.
From the outset, Dynata's routing technology was designed to ensure high–quality sampling and minimize bias. In the development of our technology, randomization was essential to minimize the risk of assignment bias. This routing technology has been tested using multiple data points to ensure that bias is limited and has been reviewed by external consultants to validate the routing technology design. In addition, reports of system statistics and conversion rates are reviewed systematically to ensure conistency.
The table below shows the unweighted and weighted distributions of the online samples for the pre- and post-campaign waves. Weighting was applied to the sample to ensure that the final data reflects the general adult population by region, age, and gender according to the 2016 Census. A Random Iterative Method (RIM) technique was applied for weighting.
| Demographics | Pre-campaign Unweighted Sample Size |
Pre-campaign Weighted Sample Size |
Post-campaign Unweighted Sample Size |
Post-campaign Weighted Sample Size |
|---|---|---|---|---|
| Canada | 2000 | 2000 | 2033 | 2033 |
| Region - BC | 227 | 271 | 259 | 271 |
| Region - Alberta | 225 | 224 | 244 | 228 |
| Region - Prairies (MB/SK) | 132 | 130 | 143 | 132 |
| Region - Ontario | 759 | 768 | 851 | 781 |
| Region - Quebec | 519 | 469 | 394 | 477 |
| Region - Atlantic | 138 | 137 | 138 | 139 |
| Gender* - Male | 921 | 977 | 954 | 990 |
| Gender* -Female | 1073 | 1017 | 1067 | 1031 |
| Gender* -Diverse | 6 | 6 | 12 | 12 |
| Age - 18-34 | 661 | 560 | 493 | 569 |
| Age - 35-54 | 852 | 680 | 692 | 691 |
| Age - 55+ | 487 | 760 | 848 | 773 |
* Denotes variables included in the weighting scheme.
The figures presented in the table above show minimal differences between the unweighted and weighted samples. The RIM weighting efficiency for the pre-campaign wave is 90.0 % and for the post-campaign wave is 97.5 %.
For this survey, a sample router was used. Therefore, a response rate cannot be calculated. However, the participation rate for the survey was 98% for the pre-campaign wave and 93% for the post-campaign wave. The average survey length was 4 minutes for the pre-campaign and 7 minutes for the post-campaign.
Participation rate is calculated as follows: (qualified completes + over quota + terminates)/click-through).
| Completions | Pre-campaign | Post-campaign |
|---|---|---|
| Click-Through | 3273 | 3202 |
| - Partial Completes | 81 | 239 |
| - Terminates | 303 | 442 |
| - Over quota | 889 | 488 |
| Qualified Completes | 2000 | 2033 |
| Participation Rate | 98% | 93% |
The results of this survey are not statistically projectable to the target population because the sampling method used does not ensure that the sample represents the target population with a known margin of sampling error. Reported percentages are not generalizable to any group other than the sample studied, and therefore no formal statistical inferences can be drawn between the sample results and the broader target population it may be intended to reflect. The sample data have been weighted to reflect the demographic composition of Canadian adults aged 18 years and over.
The table below compares the unweighted survey samples to the 2016 Census results by region, age, and gender. Overall, the sample is highly representative of the national adult population, with the exception of a few gaps, which are described below.
| Demographics | Unweighted Percentage Pre-campaign |
Unweighted Percentage Post-campaign |
Census 2016 Proportions (adults) |
|---|---|---|---|
| Region - BC | 11% | 13% | 13.6% |
| Region - Alberta | 11% | 12% | 11.2% |
| Region - Prairies (MB/SK) | 7% | 7% | 6.5% |
| Region - Ontario | 38% | 42% | 38.4% |
| Region - Quebec | 26% | 19% | 23.5% |
| Region - Atlantic | 7% | 7% | 6.8% |
| Gender* - Male | 46% | 47% | 48.5% |
| Gender* -Female | 54% | 52% | 51.4% |
| Gender* -Diverse | <1% | 1% | <1% |
| Age - 18-34 | 33% | 24% | 27.3% |
| Age - 35-54 | 43% | 34% | 34.1% |
| Age - 55+ | 24% | 42% | 38.6% |
* Denotes variables included in the weighting scheme.
Differences among the variables used in the weighting scheme are minimal; however, there are a few notable differences. Ontario is over-represented in the post-campaign unweighted sample; Quebec is under-represented in the post-campaign unweighted sample. Additionally, age skews are present in the unweighted sample: 18-34-year olds and 35-54-year olds are slightly over-represented in the pre-campaign waves, and 55+ are over-represented in the post-campaign. The remaining distribution in the sample remains consistent, with only slight differences observed between the unweighted percentages and the 2016 Census data.
Thank you for taking the time to complete this survey dealing with current issues of interest to Canadians. Si vous préférez répondre au sondage en français, veuillez cliquer sur français [SWITCH TO FRENCH VERSION].
Your participation is voluntary and your responses will be kept entirely confidential. The survey takes about 5 minutes to complete.
Click here if you wish to verify the authenticity of this survey (please type in the following project code: 20210203-IP835). To view our privacy policy, click here.
If you require any technical assistance, please contact xxx
a) Does anyone in your household work for any of the following organizations? SELECT ALL THAT APPLY
IF "NONE OF THESE ORGANIZATIONS" CONTINUE, OTHERWISE THANK AND TERMINATE.
b) Are you…
c) In what year were you born?
YYYY
ADMISSIBLE RANGE 1900- 2003
IF > 2002, THANK AND TERMINATE
ASK D IF QUESTION C IS LEFT BLANK
d) In which of the following age categories do you belong? SELECT ONE ONLY
IF "LESS THAN 18 YEARS OLD" OR "BLANK", THANK AND TERMINATE
e): Where were you born?
ASK IF E=BORN OUTSIDE CANADA
f) In what year did you first move to Canada?
YYYY
ADMISSIBLE RANGE: 1900-2020
g) In which province or territory do you live? SELECT ONE ONLY
IF NO PROVINCE OR TERRITORY IS SELECTED, THANK AND TERMINATE
h) Are you an Indigenous person? (An Indigenous person is a North American Indian or a member of a First Nation, a Métis or an Inuk (Inuit). North American Indians or members of a First Nation include status, treaty or registered Indians, as well as non-status and non-registered Indians.)
i) Which of the following categories best describes your total annual household income, including income from all household members, before taxes are deducted? SELECT ONE ONLY
j) Are you the parent of a child under the age of 18?
[if yes in Qj, ask Qk; else skip to Q1]
k) Are you the parent of a child under the age of 6?
ASK ALL RESPONDENTS
Q1:
Over the past three weeks, have you seen, read or heard any advertising from the Government of Canada?
Q2:
Think about the most recent ad from the Government of Canada that comes to mind. Where have you seen, read or heard this ad? SELECT ALL THAT APPLY
Q3:
What do you remember about this ad?
Q4:
How did you know that it was an ad from the Government of Canada?
ASK ALL RESPONDENTS
T1A:
Over the past three weeks, have you seen, read or heard any Government of Canada advertising about benefits and tax credits that are not related to COVID-19 benefits?
T1B:
Where have you seen, read or heard this Government of Canada ad about benefits and tax credits that are not related to COVID-19 benefits? SELECT ALL THAT APPLY
T1C:
What do you remember about this ad?
ASK ALL RESPONDENTS
T1D:
Are you aware of Government of Canada benefits and tax credits (not related to COVID-19 benefits) that you may qualify for?
T2D
Have you previously heard of any of the following Government of Canada benefits, tax credits or programs?
[RANDOMIZE]
Canada child benefit?
Canada training credit?
Home Buyers' Plan?
GST/HST credit?
Canada workers benefit?
T3D:
Are you aware that there are tax clinics available for those who are eligible to get their tax returns done for free by volunteers?
[IF YES:] Have you ever had your taxes done at one of these free clinics?
D1:
Which of the following categories best describes your current employment status? Are you… SELECT ONE ONLY
D2:
Are you interested in changing careers or in taking training/courses to upgrade your skills?
D3:
What is the highest level of formal education that you have completed? SELECT ONE ONLY
D4:
Are there any children under the age of 18 currently living in your household?
D5:
What is the language you first learned at home as a child and still understand? SELECT UP TO TWO
D6:
Are you considering purchasing your first home in the next 3 years?
That concludes the survey. This survey was conducted on behalf of the Canada Revenue Agency . In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to answer this survey, it is greatly appreciated.
To be conducted after the ads have been run in the media.
TV ad – English
TV ad – French
DIGITAL OUT OF HOME (DOOH) ads
CCB - English
CCB - French
CTC - English
CTC - French
HBP - English
HPB – French
BANNER ads
GST - English
GST - French
CWB - English
CWB – French
CCB - English
CCB – French
QUEBEC RESIDENTS
FR SPEAKERS OUTSIDE OF QC: SHOW FR TV AD, FR DOOH, FR BANNER ADS
Thank you for taking the time to complete this survey dealing with current issues of interest to Canadians. Si vous préférez répondre au sondage en français, veuillez cliquer sur français [SWITCH TO FRENCH VERSION].
Your participation is voluntary and your responses will be kept entirely confidential. The survey takes about 5 minutes to complete.
Click here if you wish to verify the authenticity of this survey. To view our privacy policy, click here.
If you require any technical assistance, please contact XXX.
a) Does anyone in your household work for any of the following organizations? SELECT ALL THAT APPLY
IF "NONE OF THESE ORGANIZATIONS" CONTINUE, OTHERWISE THANK AND TERMINATE.
b) Are you…
c) In what year were you born?
YYYY
ADMISSIBLE RANGE 1900-2003
IF > 2003, THANK AND TERMINATE
ASK D IF QUESTION C IS LEFT BLANK
d) In which of the following age categories do you belong? SELECT ONE ONLY
IF "LESS THAN 18 YEARS OLD" OR "BLANK", THANK AND TERMINATE
e): Where were you born?
ASK IF E=BORN OUTSIDE CANADA
f) In what year did you first move to Canada?
YYYY
ADMISSIBLE RANGE: 1900-2021
g) In which province or territory do you live? SELECT ONE ONLY
IF NO PROVINCE OR TERRITORY IS SELECTED, THANK AND TERMINATE
h) Are you an Indigenous person? (An Indigenous person is a North American Indian or a member of a First Nation, a Métis or an Inuk (Inuit). North American Indians or members of a First Nation include status, treaty or registered Indians, as well as non-status and non-registered Indians.)
i) Which of the following categories best describes your total annual household income, including income from all household members, before taxes are deducted? SELECT ONE ONLY
j) Are you the parent of a child under the age of 18?
[if yes in Qj, ask Qk; else skip to D6]
k) Are you the parent of a child under the age of 6?
D6:
Are you considering purchasing your first home in the next 3 years?
D2:
Are you interested in changing careers or in taking training/courses to upgrade your skills?
ASK ALL RESPONDENTS
Q1:
Over the past three weeks, have you seen, read or heard any advertising from the Government of Canada?
Q2:
Think about the most recent ad from the Government of Canada that comes to mind. Where have you seen, read or heard this ad? SELECT ALL THAT APPLY
Q3:
What do you remember about this ad?
Q4:
How did you know that it was an ad from the Government of Canada?
ASK ALL RESPONDENTS
T1A:
Over the past three weeks, have you seen, read or heard any Government of Canada advertising about benefits and tax credits that are not related to COVID-19 benefits?
T1B:
Where have you seen, read or heard this ad about benefits and tax credits that are not related to COVID-19 benefits? SELECT ALL THAT APPLY
T1C:
What do you remember about this ad?
ASK ALL RESPONDENTS
CAMPAIGN SPECIFIC ATTITUDINAL AND BEHAVIOURAL QUESTIONS MAY BE ADDED HERE
T1D:
Are you aware of Government of Canada benefits and tax credits (not related to COVID-19 benefits) that you may qualify for?
T2D Have you previously heard of any of the following Government of Canada benefits, tax credits or programs?
Canada child benefit?
Canada training credit?
Home Buyers' Plan?
GST/HST credit?
Canada workers benefit?
T3D:
Are you aware that there are tax clinics available for those who are eligible to get their tax returns done for free by volunteers?
[IF YES:] Have you ever had your taxes done at one of these free clinics?
T1H:
Here are some ads that have recently been broadcast on various media.
If you cannot hear the ad, please click on the ad to turn on the volume. You may also have to click on the lower right part of the ad to find the volume bar.
[FOR DIGITAL AD] Click here to watch.
[CLICK TO GO TO THE NEXT PAGE] Click here to go to next page
[PROGRAMMING INSTRUCTIONS:
For example:
TV AD - English
TV AD - French
DOOH AD
CCB - English
CCB – French
BANNER ADS
GST - English
GST - French
CWB - English
CWB – French
DOOH AD
CCB - English
CCB – French
CCB - English
CCB – French
DOOH ADS
HBP - English
HPB – French
DOOH ADS
CTC - English
CTC - French
T1H: Over the past three weeks, have you seen, read or heard these ads?
T1I:
Where have you seen, read or heard these ads? SELECT ALL THAT APPLY
T1J:
What do you think is the main point these ads are trying to get across?
T1K:
Please indicate your level of agreement with the following statements about these ads? RANDOMIZE STATEMENTS
| Statements | 1 Strongly Disagree |
2 | 3 | 4 | 5 Strongly Agree |
|---|---|---|---|---|---|
| These ads catch my attention | o | o | o | o | o |
| These ads are relevant to me | o | o | o | o | o |
| These ads are difficult to follow | o | o | o | o | o |
| These ads do not favour one political party over another | o | o | o | o | o |
| These ads talk about an important topic | o | o | o | o | o |
| These ads provide new information | o | o | o | o | o |
| These ads clearly convey that I may qualify for Government of Canada benefits, credits and programs. | o | o | o | o | o |
T1L: [ASK IF YES IN T1H]
Which of the following actions did you take as a result of seeing/hearing this advertising? [ACCEPT MULTIPLE RESPONSES]
D1:
Which of the following categories best describes your current employment status? Are you…
SELECT ONE ONLY
D3:
What is the highest level of formal education that you have completed? SELECT ONE ONLY
D4:
Are there any children under the age of 18 currently living in your household?
D5:
What is the language you first learned at home as a child and still understand? SELECT UP TO TWO
D7:
What is the language you first learned at home as a child and still understand? SELECT UP TO TWO
That concludes the survey. This survey was conducted on behalf of the Canada Revenue Agency. In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to answer this survey, it is greatly appreciated.