Prepared for the Canada Revenue Agency (CRA)
Supplier name: Ipsos Public Affairs
Contract Number: 46558-226728/001/CY
Contract value: $49,699.79
Award Date: February 2, 2021
Delivery Date: June 10, 2021
Registration number: POR 111-20
For more information, please contact: media.relations@cra-arc.gc.ca
Ce rapport est aussi disponible en français.
Prepared for Canada Revenue Agency (CRA)
Supplier name: Ipsos Public Affairs
June 2021
This public opinion research report presents the methodology of the CRA Benefits and Credits Campaign – Advertising Campaign Evaluation Tool (ACET) online survey conducted by Ipsos Public Affairs on behalf of the Canada Revenue Agency (CRA). The research study was conducted with a total sample of 4033 Canadians aged 18+: n=2000 in the pre-campaign wave between February 8th and 14th, 2021, and n=2033 in the post-campaign wave between May 7th and May 14th, 2021.
Cette publication est aussi disponible en français sous le titre : Évaluation de la campagne publicitaire des prestations et crédits de l'ARC de 2020/2021 – OECP Sommaire
This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from the Canada Revenue Agency (CRA). For more information on this report, please contact the CRA at media.relations@cra-arc.gc.ca.
101 Colonel By Drive
Ottawa, Ontario K1A 0K2
Canada
Catalogue Number: Rv4-137/2-2021E-PDF
International Standard Book Number (ISBN): 978-0-660-39640-8
Related publications (registration number: POR 111-20):
Catalogue Number (Executive Summary, French): Rv4-137/2-2021F-PDF
ISBN (French): 978-0-660-39642-2
© Her Majesty the Queen in Right of Canada, as represented by the Canada Revenue Agency, 2021
This public opinion research report presents the methodology of the CRA Benefits and Credits Campaign – Advertising Campaign Evaluation Tool (ACET) online survey conducted by Ipsos Public Affairs on behalf of the Canada Revenue Agency (CRA).
The Canada Revenue Agency (CRA) administers taxes, benefits, and related programs for Canadians. There are a number of tax benefits, credits and programs that provide additional income or tax relief to individuals which can help make life more affordable. However, vulnerable populations such as low-income households, Indigenous peoples, newcomers to Canada, and youth face a number of barriers when doing their taxes and receiving their benefits and credits. These include a lack of awareness of programs, living in remote locations, language barriers, low financial literacy and/or mental or physical health issues.
The CRA is undertaking an advertising campaign to help increase awareness amongst vulnerable populations and working Canadians to ensure that they are aware of the benefits and credits that they are entitled to when they file their taxes.. The benefits and credits advertising campaign will feature the Canada child benefit (CCB), the Canada workers benefit (CWB), the GST/HST credit, and the Canada Training Credit (CTC). This campaign will also highlight the programs that support home buyers, such as the Home Buyers' Plan (HBP), the Home buyers' amount, the GST/HST new housing rebate and the First-Time Home Buyer Incentive. The campaign will also increase awareness of the availability of free tax clinics to help individuals with modest incomes and simple tax situations do their taxes.
The main objectives of this campaign are to:
The campaign launch date was Monday February 15, 2021. The campaign ran until April 30, 2021. The research consisted of online surveys conducted in English and French, with the pre-campaign evaluation taking place between February 8th and 14th, 2021 and the post-campaign evaluation taking place between May 7th to 14th, 2021.
Ipsos was contracted to conduct the data collection and tabulation, and to provide a methodology report. Ipsos programmed, hosted and provided sample management services, while CRA provided the online questionnaires. Ipsos was responsible for data collection and data storage in Canada, data processing and data weighting. The data collection from Canadians was handled in accordance with government-wide Public Opinion Research procedures.
The total contract value of this research was $49,699.79, including HST.
The project includes data collection for pre- and post-campaign evaluations that will assist the CRA in determining the effectiveness of the advertising campaign by measuring awareness of the subject matter with the audiences. Advertising campaign evaluation is mandatory for all advertising campaigns in which media buys exceed $1 million. The pre- and post-campaign ACETs are used for all major Government of Canada (GC) advertising campaigns in order to consistently evaluate these campaigns and comply with key requirements in the GC Communications Policy.
This research has two components:
In both the pre- and post-campaign survey, the identity of the CRA as the sponsor was not revealed at the outset of the questionnaire but was detailed at the conclusion.
The research study was conducted with a total sample of 4033 Canadians aged 18+: n=2000 in the pre-campaign wave between February 8th and 14th, 2021, and n=2033 in the post-campaign wave between May 7th and May 14th, 2021.
The survey questionnaire was administered online via the usage of an online panel, therefore the survey results are not projectable to the overall Canadian populations and a margin of error cannot be provided.
The findings from the research are being used to assist in evaluating the effectiveness of the campaign.
The contract amount for this project was $49,699.79 (including HST).
I hereby certify as Senior Officer of Ipsos that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate or ratings of the performance of a political party or its leaders.
Mike Colledge
President
Ipsos Public Affairs
The research consisted of a quantitative methodology using an online non-probability panel and Computer Assisted Web Interviewing (CAWI).
The total sample of respondents reflects the Canadian adult (18+) population's demographic characteristics by age, gender and region, with sufficient quotas of:
As required in all ACET studies, the surveys for the pre and post campaign evaluation were in the form of two separate GC templated questionnaires, one for each of the pre and post-campaign evaluations. The questionnaires were not altered in any way, including the wording, order and coding of questions. No questions were deleted. A few questions were added at the end of the survey, after the core questions had been asked.
Since the survey questionnaire was administered online via the usage of an online non-probability panel, the survey results are not projectable to the overall Canadian populations and a margin of error cannot be provided.
The total sample size for the pre-campaign survey was n=2000 and post-campaign survey was n=2033. The data was weighted by age, gender, region. Within each of these general population samples, the CRA set quotas for priority groups as noted above. The following table outlines the completes obtained by target sub-population.
| Priority subgroups | Pre-campaign wave # of completes (n=) |
Post-campaign wave # of completes (n=) |
|---|---|---|
| Indigenous peoples | 134 | 104 |
| Low-income individuals (household income under 40K) | 574 | 589 |
| Parents of children <18 years old | 628 | 510 |
| Secondary groups | Pre-campaign wave # of completes (n=) |
Post-campaign wave # of completes (n=) |
| Newcomers <5 years* | 50 | 52 |
| Parents of children <6 | 336 | 252 |
| Youth 18-24 | 170 | 208 |
| Purchasing first home in the next 3 years | 430 | 421 |
| Interested in changing careers /get training or courses | 858 | 800 |
Interviews were conducted in English and French, with respondents being able to complete the survey in either official language of their choice.
Respondents were invited to participate in the survey via email, with a unique URL link. This link could only be used once, with respondents being allowed to pause during completion and return to complete it at a later time.
For the pre- and post-campaign surveys, pre-tests were conducted to confirm survey length before fully deploying the questionnaire and to determine if respondents understood the questions. For the post-campaign survey, a question was also added to ensure there were no technical issues when listening to the audio in the TV ad. No issues were flagged during the pre-tests that required revision to the original questionnaire.
For this study, Ipsos partnered with a certified partner sample provider, Dynata, to obtain the required target samples. Dynata's panels are continually refreshed and recruited through various channels including 1) loyalty program sourcing across travel, entertainment and other sectors; 2) online banners, cable TV advertising, mailings, social media influencers, mobile app, etc.; 3) integrated channels including access to online communities, social media platforms, publishers and others (this last group has not opted into a panel) but each participant has a Dynata profile.
Dynata's panel includes members who have opted in and receive a form of incentive. Incentives are based on a point-based system cashed in for rewards (electronic gift certificates, high street vouchers, charity donations, and long-term loyalty rewards), as well as sweepstakes. The amount of incentives is based on the specific requirements of each survey, depending on the length and complexity of the survey, the subject matter of the study, and the time required to complete a minimum number of interviews.
The comprehensive background profiling data gathered when respondents join a panel allows for the targeting of respondents based on key criteria, such as region, age, gender, education and income level, intention to buy a home within 12 months, profession, and other characteristics.
We ensured panel members had not participated in any Government of Canada survey in the previous 30 days as a member of that panel or in a survey on similar subject matter by any mode.
Dynata uses a router as its sampling methodology. This routing technology uses weighted randomization to assign surveys to participants. Upon entry into the system, panelists are checked to ensure they have not exceeded survey participation limits. A list of potential survey matches is determined for each panelist based upon the information we know about them. Panelists may be asked additional screening questions within the system to ensure they meet the project criteria. Priority may be given to surveys that are behind schedule; however, this is kept to a minimum as survey randomization must remain in place as a key element for preventing bias.
From the outset, Dynata's routing technology was designed to ensure high–quality sampling and minimize bias. In the development of our technology, randomization was essential to minimize the risk of assignment bias. This routing technology has been tested using multiple data points to ensure that bias is limited and has been reviewed by external consultants to validate the routing technology design. In addition, reports of system statistics and conversion rates are reviewed systematically to ensure conistency.
The table below shows the unweighted and weighted distributions of the online samples for the pre- and post-campaign waves. Weighting was applied to the sample to ensure that the final data reflects the general adult population by region, age, and gender according to the 2016 Census. A Random Iterative Method (RIM) technique was applied for weighting.
| Demographics | Pre-campaign Unweighted Sample Size |
Pre-campaign Weighted Sample Size |
Post-campaign Unweighted Sample Size |
Post-campaign Weighted Sample Size |
|---|---|---|---|---|
| Canada | 2000 | 2000 | 2033 | 2033 |
| Region - BC | 227 | 271 | 259 | 271 |
| Region - Alberta | 225 | 224 | 244 | 228 |
| Region - Prairies (MB/SK) | 132 | 130 | 143 | 132 |
| Region - Ontario | 759 | 768 | 851 | 781 |
| Region - Quebec | 519 | 469 | 394 | 477 |
| Region - Atlantic | 138 | 137 | 138 | 139 |
| Gender* - Male | 921 | 977 | 954 | 990 |
| Gender* -Female | 1073 | 1017 | 1067 | 1031 |
| Gender* -Diverse | 6 | 6 | 12 | 12 |
| Age - 18-34 | 661 | 560 | 493 | 569 |
| Age - 35-54 | 852 | 680 | 692 | 691 |
| Age - 55+ | 487 | 760 | 848 | 773 |
* Denotes variables included in the weighting scheme.
The figures presented in the table above show minimal differences between the unweighted and weighted samples. The RIM weighting efficiency for the pre-campaign wave is 90.0 % and for the post-campaign wave is 97.5 %.
For this survey, a sample router was used. Therefore, a response rate cannot be calculated. However, the participation rate for the survey was 98% for the pre-campaign wave and 93% for the post-campaign wave. The average survey length was 4 minutes for the pre-campaign and 7 minutes for the post-campaign.
Participation rate is calculated as follows: (qualified completes + over quota + terminates)/click-through).
| Completions | Pre-campaign | Post-campaign |
|---|---|---|
| Click-Through | 3273 | 3202 |
| - Partial Completes | 81 | 239 |
| - Terminates | 303 | 442 |
| - Over quota | 889 | 488 |
| Qualified Completes | 2000 | 2033 |
| Participation Rate | 98% | 93% |
The results of this survey are not statistically projectable to the target population because the sampling method used does not ensure that the sample represents the target population with a known margin of sampling error. Reported percentages are not generalizable to any group other than the sample studied, and therefore no formal statistical inferences can be drawn between the sample results and the broader target population it may be intended to reflect. The sample data have been weighted to reflect the demographic composition of Canadian adults aged 18 years and over.
The table below compares the unweighted survey samples to the 2016 Census results by region, age, and gender. Overall, the sample is highly representative of the national adult population, with the exception of a few gaps, which are described below.
| Demographics | Unweighted Percentage Pre-campaign |
Unweighted Percentage Post-campaign |
Census 2016 Proportions (adults) |
|---|---|---|---|
| Region - BC | 11% | 13% | 13.6% |
| Region - Alberta | 11% | 12% | 11.2% |
| Region - Prairies (MB/SK) | 7% | 7% | 6.5% |
| Region - Ontario | 38% | 42% | 38.4% |
| Region - Quebec | 26% | 19% | 23.5% |
| Region - Atlantic | 7% | 7% | 6.8% |
| Gender* - Male | 46% | 47% | 48.5% |
| Gender* -Female | 54% | 52% | 51.4% |
| Gender* -Diverse | <1% | 1% | <1% |
| Age - 18-34 | 33% | 24% | 27.3% |
| Age - 35-54 | 43% | 34% | 34.1% |
| Age - 55+ | 24% | 42% | 38.6% |
* Denotes variables included in the weighting scheme.
Differences among the variables used in the weighting scheme are minimal; however, there are a few notable differences. Ontario is over-represented in the post-campaign unweighted sample; Quebec is under-represented in the post-campaign unweighted sample. Additionally, age skews are present in the unweighted sample: 18-34-year olds and 35-54-year olds are slightly over-represented in the pre-campaign waves, and 55+ are over-represented in the post-campaign. The remaining distribution in the sample remains consistent, with only slight differences observed between the unweighted percentages and the 2016 Census data.