Advertising Campaign Evaluation Tool (ACET) – Methodological Report
Prepared for Canadian Revenue Agency (CRA)
Supplier name: Ipsos
Contract Number: CW2299077
Contract value: $30,284.00
Award Date: March 30th, 2023
Delivery Date: June 30th, 2023
Registration number: POR 150-22
For more information on this report, please contact Canadian Revenue Agency at:
Ce rapport est aussi disponible en français
2022-2023 CRA Benefits and Credits Campaign
Advertising Campaign Evaluation Tool (ACET) – Methodological Report
Prepared for Canadian Revenue Agency (CRA)
Supplier name: Ipsos
June 2023
This public opinion research report presents the methodology of the 2022-2023 CRA Benefits and Credits Campaign - ACET online survey conducted by Ipsos on behalf of the Canadian Revenue Agency (CRA).
The online survey was conducted with n=2,010 respondents between May 19th and 29th, 2023.
Cette publication est aussi disponible en français sous le titre : Campagne publicitaire sur les prestations et les crédits de l’Agence de 2022-2023 – Outil d’évaluation des campagnes publicitaires (OECP) – Rapport méthodologique.
This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Canada Revenue Agency. For more information on this report, please contact Canada Revenue Agency at:cra-arc.media@cra-arc.gc.ca.
Public Affairs Branch
555 MacKenzie Ave.
Ottawa, Ontario
K1A 0L5 Canada
Catalogue Number: Rv4-137/2023E-PDF
International Standard Book Number (ISBN): 978-0-660-68738-4
Related publications (registration number: POR 150-22):
Catalogue Number: Rv4-137/2023F-PDF (Methodological Report, French)
ISBN: 978-0-660-68739-1
© His Majesty the King in Right of Canada, as represented by the Minister of the Canada Revenue Agency, 2023
POLITICAL NEUTRALITY STATEMENT
I hereby certify as Senior Officer of Ipsos that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate or ratings of the performance of a political party or its leaders.
Mike Colledge
President
Ipsos Public Affairs
The Canada Revenue Agency (CRA) administers taxes, benefits, and related programs for governments across Canada. These services and programs help residents of Canada comply with tax obligations and receive benefits, which contributes to social-economic well-being. The CRA is undertaking significant efforts to increase awareness and access to CRA benefits and services, participation in the tax system, and understanding of roles and responsibilities in the tax and benefit system.
Benefits and credits provide additional income or tax relief to individuals to help make life more affordable since many depend on benefit payments and tax credits for a significant part of their household income. For modest-income households, benefits can increase income by as much as 50 percent.
However, certain segments of the vulnerable population such as modest-income households, Indigenous Peoples, those new to Canada, and youth aged 18-24 face a variety of barriers when doing their taxes and receiving their benefits and credits. These barriers include a lack of general awareness of benefits and credits, lack of support to complete their taxes, living in remote locations, language barriers, low financial literacy and/or mental or physical health issues. Given the range of backgrounds and circumstances, it is important for the Government of Canada (GC) to make proactive and targeted efforts to reach vulnerable populations with important tax-related information. Moreover, greater awareness is needed so these populations know that doing their taxes every year is necessary to be eligible for these benefits and credits. Increasing awareness of benefits and credits will help to contribute to the government priority of poverty reduction, health, and well-being, as found in the GC’s Gender Results Framework.
The Benefits and Credits advertising Campaign
This Benefits and Credits advertising campaign is part of a sustained effort by the CRA to raise awareness amongst vulnerable populations[1] of the benefits and credits that they are entitled to when they do their taxes and the availability of free tax help. The campaign featured the Canada child benefit (CCB), Canada workers benefit (CWB), GST/HST credit, the disability tax credit (DTC), and the Community Volunteer Income Tax Program’s (CVITP) free tax clinics. The campaign also promoted tax measures related to housing, including the Tax-Free First Home Savings Account, Multigenerational Home Renovation Tax Credit, Home Buyers’ Plan, GST/HST new housing rebate, home buyers’ amount and First-Time Home Buyer Incentive.
The advertising campaign aims to:
Media Tactics
An Advertising Campaign Evaluation Tool (ACET) is required for all Government of Canada (GC) advertising campaigns with a media buy over $1 million. The ACET is used to consistently evaluate these campaigns and comply with key requirements in the GC Communications Policy.
Ipsos was contracted to conduct the data collection and tabulation, and to provide a methodology report. Ipsos programmed, hosted, and provided sample management services, while the CRA provided the online questionnaire. Ipsos was responsible for data collection and data storage in Canada, data processing, and data weighting. The data collection from Canadians was handled in accordance with government-wide Public Opinion Research procedures.
The research consisted of an online post-campaign survey conducted in English and French, between May 19 and 29th, 2023.
The total contract value of this research was $30,284.00, including HST.
Advertising campaign evaluations assist in determining the effectiveness of the selected advertising campaigns by measuring (and comparing) awareness of the subject matter with the audiences. The results will be used to evaluate the effectiveness of the Benefits and Credits advertising campaign with the target audiences. The results will also help assess whether the ads were compelling / caught the audiences' attention, and whether the campaign raised awareness of the benefits and credits being promoted. Finally, the evaluation will assist the CRA in improving or modifying future ad campaigns.
For this research, CRA was seeking to evaluate the campaign which ended in May 2023. This post-campaign survey used the standardized ACET Post-Campaign online survey to assess:
The target audiences of the research were as follows:
Primary Target Audience:
Secondary Target Audience:
Respondents for the survey samples were drawn from a trusted partner panel vendor, Canadian Viewpoint Inc. The survey was conducted with a sample of n=2,010 Canadians ages 18+. Interviews were conducted in the language of their choice, either English or French.
A pre-test was launched on May 19th, 2023, which garnered 49 completes (29 English / 20 French). An open-ended question was asked at the end of the survey where any problems, questions, or unclear questions could be brought to our attention; no issues were flagged. The survey was fully launched and ran between May 20th and 29th, 2023.
Quotas were set to ensure representation by region, age, and gender, according to the latest Census information.
The sampling methodology utilized email invitations and router technology to invite participants. Each participant received a unique URL link. This link could only be used once, with respondents being allowed to pause during completion and return to complete. On average, the online interviews took 10 minutes to complete.
Ipsos partnered with sub-contractor Canadian Viewpoint Inc. Canadian Viewpoint has one of the largest consumer panels with ~300,000 active panelists. Like Ipsos, Canadian Viewpoint uses mixed-medium strategies (phone/online/in-person) to build its panel to allow more representative samples. Canadian Viewpoint’s online consumer panel is also recruited using post screeners during telephone surveys, Facebook, online vetted lists and in-person studies. Incentives were not used for recruitment purposes to ensure quality, but respondents were incentivized for completing the survey, which is directly proportionate to the amount of time taken to complete the survey and with comparable incentives offered by other online panel sources.
The comprehensive background profiling data were gathered when respondents joined the panel through screener questions, which allowed for the targeting of respondents based on key criteria, such as region, age, gender, education, and income level, intention to buy a home within 12 months, profession, and other characteristics.
The online survey was conducted using Computer Assisted Web Interviewing (CAWI). Every panelist opted-in or chose to participate in research surveys. Panelists that meet the criteria we were looking for were randomly selected and sent an email invitation to complete the survey. Participants were offered a choice to complete the survey in either English or French.
The survey platform was Accessibility for Ontarians with Disabilities Act (AODA) compliant, according to Web Content Accessibility Guidelines (WCAG2.0AA).
Sample Weighting
The table below shows the unweighted and weighted distributions of the online sample. Weighting was applied to the sample to ensure that the final data reflects the general adult population by region, age, and gender according to the 2021 Census. A Random Iterative Method (RIM) technique was applied for weighing.
Weighted and Unweighted Online Sample
|
Demographics |
Post-wave Survey |
|
|
Unweighted Sample Size |
Weighted Sample Size |
|
|
Canada |
2010 |
2010 |
|
Region |
||
|
British Columbia/Yukon |
261 |
281 |
|
Alberta/Northwest Territories |
241 |
221 |
|
Prairies (MB/SK) |
136 |
121 |
|
Ontario/ Nunavut |
766 |
784 |
|
Quebec |
464 |
462 |
|
Atlantic Canada |
142 |
141 |
|
Gender |
||
|
Male |
999 |
980 |
|
Female |
1001 |
1020 |
|
Diverse / Prefer not to answer |
10 |
10 |
|
Age |
||
|
18-24 |
200 |
203 |
|
25-34 |
333 |
339 |
|
35-44 |
333 |
331 |
|
45-54 |
316 |
312 |
|
55-64 |
354 |
353 |
|
65+ |
474 |
471 |
The figures presented in the table above show minimal differences between the unweighted and weighted samples. As a result of the strong representativeness of the unweighted data the largest weight factor that was applied for any respondent is 1.08, which is well within acceptable ranges for a survey of the general population.
Within natural fallout, we obtained our quota targets without having to oversample. The unweighted counts are shown below.
|
Primary target audience |
Target |
Achieved |
|
Indigenous |
100 |
105 |
|
Moderate-income (<$40K) |
400 - 500 |
499 |
|
Newcomers <5 years |
~50 |
61 |
|
People with disabilities |
400 - 440 |
424 |
|
Youth aged 18-24 |
200 - 250 |
200 |
|
Adults aged 65+ |
150 - 160 |
474 |
|
Secondary target audience |
||
|
Potential new homebuyers2 |
-- |
240 |
Incentives and Quality Control Measures
Canadian Viewpoint use a multi-faceted approach to quality, integrating controls and best practices into every phase of the process. Panel members come from diverse fully certified sources. A dedicated team monitors performance by source using feedback loops and continuous reporting to prevent fraud. Our dedicated team monitors the panels for potentially fraudulent behavior, and blocks these people from ever entering a survey via safeguards that includes:
Email Statistics
For this survey, a sample router was used. Therefore, a response rate cannot be calculated. However, the participation rate for the survey was 90%, which is calculated as follows: (qualified completes + over quota + terminates)/click-through).
|
Completions |
Post-wave Survey |
|
Click-Through |
3198 |
|
- Partial Completes |
307 |
|
- Terminates |
569 |
|
- Over quota |
312 |
|
Qualified Completes |
2010 |
|
Participation Rate |
90% |
Non-Response Analysis
The results of this survey are not statistically projectable to the target population because the sampling method used does not ensure that the sample represents the target population with a known margin of sampling error. Reported percentages are not generalizable to any group other than the sample studied, and therefore no formal statistical inferences can be drawn between the sample results and the broader target population it may be intended to reflect. The data have been weighted to reflect the demographic composition of Canadian adults aged 18 years and over.
The table below compares the unweighted survey samples to the 2021 Census results by region, age, and gender. Overall, the sample is highly representative of the national adult population.
|
Demographics |
Post-wave Survey |
|
|
Unweighted Percentage |
Census 2021 Proportions |
|
|
Region |
||
|
British Columbia/Yukon |
13% |
14% |
|
Alberta/Northwest Territories |
12% |
11% |
|
Prairies (MB/SK) |
7% |
7% |
|
Ontario/ Nunavut |
38% |
38% |
|
Quebec |
23% |
23% |
|
Atlantic Canada |
7% |
7% |
|
Gender |
||
|
Male |
50% |
49% |
|
Female |
51% |
51% |
|
Diverse / Prefer not to answer |
<1% |
<1% |
|
Age |
||
|
18-24 |
10% |
11% |
|
25-34 |
17% |
17% |
|
35-44 |
17% |
17% |
|
45-54 |
16% |
16% |
|
55-64 |
18% |
18% |
|
65+ |
24% |
24% |
* Denotes variables included in the weighting scheme.
Differences among the variables used in the weighting scheme are minimal. The distribution in the sample is consistent, with only slight differences observed between the unweighted percentages and the 2021 Census data.
Online Questionnaire
ADVERTISING CAMPAIGN EVALUATION TOOL
POST-CAMPAIGN SURVEY
INTRODUCTION
Thank you for taking a few minutes to complete this survey on current issues that matter to Canadians. Please select the language you would like to take this survey in /Si vous préférez répondre à ce sondage en français, veuillez cliquer sur français [SWITCH TO FRENCH VERSION].
Your participation is voluntary, and your responses will be kept entirely confidential. The survey takes about 10 minutes to complete.
START SURVEY
Click here if you wish to verify the authenticity of this survey (Project code: 20230510-IP019). To view our privacy policy, click here.
If you require any technical assistance, please contact xxxx@ipsos.com
SELECT ALL THAT APPLY
IF “NONE OF THESE ORGANIZATIONS” CONTINUE, OTHERWISE THANK AND TERMINATE.
___ ___ ___ ___
YYYY
ADMISSIBLE RANGE 1900-2005
IF > 2005, THANK AND TERMINATE
ASK D IF QUESTION C IS LEFT BLANK
IF “LESS THAN 18 YEARS OLD” OR “BLANK”, THANK AND TERMINATE
e) In which province or territory do you live?
SELECT ONE ONLYIF NO PROVINCE OR TERRITORY IS SELECTED, THANK AND TERMINATE
f): Where were you born?
Specify the country:
ASK IF g=BORN OUTSIDE CANADA
g): In what year did you first move to Canada?
___ ___ ___ ___
YYYY
ADMISSIBLE RANGE: 1900-2023
h) Are you an Indigenous person? (An Indigenous person is a North American Indian or a member of a First Nation, a Métis or an Inuk (Inuit). North American Indians or members of a First Nation include status, treaty or registered Indians, as well as non-status and non-registered Indians.)
SELECT ONE ONLY
i): Which of the following categories best describes your total annual household income, including income from all household members, before taxes are deducted?
SELECT ONE ONLY
j) Are you a person with a disability?
A person with a disability has a long-term or recurring physical, mental, sensory, psychiatric or learning impairment and:
Persons with disabilities are also those whose functional limitations owing to their impairment have been accommodated in their current job or workplace. Disabilities include:
k) Are you considering purchasing your first home in the next 3 years?
ASK ALL RESPONDENTS
Q1: Over the past three weeks, have you seen, read or heard any advertising from the Government of Canada?
Q2: Think about the most recent Government of Canada ad that comes to mind. What do you remember about this ad?
Answer:
CAMPAIGN-SPECIFIC QUESTIONS
ASK ALL RESPONDENTS
T1A: Over the past three weeks, have you seen, read or heard any Government of Canada advertising about benefits, tax credits and/or free tax help?
T1B: Where have you seen, read or heard this ad about benefits, tax credits and/or free tax help?
SELECT ALL THAT APPLY
[Randomize]
T1C: What do you remember about this ad?
Answer:
ASK ALL RESPONDENTS
CAMPAIGN SPECIFIC ATTITUDINAL AND BEHAVIOURAL QUESTIONS MAY BE ADDED HERE
T1D: Are you aware that you may qualify for Government of Canada benefits and tax credits?
T2D: Are you aware that there is free tax help for eligible people to get their taxes done by volunteers at free tax clinics?
T2DA: What motivates you to do your taxes?
T3D: Do you feel confident to do your own taxes, whether on paper or using tax software yourself?
T4D: Who does your taxes?
T5D: [IF DO ON MY OWN IN T4D ASK] Do you do taxes for anyone else in addition to yourself (select all that apply)
T6D: [IF DO ON MY OWN NOT SELECTED IN T4D] Why do you not do your own taxes?
Answer:
AD RECALL QUESTIONS
T1H: Here are some ads that have recently been broadcast on various media. Click here to watch.
[INSERT VIDEO, PRINT, AND RADIO ADS]
CCB_Sarah_15E_16x9_web
CWB_Leslie_15E_16x9_web
GST-HST_Jamal_15E_16x9_web
[CLICK TO GO TO THE NEXT PAGE]
Over the past three weeks, have you seen, read or heard these ads?
T1I: Where have you seen, read or heard these ads?
[NOTE: SELECT FROM LIST ALL MEDIA USED IN THE CAMPAIGN. YOU MAY ALSO INCLUDE OTHER MEDIA OF YOUR CHOICE. HEADINGS ARE FOR GUIDANCE ONLY AND NOT TO BE USED IN THE FINAL VERSION OF THE QUESTIONNAIRE]
SELECT ALL THAT APPLY
[Randomize]
T1J: What do you think is the main point these ads are trying to get across?
Answer:
T1K: Please indicate your level of agreement with the following statements about these ads?
RANDOMIZE STATEMENTS
|
Statements |
1 Strongly Disagree |
2 |
3 |
4 |
5 Strongly Agree |
|
These ads catch my attention |
o |
o |
o |
o |
o |
|
These ads are relevant to me |
o |
o |
o |
o |
o |
|
These ads are difficult to follow |
o |
o |
o |
o |
o |
|
These ads do not favour one political party over another |
o |
o |
o |
o |
o |
|
These ads talk about an important topic |
o |
o |
o |
o |
o |
|
These ads provide new information |
o |
o |
o |
o |
o |
|
These ads clearly convey that I may qualify for Government of Canada benefits, tax credits and/or free tax help. |
o |
o |
o |
o |
o |
T1L: [ASK IF YES IN T1H] Which of the following actions did you take as a result of seeing/hearing this advertising? [ACCEPT MULTIPLE RESPONSES]
DEMOGRAPHIC QUESTIONS
D1: Which of the following categories best describes your current employment status? Are you…
SELECT ONE ONLYD2: What is the highest level of formal education that you have completed?
SELECT ONE ONLY
D3: Are there any children under the age of 18 currently living in your household?
D4:What is the language you first learned at home as a child and still understand?
SELECT UP TO TWO
[1] The term “vulnerable” is being used in this instance to describe individuals who may be hard to reach or those who experience certain situational or systemic challenges that can negatively impact their outcomes or put them at a disadvantage.
[2] According to the 2017 Canadian Survey on Disability, more than 6 million Canadians aged 15 and over (22% of the population) identify as having a disability, and it is expected actual numbers are likely higher. (https://www.canada.ca/en/employment-social-development/programs/accessible-canada.html) The CSD definition of disability includes anyone who reported being "sometimes," "often" or "always" limited in their daily activities due to a long-term condition or health problem, as well as anyone who reported being "rarely" limited if they were also unable to do certain tasks or could only do them with a lot of difficulties.
[3] The CRA is also looking for potential new homebuyers earning $60K or more (as part of the natural fallout in the total sample of the 2000 general population; not part of set quotas). There will be questions about the intention to purchase a home among this income segment within the next three years to capture the natural fallout within the sample.