Prepared for the Canada Revenue Agency
Submitted by Narrative Research
PSPC Contract Number: 46558-227949/001/CY
Contracted Value: $23,272.35
Award Date: January 17, 2022
Delivery Date: March 31, 2022
Registration Number: POR 099-21
For more information, please contact: cra-arc.media@cra-arc.gc.ca
Ce rapport est aussi disponible en français.
Prepared for Canada Revenue Agency
Supplier Name: Narrative Research Inc.
March 2022
This report presents the methodological details for the post-campaign wave of the 2021-2022 CRA Scams Campaign study conducted by Narrative Research Inc. on behalf of the Canada Revenue Agency (CRA). The post-wave surveys were conducted with 2,002 respondents from the Canadian general public aged 18 years and older, between March 7 to 15, 2022.
Ce rapport est aussi disponible en français sous le titre: La campagne publicitaire de l’ARC de 2020-2021 – À bas l’arnaque – OECP Sommaire exécutif.
This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from the Canada Revenue Agency. For more information on this report, please contact the Canada Revenue Agency at: cra-arc.media@cra-arc.gc.ca
101 Colonel By Drive
Ottawa, Ontario K1A 0K2
Canada
Catalogue Number: Rv4-150/2-2022E-PDF
International Standard Book Number (ISBN): 978-0-660-44561-8
Related publications (registration number: POR 051-20):
Catalogue Number (Executive Summary, French): Rv4-150/2-2022F-PDF
ISBN (French): 978-0-660-44563-2
© Her Majesty the Queen in right of Canada, as represented by the Canada Revenue Agency, 2022
Narrative Research Inc.
Contract Number: 46558-227949/001/CY
POR Registration Number: 099-21
Contract Award Date: January 17, 2022
Contracted Cost: $23,272.35
Scams involving the Government of Canada (GoC) name have been prevalent in recent years, and with the introduction of emergency programs to help Canadians through the pandemic, there has been a surge of COVID-related scams.*
Since 2017, the Canada Revenue Agency (CRA) has conducted small advertising campaigns annually around tax filing season and Fraud Prevention Month in March to help increase awareness on scams involving the CRA name. Despite the initiatives the CRA has taken to distribute scam warnings to the public, daily reports in the media and conversations on social media about fraudsters posing as Government of Canada officials, scams of this nature persist.
To continue to raise awareness of scams involving the Government of Canada name and with the collaboration of other departments, the CRA lead a sustained multimedia scams campaign over the last two years. The campaign launched in Fall 2020 and a pre-wave advertising recall survey was conducted prior to the launch of the campaign. The final post-wave survey was conducted in March 2022.
This research using the Government of Canada’s Advertising Campaign Evaluation Tool (ACET) is mandatory, given that the Treasury Board requires all Government of Canada advertising campaigns that cost more than one million dollars to be evaluated.
* Canadian Anti-Fraud Center, Fraud Statistics, website https://www.antifraudcentre-centreantifraude.ca/index-eng.htm
The purpose of the quantitative research was to assess recall of and reactions to the advertising campaign. At the highest level, the purpose of the research is to evaluate the effectiveness of the advertising campaign. More specifically, the research objectives included:
Post-campaign evaluation to evaluate:
As specified in the Statement of Work (SOW), the target audience of the 2021-2022 CRA Scams Campaign is Canadians 18 years of age or older. The project also set quotas for specific groups: newcomers to Canada, seniors (age 60+), and youth (age 18-34). Data for this post-wave survey was gathered from a general public panel modelling key demographics of Canadians in various age segments (18-34, 35-59, and 60 years or older).
This post-campaign wave online survey of the Canadian general public aged 18+ years of age was undertaken from March 7 to 15, 2022. It required an average of approximately seven minutes to administer for both the French and English versions. There was a total of 2,002 useable surveys completed in the post-wave.
The email contact records for this research were drawn from panelists administered by The Logit Group of Toronto, Ontario. Given that this online survey methodology used a non-probability sample, the data collected cannot be extrapolated to the Canadian general public adult population.
As stated in the project’s Statement of Work and related communications, this pre-campaign wave research activity sought to measure recall of the 2021-22 CRA Scams Campaign media placement. Campaign placements were online (Connected TV, banner and video ads, social media, search engines) prior to this post-wave. The findings from this study are to be used by the CRA to assess the recall of the media campaign, the efficiency of the media placement, and the effectiveness of the campaign. Given that this online survey methodology used a non-probability sample, the data collected cannot be extrapolated to the Canadian general public adult population.
The project expenditure was $23,272.35, including HST, for the pre- and post-wave surveys.
Narrative Research offers this written consent allowing the Librarian and Archivist of Canada to post, in both official languages, this Methodological Report.
I hereby certify as a Senior Officer of Narrative Research that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not contain any reference to electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leader.
Peter MacIntosh
Chief Research Officer & Partner
Narrative Research
pmacintosh@narrativeresearch.ca
902-493-3832
This evaluation utilized the Government of Canada’s Advertising Campaign Evaluation Tool (ACET) and was administered to a sample of Canadian adults 18 years old and older. The online-based data collection regimen entailed post-advertising campaign data collection. The data collection was conducted between March 7 and 15 2021. When viewed alongside pre--wave data collected in the 2020-21 iteration of this research, this approach permits a comparison of awareness and opinions over time, as compared to other Government of Canada commissioned advertising campaigns.
The questions utilized in this study were based on the Government of Canada’s standard Advertising Campaign Evaluation Tool questionnaire. The primary difference between the online ACET survey questionnaire and the previously utilized telephone ACET survey questionnaire, was that in the online approach implemented in the present study, the survey respondents are aided in their ad recall by way of being shown on-screen advertisements from the recent CRA advertising campaign. In this post-campaign wave the respondents were shown three ads – two video ads and one online ad (GIF) from the campaign, and were subsequently asked a series of questions about the advertisements. This aiding of respondents by showing ads drawn from the advertising campaign is a process that is possible with an online survey methodology. Narrative Research ensured that respondents were able to complete the survey on various platforms including computers, tablets or smartphones.
As required by Government of Canada standards, English and French pre-test surveys were collected. As well, a line of questioning was included at the end of the pre-test surveys in which respondents were asked if they encountered any questions or survey wording that was difficult to understand. No pre-test respondents expressed difficulty in understanding any of the survey questions in either wave. As a result, no pre-test respondent was asked to identify which question or questions were problematic from a comprehension perspective.
The survey approach utilized in both waves was designed to be administered to an online general public panel sample of approximately 2,000 Canadian adults aged 18 or older. Narrative Research ensured that the surveys collected closely reflected the actual, true Canadian general population in terms of gender and age group (broken into 18-34, 35-59, and age 60+), and by region, as required by the project’s Statement of Work.
Attention was also given to whether respondents were immigrants to Canada, in order to survey at least 100 recent immigrants (operationalized as those who came to Canada less than five years ago), and to whether respondents were seniors (minimum of 200 surveys), or youth (minimum of 200 surveys).
Specifically, to ensure robust samples that approximate the true population parameters for age (18-34, 35-59, 60+), gender (male/female), and region (Atlantic, Quebec, Ontario, Prairies, British Columbia/ North), quotas were implemented in data collection. Age and gender quotas were implemented per region, and statistical weighting of the survey data was implemented to adjust for the small differences between the target data collection quotas, on the one hand, and the actual distribution of survey completions, on the other hand. (Immigration status also was included in this statistical weighting regimen, as discussed below in the Data Collection section of this Methodological Report).
Narrative Research utilized the services of The Logit Group for this research. The Logit Group’s online general population panel is comprised of over 600,000 Canadian residents nationally, with sound representation across regions. Logit Group panelists are recruited from a large number of sources to maximize reach and representation. The recruitment policies of The Logit Group’s partners (SSI, Toluna, Asking Canadians, and Research Now) are broad in scope. Survey data quality rests on many different factors, including sourcing of panelists who are vetted, using ongoing quality checks such as eliminating panelists who are no longer active, and so forth. The following are panel sources for Logit Group studies:
The Logit Group has established a variety of quality assurance processes to proactively identify invalid respondents. For example, the company has incorporated methods to quickly identify and flag straight-lining speedsters (i.e., respondents who give the same responses to all questions as a means of quickly finishing the survey), thereby monitoring whether panelists are able to provide thoughtful and accurate responses to survey queries.
Panel members are monitored against Statistics Canada data to gauge statistical representation. Annual profile refreshing campaigns are conducted to incentivize panelists to remain active; these can also contain new questions in order to target specific niche audiences more precisely. Panelists’ participation is rewarded with their choice of HBC Rewards bonus points, Aeroplan Miles or Petro Points, as well as various prizes.
For both the post-advertising campaign data collection wave, the online surveys were programmed by Narrative Research in both English and French, using Voxco Acuity programming software. Respondents were formally invited to the survey in the official language of their choice. As well, at any point when completing the questionnaire, respondents had the option to change the questionnaire language to the other official language. Assistance in completing the survey was available from bilingual Narrative Research staff, as required. Respondents were able to verify the legitimacy of the survey via representatives from Narrative Research, or via the survey registration system made available by the Canadian Research Insights Council (CRIC), Canada’s national research agency for this sector. The programmed surveys were tested to ensure that question order and skip patterns were properly implemented. Testing included Narrative Research researchers ensuring the accuracy of the survey delivery, text, links, and so on. CRA staff were also provided with the pre-test link for both pre- and post- wave surveys.
A total of 24 English and 13 French pre-tests were completed in March 2022 for the post-wave survey. Pre-test survey completions were conducted via a survey “soft launch” whereby a small number of panel respondents were invited to participate in the survey. The pre-testing of the surveys allowed the collected data to be reviewed to ensure accuracy and to identify any programming aspects that should be modified. Pre-test respondents were asked if they had any difficulty understanding any aspect of the surveys. No one replied in the affirmative. Thus, no substantive data quality issues arose as a result of the pre-test, and thus the pre-test data was maintained in the final data set.
Unlike telephone surveys which typically occur with new respondents being contacted throughout the specified data collection time period, in online surveys of the type implemented in this research, the preponderance of respondents are notified within a short period, that is, after the campaign for the post-wave survey.
This study consisted of a post-campaign wave that was administered between March 7 and 15, 2022. The survey invitations as well as reminder invitations were sent to panel members during the data collection period. Fieldwork was monitored and reviewed on an ongoing basis to ensure target quotas were being met. Narrative Research provided regular reports to CRA representatives regarding progress, as requested or pre-determined. No individual was able to complete the survey questionnaire more than once.
A total of 2,086 surveys were submitted by respondents, and 2,002 were ultimately used in the final data set in the post-wave survey. It is important to note that for various reasons, a small percentage of submitted online panel surveys is often removed from study data sets after submission. Such was indeed the case in this wave, as Narrative Research’s initial quota targets in each wave exceeded the overall final requirement of 2,000 questionnaires.
Thus, given the unavoidable possibility of having to remove surveys, post collection, Narrative Research as a precautionary measure collected more than the initially targeted number of surveys. Reasons for removing surveys ultimately included respondents who consistently provided non-intelligible verbatim responses, as well as duplicate IDs. Thus overall, a small number (n=44) were removed. The surveys in the post-campaign required a mean average of seven minutes to complete. A non-probability sample approach was implemented given that the study was designed to be conducted among online Canadian general public panelists. All such panels are inherently non-probability in nature, given that panelists self-select to become members of such panels, and not all adult Canadians belong to such a panel. The tables below for the post-campaign survey display regional, gender, immigration, and age data in terms of the actual distribution of adult Canadians aged 18 and older, as catalogued in the 2016 Statistics Canada Census.
As well, approximate regional, gender, and age quota targets per wave are detailed (both in terms of the actual number of surveys completed, and the percentage of all surveys completed). (Please note that for immigration status, a quota was implemented only for recent immigrants). The tables on the pages below present data with the weighted and unweighted number as well as percentage of surveys collected, for relevant demographic dimensions for each wave.
Data Tabulation: For both waves, there were a total of 30 overlapping or interlocking statistical weighting cells created from the study design using the weighting factors of: Region (5: Atlantic, Quebec, Ontario, Prairies, and British Columbia/North); Age group (3: 18–34, 35–59, 60+); Gender (2: Male, Female).
The 30 overlapping or interlocking statistical weighting cells thus were derived from Region (5) x Age (3) x Gender (2) dimensions = 30 unique statistical weighting cells. Participants were subsequently weighted based on whether they were not immigrants, immigrants who had been in Canada five years or more, or recent immigrants (less than 5 years). Population data for the 30 statistical weighting cells and immigration weights were obtained from the most recent (2016) Census of Canada, and can be found here:
With such a large number of weighting cells, it was necessary to combine specific cells due to the fact that sample sizes for specific cells were small or empty, and therefore would have led to quite large weights if left separate. Combining weighting cells is a common approach in such instances, and it explains why the final weighted data distribution differs slightly from the actual population distribution, along certain dimensions.
Nonetheless, it is suggested that the quantitative impact of implementing this approach was very modest, thereby recommending the specific statistical weighting approach as helpful. It should also be noted that a small number of individuals were not able to be placed into one of the 30 weighting cells due to the fact that they identified as gender diverse. For tabulation purposes, these individuals were given a weight value of 1.0.
Percentages may not sum exactly to 100%, owing to rounding:
| Region | 2016 Census | Quota Targets - Surveys (n=) |
Quota Targets - Surveys (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
Surveys Completed (Weighted) (n=) |
Surveys Completed (Weighted) (%) |
|---|---|---|---|---|---|---|---|
| Atlantic | 6.8% | 198 | 9.9% | 200 | 10.0% | 137 | 6.8% |
| Quebec | 23.4% | 482 | 24.1% | 501 | 25.0% | 463 | 23.1% |
| Ontario | 38.3% | 661 | 33.1% | 701 | 35.0% | 767 | 38.3% |
| Prairies | 17.7% | 324 | 16.2% | 350 | 17.5% | 352 | 17.6% |
| British Columbia/ North |
13.8% | 235 | 11.8% | 250 | 12.5% | 283 | 14.1% |
| Gender** | 2016 Census | Quota Targets - Surveys (n=) |
Quota Targets - Surveys (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
Surveys Completed (Weighted) (n=) |
Surveys Completed (Weighted) (%) |
|---|---|---|---|---|---|---|---|
| Male | 48.6% | 921 | 46.1% | 951 | 47.5% | 970 | 48.5% |
| Female | 51.4% | 979 | 49.0% | 1040 | 51.9% | 1022 | 51.0% |
| Immigrant Status*** | 2016 Census | Quota Targets - Surveys (n=) |
Quota Targets - Surveys (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
Surveys Completed (Weighted) (n=) |
Surveys Completed (Weighted) (%) |
|---|---|---|---|---|---|---|---|
| Not an immigrant | 76.6% | Not applicable | Not applicable | 1564 | 78.1% | 1534 | 76.6% |
| Immigrant, 5 years or more | 19.8% | Not applicable | Not applicable | 321 | 16.0% | 396 | 19.8% |
| Recent immigrant (less than 5 years) | 3.5% | 100 | 5.0% | 117 | 5.8% | 72 | 3.6% |
| Age (Quotas) | 2016 Census | Quota Targets - Surveys (n=) |
Quota Targets - Surveys (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
Surveys Completed (Weighted) (n=) |
Surveys Completed (Weighted) (%) |
|---|---|---|---|---|---|---|---|
| 18-34 | 27.4% | 516 | 25.8% | 543 | 27.1% | 535 | 26.7% |
| 35-59 | 43.4% | 823 | 41.1% | 878 | 43.9% | 874 | 43.7% |
| 60+ | 29.3% | 561 | 28.1% | 581 | 29.0% | 593 | 29.6% |
** Eleven respondents identified as gender diverse and are not presented in the table.
*** Census data used for immigration status was for Canadians of all ages, not just adult.
Participation Rate: The rate below was derived using the formula recommended by the Public Opinion Research Directorate of the Government of Canada:
Post-Campaign Survey
Total email addresses used: 19,580
Invalid cases
Invitations mistakenly sent to people who did not qualify for the study: 980
Incomplete or missing email addresses: 0
Unresolved (U)
Email invitations bounce back: 0
Email invitations unanswered: 14,169
In-scope non-responding units (IS)
Non-response from eligible respondents: 0
Respondent refusals 0
Language problem: 0
Selected respondent not available (illness; leave of absence; vacation; other): 0
Early break-offs: 257
Responding units (R)
Completed surveys disqualified – quota filled: 2,088
Completed surveys disqualified for other reasons: 84
Completed surveys: 2,002
Post-Campaign Wave Participation Rate = R/(U + IS + R) = 4,174/(14,169 + 257 + 4,174) = 22.4%
Reminders were distributed to potential respondents who were invited to complete a survey, but who chose not to do so. Given that the online methodology utilized a non-probability sample, a margin of error cannot be applied to the results as per the Standards for the Conduct of Government of Canada Public Opinion Research for Online Surveys.
Any survey that is conducted is potentially subject to bias or error. When a survey is conducted with a sample of the population, there are two general classes of bias or error: sampling error, which is quantifiable, and non-sampling error, which is typically not quantifiable. Sampling error necessarily arises from the fact that surveys are administered to only a subset of the targeted population, and thus is it possible that the survey results obtained from this group of respondents is not reflective of the population as a whole.
In contrast, non-sampling error encompasses a number of different types of errors including coverage error, measurement error, non-response error, and processing error. No measurement of sampling error can be attributed to the current study, given that the contact records utilized in the data collection process were derived from an online panel of the general public, which is to say, a non-probability sample source. Having stated that, measures were taken in the implementation of the data collection to ensure sufficient completed surveys were obtained from demographic groups traditionally regarded as central in quantitative survey research, such as gender, age, region/province, and immigration status. The final data set for each survey wave was statistically weighted to closely match the distribution of these dimensions as estimated in the 2016 Statistics Canada census. The statistical weights implemented were in general relatively small, given that the data collected already closely matched the actual distribution of adult Canadians along these demographic dimensions.
With respect to non-sampling error, a number of steps were taken to minimize bias due to these sources. All surveys utilized online interviewing technology to ensure proper survey skip patterns were followed and to minimize errors due to data entry and data capture. The French and English survey instruments from each campaign themselves were pre-tested with a small sample of respondents to ensure the survey material was easily understood by respondents, and that the resultant data were being captured properly. In terms of coverage, the surveys were conducted with an online panel of the Canadian general public 18 years of age or older, based on a randomized sampling of panel records for the target audience drawn from a commercially available online general public panel.