Prepared for the Canada Revenue Agency
Submitted by Narrative Research
PSPC Contract Number: 46558-227949/001/CY
Contracted Value: $23,272.35
Award Date: January 17, 2022
Delivery Date: March 31, 2022
Registration Number: POR 099-21
For more information, please contact: cra-arc.media@cra-arc.gc.ca
Ce rapport est aussi disponible en français.
Prepared for Canada Revenue Agency
Supplier Name: Narrative Research Inc.
March 2022
This report presents the methodological details for the post-campaign wave of the 2021-2022 CRA Scams Campaign study conducted by Narrative Research Inc. on behalf of the Canada Revenue Agency (CRA). The post-wave surveys were conducted with 2,002 respondents from the Canadian general public aged 18 years and older, between March 7 to 15, 2022.
Ce rapport est aussi disponible en français sous le titre: La campagne publicitaire de l’ARC de 2020-2021 – À bas l’arnaque – OECP Rapport méthodologique.
This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from the Canada Revenue Agency. For more information on this report, please contact the Canada Revenue Agency at: cra-arc.media@cra-arc.gc.ca
101 Colonel By Drive
Ottawa, Ontario K1A 0K2
Canada
Catalogue Number: Rv4-150/1-2022E-PDF
International Standard Book Number (ISBN): 978-0-660-44560-1
Related publications (registration number: POR 051-20):
Catalogue Number (Executive Summary, French): Rv4-150/1-2022F-PDF
ISBN (French): 978-0-660-44562-5
© Her Majesty the Queen in right of Canada, as represented by the Canada Revenue Agency, 2022
Narrative Research Inc.
Contract Number: 46558-227949/001/CY
POR Registration Number: 099-21
Contract Award Date: January 17, 2022
Contracted Cost: $23,272.35
Scams involving the Government of Canada (GoC) name have been prevalent in recent years, and with the introduction of emergency programs to help Canadians through the pandemic, there has been a surge of COVID-related scams.*
Since 2017, the Canada Revenue Agency (CRA) has conducted small advertising campaigns annually around tax filing season and Fraud Prevention Month in March to help increase awareness on scams involving the CRA name. Despite the initiatives the CRA has taken to distribute scam warnings to the public, daily reports in the media and conversations on social media about fraudsters posing as Government of Canada officials, scams of this nature persist.
To continue to raise awareness of scams involving the Government of Canada name and with the collaboration of other departments, the CRA lead a sustained multimedia scams campaign over the last two years. The campaign launched in Fall 2020 and a pre-wave advertising recall survey was conducted prior to the launch of the campaign. The final post-wave survey was conducted in March 2022.
This research using the Government of Canada’s Advertising Campaign Evaluation Tool (ACET) is mandatory, given that the Treasury Board requires all Government of Canada advertising campaigns that cost more than one million dollars to be evaluated.
* Canadian Anti-Fraud Center, Fraud Statistics, website https://www.antifraudcentre-centreantifraude.ca/index-eng.htm
The purpose of the quantitative research was to assess recall of and reactions to the advertising campaign. At the highest level, the purpose of the research is to evaluate the effectiveness of the advertising campaign. More specifically, the research objectives included:
Post-campaign evaluation to evaluate:
As specified in the Statement of Work (SOW), the target audience of the 2021-2022 CRA Scams Campaign is Canadians 18 years of age or older. The project also set quotas for specific groups: newcomers to Canada, seniors (age 60+), and youth (age 18-34). Data for this post-wave survey was gathered from a general public panel modelling key demographics of Canadians in various age segments (18-34, 35-59, and 60 years or older).
This post-campaign wave online survey of the Canadian general public aged 18+ years of age was undertaken from March 7 to 15, 2022. It required an average of approximately seven minutes to administer for both the French and English versions. There was a total of 2,002 useable surveys completed in the post-wave.
The email contact records for this research were drawn from panelists administered by The Logit Group of Toronto, Ontario. Given that this online survey methodology used a non-probability sample, the data collected cannot be extrapolated to the Canadian general public adult population.
As stated in the project’s Statement of Work and related communications, this pre-campaign wave research activity sought to measure recall of the 2021-22 CRA Scams Campaign media placement. Campaign placements were online (Connected TV, banner and video ads, social media, search engines) prior to this post-wave. The findings from this study are to be used by the CRA to assess the recall of the media campaign, the efficiency of the media placement, and the effectiveness of the campaign. Given that this online survey methodology used a non-probability sample, the data collected cannot be extrapolated to the Canadian general public adult population.
The project expenditure was $23,272.35, including HST, for the pre- and post-wave surveys.
Narrative Research offers this written consent allowing the Librarian and Archivist of Canada to post, in both official languages, this Methodological Report.
I hereby certify as a Senior Officer of Narrative Research that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not contain any reference to electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leader.
Peter MacIntosh
Chief Research Officer & Partner
Narrative Research
pmacintosh@narrativeresearch.ca
902-493-3832
This evaluation utilized the Government of Canada’s Advertising Campaign Evaluation Tool (ACET) and was administered to a sample of Canadian adults 18 years old and older. The online-based data collection regimen entailed post-advertising campaign data collection. The data collection was conducted between March 7 and 15 2021. When viewed alongside pre--wave data collected in the 2020-21 iteration of this research, this approach permits a comparison of awareness and opinions over time, as compared to other Government of Canada commissioned advertising campaigns.
The questions utilized in this study were based on the Government of Canada’s standard Advertising Campaign Evaluation Tool questionnaire. The primary difference between the online ACET survey questionnaire and the previously utilized telephone ACET survey questionnaire, was that in the online approach implemented in the present study, the survey respondents are aided in their ad recall by way of being shown on-screen advertisements from the recent CRA advertising campaign. In this post-campaign wave the respondents were shown three ads – two video ads and one online ad (GIF) from the campaign, and were subsequently asked a series of questions about the advertisements. This aiding of respondents by showing ads drawn from the advertising campaign is a process that is possible with an online survey methodology. Narrative Research ensured that respondents were able to complete the survey on various platforms including computers, tablets or smartphones.
As required by Government of Canada standards, English and French pre-test surveys were collected. As well, a line of questioning was included at the end of the pre-test surveys in which respondents were asked if they encountered any questions or survey wording that was difficult to understand. No pre-test respondents expressed difficulty in understanding any of the survey questions in either wave. As a result, no pre-test respondent was asked to identify which question or questions were problematic from a comprehension perspective.
The survey approach utilized in both waves was designed to be administered to an online general public panel sample of approximately 2,000 Canadian adults aged 18 or older. Narrative Research ensured that the surveys collected closely reflected the actual, true Canadian general population in terms of gender and age group (broken into 18-34, 35-59, and age 60+), and by region, as required by the project’s Statement of Work.
Attention was also given to whether respondents were immigrants to Canada, in order to survey at least 100 recent immigrants (operationalized as those who came to Canada less than five years ago), and to whether respondents were seniors (minimum of 200 surveys), or youth (minimum of 200 surveys).
Specifically, to ensure robust samples that approximate the true population parameters for age (18-34, 35-59, 60+), gender (male/female), and region (Atlantic, Quebec, Ontario, Prairies, British Columbia/ North), quotas were implemented in data collection. Age and gender quotas were implemented per region, and statistical weighting of the survey data was implemented to adjust for the small differences between the target data collection quotas, on the one hand, and the actual distribution of survey completions, on the other hand. (Immigration status also was included in this statistical weighting regimen, as discussed below in the Data Collection section of this Methodological Report).
Narrative Research utilized the services of The Logit Group for this research. The Logit Group’s online general population panel is comprised of over 600,000 Canadian residents nationally, with sound representation across regions. Logit Group panelists are recruited from a large number of sources to maximize reach and representation. The recruitment policies of The Logit Group’s partners (SSI, Toluna, Asking Canadians, and Research Now) are broad in scope. Survey data quality rests on many different factors, including sourcing of panelists who are vetted, using ongoing quality checks such as eliminating panelists who are no longer active, and so forth. The following are panel sources for Logit Group studies:
The Logit Group has established a variety of quality assurance processes to proactively identify invalid respondents. For example, the company has incorporated methods to quickly identify and flag straight-lining speedsters (i.e., respondents who give the same responses to all questions as a means of quickly finishing the survey), thereby monitoring whether panelists are able to provide thoughtful and accurate responses to survey queries.
Panel members are monitored against Statistics Canada data to gauge statistical representation. Annual profile refreshing campaigns are conducted to incentivize panelists to remain active; these can also contain new questions in order to target specific niche audiences more precisely. Panelists’ participation is rewarded with their choice of HBC Rewards bonus points, Aeroplan Miles or Petro Points, as well as various prizes.
For both the post-advertising campaign data collection wave, the online surveys were programmed by Narrative Research in both English and French, using Voxco Acuity programming software. Respondents were formally invited to the survey in the official language of their choice. As well, at any point when completing the questionnaire, respondents had the option to change the questionnaire language to the other official language. Assistance in completing the survey was available from bilingual Narrative Research staff, as required. Respondents were able to verify the legitimacy of the survey via representatives from Narrative Research, or via the survey registration system made available by the Canadian Research Insights Council (CRIC), Canada’s national research agency for this sector. The programmed surveys were tested to ensure that question order and skip patterns were properly implemented. Testing included Narrative Research researchers ensuring the accuracy of the survey delivery, text, links, and so on. CRA staff were also provided with the pre-test link for both pre- and post- wave surveys.
A total of 24 English and 13 French pre-tests were completed in March 2022 for the post-wave survey. Pre-test survey completions were conducted via a survey “soft launch” whereby a small number of panel respondents were invited to participate in the survey. The pre-testing of the surveys allowed the collected data to be reviewed to ensure accuracy and to identify any programming aspects that should be modified. Pre-test respondents were asked if they had any difficulty understanding any aspect of the surveys. No one replied in the affirmative. Thus, no substantive data quality issues arose as a result of the pre-test, and thus the pre-test data was maintained in the final data set.
Unlike telephone surveys which typically occur with new respondents being contacted throughout the specified data collection time period, in online surveys of the type implemented in this research, the preponderance of respondents are notified within a short period, that is, after the campaign for the post-wave survey.
This study consisted of a post-campaign wave that was administered between March 7 and 15, 2022. The survey invitations as well as reminder invitations were sent to panel members during the data collection period. Fieldwork was monitored and reviewed on an ongoing basis to ensure target quotas were being met. Narrative Research provided regular reports to CRA representatives regarding progress, as requested or pre-determined. No individual was able to complete the survey questionnaire more than once.
A total of 2,086 surveys were submitted by respondents, and 2,002 were ultimately used in the final data set in the post-wave survey. It is important to note that for various reasons, a small percentage of submitted online panel surveys is often removed from study data sets after submission. Such was indeed the case in this wave, as Narrative Research’s initial quota targets in each wave exceeded the overall final requirement of 2,000 questionnaires.
Thus, given the unavoidable possibility of having to remove surveys, post collection, Narrative Research as a precautionary measure collected more than the initially targeted number of surveys. Reasons for removing surveys ultimately included respondents who consistently provided non-intelligible verbatim responses, as well as duplicate IDs. Thus overall, a small number (n=44) were removed. The surveys in the post-campaign required a mean average of seven minutes to complete. A non-probability sample approach was implemented given that the study was designed to be conducted among online Canadian general public panelists. All such panels are inherently non-probability in nature, given that panelists self-select to become members of such panels, and not all adult Canadians belong to such a panel. The tables below for the post-campaign survey display regional, gender, immigration, and age data in terms of the actual distribution of adult Canadians aged 18 and older, as catalogued in the 2016 Statistics Canada Census.
As well, approximate regional, gender, and age quota targets per wave are detailed (both in terms of the actual number of surveys completed, and the percentage of all surveys completed). (Please note that for immigration status, a quota was implemented only for recent immigrants). The tables on the pages below present data with the weighted and unweighted number as well as percentage of surveys collected, for relevant demographic dimensions for each wave.
Data Tabulation: For both waves, there were a total of 30 overlapping or interlocking statistical weighting cells created from the study design using the weighting factors of: Region (5: Atlantic, Quebec, Ontario, Prairies, and British Columbia/North); Age group (3: 18–34, 35–59, 60+); Gender (2: Male, Female).
The 30 overlapping or interlocking statistical weighting cells thus were derived from Region (5) x Age (3) x Gender (2) dimensions = 30 unique statistical weighting cells. Participants were subsequently weighted based on whether they were not immigrants, immigrants who had been in Canada five years or more, or recent immigrants (less than 5 years). Population data for the 30 statistical weighting cells and immigration weights were obtained from the most recent (2016) Census of Canada, and can be found here:
With such a large number of weighting cells, it was necessary to combine specific cells due to the fact that sample sizes for specific cells were small or empty, and therefore would have led to quite large weights if left separate. Combining weighting cells is a common approach in such instances, and it explains why the final weighted data distribution differs slightly from the actual population distribution, along certain dimensions.
Nonetheless, it is suggested that the quantitative impact of implementing this approach was very modest, thereby recommending the specific statistical weighting approach as helpful. It should also be noted that a small number of individuals were not able to be placed into one of the 30 weighting cells due to the fact that they identified as gender diverse. For tabulation purposes, these individuals were given a weight value of 1.0.
Percentages may not sum exactly to 100%, owing to rounding:
| Region | 2016 Census | Quota Targets - Surveys (n=) |
Quota Targets - Surveys (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
Surveys Completed (Weighted) (n=) |
Surveys Completed (Weighted) (%) |
|---|---|---|---|---|---|---|---|
| Atlantic | 6.8% | 198 | 9.9% | 200 | 10.0% | 137 | 6.8% |
| Quebec | 23.4% | 482 | 24.1% | 501 | 25.0% | 463 | 23.1% |
| Ontario | 38.3% | 661 | 33.1% | 701 | 35.0% | 767 | 38.3% |
| Prairies | 17.7% | 324 | 16.2% | 350 | 17.5% | 352 | 17.6% |
| British Columbia/ North |
13.8% | 235 | 11.8% | 250 | 12.5% | 283 | 14.1% |
| Gender** | 2016 Census | Quota Targets - Surveys (n=) |
Quota Targets - Surveys (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
Surveys Completed (Weighted) (n=) |
Surveys Completed (Weighted) (%) |
|---|---|---|---|---|---|---|---|
| Male | 48.6% | 921 | 46.1% | 951 | 47.5% | 970 | 48.5% |
| Female | 51.4% | 979 | 49.0% | 1040 | 51.9% | 1022 | 51.0% |
| Immigrant Status*** | 2016 Census | Quota Targets - Surveys (n=) |
Quota Targets - Surveys (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
Surveys Completed (Weighted) (n=) |
Surveys Completed (Weighted) (%) |
|---|---|---|---|---|---|---|---|
| Not an immigrant | 76.6% | Not applicable | Not applicable | 1564 | 78.1% | 1534 | 76.6% |
| Immigrant, 5 years or more | 19.8% | Not applicable | Not applicable | 321 | 16.0% | 396 | 19.8% |
| Recent immigrant (less than 5 years) | 3.5% | 100 | 5.0% | 117 | 5.8% | 72 | 3.6% |
| Age (Quotas) | 2016 Census | Quota Targets - Surveys (n=) |
Quota Targets - Surveys (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
Surveys Completed (Weighted) (n=) |
Surveys Completed (Weighted) (%) |
|---|---|---|---|---|---|---|---|
| 18-34 | 27.4% | 516 | 25.8% | 543 | 27.1% | 535 | 26.7% |
| 35-59 | 43.4% | 823 | 41.1% | 878 | 43.9% | 874 | 43.7% |
| 60+ | 29.3% | 561 | 28.1% | 581 | 29.0% | 593 | 29.6% |
** Eleven respondents identified as gender diverse and are not presented in the table.
*** Census data used for immigration status was for Canadians of all ages, not just adult.
Participation Rate: The rate below was derived using the formula recommended by the Public Opinion Research Directorate of the Government of Canada:
Post-Campaign Survey
Total email addresses used: 19,580
Invalid cases
Invitations mistakenly sent to people who did not qualify for the study: 980
Incomplete or missing email addresses: 0
Unresolved (U)
Email invitations bounce back: 0
Email invitations unanswered: 14,169
In-scope non-responding units (IS)
Non-response from eligible respondents: 0
Respondent refusals 0
Language problem: 0
Selected respondent not available (illness; leave of absence; vacation; other): 0
Early break-offs: 257
Responding units (R)
Completed surveys disqualified – quota filled: 2,088
Completed surveys disqualified for other reasons: 84
Completed surveys: 2,002
Post-Campaign Wave Participation Rate = R/(U + IS + R) = 4,174/(14,169 + 257 + 4,174) = 22.4%
Reminders were distributed to potential respondents who were invited to complete a survey, but who chose not to do so. Given that the online methodology utilized a non-probability sample, a margin of error cannot be applied to the results as per the Standards for the Conduct of Government of Canada Public Opinion Research for Online Surveys.
Any survey that is conducted is potentially subject to bias or error. When a survey is conducted with a sample of the population, there are two general classes of bias or error: sampling error, which is quantifiable, and non-sampling error, which is typically not quantifiable. Sampling error necessarily arises from the fact that surveys are administered to only a subset of the targeted population, and thus is it possible that the survey results obtained from this group of respondents is not reflective of the population as a whole.
In contrast, non-sampling error encompasses a number of different types of errors including coverage error, measurement error, non-response error, and processing error. No measurement of sampling error can be attributed to the current study, given that the contact records utilized in the data collection process were derived from an online panel of the general public, which is to say, a non-probability sample source. Having stated that, measures were taken in the implementation of the data collection to ensure sufficient completed surveys were obtained from demographic groups traditionally regarded as central in quantitative survey research, such as gender, age, region/province, and immigration status. The final data set for each survey wave was statistically weighted to closely match the distribution of these dimensions as estimated in the 2016 Statistics Canada census. The statistical weights implemented were in general relatively small, given that the data collected already closely matched the actual distribution of adult Canadians along these demographic dimensions.
With respect to non-sampling error, a number of steps were taken to minimize bias due to these sources. All surveys utilized online interviewing technology to ensure proper survey skip patterns were followed and to minimize errors due to data entry and data capture. The French and English survey instruments from each campaign themselves were pre-tested with a small sample of respondents to ensure the survey material was easily understood by respondents, and that the resultant data were being captured properly. In terms of coverage, the surveys were conducted with an online panel of the Canadian general public 18 years of age or older, based on a randomized sampling of panel records for the target audience drawn from a commercially available online general public panel.
ADVERTISING CAMPAIGN EVALUATION TOOL
POST-CAMPAIGN SURVEY
To be conducted after the ads have been run in the media.
Thank you for taking the time to complete this survey dealing with current issues of interest to Canadians. Si vous préférez répondre au sondage en français, veuillez cliquer sur français [SWITCH TO FRENCH VERSION].
Your participation is voluntary and your responses will be kept entirely confidential. The survey takes about 5 minutes to complete.
START SURVEY
Click here if you wish to verify the authenticity of this survey. To view our privacy policy, click here.
If you require any technical assistance, please contact XXX. [PANEL HAS INTERNAL MESSAGING ON THIS]
a) Does anyone in your household work for any of the following organizations? SELECT ALL THAT APPLY
IF “NONE OF THESE ORGANIZATIONS” CONTINUE, OTHERWISE THANK AND TERMINATE. [ALL TERMINATES OCCUR AT END OF SCREENER QUESTIONS]
b) Are you…
c) In what year were you born?
________
YYYY
ADMISSIBLE RANGE 1900-2004
IF > 2004, THANK AND TERMINATE
ASK D IF QUESTION C IS LEFT BLANK OR IF 2004
d) In which of the following age categories do you belong?
SELECT ONE ONLY
IF “LESS THAN 18 YEARS OLD” OR “BLANK”, THANK AND TERMINATE
e) Where were you born?
ASK IF D5=BORN OUTSIDE CANADA
f) In what year did you first move to Canada?
________
YYYY
ADMISSIBLE RANGE: 1900-2022
g) In which province or territory do you live?
SELECT ONE ONLY
IF NO PROVINCE OR TERRITORY IS SELECTED, THANK AND TERMINATE
ASK ALL RESPONDENTS
Q1:
Over the past three weeks, have you seen, read or heard any advertising from the Government of Canada?
=> GO TO T1A
Q2:
Think about the most recent ad from the Government of Canada that comes to mind. Where have you seen, read or heard this ad?
SELECT ALL THAT APPLY
Q3:
What do you remember about this ad?
__________________________________________________________________
Q4:
How did you know that it was an ad from the Government of Canada?
__________________________________________________________________
ASK ALL RESPONDENTS
T1A:
Over the past three weeks, have you seen, read or heard any Government of Canada advertising about how to recognize scams?
=> GO TO T1D
T1B:
Where have you seen, read or heard this ad about how to recognize scams?
SELECT ALL THAT APPLY
T1C:
What do you remember about this ad?
__________________________________________________________________
ASK ALL RESPONDENTS
CAMPAIGN SPECIFIC ATTITUDINAL AND BEHAVIOURAL QUESTIONS MAY BE ADDED HERE
T1D:
In the last six months, have you received a message or a call that you thought was a scam?
ASK IF T1D=YES
T2D:
Was it through:
SELECT ALL THAT APPLY
ASK IF T1D=YES
T3D:
Was the message/subject matter about:
SELECT ALL THAT APPLY
T4D:
Have you ever been a victim of a scam similar to the ones mentioned previously?
ASK IF T4D=YES
T5D:
What was the nature of the scam?
__________________________________________________________________
T1H:
Next are some ads that have recently been broadcast on various media. Click 'Next' to view each ad. [INSERT VIDEO, PRINT AND RADIO ADS]
[CLICK TO GO TO THE NEXT PAGE]
Over the past three weeks, have you seen, read or heard these ads?
=> GO TO T1J
T1I:
Where have you seen, read or heard these ads?
SELECT ALL THAT APPLY
T1J:
What do you think is the main point these ads are trying to get across?
__________________________________________________________________
T1K:
Please indicate your level of agreement with the following statements about these ads? RANDOMIZE STATEMENTS
| Statements | 1 Strongly Disagree |
2 | 3 | 4 | 5 Strongly Agree |
|---|---|---|---|---|---|
| These ads catch my attention | ○ | ○ | ○ | ○ | ○ |
| These ads are relevant to me | ○ | ○ | ○ | ○ | ○ |
| These ads are difficult to follow | ○ | ○ | ○ | ○ | ○ |
| These ads do not favour one political party over another | ○ | ○ | ○ | ○ | ○ |
| These ads talk about an important topic | ○ | ○ | ○ | ○ | ○ |
| These ads provide new information | ○ | ○ | ○ | ○ | ○ |
| These ads clearly convey that you can trust your instinct to stop and think before you act | ○ | ○ | ○ | ○ | ○ |
D1:
Which of the following categories best describes your current employment status? Are you…
SELECT ONE ONLY
D2:
What is the highest level of formal education that you have completed?
SELECT ONE ONLY
D3:
Are there any children under the age of 18 currently living in your household?
D4:
Which of the following categories best describes your total annual household income, including income from all household members, before taxes are deducted?
SELECT ONE ONLY
D7:
What is the language you first learned at home as a child and still understand?
SELECT UP TO TWO
Closing (PRE-TEST ONLY):
D8:
Are there any questions in this survey that you found difficult to understand?
D9:
IF YES IN D8: Which questions did you find difficult to understand? RECORD VERBATIM
__________________________________________________________________
That concludes the survey. This survey was conducted on behalf of the Canada Revenue Agency. In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to answer this survey, it is greatly appreciated.