2023-2024 CRA Slam the Scam Campaign: Advertising Campaign Evaluation Tool (ACET) Methodologoical Report

Prepared for the Canada Revenue Agency

Submitted by Narrative Research
PSPC Contract Number: CW2348153
Contracted Value: $22,436.15
Award Date: February 1, 2024
Delivery Date: March 28, 2024

Registration Number: POR 127-23
For more information, please contact: cra-arc.media@cra-arc.gc.ca

Ce rapport est aussi disponible en français.

2023-2024 CRA Slam the Scam Campaign: Advertising Campaign Evaluation Tool (ACET)
Methodologoical Report

Prepared for Canada Revenue Agency

Supplier Name: Narrative Research Inc.
March 2024

This report presents the methodological details for the post-campaign wave of the 2023-2024 CRA Slam the Scam Campaign: Advertising Campaign Evaluation Tool (ACET) study conducted by Narrative Research Inc. on behalf of the Canada Revenue Agency (CRA). This is a follow-up to previous post-wave research conducted in 2022. The current post-wave survey was conducted with 1,356 respondents from the Canadian general public aged 18 years and older, between February 15 and March 4, 2024.

Ce rapport est aussi disponible en français sous le titre: Campagne publicitaire de l'ARC de 2023-2024 – à bas l'arnaque : Outil d'évaluation des campagnes publicitaires (OECP) – Rapport méthodologique.

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from the Canada Revenue Agency. For more information on this report, please contact the Canada Revenue Agency at: cra-arc.media@cra-arc.gc.ca or at:

101 Colonel By Drive
Ottawa, Ontario K1A 0K2
Canada

Catalogue Number: Rv4-150/1-2024E-PDF
International Standard Book Number (ISBN): 978-0-660-72623-6

Related publications (registration number: POR 127-23):
Catalogue Number (Final Methodological Report, French): Rv4-150/1-2024F-PDF
ISBN (French): 978-0-660-72624-3

© His Majesty the King in right of Canada, as represented by the Canada Revenue Agency, 2024

Table of Contents

• Executive Summary
• Background
• Research Objectives
• Target Population
• Research Usage
• Expenditure
• Consent
• Study Methodology
• Questionnaire Design
• Sampling
• Survey Administration
• Non-Response Bias Analysis
• Appendix A: Study Questionnaire

Executive Summary

Narrative Research Inc.
Contract Number: CW2348153
POR Registration Number: 127-23
Contract Award Date: February 1, 2024
Contracted Cost: $22,436.15

Background

The Statement of Work ("SOW") for this project states that scams involving the Government of Canada name have grown significantly in the last years, thereby posing a threat to individuals living in Canada regardless of age, gender, socio-economic status, or ethnicity. The SOW indicates that between 1998 and 2020, total reported crime in Canada decreased by 17%, whereas reported fraud increased by 46%. Fraud can occur using a variety of mediums, such as email, text, telephone and instant messaging, and it is known that fraud is often aimed at vulnerable persons, such as seniors, those new to Canada, and youth. The content of the fraud can range from false or misleading service advertisements, to tax refund or payment notices and requests for personal and financial information, among various other types.

The SOW also notes that the Government of Canada continues to work with a wide array of partners to ensure scam warnings are widely distributed throughout the country. Specifically, the Canada Revenue Agency (CRA), the Competition Bureau, the Royal Canadian Mounted Police (RCMP), and the Canadian Anti-Fraud Centre (CAFC) work together to provide awareness events concerning scams. The government also regularly gives interviews and issues news releases to print, radio, and television media organizations to inform individuals about how to recognize and avoid common scams in Canada.

To continue to raise awareness of scams involving the Government of Canada name and with the collaboration of other departments, the CRA has led a sustained multimedia scams campaign over the last four years. The campaign launched in Fall 2020 and a pre-wave advertising recall survey was conducted prior to the launch of the campaign, with post-wave surveys following since that time. The most recent post-wave survey was conducted in February-March 2024.

This research using the Government of Canada's Advertising Campaign Evaluation Tool (ACET) is mandatory, given that the Treasury Board requires all Government of Canada advertising campaigns that cost more than one million dollars to be evaluated.

Research Objectives

The purpose of the quantitative research was to assess recall of and reactions to the advertising campaign among three specific populations: Newcomers, Seniors, and Youth. At the highest level, the purpose of the research is to increase awareness of the advertising campaign. AS well, the study was designed to capture information concerning via which channels population members recall seeing or hearing the scams-related ads.

More specifically, the research objectives included:

• The aided and unaided awareness of the CRA advertisements and general Government of Canada advertising;
• The message recall;
• Whether the advertisements were: attention catching, relevant, difficult to follow, about an important topic, provide new information; and
• Aided and unaided awareness of the subject matter.

Target Population

As specified in the Statement of Work (SOW), the target audience of the 2024 CRA Scams Campaign was specific groups in the country: Newcomers to Canada over the past ten years (immigrants, refugees and individuals seeking Canadian citizenship) who are 18 years of age or older (minimum of n=400), seniors (age 60+) (minimum of n=400), and youth (ages 18-34) (minimum of n=400). Data for this post-wave survey was gathered from a general public panel modelling key demographics of Canadians in these various segments (18-34 year-olds referred to as Youth; Newcomers; and Seniors who are 60 years of age or older).

This post-campaign wave online survey was undertaken from February 15 to March 4, 2024. It required an average of approximately eight minutes to administer overall, with the English version taking just over eight minutes, on average, and the French version taking just over seven minutes, on average. There was a total of 1,356 useable surveys completed in this most recent post-wave. The email contact records for this research were drawn from panelists administered by The Logit Group of Toronto, Ontario. Given that this online survey methodology used a non-probability sample, the data collected cannot be extrapolated to the relevant comparator segments of the Canadian general public adult population.

Research Usage

As stated in the project's Statement of Work and related communications, this post-wave research activity aimed at measuring the target audience's post recall and reaction concerning the campaign.

Specifically, this research assessed unaided and aided awareness of the CRA advertisements and general Government of Canada advertising, the message recall, perceptions of the advertisements (i.e., whether the ads were attention catching, relevant, difficult to follow, about an important topic, provided new information, and so forth), and aided and unaided awareness of the subject matter.

Expenditure

The project expenditure was $22,436.15, including HST, for the current post-wave survey.

Consent

Narrative Research offers this written consent allowing the Librarian and Archivist of Canada to post, in both official languages, this Methodological Report.

Political Neutrality Statement and Contact Information

I hereby certify as a Senior Officer of Narrative Research that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not contain any reference to electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leader.

Peter MacIntosh
Chief Research Officer
Narrative Research
pmacintosh@narrativeresearch.ca
902-493-3832

Study Methodology

This evaluation utilized the Government of Canada's Advertising Campaign Evaluation Tool (ACET) and was administered to a sample of: 1) Newcomers to Canada over the past ten years (immigrants, refugees and individuals seeking Canadian citizenship) who are 18 years of age or older (minimum of n=400); 2) Seniors (ages 60+) (minimum of n=400), and 3) Youth (ages 18-34) (minimum of n=400). The online-based data collection regimen entailed post-advertising campaign data collection. The data collection was conducted between February 15 to March 4, 2024.

Questionnaire Design

The questions utilized in this study were based on the Government of Canada's standard Advertising Campaign Evaluation Tool questionnaire. In this post-campaign wave the respondents were exposed to five campaign components: three videos, one radio, and one gif, and were subsequently asked a series of questions about the advertisements. This aiding of respondents by showing ads drawn from the advertising campaign is a process that is possible with an online survey methodology. Narrative Research ensured that respondents were able to complete the survey on various platforms including computers, tablets or smartphones.

As required by Government of Canada standards, English and French pre-test surveys were collected. As well, a line of questioning was included at the end of the pre-test surveys in which respondents were asked if they encountered any questions or survey wording that was difficult to understand. For the current survey iteration, no pre-test respondents expressed difficulty in understanding any of the survey questions. As a result, no pre-test respondent was asked to identify which question or questions were problematic from a comprehension perspective.

Sampling

The survey approach utilized was designed to be administered to an online sample of approximately 1,200 adults with known target populations of 1) Newcomers to Canada over the past ten years (immigrants, refugees and individuals seeking Canadian citizenship) who are 18 years of age or older (minimum of n=400); 2) Seniors (age 60+) (minimum of n=400), and 3) Youth (ages 18-34).

Narrative Research ensured that the surveys collected for the 18-34 and 60+ age target groups closely reflected the actual, true Canadian general population in terms of gender and age group for Canadians 18-34 years of age, and 60 years of age or older, by region, as required by the project's Statement of Work. Newcomers were accepted in the data set if they were 18 years of age or older, and if they moved to Canada at any point in the past ten years.

Contact Records Source

Narrative Research utilized the services of The Logit Group for this research. The Logit Group's online general population panel is comprised of over 600,000 Canadian residents nationally, with sound representation across regions. Logit Group panelists are recruited from a large number of sources to maximize reach and representation. The recruitment policies of The Logit Group's partners (SSI, Toluna, Asking Canadians, and Research Now) are broad in scope.

Survey data quality rests on many different factors, including sourcing of panelists who are vetted, using ongoing quality checks such as eliminating panelists who are no longer active, and so forth. The following are panel member sources for Logit Group studies:

• Email invitations: pre-authorized to opt in lists from associations and groups
• Social Media: advertising and social groups on leading social media platforms
• Media Platforms: advertising on online media platforms both niche and mainstream
• Use of major recruiting brands
• Loyalty programs
• Targeted audiences
• Web and social networking sites
• Targeted emails by The Logit Group's online partners to their members or subscribers
• Referral programs

The Logit Group has established a variety of quality assurance processes to proactively identify invalid respondents. For example, the company has incorporated methods to quickly identify and flag straight-lining speedsters (i.e., respondents who give the same responses to all questions as a means of quickly finishing the survey), thereby monitoring whether panelists are able to provide thoughtful and accurate responses to survey queries.

Panel members are monitored against Statistics Canada data to gauge statistical representation. Annual profile refreshing campaigns are conducted to incentivize panelists to remain active; these can also contain new questions in order to target specific niche audiences more precisely. Panelists' participation is rewarded with their choice of HBC Rewards bonus points, Aeroplan Miles or Petro Points, as well as various prizes. The sampling procedure reflected a computerized randomization of online panel members, with exclusions from the randomization process being based upon, for example, whether a panelist had received his/her monthly maximum number of survey invitations.

Survey Administration

Survey Programming and Testing

The online survey was programmed by Narrative Research in both English and French, using Voxco Acuity programming software. Respondents were able to complete the survey in the official language of their choice. As well, at any point when completing the questionnaire, respondents had the option to change the questionnaire language to the other official language. Assistance in completing the survey was available from bilingual Narrative Research staff, as required. Respondents were able to verify the legitimacy of the survey via representatives from Narrative Research, or via the survey registration system made available by the Canadian Research Insights Council (CRIC), Canada's national research agency for this sector.

The programmed surveys were tested to ensure that question order and skip patterns were properly implemented. Testing included Narrative Research researchers ensuring the accuracy of the survey delivery, text, links, and so on. CRA staff were also provided with the link for this post- wave survey.

A total of 45 English and 16 French pre-tests were completed in February 2024 for the post-wave survey. Pre-test survey completions were conducted via a survey "soft launch" whereby a small number of panel respondents were invited to participate in the survey. The pre-testing of the surveys allowed the collected data to be reviewed to ensure accuracy and to identify any programming aspects that should be modified. Pre-test respondents were asked if they had any difficulty understanding any aspect of the surveys. No one replied in the affirmative. Thus, no substantive data quality issues arose as a result of the pre-test, and thus the pre-test data was maintained in the final data set.

Data Collection

This study consisted of a post-campaign wave that was administered between February 15 to March 4, 2024. Fieldwork was monitored and reviewed on a regular basis to ensure target quotas were being met. Narrative Research provided regular reports to CRA representatives regarding progress, as requested or pre-determined. No individual was able to complete the survey questionnaire more than once.

A total of 1,356 collected surveys were ultimately used in the final data set for this post-wave survey. It is important to note that for various reasons, a small percentage of submitted online panel surveys is often removed from study data sets after submission. Reasons for removing surveys ultimately included those that were over quota and respondents who consistently provided non-intelligible verbatim responses, duplicate IDs, and so forth.

Moreover, in this wave, Narrative Research's initial quota target exceeded the overall final requirement of 1,200 questionnaires, as an effort was made to complete a diverse array of respondents across relevant age (sub-)groups of each targeted population. That is, Narrative Research collected more than the initially earmarked number of surveys. The surveys in this post-campaign survey required a mean average of approximately eight minutes to complete (8.3 minutes in English and 7.2 minutes in French).

A non-probability sample approach was implemented given that the study was designed to be conducted among online Canadian general public panelists. All such panels are inherently non-probability in nature, given that panelists self-select to become members of such panels, and not all adult Canadians belong to such a panel.

The table below for the post-campaign survey displays regional, gender, age, and Newcomer data in terms of the respondents to this survey.

Post-Campaign Wave (February-March 2024)

Percentages may not sum exactly to 100%, owing to rounding:

Completed Survey Rates among Panelists Known to Have Accessed the Survey: In the current version of the Government of Canada's Standards for the Conduct of Government of Canada Public Opinion Research – Online Surveys, Section 1.2.2.7 states that "When the research design or selection procedures will not allow the calculation of a response rate or participation rate (for example, when a sample router is used to screen potential respondents and assign them to one of several surveys), describe the design/ selection procedure and any potential sampling bias that may result." As mentioned above, the current CRA Public Opinion Research study utilized the online general public panel of The Logit Group. This panel indeed uses the routing approach for panel research of the type referenced in the Government of Canada's Standards document.

Beyond the quality control measures discussed elsewhere in these standards that are employed by Narrative Research and The Logit Group, it is evident from the measures referenced in the Survey Rate formula developed below that other checks also have been introduced to enhance the quality of the study's final data set. Specifically, surveys were removed owing to captcha fails, international IP addresses, survey speeders, flatliners, and so forth.

In addition, as mentioned above, to augment the representativeness of the final data set, quotas were introduced in terms of age, gender, and region, ensuring the final data set matches as closely as possible the survey targets along those dimensions.

Please note, the Completed Survey Rate among Panelists Known to Have Accessed the Survey measure that is calculated below has been derived by Narrative Research. This is an alternate completion measure that is distinct from the standard Participation Rate formula mentioned in the Public Opinion Research Standards document (and which is used when panelists are emailed survey invitations).

Given that the online methodology utilized a non-probability sample, a margin of error cannot be applied to the results as per the Standards for the Conduct of Government of Canada Public Opinion Research for Online Surveys.

Data Tabulation: A total of 90 potential overlapping or interlocking statistical weighting cells were created from the study design using the weighting factors of: Region (5: Atlantic, Quebec, Ontario, Prairies, and British Columbia/North); Age group (3: 18–34, 35–59, 60+); Gender (2: Men, Women); and Immigrant Status (3: Non-immigrant, Recent immigrant (within the past 10 years), and Non-recent immigrant). The 90 potential overlapping or interlocking statistical weighting cells thus were derived from Region (5) x Age (3) x Gender (2) x Immigrant Status (3) dimensions = 90 potential unique statistical weighting cells. Note that the study design did not include residents aged 35-59 who were not recent immigrants, and as a result the population data for these individuals was similarly excluded (i.e., the population data for those 20 cells was set to 0).

Thus, the final weighting scheme developed for this study utilized 70 of the 90 potential weighting cells, corresponding to Region (5) x Age group (2) x Gender (2) x Immigrant Status (3) = 60 unique weighting cells (for those aged 18-34 or 60+ who were not recent immigrants) , PLUS, Region (5) x Gender (2) = 10 unique weighting cells for 35-59 year old, recent immigrants. Population data for the 90 statistical weighting cells were obtained from the most recent (2021) Census of Canada, and can be found here:

• https://www150.statcan.gc.ca/t1/tbl1/en/tv.action?pid=9810034701

With such a large number of weighting cells, it was necessary to combine specific cells due to the fact that sample sizes for specific cells were small or empty, and therefore would have led to quite large weights if left separate. Combining weighting cells is a common approach in such instances, and explains why the final weighted data distribution differs slightly from the actual population distribution, along certain dimensions.

Non-Response Bias Analysis

Any survey that is conducted is potentially subject to bias or error. When a survey is conducted with a sample of the population, there are two general classes of bias or error: sampling error, which is quantifiable, and non-sampling error, which is typically not quantifiable. Sampling error necessarily arises from the fact that surveys are administered to only a subset of the targeted population, and thus is it possible that the survey results obtained from this group of respondents is not reflective of the population as a whole.

In contrast, non-sampling error encompasses a number of different types of errors including coverage error, measurement error, non-response error, and processing error. No measurement of sampling error can be attributed to the current study, given that the contact records utilized in the data collection process were derived from an online panel of the general public, which is to say, a non-probability sample source. Having stated that, measures were taken in the implementation of the data collection to ensure sufficient completed surveys were obtained from the target groups traditionally regarded as central in quantitative survey research, such as gender, age, region/province, and immigration status.

With respect to non-sampling error, a number of steps were taken to minimize bias due to these sources. All surveys utilized online interviewing technology to ensure proper survey skip patterns were followed and to minimize errors due to data entry and data capture. The French and English survey instruments from each campaign themselves were pre-tested with a small sample of respondents to ensure the survey material was easily understood by respondents, and that the resultant data were being captured properly. In terms of coverage, the surveys were conducted with an online panel of the Canadian general public 18 years of age or older, based on a randomized sampling of panel records for the target audience drawn from a commercially available online general public panel.

Appendix A: Study Questionnaire

ADVERTISING CAMPAIGN EVALUATION TOOL POST-CAMPAIGN SURVEY

INTRODUCTION [ASK IN BASELINE AND POST-CAMPAIGN SURVEYS]

Thank you for taking a few minutes to complete this survey on current issues that matter to Canadians. Si vous préférez répondre à ce sondage en français, veuillez cliquer sur français [SWITCH TO FRENCH VERSION].

Your participation is voluntary and your responses will be kept entirely confidential.
The survey takes about 7 minutes to complete.

START SURVEY

Click here if you wish to verify the authenticity of this survey. To view our privacy policy, click here.

If you require any technical assistance, please contact XXX.

a) Does anyone in your household work for any of the following organizations?

SELECT ALL THAT APPLY

IF "NONE OF THESE ORGANIZATIONS" CONTINUE, OTHERWISE THANK AND TERMINATE.

b) What is your gender?

c) In what year were you born?

YYYY

ADMISSIBLE RANGE 1900-2006
IF > 2006, THANK AND TERMINATE
ASK D IF QUESTION C IS LEFT BLANK

d) In which of the following age categories do you belong?

SELECT ONE ONLY

IF "LESS THAN 18 YEARS OLD" OR "BLANK", THANK AND TERMINATE
e) Where were you born?

=>Specify the country:

ASK IF e=BORN OUTSIDE CANADA

f) In what year did you first move to Canada?

YYYY

ADMISSIBLE RANGE: 1900-2024

g) In which province or territory do you live? SELECT ONE ONLY

IF NO PROVINCE OR TERRITORY IS SELECTED, THANK AND TERMINATE

CORE QUESTIONS [ASK IN BASELINE AND POST-CAMPAIGN SURVEYS]

ASK ALL RESPONDENTS

Q1:

Over the past three weeks, have you seen, read or heard any advertising from the Government of Canada?

Q2:

Think about the most recent Government of Canada ad that comes to mind. What do you remember about this ad?

CAMPAIGN-SPECIFIC QUESTIONS [ASK IN BASELINE AND POST-CAMPAIGN SURVEYS]

ASK ALL RESPONDENTS

T1A:

Over the past three weeks, have you seen, read or heard any Government of Canada advertising about how to recognize scams?

T1B:

Where have you seen, read or heard this ad about how to recognize scams?

SELECT ALL THAT APPLY

T1C:

What do you remember about this ad?

ASK ALL RESPONDENTS

CAMPAIGN SPECIFIC ATTITUDINAL AND BEHAVIOURAL QUESTIONS MAY BE ADDED HERE. (T1D: T1E: ETC)

T1D: In the last six months, have you received a message or a call that you thought was a scam?

ASK IF T1D=YES

T1E: Was it through: SELECT ALL THAT APPLY

ASK IF T1D=YES

T1F:

Was the message/subject matter about: SELECT ALL THAT APPLY

T1G:

Have you ever been a victim of a scam similar to the ones mentioned previously?

ASK IF T1G=YES

T1G2:

What was the nature of the scam?

AD RECALL QUESTIONS [ASK IN POST-CAMPAIGN SURVEY ONLY]

T1H:

Next are some ads that have recently been broadcast on various media. Click 'Next' to view each ad..

[INSERT VIDEO, AND RADIO ADS]

[CLICK TO GO TO THE NEXT PAGE]

Over the past three weeks, have you seen, read or heard these ads?

T1I:

Where have you seen, read or heard these ads?

SELECT ALL THAT APPLY

T1J:

What do you think is the main point these ads are trying to get across?

T1K:

Please indicate your level of agreement with the following statements about these ads?

RANDOMIZE STATEMENTS

DEMOGRAPHIC QUESTIONS [ASK IN BASELINE AND POST-CAMPAIGN SURVEYS]

D1:

Which of the following categories best describes your current employment status? Are you…

SELECT ONE ONLY

D2:

What is the highest level of formal education that you have completed?

SELECT ONE ONLY

D3:

Are there any children under the age of 18 currently living in your household?

D4:

Which of the following categories best describes your total annual household income, including income from all household members, before taxes are deducted?

SELECT ONE ONLY

D5:

What is the language you first learned at home as a child and still understand?

SELECT UP TO TWO

That concludes the survey. This survey was conducted on behalf of [INSERT DEPARTMENT/AGENCY]. In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to answer this survey, it is greatly appreciated.