Prepared for the Canada Revenue Agency
Submitted by Narrative Research
PSPC Contract Number: CW2348153
Contracted Value: $22,436.15
Award Date: February 1, 2024
Delivery Date: March 28, 2024
Registration Number: POR 127-23
For more information, please contact: cra-arc.media@cra-arc.gc.ca
Ce rapport est aussi disponible en français.
Supplier Name: Narrative Research Inc.
March 2024
This report presents the methodological details for the post-campaign wave of the 2023-2024 CRA Slam the Scam Campaign: Advertising Campaign Evaluation Tool (ACET) study conducted by Narrative Research Inc. on behalf of the Canada Revenue Agency (CRA). This is a follow-up to previous post-wave research conducted in 2022. The current post-wave survey was conducted with 1,356 respondents from the Canadian general public aged 18 years and older, between February 15 and March 4, 2024.
Ce rapport est aussi disponible en français sous le titre: Campagne publicitaire de l'ARC de 2023-2024 – à bas l'arnaque : Outil d'évaluation des campagnes publicitaires (OECP) – Rapport méthodologique.
This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from the Canada Revenue Agency. For more information on this report, please contact the Canada Revenue Agency at: cra-arc.media@cra-arc.gc.ca or at:
101 Colonel By Drive
Ottawa, Ontario K1A 0K2
Canada
Catalogue Number: Rv4-150/1-2024E-PDF
International Standard Book Number (ISBN): 978-0-660-72623-6
Related publications (registration number: POR 127-23):
Catalogue Number (Final Methodological Report, French): Rv4-150/1-2024F-PDF
ISBN (French): 978-0-660-72624-3
© His Majesty the King in right of Canada, as represented by the Canada Revenue Agency, 2024
Narrative Research Inc.
Contract Number: CW2348153
POR Registration Number: 127-23
Contract Award Date: February 1, 2024
Contracted Cost: $22,436.15
The Statement of Work ("SOW") for this project states that scams involving the Government of Canada name have grown significantly in the last years, thereby posing a threat to individuals living in Canada regardless of age, gender, socio-economic status, or ethnicity. The SOW indicates that between 1998 and 2020, total reported crime in Canada decreased by 17%, whereas reported fraud increased by 46%. Fraud can occur using a variety of mediums, such as email, text, telephone and instant messaging, and it is known that fraud is often aimed at vulnerable persons, such as seniors, those new to Canada, and youth. The content of the fraud can range from false or misleading service advertisements, to tax refund or payment notices and requests for personal and financial information, among various other types.
The SOW also notes that the Government of Canada continues to work with a wide array of partners to ensure scam warnings are widely distributed throughout the country. Specifically, the Canada Revenue Agency (CRA), the Competition Bureau, the Royal Canadian Mounted Police (RCMP), and the Canadian Anti-Fraud Centre (CAFC) work together to provide awareness events concerning scams. The government also regularly gives interviews and issues news releases to print, radio, and television media organizations to inform individuals about how to recognize and avoid common scams in Canada.
To continue to raise awareness of scams involving the Government of Canada name and with the collaboration of other departments, the CRA has led a sustained multimedia scams campaign over the last four years. The campaign launched in Fall 2020 and a pre-wave advertising recall survey was conducted prior to the launch of the campaign, with post-wave surveys following since that time. The most recent post-wave survey was conducted in February-March 2024.
This research using the Government of Canada's Advertising Campaign Evaluation Tool (ACET) is mandatory, given that the Treasury Board requires all Government of Canada advertising campaigns that cost more than one million dollars to be evaluated.
The purpose of the quantitative research was to assess recall of and reactions to the advertising campaign among three specific populations: Newcomers, Seniors, and Youth. At the highest level, the purpose of the research is to increase awareness of the advertising campaign. AS well, the study was designed to capture information concerning via which channels population members recall seeing or hearing the scams-related ads.
More specifically, the research objectives included:
As specified in the Statement of Work (SOW), the target audience of the 2024 CRA Scams Campaign was specific groups in the country: Newcomers to Canada over the past ten years (immigrants, refugees and individuals seeking Canadian citizenship) who are 18 years of age or older (minimum of n=400), seniors (age 60+) (minimum of n=400), and youth (ages 18-34) (minimum of n=400). Data for this post-wave survey was gathered from a general public panel modelling key demographics of Canadians in these various segments (18-34 year-olds referred to as Youth; Newcomers; and Seniors who are 60 years of age or older).
This post-campaign wave online survey was undertaken from February 15 to March 4, 2024. It required an average of approximately eight minutes to administer overall, with the English version taking just over eight minutes, on average, and the French version taking just over seven minutes, on average. There was a total of 1,356 useable surveys completed in this most recent post-wave. The email contact records for this research were drawn from panelists administered by The Logit Group of Toronto, Ontario. Given that this online survey methodology used a non-probability sample, the data collected cannot be extrapolated to the relevant comparator segments of the Canadian general public adult population.
As stated in the project's Statement of Work and related communications, this post-wave research activity aimed
at measuring the target audience's post recall and reaction concerning the campaign.
Specifically, this research assessed unaided and aided awareness of the CRA advertisements and general Government of
Canada advertising, the message recall, perceptions of the advertisements (i.e., whether the ads were attention
catching, relevant, difficult to follow, about an important topic, provided new information, and so forth), and aided
and unaided awareness of the subject matter.
The project expenditure was $22,436.15, including HST, for the current post-wave survey.
Narrative Research offers this written consent allowing the Librarian and Archivist of Canada to post, in both official languages, this Methodological Report.
I hereby certify as a Senior Officer of Narrative Research that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not contain any reference to electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leader.
Peter MacIntosh
Chief Research Officer
Narrative Research
pmacintosh@narrativeresearch.ca
902-493-3832
This evaluation utilized the Government of Canada's Advertising Campaign Evaluation Tool (ACET) and was administered to a sample of: 1) Newcomers to Canada over the past ten years (immigrants, refugees and individuals seeking Canadian citizenship) who are 18 years of age or older (minimum of n=400); 2) Seniors (ages 60+) (minimum of n=400), and 3) Youth (ages 18-34) (minimum of n=400). The online-based data collection regimen entailed post-advertising campaign data collection. The data collection was conducted between February 15 to March 4, 2024.
The questions utilized in this study were based on the Government of Canada's standard Advertising Campaign Evaluation Tool questionnaire. In this post-campaign wave the respondents were exposed to five campaign components: three videos, one radio, and one gif, and were subsequently asked a series of questions about the advertisements. This aiding of respondents by showing ads drawn from the advertising campaign is a process that is possible with an online survey methodology. Narrative Research ensured that respondents were able to complete the survey on various platforms including computers, tablets or smartphones.
As required by Government of Canada standards, English and French pre-test surveys were collected. As well, a line of questioning was included at the end of the pre-test surveys in which respondents were asked if they encountered any questions or survey wording that was difficult to understand. For the current survey iteration, no pre-test respondents expressed difficulty in understanding any of the survey questions. As a result, no pre-test respondent was asked to identify which question or questions were problematic from a comprehension perspective.
The survey approach utilized was designed to be administered to an online sample of approximately 1,200 adults with known target populations of 1) Newcomers to Canada over the past ten years (immigrants, refugees and individuals seeking Canadian citizenship) who are 18 years of age or older (minimum of n=400); 2) Seniors (age 60+) (minimum of n=400), and 3) Youth (ages 18-34).
Narrative Research ensured that the surveys collected for the 18-34 and 60+ age target groups closely reflected the actual, true Canadian general population in terms of gender and age group for Canadians 18-34 years of age, and 60 years of age or older, by region, as required by the project's Statement of Work. Newcomers were accepted in the data set if they were 18 years of age or older, and if they moved to Canada at any point in the past ten years.
Narrative Research utilized the services of The Logit Group for this research. The Logit Group's online general population panel is comprised of over 600,000 Canadian residents nationally, with sound representation across regions. Logit Group panelists are recruited from a large number of sources to maximize reach and representation. The recruitment policies of The Logit Group's partners (SSI, Toluna, Asking Canadians, and Research Now) are broad in scope.
Survey data quality rests on many different factors, including sourcing of panelists who are vetted, using ongoing quality checks such as eliminating panelists who are no longer active, and so forth. The following are panel member sources for Logit Group studies:
The Logit Group has established a variety of quality assurance processes to proactively identify invalid respondents. For example, the company has incorporated methods to quickly identify and flag straight-lining speedsters (i.e., respondents who give the same responses to all questions as a means of quickly finishing the survey), thereby monitoring whether panelists are able to provide thoughtful and accurate responses to survey queries.
Panel members are monitored against Statistics Canada data to gauge statistical representation. Annual profile refreshing campaigns are conducted to incentivize panelists to remain active; these can also contain new questions in order to target specific niche audiences more precisely. Panelists' participation is rewarded with their choice of HBC Rewards bonus points, Aeroplan Miles or Petro Points, as well as various prizes. The sampling procedure reflected a computerized randomization of online panel members, with exclusions from the randomization process being based upon, for example, whether a panelist had received his/her monthly maximum number of survey invitations.
The online survey was programmed by Narrative Research in both English and French, using Voxco Acuity programming software. Respondents were able to complete the survey in the official language of their choice. As well, at any point when completing the questionnaire, respondents had the option to change the questionnaire language to the other official language. Assistance in completing the survey was available from bilingual Narrative Research staff, as required. Respondents were able to verify the legitimacy of the survey via representatives from Narrative Research, or via the survey registration system made available by the Canadian Research Insights Council (CRIC), Canada's national research agency for this sector.
The programmed surveys were tested to ensure that question order and skip patterns were properly implemented. Testing included Narrative Research researchers ensuring the accuracy of the survey delivery, text, links, and so on. CRA staff were also provided with the link for this post- wave survey.
A total of 45 English and 16 French pre-tests were completed in February 2024 for the post-wave survey. Pre-test survey completions were conducted via a survey "soft launch" whereby a small number of panel respondents were invited to participate in the survey. The pre-testing of the surveys allowed the collected data to be reviewed to ensure accuracy and to identify any programming aspects that should be modified. Pre-test respondents were asked if they had any difficulty understanding any aspect of the surveys. No one replied in the affirmative. Thus, no substantive data quality issues arose as a result of the pre-test, and thus the pre-test data was maintained in the final data set.
This study consisted of a post-campaign wave that was administered between February 15 to March 4, 2024. Fieldwork was monitored and reviewed on a regular basis to ensure target quotas were being met. Narrative Research provided regular reports to CRA representatives regarding progress, as requested or pre-determined. No individual was able to complete the survey questionnaire more than once.
A total of 1,356 collected surveys were ultimately used in the final data set for this post-wave survey. It is important to note that for various reasons, a small percentage of submitted online panel surveys is often removed from study data sets after submission. Reasons for removing surveys ultimately included those that were over quota and respondents who consistently provided non-intelligible verbatim responses, duplicate IDs, and so forth.
Moreover, in this wave, Narrative Research's initial quota target exceeded the overall final requirement of 1,200 questionnaires, as an effort was made to complete a diverse array of respondents across relevant age (sub-)groups of each targeted population. That is, Narrative Research collected more than the initially earmarked number of surveys. The surveys in this post-campaign survey required a mean average of approximately eight minutes to complete (8.3 minutes in English and 7.2 minutes in French).
A non-probability sample approach was implemented given that the study was designed to be conducted among online Canadian general public panelists. All such panels are inherently non-probability in nature, given that panelists self-select to become members of such panels, and not all adult Canadians belong to such a panel.
The table below for the post-campaign survey displays regional, gender, age, and Newcomer data in terms of the respondents to this survey.
Percentages may not sum exactly to 100%, owing to rounding:
| Region | Quota Targets – Surveys (n=) |
Quota Targets – Surveys (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
|---|---|---|---|---|
| Atlantic | 125 | 10.4% | 91 | 6.7% |
| Quebec | 300 | 25.0% | 388 | 28.6% |
| Ontario | 400 | 33.3% | 479 | 35.3% |
| Prairies | 175 | 14.6% | 252 | 18.6% |
| British Columbia/North | 200 | 16.7% | 146 | 10.8% |
| Gender [1] | Quota Targets – Surveys (n=) |
Quota Targets – Surveys (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
| Male | 600 | 50.0% | 597 | 44.0% |
| Female | 600 | 50.0% | 753 | 55.5% |
| Age (Quotas) | Quota Targets – Surveys (n=) |
Quota Targets – Surveys (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
| 18-34 | 525 | 43.7% | 729 | 53.8% |
| 35-59 | 175 | 14.6% | 167 | 12.3% |
| 60+ | 500 | 41.7% | 460 | 33.9% |
Completed Survey Rates among Panelists Known to Have Accessed the Survey: In the current version of the Government of Canada's Standards for the Conduct of Government of Canada Public Opinion Research – Online Surveys, Section 1.2.2.7 states that "When the research design or selection procedures will not allow the calculation of a response rate or participation rate (for example, when a sample router is used to screen potential respondents and assign them to one of several surveys), describe the design/ selection procedure and any potential sampling bias that may result." As mentioned above, the current CRA Public Opinion Research study utilized the online general public panel of The Logit Group. This panel indeed uses the routing approach for panel research of the type referenced in the Government of Canada's Standards document.
Beyond the quality control measures discussed elsewhere in these standards that are employed by Narrative Research and The Logit Group, it is evident from the measures referenced in the Survey Rate formula developed below that other checks also have been introduced to enhance the quality of the study's final data set. Specifically, surveys were removed owing to captcha fails, international IP addresses, survey speeders, flatliners, and so forth.
In addition, as mentioned above, to augment the representativeness of the final data set, quotas were introduced in terms of age, gender, and region, ensuring the final data set matches as closely as possible the survey targets along those dimensions.
Please note, the Completed Survey Rate among Panelists Known to Have Accessed the Survey measure that is calculated below has been derived by Narrative Research. This is an alternate completion measure that is distinct from the standard Participation Rate formula mentioned in the Public Opinion Research Standards document (and which is used when panelists are emailed survey invitations).
| Number of panelists who commenced the survey process (total clicks): | 3,604 |
| Invalid cases (outside of target population; sensitive occupations): | 494 |
| Blocked from process for quality reasons (e.g., invalid IP address/international; captcha fails; professional survey takers; etc.): | 89 |
| Incomplete surveys: | 1,091 |
| Quota full (age/gender/region): | 499 |
| Completed survey but removed for quality reasons (e.g., speeders; flatliners, etc.): | 75 |
| Completed surveys/Qualified completes: | 1,356 |
| Post-Campaign Wave Completed Survey Rate among Panelists Known to Have Accessed the Survey = 1,356/3,604 = 37.6% |
|
Given that the online methodology utilized a non-probability sample, a margin of error cannot be applied to the results as per the Standards for the Conduct of Government of Canada Public Opinion Research for Online Surveys.
Data Tabulation: A total of 90 potential overlapping or interlocking statistical weighting cells were created from the study design using the weighting factors of: Region (5: Atlantic, Quebec, Ontario, Prairies, and British Columbia/North); Age group (3: 18–34, 35–59, 60+); Gender (2: Men, Women); and Immigrant Status (3: Non-immigrant, Recent immigrant (within the past 10 years), and Non-recent immigrant). The 90 potential overlapping or interlocking statistical weighting cells thus were derived from Region (5) x Age (3) x Gender (2) x Immigrant Status (3) dimensions = 90 potential unique statistical weighting cells. Note that the study design did not include residents aged 35-59 who were not recent immigrants, and as a result the population data for these individuals was similarly excluded (i.e., the population data for those 20 cells was set to 0).
Thus, the final weighting scheme developed for this study utilized 70 of the 90 potential weighting cells, corresponding to Region (5) x Age group (2) x Gender (2) x Immigrant Status (3) = 60 unique weighting cells (for those aged 18-34 or 60+ who were not recent immigrants) , PLUS, Region (5) x Gender (2) = 10 unique weighting cells for 35-59 year old, recent immigrants. Population data for the 90 statistical weighting cells were obtained from the most recent (2021) Census of Canada, and can be found here:
With such a large number of weighting cells, it was necessary to combine specific cells due to the fact that sample sizes for specific cells were small or empty, and therefore would have led to quite large weights if left separate. Combining weighting cells is a common approach in such instances, and explains why the final weighted data distribution differs slightly from the actual population distribution, along certain dimensions.
Any survey that is conducted is potentially subject to bias or error. When a survey is conducted with a sample of the population, there are two general classes of bias or error: sampling error, which is quantifiable, and non-sampling error, which is typically not quantifiable. Sampling error necessarily arises from the fact that surveys are administered to only a subset of the targeted population, and thus is it possible that the survey results obtained from this group of respondents is not reflective of the population as a whole.
In contrast, non-sampling error encompasses a number of different types of errors including coverage error, measurement error, non-response error, and processing error. No measurement of sampling error can be attributed to the current study, given that the contact records utilized in the data collection process were derived from an online panel of the general public, which is to say, a non-probability sample source. Having stated that, measures were taken in the implementation of the data collection to ensure sufficient completed surveys were obtained from the target groups traditionally regarded as central in quantitative survey research, such as gender, age, region/province, and immigration status.
With respect to non-sampling error, a number of steps were taken to minimize bias due to these sources. All surveys utilized online interviewing technology to ensure proper survey skip patterns were followed and to minimize errors due to data entry and data capture. The French and English survey instruments from each campaign themselves were pre-tested with a small sample of respondents to ensure the survey material was easily understood by respondents, and that the resultant data were being captured properly. In terms of coverage, the surveys were conducted with an online panel of the Canadian general public 18 years of age or older, based on a randomized sampling of panel records for the target audience drawn from a commercially available online general public panel.
[1] Six respondents identified as gender diverse and are not presented in the table.