Inclusive Communities Campaign Creative Testing & Advertising Campaign Evaluation Tool (ACET)

Executive Summary

Prepared for: Immigration, Refugees and Citizenship Canada

Supplier Name: Leger Marketing Inc
Contract Number: B8815-23-0100
Contract Value: $107,143.21 (including HST)
Award Date: 2022-08-08
Delivery Date: 2023-03-29

Registration Number: POR 026-22

For more information on this report, please contact IRCC at:
IRCC.COMMPOR-ROPCOMM.IRCC@cic.gc.ca

Ce rapport est aussi disponible en français.

Copyright

This public opinion research report presents the results of two studies conducted by Léger Marketing Inc. on behalf of Immigration, Refugees and Citizenship Canada (IRCC). The first study consisted in a qualitative research with nine online focus groups and was conducted with 55 Canadians between October 18 and October 20, 2022. The second study was a quantitative study consisting in a baseline survey and a post-campaign survey conducted with 2,502 respondents each.

Cette publication est aussi disponible en français sous le titre : Évaluation des exécutions publicitaires de la campagne Communautés inclusives & Outil d’évaluation de campagnes publicitaires (OECP).

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Immigration, Refugees and Citizenship Canada. For more information on this report, please contact Immigration, Refugees and Citizenship Canada at IRCC.COMMPOR-ROPCOMM.IRCC@cic.gc.ca or at:

Communications Branch
Immigration, Refugees and Citizenship Canada
Jean Edmonds Tower South
365 Laurier Ave W
Ottawa ON K1A 1L1

Catalogue Number: Ci4-248/2023E-PDF
International Standard Book Number (ISBN): 978-0-660-67843-6

Related publications (registration number: POR 026-22):
Catalogue Number: Ci4-248/2023F-PDF (Final Report, French)
ISBN: 978-0-660-67844-3

© His Majesty the King in Right of Canada, as represented by the Minister of Immigration, Refugees and Citizenship Canada, 2023

Executive Summary

Leger is pleased to present IRCC with this report on findings from virtual discussion groups which aimed to collect the participants' impressions and comments on three advertising concepts. The report also details the methodology of the quantitative survey designed to evaluate IRCC’s Inclusive Communities campaign using the Advertising Campaign Evaluation Tool (ACET). This report was prepared by Léger Marketing Inc. who was contracted by IRCC (contract number B8815-23-0100 awarded August 8, 2022).

Background and Objectives

The Inclusive Communities campaign first launched in 2020-21. This year’s advertising campaign aimed to encourage Canadians to engage with immigrants, understand their contributions to the Canadian economy and local communities and get involved in welcoming newcomers into their communities for everyone’s benefit. The tactics used included digital, search engine marketing (SEM), audio ads (e.g., radio and Spotify), print ads, and out-of-home ads.

By being inclusive, communities can help newcomers (immigrants and refugees) feel welcomed. Demonstrating the benefits of immigration at the local, community level, dispelling common myths about immigration and refugees, and promoting positive engagement between Canadians and newcomers is vital to ensuring that newcomers are welcomed, and communities are inclusive.

The main objective of the qualitative phase of this research was to determine which creatives (and elements of those creatives) most positively resonate with the target audience. The feedback from this research provided guidance to IRCC on how best to construct effective communications and marketing material which will increase awareness of the advertising campaign and increased intention or action to welcome newcomers. Specifically, the creative testing aimed to:

• Determine which creatives the target audience prefer and what gets their attention.
• Identify which calls to action and key messages are the most effective.
• Test specific aspects and variations of creative concepts (colours, fonts, patterns, pictures, etc.).
• Provide feedback which will guide the development and implementation of communications with the target audience.

The main objective of the quantitative study was to measure awareness, recall, and impact of the ads among the target audience. The surveys used the Government of Canada’s Advertising Campaign Evaluation Tool and consisted of two waves: a baseline wave conducted before the campaign launched and a wave after the campaign had been running for at least three weeks. The ACET assessed:

• Unaided and aided recall of the ads
• Awareness and use of information channel (i.e. website) promoted in the ads
• The perceived messages of the ads, as well as the effectiveness of the ads in increasing intention or action to welcome newcomers
• Receptivity to government advertising

Methodology

Qualitative Research – Online Focus Groups

Leger conducted a series of nine virtual discussion group sessions with French-speaking and English-speaking Canadians recruited from all the regions in Canada aside from the province of Quebec. The groups were conducted among two targets: Canadians from 30 to 75 years of age, who are likely to engage in community improvement, and established immigrants. Eight participants were recruited by our professional recruiters for each discussion group session (see Appendix 2 under separate cover for screening guide).

The main purpose of the focus groups was to test three different ad concepts and to collect comments and impressions (see Appendix 2 under separate cover for tested materiel). Each group saw the three concepts in a different order to reduce fatigue bias. A total of 55 recruits participated in the virtual discussion groups. All participants received an honorarium of $100.

The virtual discussion group sessions were approximately 60 minutes in duration and were conducted by a moderator using the CMNTY online platform. The choice of the platform helped to facilitate the moderation, ensure an optimal interface between moderator and participants, and enable interaction as the discussion unfolded. The platform also allowed the moderator to share advertising concepts with participants to lead each part of the discussion. A polling question on the preferred advertising concept was also integrated. The online platform also allowed for remote viewing of each session by Leger and IRCC observers.

All focus group sessions were moderated and supervised by a senior Leger researcher assisted by a research analyst. The discussion guide (see Appendix 2 under separate cover) consisted of a semi-structured discussion guide. It allowed the moderator to follow the thread of the discussion while leaving sufficient room for the participants to express themselves and develop in detail their ideas, opinions and perceptions.

Quantitative Research – Advertising Campaign Evaluation Tool (ACET)

Quantitative research was conducted through online surveys, using Computer Aided Web Interviewing (CAWI) technology. The questionnaires used for both phases of the study are the Government of Canada ACET questionnaires. Leger Marketing Inc. used these questionnaires without making any modifications to their format nor to their content, except for the modifications necessary to adapt the questions to this present study on the Inclusive Communities ad campaign requested by Immigrations, Refugees and Citizenship Canada.

The targeted audience of this study included a national sample reflecting the 30-75 year old Canadian population outside the Province of Quebec. Both the baseline and post-campaign surveys were conducted with a sample of around 2,500 Canadian adults. The respondents were recruited via Leger’s web panel using a random selection stratified by region. The general adult population sample was distributed as follows:

1.2.1 Baseline Study

This public opinion research was conducted via online surveys, using Computer Aided Web Interviewing (CAWI) technology. Fieldwork for the survey was carried out from December 12 to 23, 2022. A total of 2,502 Canadians aged between 30 and 75 living outside the province of Quebec were surveyed. The sample was drawn randomly from the Leo panel and the overall response rate for the survey was 18.5% (see Table 2 in the Appendix for the calculation details).

Using data from the most recent Canadian census, the weighting was done within each region by gender, age, language, presence of children in the household and education to ensure the best possible representativeness of the sample within each region and overall. The weight of each region was adjusted to be equivalent to its actual weight in relation to the distribution of the Canadian population. The weighting factors are presented in detail in the appendix of this report.

A pre-test of 49 interviews was completed before launching data collection to validate the programming of the questionnaire in both English and French. An average of two minutes and thirty one seconds was required for the respondents to complete the survey.

Leger adheres to the most stringent guidelines for quantitative research. The survey instrument was compliant with the Standards of Conduct of Government of Canada Public Opinion Research.

A complete methodological description is provided in the Appendices section of this document (please see Appendix A).

1.2.2 Post-Campaign Study

This public opinion research was conducted via online surveys, using Computer Aided Web Interviewing (CAWI) technology. Fieldwork for the survey was carried out from March 6^th to 24^th, 2023. A total of 2,502 Canadians aged between 30 and 75 living outside the province of Quebec were surveyed. The sample was drawn randomly from the Leo panel and the overall response rate for the survey was 17.8% (see Table 11 in the Appendix for the calculation details).

Using data from the most recent Canadian census, the weighting was done within each region by gender, age, language, presence of children in the household and education to ensure the best possible representativeness of the sample within each region and overall. The weight of each region was adjusted to be equivalent to its actual weight in relation to the distribution of the Canadian population. The weighting factors are presented in detail in the appendix of this report.

A pre-test of 33 interviews was completed before launching data collection to validate the programming of the questionnaire in both English and French. An average of four minutes and fifty-five seconds was required for the respondents to complete the survey.

Leger adheres to the most stringent guidelines for quantitative research. The survey instrument was compliant with the Standards of Conduct of Government of Canada Public Opinion Research.

A complete methodological description is provided in the Appendices section of this document (please see Appendix A).

Overview of Findings – Qualitative Research

Leger recommends moving forwards with the concept: Canadian Welcome. This was the concept that was most appreciated by the participants and that was the easiest to understand. However, some changes should be made to improve the concept. The scene in the airport should be changed, because it did not convince the participants and they did not seem to understand what the government was asking them to do. The images used for social media were also criticized for looking old and unappealing. Nevertheless, the concept was the one that spoke most to the Canadian-born and immigrant participants.

Based on the discussions, it seems that the key to the appreciation of the campaign is:

• A short and straight to the point message that clearly indicates what would be expected from Canadians.
• A campaign where the script translates into the video in a clear way
• Static images that show warm interactions between Canadians and newcomers
• A campaign where the word Donate is used carefully.

More details on the different concepts are presented below:

Warmest welcome

• A lot of participants agreed with the message that Canadians are very warm people.
• This concept was misunderstood by some participants who were not sure or didn’t get the key message of the campaign. Some others were uncertain if it was targeted to Canadians or immigrants.
• Some felt that there was too much focus on the weather.
• Some participants didn’t like the text style as it made them feel like the government is trying to sell them something by using a smart text.
• Some others felt that there was a disconnect between the script and the storyboard: for them it should show how Canadians are welcoming during wintertime and not only during spring.
• Many mentioned that they would like more details on the different ways they could help (the words Volunteer and Donate were not clear in the radio version)
• Static images of the concept were generally appreciated as they show diversity and the warmth of the Canadian welcome. The image showing 5 people in a half circle was less appreciated as participants didn’t feel it was natural.

Canadian Welcome

• Radio script of this concept was the most preferred as it was simple, efficient and straight to the point: they could understand the message quickly and what is expected from them.
• Some participants mentioned wanting to have other examples of actions they could take to help, other than going to the airport.
• Many disagreed with the choice of highlighting the arrival experience in the video ad: Some thought that this what would be expected from them, and many thought that it doesn’t show all the ways by which Canadians can welcome immigrants in their community.
• Others mentioned that it is only a part of the newcomer experience, and that the ad needs to show what happens after the group leaves the airport and how the newcomers are welcomed in their community.
• Among all the static images shown to participants, images of this concept were the least appreciated: they were perceived as of bad quality, some mentioned they were of an outdated ad style, others that it only shows immigrants and not the experience of immigrants being welcomed by Canadians.

Everyday Heroes

• The concept was well appreciated by the participants who found the visuals appealing.
• There was, however, confusion over who was the hero. Is it the Canadian welcoming newcomers or the newcomer’s child, as in the storyboard?
• Many felt that the word Hero was overused during the pandemic, so they are not paying attention to that kind of message anymore.
• In the radio script, the sentence: Now that’s super was not appreciated by participants who stated that it was a little tacky.
• This concept was seen more as a message for children or at least for a younger audience.
• Some reacted negatively to the word Donate: they disliked the fact that the Government of Canada would be asking Canadians who are suffering from the high cost of living to donate money. They were open to make non-monetary donations and volunteer. Others reacted negatively to the sentence Donate, volunteer, learn, celebrate, as they felt that the government was giving them orders.
• The static image showing a mom and her girl was seen as not fitting with the concept.

Website Landing Page

• In general, the website content matches with what participants would have expected by clicking on the link.
• Few participants mentioned that it could be designed in a more appealing way
• Some participants were not sure why the website included specific boxes to help newcomers from Ukraine and Afghanistan.

Contract value

The contract value was $107,143.21 (HST included).

Notes on Interpretation of the Research Findings

The views and observations expressed in this document do not reflect those of IRCC. This report was compiled by Leger based on the research conducted specifically for this project.

Qualitative research is designed to reveal a rich range of participants’ opinions, perceptions and interpretations. It does not and can not measure what percentage of the target population holds a given opinion or perception. Findings are qualitative in nature and cannot be used quantitatively to estimate the numeric proportion or number of individuals in the population who hold a particular opinion.

The results of the quantitative research use a sample drawn from an internet panel which is not probabilistic in nature. As a result, the margin of error cannot be calculated for this survey and the results cannot be described as statistically projectable to the target population.

Political Neutrality Statement and Contact Information

Leger certifies that the final deliverables fully comply with the Government of Canada’s political neutrality requirements outlined in the Policy on Communications and Federal Identity and the Directive on the Management of Communications.

Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leaders.

Signed:

Christian Bourque, Senior Researcher
Léger