2021-22 Newcomer Services Advertising Campaign – ACET

Methodological Report

Prepared for Immigration, Refugees and Citizenship Canada

Supplier name: Advanis Inc.

Contract number: B8815-220330/001/CY

Contract value: $41,329.57 (including HST)

Award date: December 31, 2021

Delivery date: March 31, 2022

Registration number: POR 088-21

For more information on this report, please contact IRCC at:
IRCC.COMMPOR-ROPCOMM.IRCC@cic.gc.ca

Ce rapport est aussi disponible en français

Copyright

This public opinion research report presents the methodological details for the 2021-22 Newcomer Services Advertising Campaign ACET online study conducted by Advanis Inc. on behalf of Immigration, Refugees and Citizenship Canada (“IRCC”). The survey for the baseline (pre-advertising campaign) wave was administered with 626 newcomers to Canada between January 13^th and 31^st, 2022, while the post-advertising campaign wave was conducted with 599 newcomers March 10^th to March 24^th, 2022.

Cette publication est aussi disponible en français sous le titre : Campagne publicitaire sur les services à l’intention des nouveaux arrivants 2021-22 - OECP

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Immigration, Refugees and Citizenship Canada. For more information on this report, please contact Immigration, Refugees and Citizenship Canada at IRCC.COMMPOR-ROPCOMM.IRCC@cic.gc.ca or at:

Communications Branch
Immigration, Refugees and Citizenship Canada
Jean Edmonds Tower South
365 Laurier Ave W
Ottawa ON K1A 1L1

Catalogue Number: Ci4-226/2022E-PDF
International Standard Book Number (ISBN): 978-0-660-45157-2

Related publications (registration number: POR 088-21):
Catalogue Number: Ci4-226/2022F-PDF (Methodological Report, French)
ISBN: 978-0-660-45158-9

Copyright: Her Majesty the Queen in Right of Canada, as represented by the Minister of Immigration, Refugees and Citizenship Canada, 2022

Political neutrality certification

I hereby certify as Senior Officer of Advanis that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Policy on Communications and Federal Identity and the Directive on the Management of Communications.

Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leaders.

Signed: Nicolas Toutant, Vice-President, Research and Evaluation, Advanis Inc.

Executive Summary

Background

The Government of Canada wishes to make it easier for immigrants to build successful lives in Canada. In order to raise awareness among newcomers about services permanent residents can access, the Government of Canada launched the Newcomer Services advertising campaign in 2009, to inform newcomers to Canada about the settlement services available to help them succeed in their new country.

The Government of Canada requires a campaign evaluation using the Advertising Campaign Evaluation Tool (ACET) for all advertising campaigns with a media buy exceeding $1,000,000 (See the Directive on the Management of Communications: https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=30682#appB)

Objectives

The purpose of this quantitative study is to ensure that advertising aimed at newcomers to Canada was effective and appropriate and that it accomplished its objective to build awareness and understanding of the settlement services available during the pandemic.

The focus of the survey was newcomers to Canada since 2017 across the country (outside Quebec). This study measured awareness, recall and impact of the ads among the target population. It consisted of two survey waves: a baseline wave conducted before the campaign launch and a wave at the end of the campaign. More precisely the study aimed to assess:

• Unaided and aided recall of the ads
• Awareness and use of information channel (i.e. website) promoted in the ads
• The perceived messages of the ads, as well as the effectiveness of the ads in increasing awareness of IRCC and newcomer services
• Receptivity to government advertising and ratings of government performance and priorities

The results of this research provided feedback on the effectiveness of the ads in raising awareness among the target audience about the services available to them. The results will also be used to guide and inform the design and implementation of future public awareness and outreach activities aimed at newcomers.

Methodology

The baseline wave of the survey was conducted online with 626 newcomers to Canada (since 2017, aged 18 and older) outside of Quebec from January 13 to January 31, 2022. The post-campaign wave of the survey was conducted with 599 newcomers to Canada outside of Quebec from March 10 to March 24, 2022. Respondents from a pre-profiled database recruited randomly via telephone were sent SMS or email invites to participate in the study. The average survey length of the baseline survey was 5 minutes, and was 8 minutes for the post-campaign survey. The online survey sample was non-probability, meaning that a margin of error cannot be calculated. The data reported was weighed to replicate, as closely as possible, actual newcomer population distribution by age and gender within Canada according to IRCC data. Respondents were offered the survey in the official language of their choice.

A full quantitative methodology report, including all information about the execution of the fieldwork that is needed to replicate the research initiative, can be found in Appendix A. The quantitative survey research instruments and a set of tabulated results from the quantitative surveys are provided in English and French under separate covers.

The value of this contract, including HST, is $41,329.57.

Appendix A: Methodology

Advanis used its General Population Random Sample (GPRS), which is a pre-profiled database to find potential respondents. Our GPRS recruits people living in Canada using random digit telephone dialing (including both landline and cell phone numbers) to collect a random sample of potential respondents. Advanis identified newcomers from all Canadian provinces (except Quebec) who have been in Canada for less than 10 years to populate its sample. Potential respondents were selected to attain the targeted number of completed surveys for each province/region of Canada.

Questions within the survey further filtered out responses from anyone under the age of 18, anyone who mentioned having been born in Canada, anyone who mentioned having arrived in Canada prior to 2017 and anyone working for a market research firm, a magazine or a newspaper, an advertising agency, a graphic design firm, a political party, a radio or television station, a public relations company or the federal/provincial governments.

CATI recruit

For both waves of the survey, Advanis used a CATI (Computer Assisted Telephone Interviewing) survey to recruit potential participants to participate in the web survey. Potential participants pulled from our GPRS sample were randomly called and were asked if they agreed to participate in an online study. Those who agreed were sent an invitation via SMS and via email in a few cases. Overall, 5,211 people agreed to receive an invitation to participate to the web survey for the baseline wave and 6,550 for the post-campaign wave.

Email and SMS invites/reminders

Invitations were mainly sent by SMS. Emails were sent only if it was requested specifically by the potential participant. After sending the initial invitation, a reminder message was sent every three days to applicants who did not complete a survey or were not screened out of the survey. A maximum of three reminder messages were sent.

A test pre-launch was conducted on January 13 for the baseline survey and on March 10 for the post-campaign survey. Respondents were asked to provide feedback on the survey. Neither pretest highlighted any difficulties with the survey itself or data collection.

A support email address and phone number were provided in the survey itself for any technical issues related to the survey. Surveys had a unique number embedded in the hyperlink to eliminate the possibility of duplicate responses from one participant.

Results

The table below shows the details of initial invitations sent, unopened surveys, screened out applicants, as well as complete and incomplete surveys. The response rate, calculated as the number of in-scope – responding units divided by the sum of unresolved units, in-scope – non-responding units, and in-scope – responding units, was 28.2% for the baseline pre-campaign wave and 20.0% for the post-campaign wave. This response rate is within normal ranges for a survey of this type.

Non-response bias analysis

Since no census data exists of the demographic make-up of the newcomer target population (those who have come to Canada since 2017, over the age of 18), an analysis of non-response bias is not possible.

Margin of Error

Although recruitment to Advanis’ GPRS panel was probabilistic in nature, the sample drawn from it for our survey cannot be considered probabilistic. As such, this would be considered a panel project and no margin of error can be reported. The data reported was weighed to replicate, as closely as possible, actual newcomer population distribution by age and gender within Canada according to IRCC data.

Completes and Targets

Overall, 5,211 were invited to participate to the web survey during the first wave and 6,550 during the second wave. The average survey length of the baseline survey was 5 minutes and 8 minutes for the post-campaign survey. In total, 626 web questionnaires were completed during the baseline wave and 599 during the post-campaign wave. For both waves, the number of completes per province matched approximately the number of completes targeted by IRCC.

Weighting

Overall, 1,225 web interviews were conducted during data collection. The final data was weighted within each survey wave by age and gender. The weighting scheme was provided by IRCC based on internal estimates of the demographic characteristics of the newcomer population within Canada (encompassing both permanent and temporary residents) and Advanis applied the weights to the database. Those whose gender was recorded as ‘other’ (n=6 in the Baseline Survey and n=4 in the Post-Campaign Survey) received a weight of 1.0.

Appendix B: Quantitative Instruments

English and French quantitative survey instruments for the Baseline and Post-Campaign waves are provided under a separate cover.

Appendix C: Full set of tabulated data

A full set of tabulated data for the Baseline and Post-Campaign waves are provided under a separate cover.