Post - ACET - COVID Public Health Measures Campaign Evaluation
Methodological Report
Prepared for Health Canada
Supplier name: Ipsos Public Affairs
Contract Number: HT372-204271/001/CY
Contract value: $33,486.22
Award Date: January 15, 2021
Delivery Date: March 5, 2021
Registration number: POR 105-20
For more information on this report, please contact Health Canada at:
hc.cpab.por-rop.dgcap.sc@canada.ca.
Ce rapport est aussi disponible en français
POLITICAL NEUTRALITY STATEMENT
I hereby certify as Senior Officer of Ipsos that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate or ratings of the performance of a political party or its leaders.
Mike Colledge
President
Ipsos Public Affairs
Post-ACET - COVID Public Health Measures Campaign Evaluation
Methodological Report
Prepared for Health Canada and the Public Health Agency of Canada
Supplier name: Ipsos Public Affairs
January 2021
This public opinion research report presents the methodology of the COVID Public Health Measures Campaign Post-ACET online survey conducted by Ipsos Public Affairs on behalf of Health Canada and the Public Health Agency of Canada. The research study was conducted with n=2,288 Canadians aged 18+ between February 9th and 22nd, 2021.
Cette publication est aussi disponible en français sous le titre : Évaluation de la campagne de mesures de santé publique contre la COVID – Post OECP.
This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Health Canada. For more information on this report, please contact Health Canada at hc.cpab.por-rop.dgcap.sc@canada.ca or at:
Communications Branch
Health Canada
200 Eglantine Driveway
AL 1915C, Tunney’s Pasture
Ottawa, ON K1A 0K9
Catalogue Number:
H14-364/2021E-PDF
International Standard Book Number (ISBN):
978-0-660-38535-8
Related publications (registration number: POR 105-20)
Catalogue Number H14-364/2021F-PDF (Final Report, French)
ISBN 978-0-660-38536-5
© Her Majesty the Queen in Right of Canada, as represented by the Minister of Health, 2021
In December 2019, a cluster of cases of pneumonia was reported in Wuhan, China; the cause was confirmed as a new coronavirus (COVID-19) that had not previously been identified in humans. The situation quickly evolved into a worldwide pandemic. By March 2021, there were more than 899,000 confirmed cases in Canada and over 22,000 deaths.
To help reduce the spread of COVID-19, Health Canada (HC) and the Public Health Agency of Canada (PHAC) launched a multi-phase $50 million public education advertising campaign from February 2020 to present. A mix of traditional and digital media has been instrumental to the success of the campaign.
Overall, the campaign has sought to achieve the following objectives:
Both phases of this campaign included a number of sub-campaigns with messaging and creatives on the following:
HC and PHAC’s campaign have included television, radio, print, digital, social media, out-of-home and mobile advertising using a wide range of graphic, video, and audio formats. Digital efforts alone have generated over 1 billion impressions, 10 million clicks and 122 million completed video views.
The research consisted of online surveys conducted in English and French, taking place between February 9th and 22nd, 2021.
Ipsos programmed, hosted and provided sample management services, while HC and PHAC provided the online questionnaires. Ipsos was responsible for data collection and data storage in Canada, data processing and data weighting. The total contract value of this research was $33,486.22, including HST.
HC and PHAC is looking to assess the impact of its COVID-19 Public Education campaign to date in raising awareness, motivating interest, and generating engagement with federal information and resources about the virus at Canada.ca/coronavirus through various sub-campaigns.
This research will help to evaluate the impact of this public education campaign, such as:
The results of this research will allow PHAC to evaluate the performance of this multi-phase ad campaign, especially in context of public health measures ads by other levels of government across Canada. It will also help PHAC to improve future phases of this ongoing public education campaign.
The Government of Canada’s Policy on Communications and Federal Identity requires the evaluation of all advertising campaigns exceeding $1 million in media through the Advertising Campaign Evaluation Tool (ACET). The campaign launched in April 2020
The specific research objectives were as follows:
Respondents for the survey samples were drawn from the Ipsos i-Say panel, consisting of over 200,000 Canadians recruited in all provinces and territories. The survey sample of n=2,288 was drawn from Canadians 18+ and included an oversample of n=88 (see detailed below regarding oversampling).
A pre-test was conducted on February 9th on a sample of n=45 respondents (27 in English and 18 in French), to confirm survey length before fully deploying the questionnaire. In the pre-test, an open-ended question was asked at the end of the survey where any problems, questions, or unclear questions could be brought to our attention. No issues were flagged during the pre-test that required revision to the original questionnaire.
The general population sample consisted of n=2,200 respondents. Quotas were set to ensure representation by region, age and gender, according to the latest Census information.
In addition to the natural fallout from the general population sample, oversamples in each of the following four target groups were carried out in order to reach a minimum of n=200 in each of these groups except indigenous communities n=100. In the end, we obtained the following number of completes, including both natural fallout and oversample (note that one person can fall into more than one category):
Interviews were conducted in English and French, with respondents being able to complete the survey in either official language of their choice.
Respondents were invited to participate in the survey via email, with a unique URL link. This link could only be used once, with respondents being allowed to pause during completion and return to complete it at a later time. On average, the online interviews took 8 minutes to complete.
The Ipsos i-Say panel was the primary source for recruiting respondents and, where necessary, a supplemental sample from a third-party vendor, Dynata, was used for additional completes. Ipsos lead the data collection process in conjunction with Dynata. Outgoing sample lists were coordinated between Ipsos and Dynata to ensure that an individual was not allowed to complete the same survey twice, in the event that they are members of both companies’ panels. The Ipsos i-Say online panel is a diversely-sourced and actively maintained panel of over 200,000 Canadian adults. Using this panel allowed us to confidently build online samples based on known population characteristics, such as ones based on Census data. Invitations to participate in each survey are adjusted to reflect the most recent current population data provided by Statistics Canada.
Online panel participants were recruited and screened using rigorous double and triple opt-in checksFootnote 1 to ensure that respondents want to receive contact, such as invitations to participate in research surveys. The Ipsos i-Say panel’s Terms and Conditions of Membership details the respondents’ rights to, at any time and for any reason, ask to be removed from the panel and stop receiving emails from Ipsos and its clients.
Panel recruitment consists of the following steps:
It is important to note that any panel – no matter how well recruited – does not enable random selection in the sense that not all members of the public are likely to join an online survey panel. This applies to panels recruited through online approaches as well as panels recruited via traditional telephone Random Digit Dialing (RDD) in that the sampling naturally skews towards those who are amenable to answering a survey.
Rigorous panel maintenance procedures mean only those participants who actively participate in online surveys at least once every six months are included. Participants can choose whether to enter a survey or not. While participating in a survey, the respondent can simply exit the survey by closing the survey Internet browser window at any time without submitting their responses if they no longer want to participate.
In order to capture responses from low incidence individuals, such as pregnant women who are vaccine-hesitant, we use what is called a Webcruiser on our iSay Panel. A Webcruiser asks a set number of questions to our entire panel and typically takes approximately three weeks to complete. Panelists who meet the criteria we are looking for are then flagged and can be contacted for further research.
Incentives and Quality Control Measures
Respondents to Ipsos’s online surveys are offered a number of innovative incentive programs in the forms of a points-based system where participants can redeem points for various items. We do not reward our panelists using cash payments.
Extensive quality-control procedures are in place within IIS ( Ipsos Interactive Services, who manage our panel) to ensure that the survey inputs (sample and questionnaire design) allow for high-quality survey data. These processes span the life cycle of a panelist and are in place for all Ipsos online surveys. IIS experts are constantly monitoring and reviewing the performance of our quality measures and updating and integrating new ones as respondents’ behaviors and the online landscape evolve.
Are panelists who they say they are?
Have they recently participated in similar surveys?
Are they completing surveys seriously?
Can they only take the survey once?
Sample Weighting
The table below shows the unweighted and weighted distributions of the online samples. Weighting was applied to the sample to ensure that the final data reflects the general adult population by region, age, and gender according to the 2016 Census. A Random Iterative Method (RIM) technique was applied for weighting.
| Unweighted Sample Size | Weighted Sample Size | |
|---|---|---|
| Canada | 2288 | 2288 |
| Region | ||
| British Columbia/Territories | 281 | 310 |
| Alberta | 276 | 256 |
| Prairies (MB/SK) | 165 | 149 |
| ON | 958 | 879 |
| QC | 455 | 537 |
| Atlantic Canada | 153 | 156 |
| Gender | ||
| Male | 956 | 1116 |
| Female | 1321 | 1161 |
| Diverse | 11 | 11 |
| Age | ||
| 18-34 | 761 | 641 |
| 35-54 | 705 | 778 |
| 55+ | 822 | 869 |
The figures presented in the table above show minimal differences between the unweighted and weighted samples. As previously noted, the research was drawn from a sample of adult Canadians, aged 18 years an over. This resulted in a weight ratio of 1.11:1, which remains within acceptable ranges for a survey of the general population.
Email Statistics
For this survey, a sample router was used. Therefore, a response rate cannot be calculated. However, the participation rate for the survey was 93%. Participation rate is calculated as follows: (qualified completes + over quota + terminates)/click-through).
| Completions | Survey |
| Click-Through | 3674 |
| - Partial Completes | 264 |
| - Terminates | 401 |
| - Over quota | 721 |
| Qualified Completes | 2288 |
| Participation Rate | 93% |
Non-Response Analysis
The results of this survey are not statistically projectable to the target population because the sampling method used does not ensure that the sample represents the target population with a known margin of sampling error. Reported percentages are not generalizable to any group other than the sample studied, and therefore no formal statistical inferences can be drawn between the sample results and the broader target population it may be intended to reflect. The sample data have been weighted to reflect the demographic composition of Canadian adults aged 18 years and over.
The table below compares the unweighted survey samples to the 2016 Census results by region, age, and gender. Overall, the sample is highly representative of the national adult population, except for a few gaps, which are described below.
| Unweighted Percentage | Census 2016 Proportions (adults) | |
|---|---|---|
| Region* | ||
| BC | 12.3% | 13.6% |
| Alberta | 12.1% | 11.2% |
| Prairies (MB/SK) | 7.2% | 6.5% |
| Ontario | 41.9% | 38.4% |
| Quebec | 19.9% | 23.5% |
| Atlantic | 6.7% | 6.8% |
| Gender* | ||
| Male | 41.8% | 48.5% |
| Female | 57.7% | 51.4% |
| Diverse | <1% | <1% |
| Age* | ||
| 18-34 | 33.3% | 27.3% |
| 35-54 | 30.8% | 34.1% |
| 55+ | 35.9% | 38.6% |
* Denotes variables included in the weighting scheme.
Differences among the variables used in the weighting scheme are minimal; however, there are a few notable differences. Young adults (18-34) are overrepresented in the unweighted sample. This is due to the intentional oversampling of this population and is to be expected. Additionally, females are also overrepresented in the sample, particularly among young adults. This is because gender quotas were not interlocked with age among the young adult oversample. The remaining distribution in the sample remains consistent, with only slight differences observed between the unweighted percentages and the 2016 Census data. Regional distributions similarly demonstrate slight, but not significant, differences between the unweighted percentages and the 2016 Census data.
INTRODUCTION
Thank you for taking the time to complete this survey dealing with current issues of interest to Canadians. Do you prefer to continue in English or French? / Préférez-vous continuer en français ou en anglais ?
Your participation is voluntary, and your responses will be kept entirely confidential . The survey takes about 5 minutes to complete.
START SURVEY
To view our privacy policy, click here.
If you require any technical assistance, please contact Daniel Kunasingam at daniel.kunasingam@ipsos.com
a) Does anyone in your household work for any of the following organizations? Select all that apply.
IF "NONE OF THESE ORGANIZATIONS" CONTINUE, OTHERWISE THANK AND TERMINATE.
b) Are you…
c) In what year were you born?
YYYY
ADMISSIBLE RANGE 1900-2003
IF > 2003, THANK AND TERMINATE
ASK D IF QUESTION C IS LEFT BLANK
d) In which of the following age categories do you belong?
SELECT ONE ONLYIF "LESS THAN 18 YEARS OLD" THANK AND TERMINATE
E) In which province or territory do you live?
SELECT ONE ONLYQF. Do you identify as any of the following? Select all that apply
[RANDOMIZE]
[If anyone select option 2 consider Indigenous . If anyone select options 3 to 13 consider Ethnic.]
CORE QUESTIONS
ASK ALL RESPONDENTS
Q1:
Over the past year, have you seen, read or heard any advertising from
the Government of Canada?
Q2:
Think about the most recent ad from the Government of Canada that comes to mind. Where have you seen, read or heard this ad? Select all that apply.
RANDOMIZE
Q3:
What do you remember about this ad?
Q4:
How did you know that it was an ad from the Government of Canada?
CAMPAIGN SPECIFIC QUESTIONS
ASK ALL RESPONDENTS
T1A:
Over the past year, have you seen, read or heard any Government of Canada advertising about
protecting yourself and others from COVID-19?
T1B:
Where have you seen, read or heard this ad about COVID-19? Select all that apply.
RANDOMIZE
T1C:
What do you remember about this ad?
ASK ALL RESPONDENTS
Pretest1: Was anything in the survey today confusing?
T1H:
Here are some ads that have recently been broadcast on various media.
Click next to watch.
[CLICK TO GO TO THE NEXT PAGE]
Over the past year have you seen these ads?
T1I:
Where have you seen, these ads? Select all that apply.
RANDOMIZE
T1J:
What do you think is the main point these ads are trying to get across?
T1K:
Please indicate your level of agreement with the following statements about these ads?
| 1 Strongly Disagree | 2 | 3 | 4 | 5 Strongly Agree | |
| These ads catch my attention | o | o | o | o | o |
| These ads are relevant to me | o | o | o | o | o |
| These ads are difficult to follow | o | o | o | o | o |
| These ads do not favour one political party over another | o | o | o | o | o |
| These ads talk about an important topic | o | o | o | o | o |
| These ads provide new information | o | o | o | o | o |
| These ads clearly convey how to help stop the spread of COVID-19 | o | o | o | o | o |
| This ad prompted me to download the COVID Alert app | o | o | o | o | o |
| These ads prompted me to stay home | o | o | o | o | o |
| These ads prompted me to wear a face mask when going out | o | o | o | o | o |
| These ads prompted me to wash my hands more frequently | o | o | o | o | o |
| These ads prompted me to stay 2 meters apart from others | o | o | o | o | o |
[Ask all]
T1L. Did you do any of the following as a result of seeing the ads? Please select all that apply
[RANDOMIZE]
Pretest2: Was anything in the survey today confusing?
DEMOGRAPHIC QUESTIONS
D1:
Which of the following categories best describes your current
employment status? Are you…
D2:
What is the highest level of formal education that you have completed?
SELECT ONE ONLY
D3:
Are there any children under the age of 18 currently living in your household?
D4:
Which of the following categories best describes your total annual
household income, including income from all household members, before
taxes are deducted?
SELECT ONE ONLY
D5: Where were you born?
Ê Specify the country:
ASK IF D5=BORN OUTSIDE CANADA
D6:
In what year did you first move to Canada?
YYYY
ADMISSIBLE RANGE: 1900-2021
D7:
What is the language you first learned at home as a child and still understand?
SELECT UP TO TWO
That concludes the survey. This survey was conducted on behalf of Health Canada. In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to answer this survey, it is greatly appreciated.
The Ipsos online panel is recruited utilizing a double and triple opt-in screening process to ensure maximum return from an engaged and representative audience. After the initial opt-in to the Ipsos online panel, respondents are sent a second and third email to confirm they want to join our panel.