Methodological Report
Prepared for Health Canada
Supplier name: Ipsos Public Affairs
Contract Number: HT372-214614/001/CY
Contract value: $73,425.59
Award Date: November 26th, 2021
Delivery Date: March 31st, 2022
Registration number: POR 045-21
For more information on this report, please contact Health Canada at:
hc.cpab.por-rop.dgcap.sc@canada.ca.
Ce rapport est aussi disponible en français
POLITICAL NEUTRALITY STATEMENT
I hereby certify as Senior Officer of Ipsos that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate or ratings of the performance of a political party or its leaders.
Mike Colledge
President
Ipsos Public Affairs
Baseline and Post-Campaign ACET Survey for the COVID-19 Mental Health Campaign
Methodological Report
Prepared for Health Canada and the Public Health Agency of Canada
Supplier name: Ipsos Public Affairs
March 2022
This public opinion research report presents the methodology of the Baseline and Post-Campaign ACET Survey for the COVID-19 Mental Health Campaign online survey conducted by Ipsos Public Affairs on behalf of Health Canada and the Public Health Agency of Canada. The baseline survey was conducted with a sample of n=2,137 Canadians ages 18+ between December 10th, 2021, and January 10th, 2022, and the post-campaign survey was conducted with a sample of n=2,101 Canadians ages 18+ between March 2nd and March 16th, 2022.
Cette publication est aussi disponible en français sous le titre : Sondage de référence et post campagne OECP pour la campagne COVID-19 sur la santé mentale.
This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Health Canada. For more information on this report, please contact Health Canada at hc.cpab.por-rop.dgcap.sc@canada.ca or at:
Communications Branch
Health Canada
200 Eglantine Driveway
AL 1915C, Tunney's Pasture
Ottawa, ON K1A 0K9
Catalogue Number:
H14-400/2022E-PDF
International Standard Book Number (ISBN):
978-0-660-43754-5
Related publications (registration number: POR 20-30)
Catalogue Number H14-400/2022F-PDF (Final Report, French)
ISBN 978-0-660-43755-2
©Her Majesty the Queen in Right of Canada, as represented by the Minister of Health, 2022
The COVID-19 pandemic has had significant impacts on mental health across the population, with concerns about the virus itself compounded by impacts on work and routines, financial stress, social isolation, substance use, anxiety and depression, grief and bereavement, and reduced access to services and supports.
In light of ongoing uncertainties, as well the lasting effects of the pandemic thus far, it is imperative that we continue to offer ongoing mental health and substance use supports to those who need it. The effects of the pandemic have been higher among certain populations, including youth, seniors, parents, Indigenous Peoples, and ethnic and racialized populations, as observed by Statistics Canada, leading Canadian health organizations and associations, academia, research organizations, and governments at all levels. Even prior to the pandemic, many of these at-risk groups reported lower rates of positive mental health and higher rates of mental illness than the general population.
In response, Health Canada (HC) and the Public Health Agency of Canada (PHAC) launched a national awareness ad campaign to support the mental health of these Canadians most affected by COVID-19. The campaign aimed to drive these audiences to Canada.ca/covid-mental-health to motivate interest and engagement with free resources, tools and professional support services.
The objective of the research was to assess the impact of the COVID-19 Mental Health Campaign in raising awareness of the online health-related resources that at-risk and vulnerable Canadians (5 groups) could immediately access to get help. The research involved a baseline survey to collect information on status of awareness and use of these resources prior to the campaign and a post-campaign survey to measure the impact of the campaign on awareness and usage.
The objectives of the research were as follows:
The results of this research will allow HC and PHAC to evaluate the performance of this campaign. It will also help HC and PHAC to improve any future phases of the campaign (or similar).
The Government of Canada's Policy on Communications and Federal Identity requires the evaluation of all advertising campaigns exceeding $1 million in media through the Advertising Campaign Evaluation Tool (ACET).
The target audience of the research was as follows:
The target audiences of the campaign represent five (5) at-risk and vulnerable groups of Canadians.
This project involved two (2) online surveys – a baseline survey and a post-campaign survey. Both surveys were executed online using a non-probability online panel. This is the standard approach for all Government of Canada advertising evaluation surveys. The initial baseline survey was conducted before the campaign launched and the post-campaign survey was conducted following the completion of the campaign.
Respondents for the survey were drawn from the Ipsos' i-Say panel and a trusted partner panel vendor, Dynata. The baseline survey was conducted with a sample of n=2,137 Canadians ages 18+ and the post-campaign survey was conducted with a sample of n=2,101 Canadians ages 18+. Survey respondents took the survey in the official language of their choice, either English or French.
For the baseline survey a pre-test was conducted on December 10th, 2021, with 40 completes (24 English / 16 French), to confirm survey length before fully deploying the questionnaire. An open-ended question was asked at the end of the survey where any problems with the clarity of the survey questions could be brought to our attention; no issues were flagged. The survey was fully launched and ran between December 10th, 2021, and January 10th, 2022.
For the post-campaign survey, a pre-test was conducted on March 2, 2022, with 96 completes (62 English / 34 French). No issues were flagged. The survey was fully launched and ran between March 2nd and March 16th, 2022.
For both surveys, quotas were set to ensure representation by region, age and gender, according to the latest Census information. In the end, within natural fallout we obtained our targets without having to oversample except Indigenous (oversampled n=100 to ensure we achieved a minimum sample of n=200 Indigenous Canadians). The unweighted counts are shown below.
| Target audience | Baseline Survey Sample Size | Post-campaign Survey Sample Size |
|---|---|---|
| Young adults 18-24 years of age | 194 | 223 |
| Young adults 25-34 years of age (n=326) | 365 | 333 |
| Ethnic communities (n=200) | 396 | 423 |
| Indigenous communities (n=200) | 200 | 200 |
| Parents of children 12-17 (n=300) | 430 | 385 |
The sampling methodology utilized email invitations + router technology to invite participants. Each participant received a unique URL link. This link could only be used once, with respondents being allowed to pause during completion and return to complete. On average, each survey was completed within 10 minutes.
Ipsos partnered with Dynata on the fieldwork and in obtaining the required sample. Ipsos and Dynata have over 300,000 active panelists. Dynata's panels are continually refreshed and recruited through various channels including 1) loyalty program sourcing across travel, entertainment and other sectors; 2) online banners, cable TV advertising, mailings, social media influencers, mobile app, etc.; 3) integrated channels including access to online communities, social media platforms, publishers and others (this last group has not opted into a panel) but each participant has a Dynata profile.
Dynata's panel includes members who have opted in and receive a form of incentive. Incentives are based on a point-based system cashed in for rewards (electronic gift certificates, high street vouchers, charity donations, and long-term loyalty rewards), as well as sweepstakes. The amount of incentives is based on the specific requirements of each survey, depending on the length and complexity of the survey, the subject matter of the study, and the time required to complete a minimum number of interviews.
The comprehensive background profiling data gathered when respondents join a panel allows for the targeting of respondents based on key criteria, such as region, age, gender, education and income level, intention to buy a home within 12 months, profession, and other characteristics.
The data excludes any duplicate respondents based on IP capture and excludes panelists who have completed another Government of Canada survey in the previous 30 days as a member of that panel.
The table below shows the unweighted and weighted distributions of the online sample. Weighting was applied to the sample to ensure that the final data reflects the general adult population by region, age, and gender according to the 2016 Census. A Random Iterative Method (RIM) technique was applied for weighting. Note: totals may not add to 100% due to rounding.
| Baseline Survey | Post-campaign Survey | |||
|---|---|---|---|---|
| Unweighted Sample Size | Weighted Sample Size | Unweighted Sample Size | Weighted Sample Size | |
| Canada | 2137 | 2137 | 2101 | 2101 |
| Region | ||||
| British Columbia/Territories | 293 | 290 | 294 | 285 |
| Alberta | 247 | 240 | 237 | 236 |
| Prairies (MB/SK) | 178 | 139 | 158 | 137 |
| Ontario | 810 | 821 | 770 | 807 |
| Quebec | 447 | 502 | 489 | 493 |
| Atlantic Canada | 162 | 146 | 153 | 143 |
| Gender | ||||
| Male | 1037 | 1033 | 1014 | 1015 |
| Female | 1088 | 1092 | 1074 | 1073 |
| Diverse | 12 | 12 | 13 | 13 |
| Age | ||||
| 18-34 | 559 | 586 | 556 | 578 |
| 35-54 | 758 | 727 | 703 | 716 |
| 55+ | 820 | 824 | 842 | 807 |
| Gender by Age | ||||
| Male 18 - 24 | 92 | 119 | 110 | 117 |
| Male 25 - 34 | 168 | 173 | 161 | 170 |
| Male 35 - 44 | 171 | 168 | 168 | 165 |
| Male 45 - 54 | 196 | 187 | 176 | 184 |
| Male 55 - 64 | 196 | 182 | 189 | 179 |
| Male 65 - 100 | 214 | 204 | 210 | 200 |
| Female 18 - 24 | 101 | 114 | 110 | 112 |
| Female 25 - 34 | 193 | 175 | 168 | 172 |
| Female 35 - 44 | 184 | 175 | 165 | 172 |
| Female 45 - 54 | 204 | 193 | 189 | 190 |
| Female 55 - 64 | 210 | 190 | 192 | 187 |
| Female 65 - 100 | 196 | 244 | 250 | 240 |
| Gender Diverse | 12 | 12 | 13 | 13 |
The figures presented in the table above show minimal differences between the unweighted and weighted samples. As a result of the strong representativeness of the unweighted data the largest weight factor that was applied for any respondent is 1.29, which is well within acceptable ranges for a survey of the general population.
For this survey, a non-probability sample was used. Therefore, a response rate cannot be calculated. However, the participation rate for the baseline survey was 97% and post-campaign survey was 93%. The post-campaign rate is lower because more cases were excluded based on having completed a similar survey within 30 days or the baseline ACET wave. Participation rate is calculated as follows: Participation Rate = R/(R+IS+U).
| Completions | Baseline Survey | Post-campaign Survey |
|---|---|---|
| Invalid Cases | 357 | 1621 |
| Unresolved (U) | 0 | 0 |
| In-scope non-responding (IS) | 89 | 277 |
| Responding units (R) | 2571 | 3673 |
| Participation Rate | 97% | 93% |
Online survey cases can be broken down into four broad categories:
Invalid Cases
These can include only clearly invalid cases (for example, invitations mistakenly sent to people who did not qualify for the study, incomplete or missing email addresses in a client-supplied list).
Unresolved (U)
These include all the cases where it cannot be established whether the invitation was sent to an eligible or an ineligible respondent or unit (for example, when email invitations bounce back or remain without an answer before the candidate could be qualified).
In-scope non-responding (IS)
These include all refusals, either implicit or explicit; all non-contacts and early break-offs of known eligible cases; and other eligible non-respondents (due to illness, leave of absence, vacation or other).
Responding units (R)
These include cases who have participated but who were disqualified afterwards (for example, when admissible quotas have been reached). It also includes all completed surveys or partially completed surveys that meet the criteria set by the researcher to be included in the analysis of the data.
Unresolved (U), in-scope (IS), and responding units (R) are all included in the broad category of "potentially eligible" cases. However, invalid cases are not included in the calculation of outcome rates.
For this survey a router was used to screen potential respondents and assign them to one of several surveys. Given this, it is not possible to estimate the number of cases "invited" to participate and whether they were eligible or not. Therefore, it is not possible to estimate the "unresolved" cases. For this survey, responding units is broken out as follows.
| Completions | Baseline Survey | Post-campaign Survey |
|---|---|---|
| - Over quota | 434 | 1572 |
| - Qualified Completes | 2137 | 2101 |
| Responding units (R) | 2571 | 3673 |
The sample routing technology uses weighted randomization to assign surveys to participants. Upon entry into the system, panelists are checked to ensure they have not exceeded survey participation limits. A list of potential survey matches is determined for each panelist based upon the information we know about them. Panelists may be asked additional screening questions within the system to ensure they meet the project criteria. Priority may be given to surveys that are behind schedule; however, this is kept to a minimum as survey randomization must remain in place as a key element for preventing bias. In this case, limited prioritization was applied during the field window, therefore there is a low chance of sample bias.
The results of this survey are not statistically projectable to the target population because the sampling method used does not ensure that the sample represents the target population with a known margin of sampling error. Reported percentages are not generalizable to any group other than the sample studied, and therefore no formal statistical inferences can be drawn between the sample results and the broader target population it may be intended to reflect. The data have been weighted to reflect the demographic composition of Canadian adults aged 18 years and over.
The table below compares the unweighted survey samples to the 2016 Census results by region, age, and gender. Overall, the sample is highly representative of the national adult population.
| Baseline Survey | Post-campaign Survey | |||
|---|---|---|---|---|
| Unweighted Percentage | Census 2016 Proportions | Unweighted Percentage | Census 2016 Proportions | |
| Region* | ||||
| British Columbia /Territories | 14% | 14% | 14% | 14% |
| Alberta | 12% | 11% | 11% | 11% |
| Prairies (MB/SK) | 8% | 7% | 8% | 7% |
| Ontario | 38% | 38% | 37% | 38% |
| Quebec | 21% | 23% | 23% | 23% |
| Atlantic | 8% | 7% | 7% | 7% |
| Gender | ||||
| Male | 49% | 49% | 48% | 49% |
| Female | 51% | 51% | 51% | 51% |
| Diverse | 1% | <1% | 1% | <1% |
| Age | ||||
| 18-34 | 26% | 27% | 26% | 27% |
| 35-54 | 35% | 34% | 33% | 34% |
| 55+ | 38% | 40% | 40% | 40% |
| Gender by Age* | ||||
| Male 18 - 24 | 4% | 6% | 5% | 6% |
| Male 25 - 34 | 8% | 8% | 8% | 8% |
| Male 35 - 44 | 8% | 8% | 8% | 8% |
| Male 45 - 54 | 9% | 9% | 8% | 9% |
| Male 55 - 64 | 9% | 9% | 9% | 9% |
| Male 65 - 100 | 10% | 10% | 10% | 10% |
| Female 18 - 24 | 5% | 5% | 5% | 5% |
| Female 25 - 34 | 9% | 8% | 8% | 8% |
| Female 35 - 44 | 9% | 8% | 8% | 8% |
| Female 45 - 54 | 10% | 9% | 9% | 9% |
| Female 55 - 64 | 10% | 9% | 9% | 9% |
| Female 65 - 100 | 9% | 12% | 12% | 12% |
| Gender Diverse | 1% | <1% | 1% | <1% |
| * Denotes variables included in the weighting scheme. | ||||
Differences among the variables used in the weighting scheme are minimal. The distribution in the sample is consistent, with only slight differences observed between the unweighted percentages and the 2016 Census data.
Thank you for taking the time to complete this survey dealing with current issues of interest to Canadians. Si vous préférez répondre au sondage en français, veuillez cliquer sur français [SWITCH TO FRENCH VERSION].
Your participation is voluntary, and your responses will be kept entirely confidential. The survey takes about 10 minutes to complete.
START SURVEY
To view our privacy policy, click here.
If you require any technical assistance, please contact Ipsos at pasurveys @ ipsosasks.ca
a) Does anyone in your household work for any of the following organizations?
SELECT ALL THAT APPLY
IF "NONE OF THESE ORGANIZATIONS" CONTINUE, OTHERWISE THANK AND TERMINATE.
b) Are you…?
c) YEAR/MONTH (Drop down menus). What is your date of birth?
YEAR (1910 – 2003)
MONTH
ADMISSIBLE RANGE 1910-2003
IF > 2003, THANK AND TERMINATE
ASK D IF QUESTION C IS LEFT BLANK
d) In which of the following age categories do you belong?
SELECT ONE ONLY
IF "LESS THAN 18 YEARS OLD" OR "BLANK", THANK AND TERMINATE
e) In which province or territory do you live?
SELECT ONE ONLY
SCREENER QUESTIONS TO IDENTIFY IF HAVE CHILDREN UNDER 18 AND THEIR AGE
f) Are you the parent or guardian of a child or children under 18 years of age living in your household?
[ASK IF 1 at F]
g) How old is/are the child/children? Select all that apply
SCREENER QUESTIONS TO IDENTIFY IF INDIGENOUS POPULATIONS OR ETHNIC/RACIALIZED POPULATIONS
h) Do you identify as any of the following? Select all that apply
ASK ALL RESPONDENTS
Q1: Over the past three weeks, have you seen, read or heard any advertising from the Government of Canada?
Q2: Think about the most recent ad from the Government of Canada that comes to mind. Where have you seen, read or heard this ad?
SELECT ALL THAT APPLY
[RANDOMIZE]
Q3: What do you remember about this ad?
Q4: How did you know that it was an ad from the Government of Canada?
ASK ALL RESPONDENTS
T1A: Over the past six weeks, have you seen, read or heard any Government of Canada advertising about mental health and substance use resources or services?
T1B: Where have you seen, read or heard this ad about mental health and substance use linked to the COVID-19 pandemic?
SELECT ALL THAT APPLY
[RANDOMIZE]
T1C: What do you remember about this ad?
ASK ALL RESPONDENTS
CAMPAIGN SPECIFIC ATTITUDINAL AND BEHAVIOURAL QUESTIONS MAY BE ADDED HERE
Q1 Generally speaking, how would you rate your overall mental health?
Q2. Has the COVID pandemic affected your mental health?
Q3 For each of the items listed below, please indicate whether you have experienced any changes since the start of the pandemic by selecting increased, decreased, or stayed the same/no change.
[RANDOMIZE]
Q4 Please indicate if the situations listed below changed for you since the beginning of the pandemic by selecting increased, decreased or the same/no change.
[RANDOMIZE]
Q5 Have you recently received help for mental health, if ever? (select one only)
[If not currently AND rate average/poor mental health: Q1 = 1, 2 or 3, AND Q5 = 2 or 5 ASK Q6]
Q6 Why are you not currently receiving help? (Select all that apply)
Q7 How familiar would you say you are with the different mental health services available?
Q8 Did you know the Government of Canada offers free mental health and substance use help? (select one only)
D1: Which of the following categories best describes your current employment status? Are you…
SELECT ONE ONLY
D2: What is the highest level of formal education that you have completed?
SELECT ONE ONLY
D4: Which of the following categories best describes your total annual household income, including income from all household members, before taxes are deducted?
SELECT ONE ONLY
D5: Where were you born?
ASK IF D5=BORN OUTSIDE CANADA
D6: In what year did you first move to Canada?
YYYY
ADMISSIBLE RANGE: 1900-2021
D7: What is (are) the language(s) you first learned at home as a child and still understand?
SELECT UP TO TWO
That concludes the survey. This survey was conducted on behalf of Health Canada. In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to answer this survey, it is greatly appreciated.
YOU ARE NOT ALONE. IF YOU NEED HELP, PLEASE VISIT Canada.ca/covid-mental-health
Thank you for taking the time to complete this survey dealing with current issues of interest to Canadians. Si vous préférez répondre au sondage en français, veuillez cliquer sur français [SWITCH TO FRENCH VERSION].
Your participation is voluntary, and your responses will be kept entirely confidential. The survey takes about 7 minutes to complete.
START SURVEY
To view our privacy policy, click here.
If you require any technical assistance, please contact Ipsos at pasurveys @ ipsosasks.ca
i) Does anyone in your household work for any of the following organizations?
SELECT ALL THAT APPLY
IF "NONE OF THESE ORGANIZATIONS" CONTINUE, OTHERWISE THANK AND TERMINATE.
j) Are you…?
c) In what year were you born?
YYYY
ADMISSIBLE RANGE 1900 -2004
IF > 2003, THANK AND TERMINATE
ASK D IF QUESTION C IS LEFT BLANK
k) In which of the following age categories do you belong?
SELECT ONE ONLY
IF "LESS THAN 18 YEARS OLD" THANK AND TERMINATE
l) In which province or territory do you live?
SELECT ONE ONLY
SCREENER QUESTIONS TO IDENTIFY IF HAVE CHILDREN UNDER 18 AND THEIR AGE
m) Are you the parent or guardian of a child or children under 18 years of age living in your household?
[ASK IF 1 at F]
n) How old is/are the child/children? Select all that apply
SCREENER QUESTIONS TO IDENTIFY IF INDIGENOUS POPULATIONS OR ETHNIC/RACIALIZED POPULATIONS
o) Do you identify as any of the following? Select all that apply
ASK ALL RESPONDENTS
Q1: Over the past six weeks, have you seen, read or heard any advertising from the Government of Canada?
Q2: Think about the most recent ad from the Government of Canada that comes to mind. Where have you seen, read or heard this ad?
SELECT ALL THAT APPLY
[RANDOMIZE]
Q3: What do you remember about this ad?
Q4: How did you know that it was an ad from the Government of Canada?
ASK ALL RESPONDENTS
T1A: Over the past six weeks, have you seen, read or heard any Government of Canada advertising about free resources or services in support of mental health and substance?
T1B: Where have you seen, read or heard this ad about mental health and substance use supports linked to the COVID-19 pandemic?
SELECT ALL THAT APPLY
[RANDOMIZE]
T1C: What do you remember about this ad?
ASK ALL RESPONDENTS
Q1 Generally speaking, how would you rate your overall mental health?
Q2. Has the COVID pandemic affected your mental health?
Q3 For each of the items listed below, please indicate whether you have experienced any changes since the start of the pandemic by selecting increased, decreased, or stayed the same/no change.
[RANDOMIZE]
Q4 Please indicate if the situations listed below changed for you since the beginning of the pandemic by selecting increased, decreased or the same/no change.
[RANDOMIZE]
Q5 Have you recently received help for mental health? (select one only)
T1H: Here are some ads that have recently been broadcast on various media. Click next to watch.
[show banner ad first (jpg) then video ad show each ads separate screen]
[Video: https://www.youtube.com/watch?v=Em6w5SxF-c8]
[Banner to be added]
[CLICK TO GO TO THE NEXT PAGE]
Over the past six weeks, have you seen these ads?
T1I: Where have you seen, read or heard these ads? (Select all that apply)
T1J: What do you think is the main point these ads are trying to get across?
T1K: Please indicate your level of agreement with the following statements about these ads?
RANDOMIZE STATEMENTS
| 1 Strongly Disagree | 2 | 3 | 4 | 5 Strongly Agree | |
|---|---|---|---|---|---|
| These ads catch my attention | |||||
| These ads are relevant to me | |||||
| These ads are difficult to follow | |||||
| These ads do not favour one political party over another | |||||
| These ads talk about an important topic | |||||
| These ads provide new information | |||||
| These ads clearly convey that free mental health and substance use supports and resources are available | |||||
| These ads prompt me to visit a Government of Canada website to learn more | |||||
| These ads prompt me to visit additional websites for free mental health and substance use supports and resources | |||||
| These ads prompt me to search for free mental health and substance use supports and resources at a later time |
[Ask all]
T1L. Did you [If NO in T1H: Will you] do any of the following as a result of seeing the ads? Please select all that apply
[RANDOMIZE]
D1: Which of the following categories best describes your current employment status? Are you…
SELECT ONE ONLY
D2: What is the highest level of formal education that you have completed?
SELECT ONE ONLY
D4: Which of the following categories best describes your total annual household income, including income from all household members, before taxes are deducted?
SELECT ONE ONLY
D5: Where were you born?
ASK IF D5=BORN OUTSIDE CANADA
D6: In what year did you first move to Canada?
YYYY
ADMISSIBLE RANGE: 1900-2022
D7: What is the language you first learned at home as a child and still understand?
SELECT UP TO TWO
That concludes the survey. This survey was conducted on behalf of Health Canada. In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to answer this survey, it is greatly appreciated.