Evaluation of the 2021 Remembrance Advertising Campaign: Methodology Report

Prepared for Veterans Affairs Canada

Supplier Name: Phoenix SPI
Contract Number: 51019-220187/001/CY
Contract Value: $55,324.24 (including HST)
Award Date: 2021-06-22
Delivery Date: 2021-11-01
Registration Number: POR 018-21

For more information on this report, please contact Veterans Affairs Canada at: Commsresearch-commsrecherche@veterans.gc.ca

Ce rapport est aussi disponible en français.

This public opinion research report presents the results of two online surveys conducted on behalf of Veterans Affairs Canada using the Government of Canada’s Advertising Campaign Evaluation Tool (ACET).

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Veterans Affairs Canada. For more information on this report, please contact Veterans Affairs Canada at:

Commsresearch-commsrecherche@veterans.gc.ca

Catalogue number: V44-8/2021E-PDF

International Standard Book Number (ISBN): 978-0-660-41488-1

Related publications (POR registration number: POR 018-21):
Catalogue number (Final report, French): V44-8/2021F-PDF
ISBN: 978-0-660-41489-8

Aussi offert en français sous le titre Campagne du Souvenir 2021: Évaluation de concepts

Summary
Introduction
Appendix
- 1. ACET: Baseline Survey
- 2. ACET: Post-campaign Survey

Summary

1. Background

Within VAC, the core responsibility of commemoration is unique in that it serves Veterans and their families, recognizes the contributions of those who made the ultimate sacrifice, along with engaging the broader Canadian public in remembrance. A national advertising campaign is one way VAC seeks to engage Canadians in remembrance. The 2021 Remembrance advertising campaign ran from November 1 to 11, 2021 with ads appearing on television, digital, and online.

2. Research Objectives

The purpose of this research was to evaluate the 2021 Remembrance campaign using the Government of Canada’s Advertising Campaign Evaluation Tool (ACET), a standardized survey questionnaire administered before and after the campaign’s paid media placement. The specific objectives were to measure:

unaided recall of Government of Canada advertising about Veterans;
Canadians’ participation in commemorative events;
aided recall of the Remembrance campaign by media;
recall of key messages of the ads;
perceived importance of VAC’s commemorative programming
satisfaction with VAC’s communications about the services and benefits available to Veterans and their families; and
knowledge of the countries in which Veterans and members of the Canadian Armed Forces have served over the years.

The target population for the pre-campaign baseline and post-campaign surveys were Canadians aged 18 and older.

3. Methodology

Evaluation of the 2021 Remembrance campaign involved two online surveys: a pre-campaign baseline survey and a post-campaign survey. The baseline survey was conducted with a sample of 2,312 Canadian adults and took place from October 21 to 29, 2021. Following the pre-campaign baseline survey, the Remembrance campaign was launched and ran from November 1 to 11, 2021. The post-campaign survey was conducted following Remembrance Day to assess the reach of the department’s media campaign. The post-campaign survey was in field November 12 to 22, 2021 and was completed by 2,276 Canadian adults. The baseline questionnaire had an average duration of approximately seven minutes while the post-campaign questionnaire averaged 13 minutes to complete.

4. Political neutrality certification

I hereby certify as a Senior Officer of Phoenix Strategic Perspectives that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Policy on Communications and Federal Identity of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not contain any reference to electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leader.

(original signed by)

Alethea Woods
President
Phoenix SPI

5. Contact value

The contract value was $55,324.24 (including HST).

Introduction

Phoenix Strategic Perspectives Inc. (Phoenix SPI) was commissioned by Veteran's Affairs Canada (VAC) to conduct quantitative public opinion research to measure Canadians’ reactions to the department’s 2021 Remembrance campaign.

1. Background

Within VAC, the core responsibility of commemoration is unique in that it serves Veterans and their families, recognizes the contributions of those who made the ultimate sacrifice, along with engaging the broader Canadian public in remembrance. A national advertising campaign is one way VAC seeks to engage Canadians in remembrance. The Remembrance campaign focuses on engaging and honouring all who have served our country in times of war, military conflict and peace, and who made life better for Canadians and the world.

The 2021 Remembrance advertising campaign ran from November 1 to 11, 2021 with ads appearing on television, digital, and online. Specifically: TV (November 5 to 11); digital (CBC/SRC), Spotify (November 11) and Apex (November 1 to 11); programmatic (YouTube, CBC/SRC, Connected TV, Spotify, CPPM – top 1000); and social media (Twitter, Snapchat, Facebook, Instagram, Tiktok). The campaign used a combination of words and pictures to dramatize the various experiences of those who served from the First World War through to modern day missions. The call to action of the campaign was for Canadians to remember Canada’s Veterans and visit Veterans.gc.ca/CanadaRemembers.

2. Research Purpose

According to the Government of Canada’s Communications Policy, federal government advertising valued at more than $1 million must be assessed to determine its impact and value. The purpose of this research was to evaluate the 2021 Remembrance campaign using the Government of Canada’s Advertising Campaign Evaluation Tool (ACET), a standardized survey questionnaire administered before and after the campaign’s paid media placement. This data collection supports the Government of Canada’s goal of gauging Canadians’ knowledge of the efforts of Veterans and those who died in service to Canada, and ensuring Veterans who participated in Canadian military milestones are publicly recognized for their service. The information obtained through this public opinion research will allow VAC to determine the impact of its advertising campaign and provide direction on areas where the campaign can be adjusted to reach a broader Canadian public in future years.

3. Research Objectives

The purpose of the research was to evaluate the effectiveness of the advertising campaign. The specific objectives were to measure:

unaided recall of Government of Canada advertising about Veterans;
Canadians’ participation in commemorative events;
aided recall of the Remembrance campaign by media;
recall of key messages of the ads;
perceived importance of VAC’s commemorative programming
satisfaction with VAC’s communications about the services and benefits available to Veterans and their families; and
knowledge of the countries in which Veterans and members of the Canadian Armed Forces have served over the years.

The target population for the pre-campaign baseline and post-campaign surveys were Canadians aged 18 and older.

4. Sample Source and Sampling Plan

The sample source used was Advanis’ proprietary General Population Random Sample (GPRS) ^{Footnote 1}, which has been developed using probability-based recruitment; specifically, RDD via Interactive Voice Response (IVR) and via live Computer Assisted Telephone Interviewing (CATI). The pre-campaign baseline and post-campaign survey samples were regionally disproportionate to increase the accuracy of regional results. The table below presents the target and actual number of completes by province or region of Canada for each wave: the pre-campaign baseline and post-campaign survey.

	Target # of Interviews	Pre-Campaign Completed Interviews	Post-Campaign Completed Interviews
Atlantic provinces	200	272	252
Quebec	400	413	407
Ontario	600	674	652
Manitoba/Saskatchewan/Nunavut	200	226	237
Alberta/Northwest Territories	250	302	270
British Columbia/Yukon Territories	350	425	485
Total	2,000	2,312	2,276

The margin of error for both the pre-campaign baseline survey and post-campaign survey was ±2.0% (19 times out of 20). The margin of error is greater for results pertaining to subgroups of the total samples.

5. Questionnaire

The questionnaire for the pre-campaign baseline survey and the post-campaign survey was the Government of Canada’s ACET. In the ACET, the question order, wording and coding is standardized. The only alteration to the questionnaire may be the addition of campaign-specific questions to obtain additional data of particular interest to the Department. For both waves, additional department-specific questions followed the core ACET questions. Both questionnaires contained primarily closed-ended questions, as well as several open-ended questions; there were five in the pre-campaign baseline survey and six in the post-campaign survey. The pre-campaign baseline survey questionnaire had an average duration of seven minutes and the post-campaign survey questionnaire averaged 13 minutes to complete. The final questionnaires can be found in the Appendix of this report.

6. Survey Programming and Testing

Both surveys were programmed using a mobile-optimized Computer Assisted Web Interviewing (CAWI) platform (which allows respondents to complete the survey on various platforms, including computers, tablets, and smartphones). The questionnaires and collateral materials (survey invitation and reminders) were available in both official languages. In advance of the fieldwork, both online questionnaires were thoroughly tested by team members to ensure the skip patterns and questions were correctly programmed.

Following the internal testing, the surveys were deployed in the form of a soft launch. SMS invitations to complete the surveys were sent to a small number of panelists. For each pre-test, after at least 20 surveys were completed, 10 in each language, the results were reviewed to assess data quality and general functioning of the questionnaire. In addition to responding to the survey questions, pre-test respondents were invited to offer comments on each questionnaire. No substantive issues were identified during the pre-tests. Once the reviews were completed for each survey, the online questionnaires were launched in full.

7. Survey Administration

As noted, the surveys were administered online. The following specifications applied:

Panelists were invited to complete each online survey via telephone and SMS.
Both online surveys were accessed through a unique link with an embedded code.
The survey invitations and survey introduction/landing pages identified the Government of Canada as the sponsor.
The surveys were available for completion in the respondent’s official language of choice.
The online platform automatically identified respondents’ device type and presented the survey in a format appropriate to that device.
Respondents were able to stop at any point in the questionnaire and return at a later time to the last question completed before the session was terminated.
Two reminders were sent to respondents to help maximize the response rate.
The fieldwork dates were as follows:

Pre-campaign baseline survey: October 21st to 29th, 2021; and
Post-campaign survey: November 12th to 22nd 2021.

The fieldwork was conducted in accordance with the Government of Canada’s Standards for the Conduct of Government of Canada Public Opinion Research for online surveys, the standards set out by the Canadian Research Insights Council (CRIC), and applicable federal legislation, including the Personal Information Protection and Electronic Documents Act (PIPEDA), Canada’s private sector privacy law.

8. Participation Rate

8.1. Precampaign baseline survey

	Total
Total survey invitations (U+IS+R)	19,833
Phone Invites	13,178
SMS Invites	6,655
Unresolved (U)	4,658
No answer/line busy/voice mail	4,658
In-scope non-responding units (IS)	12,311
Phone refusal	5,243
Language barrier/illness/incapacity	67
Respondent break-off/did not complete the recruitment call	112
Invited by phone: did not complete online survey	1,712
Invited by SMS: did not complete online survey	5,177
Responding units (R)	2,864
Completed survey	2,312
Disqualified – industry	552
Response Rate = R/(U+IS+R)	14.4%

8.2. Post-campaign survey

	Total
Total survey invitations (U+IS+R)	13,549
Phone Invites	6,894
SMS Invites	6,655
Unresolved (U)	3,636
No answer/line busy/voice mail	3,636
In-scope non-responding units (IS)	7,463
Phone refusal	1,599
Language barrier/illness/incapacity	54
Respondent break-off/did not complete the recruitment call	49
Invited by phone: did not complete online survey	619
Invited by SMS: did not complete online survey	5,142
Responding units (R)	2,713
Completed survey	2,276
Disqualified – industry	437
Response Rate = R/(U+IS+R)	19.6%

9. Weighting

The pre-campaign baseline and post-campaign survey datasets were weighted to ensure the responses accurately represent the distribution of the adult Canadian population. The weighting of the datasets was based on three demographic variables: age, gender, and region (to adjust for the geographically disproportionate sampling). Respondents who did not provide a response to the question regarding gender, or who indicated that they identify as neither male nor female, were assigned a neutral adjustment factor of 1. The population data used to construct the weights was drawn from the most recent Census data available from Statistics Canada (2016).

10. Non-response Bias Analysis

A non-response analysis was conducted to assess the potential for non-response bias. Survey non-response can bias results when there are systematic differences between survey respondents and non-respondents. To undertake the analysis, characteristics of survey respondents (before weights were applied) were compared with those of the target population.

The following table compares the unweighted and weighted survey samples by age and gender for each of the surveys (geographic comparisons are not included because of the sample design, which provided for regionally disproportionate sampling):

Age	Pre-Campaign Survey				Post-Campaign Survey
	Male		Female		Male		Female
	Unweighted	Weighted	Unweighted	Weighted	Unweighted	Weighted	Unweighted	Weighted
18 to 24	4.6%	5.6%	3.1%	4.5%	5.1%	5.7%	3.3%	4.7%
25 to 34	20.0%	23.9%	16.2%	22.7%	21.3%	23.8%	15.9%	22.5%
35 to 44	16.1%	14.9%	14.0%	14.4%	13.8%	15.3%	17.4%	18.1%
45 to 54	18.7%	18.0%	17.5%	17.9%	16.4%	17.5%	13.5%	14.2%
55 to 64	19.5%	18.2%	24.9%	21.9%	20.5%	18.0%	22.5%	18.6%
65+	21.2%	19.3%	24.3%	18.6%	22.9%	19.6%	27.4%	21.9%

As is often the case with surveys of the general population, Canadians aged 55 and older, in particular females, are over-represented in the survey sample and those under 35 years of age, in particular 25- to 34-year-olds, are under-represented in the survey sample. In order for the results to be representative of the population of Canada, weights were applied to correct for this discrepancy between the sample proportions and the population.

Appendix

ACET: Pre-campaign Baseline Survey

INTRO

Thank you for taking the time to complete this survey dealing with current issues of interest to Canadians.

Si vous préférez répondre au sondage en français, veuillez cliquer sur « Français » dans le coin supérieur droit.

Your participation is voluntary and your responses will be kept entirely confidential and anonymous. The survey takes about 7 minutes to complete. This survey is being directed by Phoenix SPI and is being administered according to the requirements of the Privacy Act and is registered with the Canadian Research Insights Council's (CRIC) Research Verification Service (20211014-PH079).

D1A [1,7]

Does anyone in your household work for any of the following organizations?

SELECT ALL THAT APPLY

A marketing research firm........................................................................ 1

A magazine or newspaper......................................................................... 2

An advertising agency or graphic design firm............................................. 3

A political party........................................................................................ 4

A radio or television station...................................................................... 5

A public relations company....................................................................... 6

Federal or provincial government............................................................. 7

None of the above................................................................................. 98 SX

Prefer not to say.................................................................................... 99 SX

D1B

Are you...

Male gender............................................................................................ 1

Female gender......................................................................................... 2

Gender diverse........................................................................................ 3 B

Prefer not to say.................................................................................... 99 BX

D1C

In which of the following age categories do you belong?

Less than 18 years old.............................................................................. 1 ->THNK2

18 to 24................................................................................................... 2

25 to 34................................................................................................... 3

35 to 44................................................................................................... 4

45 to 54................................................................................................... 5

55 to 64................................................................................................... 6

65 or older............................................................................................... 7

D1E

In which province or territory do you live?

Newfoundland and Labrador.................................................................... 1

Prince Edward Island................................................................................ 2

Nova Scotia.............................................................................................. 3

New Brunswick........................................................................................ 4

Quebec.................................................................................................... 5

Ontario.................................................................................................... 6

Manitoba................................................................................................. 7

Saskatchewan.......................................................................................... 8

Alberta.................................................................................................... 9

British Columbia..................................................................................... 10

Yukon.................................................................................................... 11

Northwest Territories............................................................................. 12

Nunavut................................................................................................ 13

None of the above................................................................................. 99 SX ->THNK2

Over the past three weeks, have you seen, read or heard any advertising from the Government of Canada?

Yes.......................................................................................................... 1

No / Do not remember............................................................................. 2 ->T1A

Prefer not to say.................................................................................... 99 SX ->T1A

Q2 [1,16]

Think about the most recent ad from the Government of Canada that comes to mind. Where have you seen, read or heard this ad?

Select all that apply

Cinema.................................................................................................... 1

Facebook............................................................................................... 12

Internet website...................................................................................... 2

Magazines............................................................................................... 3

Newspaper (daily).................................................................................... 4

Newspaper (weekly or community)........................................................... 5

Outdoor billboard.................................................................................... 6

Pamphlet or brochure in the mail.............................................................. 7

Public transit (bus or subway)................................................................... 8

Radio....................................................................................................... 9

Television.............................................................................................. 10

Twitter.................................................................................................. 11

YouTube................................................................................................ 13

Instagram.............................................................................................. 14

LinkedIn................................................................................................. 15

Other, specify:....................................................................................... 77

Prefer not to say.................................................................................... 99 SX

What do you remember about this ad?

77......................................................................................................... 77

Don't remember.................................................................................... 97 SX

T1A

Over the past three weeks, have you seen, read or heard any Government of Canada advertising about Canadian Veterans?

Yes.......................................................................................................... 1

No........................................................................................................... 2 ->T1D

I don't remember / Not sure..................................................................... 3 ->T1D

Prefer not to say.................................................................................... 99 SX ->T1D

T1B [1,16]

Where have you seen, read or heard this Government of Canada ad about Canadian Veterans?