Executive summary

A. Background and objectives

Veterans Affairs Canada (VAC) supports the well-being and recognition of Canada’s Veterans and their families and caregivers. Budget 2019 highlighted that the Government will continue to recognize its solemn duty to those who have chosen to serve in the Canadian Armed Forces. The Government will build on the $10 billion previously invested to deliver better outcomes for Canada’s Veterans by improving mental health care supports and helping ensure that every homeless Veteran has a place to call home.

Mental health is a key area of focus in the services delivered by Veterans Affairs Canada. VAC’s Mental Health Strategy (2015–2020) establishes that “Veterans with a service-related mental health condition, served by Veterans Affairs Canada, will have timely access to evidence-based mental health care.” For 2020-21, a newly developed Mental Health Services for Veterans and their Families national advertising campaign will highlight the services and supports available.

This research program required the pretesting of messages and concepts, and the administration of evaluation surveys to test the effectiveness of the campaign (based on the Government of Canada’s Advertising Campaign Evaluation Tool, or ACET). As this is a large publicity campaign with a budget of over $1 million, it is critical that VAC ensure it resonates with the intended audiences. By conducting qualitative and quantitative research, the Department will be able to assess the effectiveness of the creative treatments and make adjustments as required based on feedback.

The goal of the concept and message-testing qualitative phase was to determine whether the creative treatments resonate with the target audiences and determine what changes, if any, are needed in the draft creative to meet the advertising campaign objectives.

Pre- and post-campaign evaluation were used to assess and track campaign performance using the advertising campaign evaluation tool (ACET), in accordance with the Communications Policy of the Government of Canada, and mandatory for all advertising campaigns with a media component of $1 million or more. The ACET testing, in the second phase of this project, were designed to gauge:

· the effectiveness of the campaign to drive traffic to veterans.gc.ca/mental health and engage on social media sites, or call VAC’s 1-866 number.

· Veterans’ and Canadians’ awareness/knowledge of the campaign (as heard on radio or seen in print, on the web, and in social media).

· personal relevance of the campaign.

B. Methodology

This research study consisted of two separate but complementary phases, described below.

Phase 1: Creative testing

Environics Research conducted a series of twelve online focus groups with Veterans and family members/caregivers of Veterans. Environics recruited participants via a combination of industry-standard methods to ensure participation in all the sessions across all provinces. Veterans were defined as those who had served in either the Canadian Armed Forces or the RCMP in the past and who had been honourably discharged. Participants were offered a $100 incentive to thank them for their time. Environics invited 7 participants to each session.

The sessions were hosted using the Zoom conferencing platform. The twelve focus groups were conducted in two phases. The first phase of six focus groups (October 28-31, 2020) was conducted to explore reactions to a series of storyboard concepts and messages on mental health services. The second phase of six focus groups (November 12 – 16, 2020) was conducted to explore reactions to radio concepts and animatized versions of the ad concepts that had been tested in the initial phase. Eight of the twelve sessions were conducted in English and four were conducted in French.

The sessions were distributed as follows:

Location of participants	Dates and times	Composition and language
Prairies/NWT	Wednesday, October 28, 2020; 4:00-5:30pm MST	Veterans (English)
Prairies/NWT	Wednesday, October 28, 2020; 6:00-7:30pm MST	Family members of Veterans (English)
Atlantic provinces	Thursday, October 29, 2020; 4:00-5:30pm AST	Veterans (English)
Atlantic provinces	Thursday, October 29, 2020; 6:00-7:30pm AST	Family members of Veterans (English)
Québec/Atlantic	Saturday, October 31, 2020 11:00-12:30pm EST	Veterans (French)
Québec/Atlantic	Saturday, October 31, 2020 1:00-2:30pm EST	Family members of Veterans (French)
B.C./Yukon	Thursday, November 12, 2020; 3:00-4:30pm PST	Veterans (English)
B.C./Yukon	Thursday, November 12, 2020; 5:00-6:30pm PST	Family members of Veterans (English)
Ontario	Saturday, November 14, 2020 11:00-12:30pm EST	Veterans (English)
Ontario	Saturday, November 14, 2020 1:00-2:30pm EST	Family members of Veterans (English)
Québec/rest of Canada (ex. Atlantic)	Monday, November 16, 2020; 4:00-5:30pm MST	Veterans (French)
Québec/rest of Canada (ex. Atlantic)	Monday, November 16, 2020; 6:00-7:30pm MST	Family members of Veterans (French)

Statement of limitations: Qualitative research provides insight into the range of opinions held within a population, rather than the weights of the opinions held, as would be measured in a quantitative study. The results of this type of research should be viewed as indicative rather than projectable to the population.

More information about the methodology for the creative testing phase is included in Appendix A.

Phase 2: Pre- and post-campaign ACET surveys

Environics conducted bilingual, national, online surveys: one as a baseline, in advance of the campaign, and one following the end of the campaign to measure change in awareness and intent to visit over time, to assess impact of the campaign. The baseline included 2,000 Canadians 18 years of age or older, and the post campaign sample also included 2,000 Canadians, also 18 years of age or older. Each survey instrument was based on the Advertising Campaign Evaluation Tool questions used by the Government of Canada in all of its advertising evaluation research, adding a few campaign-specific questions testing the impact on perceptions and behaviour targeted though the campaign. Each survey instrument included a sample of randomly selected respondents from across the country.

The pre-campaign ACET field period was from December 7-15, 2020 and the post-campaign ACET was undertaken from March 9-18, 2021. Data were weighted by region based on 2016 Census proportions.

Because respondents are recruited from a panel, this is a non-probability survey, and no formal estimates of sampling error can be calculated. Although opt-in panels are not random probability samples, online surveys can be used for general population surveys provided they are well-designed and employ a large, well-maintained panel, as was the case here.

C. Contract value

The contract value was $129,957.55 (HST included).

D. Report

This report begins with an executive summary outlining key findings and conclusions of the creative testing qualitative research, followed by a detailed analysis of the creative testing qualitative research and a methodology report for both phases. Provided under separate cover are the detailed ACET “banner tables” presenting the results of the surveys for all questions by population segments as defined by region and demographics. These tables are referenced by survey question in the detailed analysis.

E. Use of findings of the research

This research supports the Government’s and Veterans Affairs Canada’s commitments to continue to improve awareness of mental health services, by ensuring the opinions of the Veteran community are considered. It also supports the continuing Government priority of accountability and performance measurement. The research informed final creative development by determining the potential for effectiveness and the resonance of messages and concepts with target audiences, and the information gathered will support decisions on how to communicate and better inform Veterans, their families and Canadians. In addition, ensuring the messages resonate with service providers and community stakeholders will help reach those who work with Veterans and their families not currently served by the Department.

Participants provided feedback on specific creative messages and concepts., which helped VAC gauge how they perform in terms of meeting the campaign objectives. The research informed final creative development by determining the potential for effectiveness and the resonance of messages and concepts with target audiences. Additionally, for the campaign launch in mid-December, both baseline and post-testing (ACET) measured aided and unaided recall, message retention, and effectiveness.

F. Creative testing - key findings

General observations

· Mentions of the issues Veterans currently face included: mental health problems, including PTSD, not knowing where and how to get help, difficulty submitting and receiving disability claims, feelings of loneliness, and transition to and integration into civilian life. Some also spoke of poverty among Veterans and the inadequacy of pensions and supports.

· When participants were prompted more specifically about mental health issues among Veterans, virtually all of them acknowledged this is a serious issue – particularly PTSD.

· When asked how they would seek help for mental health issues, participants said they would most likely speak to their general physician. Some mentioned seeking help from the Royal Canadian Legion. Others said they would search online for mental health services. Only a few said they would reach out to Veterans Affairs Canada.

Ad concept storyboards

· Participants appreciated all three of the concepts they were shown in storyboard format, in that they virtually all approved of the fact Veterans Affairs Canada was trying to raise awareness of services and programs available to help Veterans deal with mental health issues.

· There were some common key overall observations from the participants applicable to all three concepts: any ads must have a toll-free telephone number as well as a URL; inclusion of a 1-800 number for VAC mental health services may lead to an expectation there will be someone answering that number 24 hours a day who is able to provide counselling; ad content should be inclusive of RCMP Veterans and their families, as they often feel excluded from VAC communications.

· The “Faces” concept tested quite well with most participants. They appreciated the personal touch of facial images they could identify with, the diversity, the inclusion of a phone number and that it was a message intended “for those who have served.” Many participants picked up on the notion that someone can be smiling on the outside but be struggling with mental health issues on the inside. Several participants noted there is a common stereotype that Veterans are all older men who fought in the Second World War, and that it was important to show the reality of Veterans who are much younger and more diverse.

· Some felt the Faces concept recognized mental health “can take on many forms” and “can affect Veterans young and old.” The phrase “you are not alone” also resonated with some people, who noted how the experience of mental challenges can be very lonely and that this can help to destigmatize it.

· The “Awake at 3 a.m.” concept also tested quite well. Many participants noted they could personally relate to being awake at in the middle of the night and feeling depressed or anxious. Some spouses of Veterans noted they have experienced this indirectly through their spouse. The idea of the feeling of solitude and helplessness when awake in the middle of the night touched a chord. However, many felt that “Awake at 3 a.m.” was giving them the impression they could call a 24-hour help line where they could get counselling.

· The word “hope”/”l’espoir” was noted as being a powerful one that added to the effectiveness of the ad, and many appreciated the imagery of the old fashioned alarm clock having its numerals go from red to green and from 3 a.m. to hope.

The term “struggling” was appreciated when talking about mental health, since part of being in the armed forces or the RCMP involves being challenged and having to struggle.

· Reaction to the “Upside down” concept was initially mixed, at least partly because this concept may not lend itself as well to being a drawing as was the case with the other two concepts. Most appreciated the idea of the upside-down world as a metaphor for mental health challenges. Many family members in particular liked this metaphor and imagery, because it put them in the shoes of the Veteran and made them feel like they could be seeing the world through his or her eyes.

· Some appreciated how this upside-down image was a more subtle and less stigmatizing depiction of mental health challenges. Some wanted more “resolution” at the end, such as adding people to the final right side up screen to make it look like “you’re less alone,” or maybe having the images go from black and white to colour.

Messages

· When participants were shown a list of eight phrases or messages on the topic of mental health services certain terms such as “if you have a loved one that has served,” “mental health challenges,” and “tailored” resonated with participants across the groups.

· Messages H and F resonated most frequently with participants, as they specifically mentioned the military AND the RCMP and referred to mental health as a “challenge” and not as an “illness.” They also liked that G was inclusive of “any mental health challenge.” In addition, “tailored” resonated with participants because it made them feel counsellors would not take a one-size-fits-all approach. Instead, their unique challenges would be assessed and cared for. H was almost universally appreciated because it spoke to family members directly and not only to Veterans.

· Many were attracted to message G since it specifically addressed to whom the message is directed, and because of its use of the word “tailored.” “Tailored” resonated with participants because it made them feel counsellors would assess and care for their unique challenges as Veterans.

Radio and digital executions

· The “Faces” concept tested well in both radio and digital formats and was regarded as “relatable”. There was somewhat of a disconnect between the radio and digital concepts, as the diversity represented by the faces was not easily translated to a radio execution. The radio ad was considered relatable because participants said they themselves, or someone they knew, often put up a front, or knew that “sometimes a smile isn’t a smile”. The digital ad was considered relatable because participants felt the diversity of the faces made it easy for them to recognize themselves, or the Veteran in their life, and helped to erase the stereotype of who a Veteran is.

· Reactions to the radio and digital executions of the “Awake at 3 a.m.” concept were more mixed. While some appreciated how it presented a scenario many could relate to – being awake and experiencing anxiety in the middle of the night – others found it to be too specific and repetitive. They also often interpreted the message of both the radio and digital executions to be a promotion of a 24/7 helpline for Veterans in distress. Some felt the messaging was too specific and wasn’t inclusive of the range of mental health issues Veterans face. Others felt that because the ad was focused on “being awake at 3 a.m.,” it was a promotion for insomnia remedies. French participants were more critical of the radio execution for the use of the word “perdu,” as they felt it focused too much on the problem as opposed to the solution.

· The radio and digital executions of the “Upside down” concept had a better reception in this second phase. While some participants found the sight of an upside-down streetscape to be jarring, they also found it drew them in because it was something they don’t usually see. The radio execution of this concept did not always resonate as well as the digital execution, but specific phrases in the radio ad, such as “tailored support for you and your family” and “you know when you’re not yourself” were well received. The “upside down” metaphor and image was generally liked, especially by Veterans’ family participants, as they felt it was situational and therefore relatable. It was also a visual image that would make people curious and want to know more.

G. Political neutrality statement and contact information

I hereby certify as senior officer of Environics that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada, and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leaders.

Derek Leebosh

Vice President, Public Affairs, Environics Research Group

derek.leebosh@environics.ca

(416) 820-1963

Contract Number: 51019-184024/001/CY

Supplier name: Environics Research Group

PWGSC contract number: 51019-201019/001/CY

Original contract date: 2020-10-20

For more information, contact the Department at Commsresearch-commsrecherche@veterans.ca