Employment Insurance Client Experience Transformation Survey - Methodology Report
Prepared by Employment and Social Development Canada
Supplier name: Angus Reid Group Inc.
Contract value: $30,849.00 (tax in)
Contract award date: September 22, 2023
For more information on this report, please contact ESDC at: nc-por-rop-gd@hrsdc-rhdcc.gc.ca
Ce rapport est aussi disponible en français.
Employment Insurance Client Experience Transformation Survey - Methodology Report
Prepared by Service Canada’s Benefits Delivery Modernization on Employment Insurance (BDM on EI) team at Employment and Social Development Canada
The reproduction of this publication is permitted for personal or public non-commercial purposes. For all other uses, prior written permission must be obtained from ESDC. For more information, please email the droitdauteur.copyright@HRSDC-RHDCC.gc.ca.
Catalogue Number:
SG5-104/2024E-PDF
ISBN:
978-0-660-71367-0
Aussi disponible en français sous le titre : Sondage sur la transformation de l'expérience client de l'assurance-emploi - Rapport méthodologique
Nº de cat. :
SG5-104/2024F-PDF
ISBN:
978-0-660-71368-7
© His Majesty the King in Right of Canada, as represented by the Minister of Citizens’ Services, 2024.
Research Purpose and Objectives
Contract value of the POR study
Appendix C: Research Instruments
The Benefits Delivery Modernization (BDM) Programme is a large-scale transformation programme designed to deliver an improved client experience for the Employment Insurance (EI), Old Age Security (OAS), Canada Pension Plan (CPP) and other benefits through a modern technology platform streamlining benefits processing, client centric digital services, and enhanced service management capabilities. Central to the success of BDM is user research with Service Canada clients, enabling a deeper understand the pain points in the client experience, the ability to validate design hypotheses, and co-create and test potential solutions.
In October 2023, a mixed-method study with people who considered or otherwise engaged with the EI program in the last 24 months. The purpose of the Client Experience Transformation (CXT) survey was to better understand the EI journey, with a particular focus on experiences of vulnerable and under-served individuals.
Building on past research efforts within BDM, the CXT survey sought to explore the end-to-end EI experience; for example, how individuals become aware of EI and their eligibility, how they apply for benefits, how they seek out help, and how they feel about the overall service experience. The survey also explored barriers to access within the client experience, which enabled researchers to identify several opportunity areas for future transformation.
This report presents the methodology for the conduct of the online survey designed by Service Canada’s BDM on EI team, with data collection conducted by the Angus Reid Group (ARG) Inc.
The CXT survey data collection was conducted by the Angus Reid Group (ARG) Inc. using an online survey from October 4th to October 31st, 2023. The survey was completed by a total of 358 respondents.
Initial outreach through Angus Reid’s online platform consisted of a national online panel survey of people who considered or otherwise engaged with the EI program in the last 24 months, including those who applied for EI, were eligible and not eligible, and those who did not apply.
Since the incidence rate was low following the initial outreach, ARG conducted a series of additional outreach efforts, increasing participation amongst vulnerable client segments. This targeted “boost” sample included Indigenous people, people with disabilities, newcomers, and self-employed individuals. The boost sample served to capture the experiences of communities who have been historically under-served and to create a channel for recruitment of qualitative interviews. Of the total 358 survey respondents, 50% represented one or more of the targeted vulnerable client segments (or “boost” sample).
The results from the CXT survey will be used to inform future product and service design strategy within the BDM programme. The survey, which consisted of 358 respondents, also helped to drive the qualitative research methodological approach (one-on-one interviews with underserved client groups). The findings from the CXT survey and qualitative research activities highlighted areas of optimization and transformation for improving EI client experience.
Given that the online survey methodology entailed a non-probability sampling, and the specific demographic proportions of the universe of the target audience are unknown, the data collected cannot be extrapolated to the broader population. Results should be understood as indicative to inform researchers’ understanding of the EI experience in a more qualitative manner.
Contract value: $30,849.00 (tax in)
Contract award date: September 22, 2023
A full set of tabulated data is available under a separate .CSV file. To ensure respondent confidentiality, the data has hidden demographic or respondent characteristic categories with a column or row total of fewer than 10 respondents. Any tables containing information with fewer than 10 respondents have been replaced with a dash and greyed out.
Below details how the Angus Reid Group (ARG) Inc. approached recruitment for the CXT survey.
· ARG participant panel
o The Angus Reid Group Inc. manages via market research panel experts an online research community. They employ a number of methods and varying techniques to ensure that participants within the community are genuine, engaged, and representative of the Canadian population. A variety of advertising messages and sources of recruitment (e.g., social media, digital advertising, media partnerships and select affiliate partners) are used to attract both broad and targeted groups of people to reach a diverse range of audiences. ARG clients include commercial enterprises, not-for-profits, academic institutions, and non-governmental organizations. ARG adheres to the highest standards of market research practices to ensure true representativity and an accurate representation of what Canada, in all of its diversity, think, believe and feel on a variety of subjects. The scale of the online community enables ARG to draw random samples that represent the population as a whole.
· Sample size
o A total sample of 358 respondents participated in the CXT survey.
· Sampling procedures
o The CXT survey respondents were selected based on the following target demographics (1) Indigenous, (2) People with disabilities, (3) Newcomers to Canada within less than 5 years and (4) Self-employed. These demographics were collected during participant onboarding to the ARG panel and therefore part of their panel database. Additional criteria were included in the screening questionnaire that aligned with the study’s objectives (e.g., engagement with the EI program in the last 24 months).
· Dates of research field work
o The CXT survey was fielded from October 4th to October 31st, 2023.
· Potential for bias in the research
o This study utilized a convenience sample of people with very specific characteristics (Indigenous, disabled, newcomers and self-employed who had experience with employment insurance). Noteworthy possible sample biases include selection bias, non-response bias, and sample frame error arising from choosing to be a member of the ARG research community. There is also a wide array of potential non-sample-related biases including measurement error, respondent error, and a variety of biases including social desirability, recall, acquiescence, and question order/anchoring.
· Weighting procedures, the confidence interval, and the margin of error
o The sample was sourced from ARG’s online panel community. Since the research was focused on people who have considered or engaged with EI, and aimed to study specific subgroups, people were screened to ensure they meet the criteria for the study. However, in the absence of an available profile of the total pool of current or recent EI clients in Canada, the research cannot assume that the resulting sample was proportionally representative, and therefore cannot apply weighting or establish a confidence interval. Since a convenience sample was used and is not guaranteed to be representative of the general population, the confidence interval and margin of error are not applicable.
· Response rate and method of calculation
o Given the CXT survey results are not representative of the general population or the EI client population, the response rate and method of calculation should not apply. The final incidence rate of the study was 6%; some low-incidence groups in the survey brought the overall incidence rate down.
The CXT survey questionnaire was designed by BDM and was conducted online through ARG’s survey platform. Below you will find the CXT survey questions provided to survey participants.
Q1 - [MULTI CHOICE]
Do you or anyone in your household work for any of the following industries or occupations?
· Government
· Banking or insurance
· Automotive industry
· Marketing research
· Public relations or Advertising
· Media/Journalism
· Packaged food manufacturer
· None of the above
Q1a - Base: SELECTED GOVERNMENT AT Q1 - [SINGLE CHOICE]
You mentioned you or someone in your household works in the government. Do you or the person in your household work for Service Canada?
· Yes [TERMINATE]
· No
Q2 - Base: TOTAL - [MULTI CHOICE QUESTION]
For which of the following brands have you participated in any research or testing in the past 12 months?
· ROWS [RANDOMIZE]
· Service Canada [PN: TERMINATE IF SERVICE CANADA = YES]
· Mazda
· McDonalds
· None of the above [ANCHOR, EXCLUSIVE]
QETHNICITY - Base: TOTAL - [MULTI CHOICE]
What is your own ethnic or racial background? Are you:
(Many people come from a mixed background. In that case, please just select all that may apply.)
· Indigenous Canadian
· White (Caucasian) (e.g., English, French, German, Polish)
· Black (e.g., African, Haitian, Nigerian, Jamaican)
· South Asian (e.g., Indian, Pakistani, Bangladeshi, Nepalese)
· Chinese, Hongkonger or Taiwanese
· Filipino
· Other East Asian (e.g., Korean, Japanese, Vietnamese, Singaporean)
· Middle Eastern/West Asian (e.g., Arab, Iranian, Afghan, Iraqi)
· Latin American (e.g., Mexican, Colombian, Peruvian, Guatemalan)
· Other (Specify): [ANCHOR]
· Prefer not to say [EXCLUSIVE, ANCHOR]
Q3 - Base: TOTAL - [MULTI CHOICE]
Which of the following Federal Government services have you considered, applied for, or utilized in the last 24 months?
· [PN: RANDOMIZE]
· Canadian Pension Plan (CPP) or Old Age Security (OAS)
· Canadian Child Benefit (CCB)
· Passport Services
· Immigration or Citizenship services
· Veterans Affairs Canada (VAC) services
·
Employment Insurance (EI) Benefits and Leave
(i.e., regular benefits, sick leave,maternal/parental leave, caregiver leave,
compassionate caregiver leave, fishing benefits etc.) [MUST SELECT, ELSE TERMINATE]
· International Benefits (Lived or Living Outside Canada)
· Federal or Provincial Disability Support Programs
· Indigenous Services Canada (ISC)
· None of the above [EXCLUSIVE, ANCHOR PN: TERMINATE]
Q4 - Base: IF Employment Insurance (EI) was selected in Q3 - [SINGLE CHOICE]
Not including vacation, has there been a 7-day period in the last 24 months where you didn’t work?
· Yes
· No [PN: TERMINATE]
Q5 - Base: TOTAL - [SINGLE CHOICE]
When was the last time you engaged with the Employment Insurance (EI) program in any way?
· 1 – 5 months ago
· 6 – 12 months ago
· 13 – 23 months ago
· More than 2 years ago [PN: TERMINATE]
Q6 - Base: TOTAL - [SINGLE CHOICE]
What is your current employment status?
[RANDOMIZE]
· Employed full-time
· Employed part-time
· Unemployed
· On leave of absence from employment
· Self-employed (i.e., freelancer or independent business owner registered as self-employed and/or applied to collect benefits within the last 12 months)
· Mixed employment (e.gstudent + part-time work)
· Student
· Retired
· Other [ANCHOR]
Q7 - Base: TOTAL - [SINGLE CHOICE]
What is personal annual income before tax?
[SAMPLE NOTE: Skewed towards low-income; 65% below $54K, 25% between $54 - $100K, 10% above $100K]
· Under $20,000
· $20,000 - $39,999
· $40,000 - $53,999
· $54,000 - $79,999
· $80,000 - $99,999
· $100,000 - $149,999
· $150,000 and over
Q7ROLLUP - Base: TOTAL - [HIDDEN SINGLE CHOICE]
· Below $54K [PN: CODES 1,2,3 AT Q7]
· Between $54K-$100K [PN: CODES 4 AND 5 AT Q7]
· $100K+ [PN: CODES 6 AND 7 AT Q7]
Q8 - Base: TOTAL - [SINGLE CHOICE]
When did you first move to Canada?
· 0 – 4 years ago
· 5 – 10 years ago
· 10+ years ago
· I have always lived in Canada.
Q9 - Base: TOTAL - [SINGLE CHOICE]
Do you self-identify as a person with disabilities? (i.e., physical, sensory or mobility impairment that is permanent or episodic)
· Yes, I have a physical impairment.
· Yes, I have a sensory impairment.
· Yes, I have a mobility impairment.
· Yes, I have a combination of the above impairments.
· No, I do not self-identify as a person with disabilities.
BOOST SAMPLE MUST QUALIFY FOR ONE OF THE FOLLOWING, ELSE TERMINATE:
INDIGENOUS: QETHNICITY = CODE 1
DISABILITY: Q9 = CODES 1,2,3 OR 4
NEWCOMERS: Q8 = CODE 1
SELF EMPLOYED: Q6 = CODE 6
BOOSTFLAG - Base: SAMPLEFLAG = 2 - [HIDDEN MULTI CHOICE]
INDIGENOUS: QETHNICITY = CODE 1
DISABILITY: Q9 = CODES 1,2,3 OR 4
NEWCOMERS: Q8 = CODE 1
SELF EMPLOYED: Q6 = CODE 6
NONE OF THE ABOVE
QVIDEO - Base: TOTAL - [SINGLE CHOICE]
In this survey we’d like to do something a little different! We find that hearing people explain their answer directly helps us better understand their views. If you are interested, we will ask you to record a video response to one question later in this survey. To thank you for sharing your thoughts with us, you’ll receive 500 Angus Reid Forum Points. Your points will be automatically awarded to your account within 10 business days if your video is clear and answers our question. Please make sure your video and audio are clear so we can understand your opinion about our question!
Before we get started, we just need your consent on a few things. By participating in this survey, and future surveys with us, you acknowledge and agree that:
(1) you are 18 years of age or older,
(2) all ideas, images, videos, or voice recordings (the “responses”) shared by you during the survey may be used, for research, marketing and other purposes, without any further obligations or payment to you,
(3) all rights, title and interest in and to the intellectual property in the responses is assigned to the end client, and
(4) you hereby release us from all claims by you in connection with any such usage of the responses, your name and/or likeness.
Yes, I consent
No, I’d rather type my answer
[PN: RESPONDENTS TO ANSWER ALL OPEN-ENDS QUALIFIED FOR. IF YES AT QVIDEO, ASSIGN RESPONDENT TO ANSWER ONE OPEN-END AS VIDEO RESPONSE Q18/Q21/Q26/Q28. LEAST FILL ON BOOSTFLAG]
VIDEO INSTRUCTION TEXT [TO SHOW ON VIDEO OPEN-END PAGE]
To record:
1. Click “Record video” and ensure camera is ready to record.
2. Start recording by clicking white button underneath video frame.
3. Click white button when you are done and select “Use Video”.
Q10 - Base: TOTAL - [MULTI CHOICE]
Earlier you noted you considered, applied for, or utilized Employment Insurance (EI) in the last 24 months. There are multiple types of EI benefits that people can apply for, depending on their need (i.e., regular benefits, sickness benefits, parental leave, caregiving or compassionate care leave, fishing benefits, etc.). Which of the following reasons did you have for considering, applying for, or utilizing using the Employment Insurance (EI) program?
[PN: RANDOMIZE]
· I was laid off from work due to shortage of work, mass lay-offs or terminated without cause.
· I quit my job due to retirement, health reasons, or moving to accompany a spouse or dependent.
· My seasonal, temporary, or part-time job has now ended.
· I was unable to work due to illness, injury, or quarantine.
· I was pregnant, gave birth, adopted a child or was caring for a newborn.
· I was away from work to care for or support a critically ill or injured person or someone needing end-of-life care.
· Other (Please specify) [ANCHOR]
· I was out of work for a period of time but did not apply for EI. [EXCLUSIVE, ANCHOR]
Q11 - Base: TOTAL - [SINGLE CHOICE]
Think about your most recent benefits. What is/was the status of your EI benefits?
· I never applied for EI benefits. (PN: If this is selected, only complete questions through to Q18 then skip to Q28)
· I applied for EI benefits, and I am awaiting approval.
· I applied but was refused EI benefits.
· I have been approved to receive EI benefits but have not received a payment yet.
· I am currently receiving EI benefits.
· I have previously received EI benefits but am I am not receiving them now.
· Other (Please specify) [ANCHOR]
Q12 - Base: SELECTED CODE 1 AT Q11 - [MUTLI CHOICE]
What are the reasons you decided not to apply for EI benefits? Select all that apply.
[PN: RANDOMIZE]
· I assumed I was not eligible or would not be approved.
· I felt confused by the application process.
· I found the application process took too long.
· I couldn’t find the help I needed from customer support to complete the application.
· The customer support I got was not helpful.
· I had another income coming from someplace else (e.g., my employer)
· I am uncomfortable dealing with government organizations or representatives.
· A family member, friend or colleague told me I was not eligible to apply for benefits.
· Other (Please specify) [ANCHOR]
Q13 - Base: TOTAL - [SINGLE CHOICE]
Reflect on your personal Employment Insurance (EI) experience(s), regardless of benefit status (i.e., did not apply, applied, pending decision, received benefits, etc.). Click on the photo that best describes the feeling you get when you think about the overall experience.
[PN: RANDOMIZE]
· Q13 – Image 1
· Q13 – Image 2
· Q13 – Image 3
· Q13 – Image 4
· Q13 – Image 5
· Q13 – Image 6
Q14 - Base: TOTAL - [MULTI CHOICE]
You picked this picture to represent your experiences with EI benefits.
[INSERT IMAGE]
What 3 emotions do you associate with this picture and your experience?
[PN: SHOW IN ALPHA ORDER]
· Accepted
· Afraid
· Alone
· Angry
· Appreciative
· Confident
· Confused
· Embarrassed
· Sad
· Supported
· Respected
· Other (Please Specify) [PN: ANCHOR]
Q15 - Base: TOTAL - [SINGLE CHOICE]
Before you had a need to apply for Employment Insurance (EI) benefits, did you know anything about them?
· Yes
· No
Q16 - Base: TOTAL - [RANK ORDER]
Below is a list of questions people think about before applying for EI benefits. Rank each one in terms of its importance to you, where 1 represents the most important and 8 represents the least important.
[PN: RANDOMIZE]
· How much ($) benefits will I receive?
· When will I get paid?
· How long will I be able to be on EI?
· What support will I get from my employer?
· How long is the application process?
· How easily can I find information on my eligibility?
· How easy will it be to collect the required information and documents for my application?
· How difficult will it be to satisfy the application requirements in order to collect benefits on an ongoing basis?
· Does not apply to me [PN: EXCLUSIVE, ANCHOR]
Q17 - Base: TOTAL - [MULTI CHOICE]
Prior to completing the application, how did you find out if you were eligible for EI benefits? Select all that apply.
[PN: RANDOMIZE]
· I reviewed information given to me by my employer and/or unions.
· I used a search engine that brought me to a non-government website.
· I used a search engine that brought me to the Federal Government website (i.e., Canada.ca).
· I contacted Service Canada through a call center.
· I visited a Service Centre in person.
· I talked to family, friends, community groups and/or online forums.
· I looked on social media.
· Other (Please specify) [ANCHOR]
· I did not seek information and went straight to the application. [EXCLUSIVE, ANCHOR]
Q18 - Base: CODES 1,2,3,4,5, OR 6 AT Q17 - [OPEN END – TEXT OR VIDEO]
Can you describe your experience searching for information on EI through these sources? For instance, how useful and accurate was the information? Did you experience any frustrations? Did you understand what you needed to do next?
[PN: AFTER Q18, ANYONE WHO SELECTED CODE AT Q11, SHOULD SKIP TO Q28]
Q19 - Base: TOTAL - [SINGLE SELECT]
Before applying for EI, did you know how much you would receive in payment if your application was approved?
· Yes
· No
Q20 - Base: TOTAL - [MULTI SELECT]
How did you complete your application?
[PN: RANDOMIZE]
· Phone or tablet
· Personal computer
· In-person at a Service Canada Centre.
· Fax
·
None of the above [PN: EXCLUSIVE, ANCHOR]
Q21 - Base: TOTAL - [MULTI SELECT]
Which of the following steps in the application process stood out to you as particularly frustrating?
[PN: RANDOMIZE]
· Filling out the application
· Providing required documents
·
Completing a biweekly benefits report
· Updating personal information
· None of the above
Q21A - Base: Base: DID NOT SELECT NONE OF THE ABOVE AT Q21 - [OPEN END – TEXT OR VIDEO]
You selected [ANSWER FROM Q21] as being a frustrating step in the application process. Can you tell us a bit more about what you found frustrating?
Q22 - Base: TOTAL - [SINGLE CHOICE]
Did you complete your application all in one go or stop mid-application at any point? Select the option below that applies to you.
· I completed the application in one go.
· I stopped mid-application, saved, and returned to my application.
· I stopped mid-application, ended an application, and started a new one.
· I do not recall.
Q23 - Base: SELECTED CODE 2 OR 3 AT Q22 - [MULTI CHOICE]
Why did you stop mid-application and either save or end it?
[PN: RANDOMIZE]
· The application form was too long to complete in one sitting.
· I felt confused by the application process.
· I felt overwhelmed by the amount of information.
· I didn’t have my documents ready to submit with my application.
· I contacted Service Canada and needed their support.
· I experienced technology issues during the application process.
· I had to provide my Record of Employment (ROE) but my employer did not produce it
· Other (Please specify) [ANCHOR]
Q24 - Base: TOTAL - [MULTI CHOICE]
After you submitted your application, did you experience any of the following problems? Select all that apply.
[PN: RANDOMIZE]
· I didn’t receive any updates from Service Canada.
· I received confusing updates from Service Canada.
· I didn’t know how to track my application status.
· My payments were delayed.
· It wasn’t clear to me when I would get paid after I submitted my application.
· I had to provide additional information, and it was hard to understand what I needed to do.
· The waiting period between my application submission and benefits decision was longer than I expected.
· Completing the biweekly report was confusing.
·
I did not receive my 4-digit access code in the
mail, so I couldn’t submit my biweekly benefits report.
· Complications submitting my tax forms.
· Other (Please specify) [ANCHOR]
· None of the above [EXCLUSIVE, ANCHOR]
Q25 - Base: TOTAL - [MULTI CHOICE]
If you needed help applying, how did you contact Service Canada?
[RANDOMIZE]
· In-person (i.e., Service Center)
· Phone (i.e., Call Center)
· I looked online for an email or chat option
· Other (Please specify) [ANCHOR]
· I did not need help with my application [EXCLUSIVE, ANCHOR]
Q26 - Base: TOTAL - [OPEN END – TEXT OR VIDEO]
Reflect on your experience seeking support from Service Canada throughout your EI journey. What was most memorable about your experience? For instance, was there any interaction that was particularly helpful or unhelpful? Were you able to resolve any issues you had?
Q27 - Base: TOTAL - [SINGLE CHOICE]
Overall, which of these images best represents your most recent journey with Employment Insurance?
[PN: RANDOMIZE]
· Q27 – Image 1
· Q27 – Image 2
· Q27 – Image 3
Q28 - Base: TOTAL - [OPEN END – TEXT OR VIDEO]
Regardless of where you are in the EI journey from application to payment, what is one thing that could be changed or improved about the process and how would you like to see it change?
Q29 - Base: TOTAL - [SINGLE CHOICE]
We are looking for participants to chat more with us about their experience with EI. We are looking to recruit individuals to participate in a 60-minute interview between Tuesday, October 10 and Friday, October 27, 2023.
Would you be interested in participating in this 60-minute interview?
· Yes, I would like to participate!
· Yes, I’d like to participate but the above dates don’t work for me. Please keep me informed of future opportunities!
· No, I would not like to participate
Q29a - Base: SELECTED YES, I WOULD LIKE TO PARTICIPATE AT Q29 - [SINGLE CHOICE]
We’d like to confirm your contact information where we can reach you to schedule the interview.
· Full Name:
· Phone number:
· Email:
· I’ve changed my mind and would no longer like to participate [EXCLUSIVE]
Q30 - Base: IF YES IN Q29 AND DID NOT SELECT “I’VE CHANGED MY MIND…” AT Q29a - [SINGLE CHOICE]
Do you have a laptop or desktop equipped with a webcam on which you can conduct the interview?
· Yes
· No
Q31 - Base: IF YES IN Q30 - [SINGLE CHOICE]
Do you have a high-speed stable internet connection?
· Yes
· No
Q32 - Base: IF YES IN Q29 AND DID NOT SELECT “I’VE CHANGED MY MIND…” AT Q29a - [SINGLE CHOICE]
Do you have access to a phone to conduct the interview?
· Yes
· No
Q32a - Base: IF YES IN Q29 AND DID NOT SELECT “I’VE CHANGED MY MIND…” AT Q29a - [MULTI CHOICE]
Thank you for your willingness to participate in an interview. Interviews will be conducted Monday through Friday, October 10-27. What time(s) of day can you be available?
[Columns]
· Mornings
· Afternoon
· Evening
· None [EXCLUSIVE]
Q33 - Base: TOTAL - [SINGLE CHOICE]
Do you require the use of an assistive technology to support in accessing the internet and using digital devices (i.e., screen readers, screen magnifiers and alt navigation)?
· Yes (Please specify)
· No
Q34 - Base: TOTAL - [SINGLE CHOICE]
How comfortable are you with navigating digital devices and platforms (e.g., smartphones, laptops, websites, applications)?
· Very comfortable
· Somewhat comfortable
· Neutral
· Somewhat uncomfortable
· Very uncomfortable