Employment Insurance Service Quality Review Survey Final Report

Executive Summary

• The survey of Employment Insurance (EI) clients examined the quality of service delivery from the client’s perspective, and the degree to which clients can easily and efficiently move through the stages of the process from gathering information about the program to receiving a benefits decision.
• Most clients had a positive service experience.  They were satisfied with the quality of the service (78%), and, in this era in which EI applications are completed online, they felt that being able to do so made the overall service experience easier for them (84%).  When clients required assistance, almost all found staff to be respectful (91% for service by phone, and 90% for in-person service) and helpful (82% for service by phone, and 87% for in-person service).
• However, about one in five clients experienced difficulty with aspects of the process such as finding the information they were looking about the EI program, figuring out whether they were eligible and the steps to follow to apply, and using the online channel in general.  These clients were more likely to be seniors, to have less than high school education, to be unfamiliar with online services, or to be applying for special benefits, such as the EI Sickness Benefit, for which the application processes are more complex. 
• The desire to complete applications online is high, but assistance is required. A large majority of clients began their application online from home (72%). Three in ten needed assistance, half of whom found this at an in-person office, and 14% of whom got assistance via the phone.  Clients often had difficulty getting through on the phone, which was most often a call to an EI Specialized Call Centre.  Only 38% felt that the time they waited on the phone was reasonable, and nearly half tried two to five times before they reached an agent.
• During the stage between submitting an application and receiving a decision, about 1 in 5 clients who followed up on their application to, for example, check whether a decision had been made about their benefits encountered difficulty and, further, were often not sure what to do if they had a problem (29%).  While almost half the clients reported receiving a decision within two weeks, one third waited for what they felt to be an unreasonable amount of time.  The timeliness of a decision was found to have a notable impact on overall satisfaction, reflecting its importance to clients.
• While three quarters of the clients were able to move smoothly through the steps of the client journey one quarter was not. As the single service dimension with the greatest impact on overall satisfaction, this ability to move forward, easily completing steps and accessing assistance to address difficulties as they are encountered, is key to maintaining the quality of service for all EI clients and improving service for those experiencing difficulties. 

Background and Objectives

Background

Through Budget 2016, the Government of Canada committed to improve services for Canadians.  To support this commitment, the Government launched the EI Service Quality Review,  a nationwide consultation process to seek input from key stakeholders and the public on ways Service Canada can improve services to EI claimants.

Objectives

As part of this consultation process, Service Canada conducted a survey of EI clients to assess the overall client experience across the EI service delivery experience and all channels, with a focus on the following service dimensions:

• Ease of use
• Timeliness/Efficiency
• Satisfaction

The results provide representative data about the experiences and perspectives of the EI clientele that could be used for decision-making in accordance with the requirement to improve service delivery and to base such decisions on complete information.

Note: The client experience measurement model and questionnaire is based in part on 1) the Common Measurements Tool (CMT), owned and licensed by the Institute for Citizen-Centred Service (ICCS), 2) the model used by the Government of Quebec, and 3) the service issues identified in the consultation process of the EI Service Quality Review.

Research Methodology

A 15-minute survey was conducted by telephone with a sample of 1,528 EI clients. Interviewing was conducted in English and French between September 2 and 16, 2016. The following specifications applied:

• Interviews were conducted by Elemental Data Collection (EDCI) using Computer Assisted Telephone Interviewing (CATI) technology.
• Respondents were randomly selected from EI administrative data. This included clients who, between April 1^st and June 30^th, 2016:
  • Received a decision and/or
  • Began receiving benefits or
  • Submitted a partial application that was never completed.
• Sampling was stratified by the following criteria:
  • Claim status – approved (71%) and denied in part or in full (includes partial applications) (29%)
  • Benefit type – regular (64%) and special (36%)
  • Service delivery region -Atlantic Canada (13%), Quebec (23%), Ontario (25%) and Western Canada + the Territories (35%).
• The survey results were weighted to reflect the actual distribution of these characteristics in the EI population.
• The margin of error is ±2.5%, 19 times out of 20 (finite population factor applied). The results can be extrapolated to the entire EI population.
• The response rate was 15% (calculated using the Marketing Research and Intelligence Association [MRIA] formula).

Profile of Survey Respondents

• Representation of vulnerable clients:
  • Low education: 8% with less than high school diploma
  • Immigrants:
    • 22% not born in Canada
    • 14% of immigrants arrived in the last 3 years
  • First Nations, Métis or Inuit: 5%
  • Disability: 7%
  • E-Vulnerable: 21% rarely/never, 35% sometimes use online services
• Frequent claimants: 10% of survey respondents
• Age:
  • Youth (18 to 25 years): 8%
  • Adults (26 to 54 years): 65%
  • Seniors (55+ years): 27%
• Gender:
  • Male: 41%
  • Female: 59%
• Language: 79% completed the survey in English, 21% in French

Draft Service Canada Client Experience Survey Model

• The EI Client Survey was structured in part, using the client experience survey model that Service Canada is currently developing.
• In accordance with the model below, the survey assesses the multi-channel service experience across the stages of the client journey. Note: questions on channel use were limited to allow space for diagnostic questions about the EI service delivery process.
• It then assesses the overall client experience and the resulting client perception of the experience.
• In this survey, the experience with each individual service channel is also assessed to meet EI SQR diagnostic needs.

Text description of Figure 1: Service Experience

Performance

Overall Service Experience

*Satisfaction with Overall Quality of Service

Text description of Figure 2: Majority of Clients Satisfied with Overall Quality of Service

Overall Performance

• 78% were satisfied or very satisfied with the overall quality of service they received
• 89% would probably or definitely speak positively about the services received

Differences in Overall Satisfaction by Client Groups

Text description of Figure 4: Differences in Overall Satisfaction by Client Groups

Differences are statistically significant.

Performance by Stage of Journey

Clients rated performance very positively over their entire journey

Text description of Figure 5: Performance by Stage of Journey

Sample sizes for each variable vary

Service Experience for Overall Client Journey

Roughly 2 in 5 Had to Keep Following Up and 1 in 5 Got Bounced Around

Text description of Figure 6: Service Experience for Overall Client Journey

Performance - Ease and Effectiveness of Client Experience by Stage of Client Journey

Stage 1: Aware - Clients seek general information

Ease and Effectiveness of Gathering Information about EI Program

Majorities Found it Easy to Access Information about EI

• Three quarters or more found it easy or somewhat easy to get information about EI
• Aspects that caused some difficulty for 19-24% of clients included:
  • Determining eligibility
  • Determining steps in the application process
  • Finding the information they were looking for
• Top two reasons for difficulty were:
  • website navigation (37%)
  • lack of clarity of the eligibility requirements (27%)
• 88% said they got the information they needed in a reasonable amount of time

Text description of Figure 7: Ease and Effectiveness of Gathering Information about EI Program

Profile of Clients Having Difficulty Gather Information on EI Program

Note: only statistically significant difference from the overall EI clientele are presented.

Preferred Changes to Improve Ability to Gather EI Information

Text description of Figure 8: Preferred Changes to Improve Ability to Gather EI Information

Areas to Improve Stage of Gathering EI Information

• Ease of gathering information about EI program
  • Make it easier to find information about EI program online
    • 22% of clients who used the online channel needed help searching for information;
    • 37% of those who had difficulty finding information attributed this to the website—specifically, they said the website was difficult to navigate; and
    • 27% of clients are still using the in-person channel first to gather general information about EI.
• Clarity of information about EI program
  • Make it easier to understand information about EI
    • 24% found it somewhat or very difficult to determine eligibility for EI benefits; and
    • 27% of those who had difficulty finding information attributed this to lack of clarity—specifically, they said the eligibility information was not clear.
• Quick access by phone

Stage 2: Apply - Clients submit application

Ease and Effectiveness of Applying for EI Benefits

Majorities Found it Easy to Apply for EI Benefits

• 4 in 5 or more found the process of applying for EI to be  somewhat or very easy
• Completing the online application caused difficulty for the smallest proportion of clients – 12%
• The top two reasons for difficulty:
  • the language used in application was not clear (39%)
  • questions were not clear/were difficult to answer (31%)
• 96% considered time to complete application to be reasonable
• 32% said they needed help or assistance to complete their application

Text description of Figure 9: Ease and Effectiveness of Applying for EI Benefits

Profile of Clients Having Difficulty Applying for EI Benefits

Note: only statistically significant differences from the overall EI clientele are presented.

Preferred Changes to Improve EI Application Experience

Quicker access to an agent—change viewed as having the most positive impact on EI application experience

Text description of Figure 10: Preferred Changes to Improve EI Application Experience

Areas to Improve Stage of Submitting Application

• Ease of online process
  • Consider ease of completing the online application
    • 12% had difficulty completing the application
    • 32% of clients who submitted an application needed help or assistance to complete it.
• Clarity of information
  • Make it easier to understand the application
    • 39% of those who had difficulty completing the online application said the terminology used in the application was not clear; and
    • 31% of those who had difficulty completing the online application said the questions in the application were not clear or difficult to answer.
• Easier phone access to get assistance (1^st priority) and online real-time support (2^nd priority)

Stages 3 + 4: Follow-up and Decision

Clients seek/receive/provide information after application

Clients receive a service outcome

Ease of Understanding Follow-Up Information

Among clients who recalled receiving a letter from Service Canada, few had difficulty understanding the information or next steps

• When asked how easy or difficult it was to communicate with the government about EI, most said it was somewhat or very easy to understand the information in  the letter they received about their application and the next steps.

Among clients who had to provide additional information, almost one-quarter had some difficulty

• When asked how easy or difficult it was to provide missing information regarding their application, three-quarters said it was somewhat or very easy to understand what information was missing. Conversely, nearly one-quarter (24%) experienced some difficulty.

Text description of Figure 11: Ease of Understanding Follow-Up Information

Effectiveness of Follow-Up Process

More than one-quarter of clients who had to provide additional information had difficulty doing so

• Nearly three in 10 found it was somewhat or very difficult to submit missing information (26%) and to know what to do if they had a problem submitting the information (29%)

1 in 5 Who Checked Application Status had Difficulty

• Roughly four in five (79%) found it was somewhat or very easy to get information on the status of their application
• Conversely, one in five (20%) experienced some or a lot of difficulty getting this information

Text description of Figure 12: Effectiveness of Follow-Up Process

Profile of Clients Having Difficulty Applying for EI Benefits

Note: Only statistically significant differences from the general EI population are presented.

Only variables with a sufficient sample size are presented.

Amount of Time Considered Reasonable for Decision

Three-quarters received a decision about benefits within four weeks

• The plurality (46%) of clients said it took up to two weeks to be informed about whether they would receive benefits. Following this, three in 10 (31%) reported waiting three to four weeks for a decision about their EI application. In total, Seventeen percent reported waiting five or more weeks.

Clients most likely to consider 2 weeks or less to be a reasonable amount of time to receive a decision

• When asked if the length of time they waited to receive a decision was reasonable, 86% of those who waited up to two weeks felt it was reasonable. Among those who waited three to four weeks, 58% considered this a reasonable wait time. Seventeen percent waited five or more weeks for a decision; most did not consider this reasonable.

Text description of Figure 13: Amount of Time Considered Reasonable for Decision

Preferred Changes to Improve EI Follow-up Experience

Shorter Phone Wait, Ability to Communicate Online—Changes Viewed as Having Most Positive Impact

Text description of Figure 14: Preferred Changes to Improve EI Follow-up Experience

Areas for Improvement for Follow-up Stage

• Clarity of communications
  • Use clearer language to communicate with clients about their EI application
    • 23% found it somewhat or very difficult to determine what information was missing regarding their application; and
    • 29% had at least some difficultly understanding what to do if they had a problem submitting the missing information to Service Canada.
• Ease of follow-up
  • Make it easier to check the status of EI applications
    • 20% experienced some or a lot of difficulty getting information on the status of their application
• Reduce time to deliver a decision
  • Almost half waited more than two weeks for a decision, including 17% who waited five or more weeks. Of those who waited five or more weeks, most did not consider this a reasonable amount of time.
• Quicker to get through to an agent and ability to communicate online

Performance - Clients’ Experience by Service Channel

Timeliness/ Efficiency

• In-person Wait times 90% found wait time reasonable
  • A wait time of less than 10 minutes was most often identified as reasonable. However, most clients who waited up to 25 minutes for service said this was reasonable.
  • The majority (61%) of clients who visited a Service Canada Centre reported having to wait less than 10 minutes for service. Conversely, 37% waited 10 minutes or longer for service, with one in 10 saying they had to wait for more than 25 minutes.
• Telephone Wait times 38% found wait time reasonable
  • A wait time of LESS than five minutes was most often identified as reasonable, although a considerable majority considered a five to 10 minute wait acceptable.
  • More than two-thirds (69%) of clients who received phone service reported having to wait MORE than 10 minutes.
  • Eighteen percent said they waited between five and 10 minutes, while one in 10 waited, on average, less than five minutes to speak to an agent.
• Telephone Attempts
  • Nearly half of clients (48%) who attempted to contact the EI Specialized Call Centre had to try two to five times before they  reached an agent.
  • Two-thirds (65%) of clients said that, on average, they had to call more than once to reach an agent. An additional 9% reported that they never reach an agent.

Effectiveness

• Online
  • Seven in 10 clients who used the website (71%) agreed that the process to create their My Service Canada Account was straightforward. Of the rest, 17% were neutral and 11% disagreed that the process was straightforward.
  • Two-thirds of clients who used the website (66%) agreed that they were quickly able to find what they were looking for. Those who did not agree were more likely to be neutral (20%) than to disagree (13%).
• Telephone
  • The majority of clients who spoke to an agent (79%) agreed that their questions were answered completely (57% agreed strongly).
  • Nearly three in 10 clients who spoke to an agent (28%) indicated that they received conflicting information from different agents. Conversely, just over half (58%) reported that this did not happen to them.
• In-person
  • More than four in five clients who visited an office (84%) agreed that their questions were answered completely (of this majority, 66% agreed strongly).

Ease

• Online
  • Most (84%) felt that being able to complete steps online made the overall service experience easier for them.
• Telephone
  • Four in five clients who spoke to an agent (80%) agreed that the information they received was easy to understand (57% agreed strongly).
• In-person
  • The majority of clients who visited an office (84%) agreed that the information they received was easy to understand (61% agreed strongly).

Treatment by Staff

• Telephone
  • The vast majority of clients who spoke to an agent (91%) agreed that they were treated with respect (77% agreed strongly).
  • In addition, 82% felt that staff were helpful. Once again, the majority of clients agreed strongly.
• In-person
  • Ninety-percent of clients who visited an office agreed that they were treated with respect (82% agreed strongly that this was the case).
  • Almost as many (87%) agreed that staff were helpful—nearly three-quarters agreed strongly.

Overall Satisfaction

Majorities of clients were satisfied with the overall quality of service they received using the online, phone and in-person channels

• 82% were satisfied with the overall quality of in-person service (59% were very satisfied).
• 76% were satisfied with the overall quality of the online channel. Clients were evenly split between those who were intensely and moderately satisfied.
• 72% were satisfied with the overall quality of service they received through the phone channel. Notably, clients who were satisfied were more likely to say they were strongly, as opposed to moderately, satisfied.

Satisfaction is lower for the phone than the other channels

Text description of Figure 15: Overall Satisfaction

Omni-channel Experience

Understanding the Client Journey

Levels of Omni-channel Use

For the purposes of this examination of the client experience across the client journey, the following categories of levels of service are used:

Self-service:

• clients used only online and mail channels throughout the client journey

Self-service and Assistance:

• Clients used online, mail and phone, either 1-800 O Canada or EI Specialized Call Centre, as they progressed through the client journey

Full-service:

• Clients visited an in-person office at one point during the client journey

Service Level by Client Experience

Text description of Figure 16: Service Level by Client Experience

Service Channels Used During Client Journey

1 in 10 self-serve across the client journey; in-person (full service) used by over half

• Twelve percent of clients used only self service channels (mail and online) through the entire client journey, from gathering information about EI to receiving a decision about their application
• Nearly one-third (32%) said they used self service and telephone assistance (either 1-800 O Canada or EI Specialized Call Centre) as they progressed through the stages of the client journey
• The majority (55%) of clients made an in-person visit at one point during the client journey (full service).
• Among the 55% who visited an office, 23% live in rural communities, 18% are aged 60 and older, and 10% have completed some high school or less

Text description of Figure 17: Service Channels Used During Client Journey

Text description of Figure 18: Service Channels Used During Client Journey

Purpose of Most Recent Visit to a Service Canada Office

Get and Provide Information—Top Reasons for Visiting an Office

Channels Used by Stage of Client Journey

Significant proportion used two channels for service tasks within the client journey

• Roughly four in 10 clients said they used two channels when gathering information about EI, applying for benefits, and following up about their application
• Three in 10 used one service to gather information and to apply, while fewer (25%) used only one channel to follow-up with Service Canada about their application
• The likelihood* of using more than two channels was higher among clients following up with Service Canada about their application than among clients gathering general information about EI or applying for benefits

Majority Use Online Channel First for Each Service Task

• The Government of Canada website was the first service channel used by a majority of clients across the service tasks
• The typical pattern of service channel use for each service task (among those who used multiple channels) was online followed by visiting a SCC

Text description of Figure 20: Channels Used by Stage of Client Journey

Typical Channel Patterns by Stage of Client Journey

• As previously noted, clients are most likely to use the online service channel first.
• Among clients who used more than one service channel for a service task, a typical pattern of channel use is evident.
• By stage of client journey, this pattern is described in the table below

Text description of Figure 21: Typical Channel Patterns by Stage of Client Journe

Differences circled are statistically significant.

• There are also differences in service channel use by type of claim (frequent and non-frequent claimants) and benefit type (regular and special benefits).
• Frequent claimants are less likely to search online for general information and more likely to visit a Service Canada Centre.
• Clients applying for Special Benefits were more likely to visit a Service Canada Centre when following up about their application.

Text description of Figure 22: Typical Channel Patterns by Stage

Differences circled are statistically significant.

Channel Use and Timeliness

Gather Information

Got the information needed in a reasonable amount of time?

• Used 1 channel: 93% found amount of time reasonable
• Used 2 channels: 84% found amount of time reasonable
• Used 3 or more channels: 83% found amount of time reasonable

Apply

Able to complete the application in a reasonable amount of time?

• Used 1 channel: 98% found amount of time reasonable
• Used 2 channels: 92% found amount of time reasonable
• Used 3 or more channels: 95% found amount of time reasonable

Follow-up

Was this a reasonable amount of time to wait [to hear about decision]?

• Used 1 channel during follow-up: 69% found amount of time reasonable
• Used 2 channels: 61% found amount of time reasonable
• Used 3 channels: 58% found amount of time reasonable

Multi-channel Use and Client Experience by Stage of Journey

Aware: Clients Seeking General Information

Those who used more than two channels were MORE likely to say:

• it was difficult to find information;
• they required assistance using the website; 
• the assistance was not helpful in getting what they needed;
• they did not get what they needed in a reasonable amount of time; and
• quicker access through the phone would have had the most positive impact on this part of the process.

Apply: Clients Submitting Online Application

Those who used more than two channels were LESS likely to say:

• it was easy to:
  • know what documents are needed to apply for EI; and
  • determine the steps in the application process.
• they were able to complete the application in a reasonable amount of time.

Those who used more than two channels were MORE likely to say:

• quicker access through the phone would have had the most positive impact.

Follow-up: Clients Seeking, Receiving, Providing Information

Those who used more than two channels were MORE likely to say:

•  it was difficult to:
  • understand the next steps; and
  • get information on the status of their application.
• shorter phone wait times and the ability to get a call back would have had the most positive impact.

Those who used more than two channels were LESS likely to say:

•  they received a decision in a reasonable amount of time.

All differences reported are statistically significant.

Predictors of Satisfaction

Strongest predictors of satisfaction, indicating importance to clients:

• Smooth progression through the steps of the client journey
• Overall effectiveness* (includes wait times, completing stages in reasonable amount of time, and ease of following the service delivery process and getting information)
• Overall ease*(includes all measures of ease of finding and understanding information)
• When taken together, combined ease and effectiveness measures, explain nearly 44% of the variance in satisfaction
• Therefore changes in any one of these measures would be expected to produce a change in clients’ overall satisfaction rating
• It would be reasonable to expect that dimensions not included in the model due to smaller sample sizes, such as respectful treatment by staff, also drive satisfaction**

Individual dimensions that also have a notable impact on overall satisfaction:

• Ease of finding the information about the EI program sought during the “awareness” stage of gathering information
• Ease with which clients can determine whether they are eligible during the “awareness” stage
• Receiving complete and consistent information by phone and in-person
• Wait time for a decision

*See Appendix 2 for all variables included in overall ease and overall effectiveness composite variables

**Previous research (2006, 2008 and 2011 Service Canada Client Satisfaction Surveys) have found that people skills of staff have a significant impact on overall satisfaction.

Impact of Service Dimensions by Performance

Text description of Figure 23: Impact of Service Dimensions by Performance

Note:  Only dimensions for which unique coefficients are available are presented in the grid. This excludes dimensions with smaller sample sizes.

• The results of the multivariate analysis are available as an Annex under separate cover.
• Results presented above reflect performance/importance for the total EI client population; improvements for those experiencing the most difficulty are assessed in the measures of ease/difficulty presented throughout the deck.

Ability to move smoothly through the steps

74% were able to move smoothly through all steps related to EI claim, the strongest single predictor of clients’ overall satisfaction among those examined

Text description of Figure 24: Ability to move smoothly through the steps

Diagnostics

What to Improve and for Whom?

Vulnerable Groups: Client Experience by Service Task

Aware: Clients Seek General Information

• With the exception of clients receiving Special Benefits, all vulnerable groups were MORE likely to visit a Service Canada Centre than non-vulnerable clients.
• Clients with disabilities were MORE likely to find it difficult to determine eligibility for EI.
• Clients receiving Special Benefits and new Canadians were MORE likely to use the website when seeking information about EI.  Conversely, clients with lower education and clients who rarely or never use online services were LESS likely to use the website.

Apply: Clients Submit Online Application

• Vulnerable clients generally were MORE  likely to:
  • complete their application in an SCC (new Canadians were the exception);
  • require assistance completing their application (low education clients were the MORE likely to need help); and
  • seek in-person assistance to complete their application (Aboriginal clients were the MOST likely to seek help at an SCC).
• Compared to non-vulnerable clients, rural clients and clients with disabilities were MORE likely to find it difficult to put together the information needed for the EI application.

Follow-up: Clients Seek, Receive, Provide Information

• Clients with disabilities were MORE likely to have had difficulty understanding the next steps.
• Clients with disabilities were LESSlikely to have found it easy to check their application status.

To follow-up about their application, vulnerable clients were generally LESS likely to have used the website first (compared to non-vulnerable clients).

Only differences compared to the general EI population that are statistically significant or indicative of a pattern are presented.

Vulnerable Groups: Client Experience by Service Channel

The following differences by service channel are noteworthy for service improvement:

Telephone

• Clients with disabilities were MORE likely to report longer wait times to speak with an agent.
• Clients with disabilities were LESS likely to say their questions were fully answered.

In-person

• Compared to other vulnerable clients, clients with low education were MORE  likely to say that having computers available to complete the EI application with assistance is most important to them when it comes to in-person service. [Recall that these clients were the MOST likely to need assistance completing their application.]

Online

• Clients who rarely or never use online services were MORE likely to have disagreed that the process to create a My Service Canada Account was straightforward.
• Clients with disabilities were MORE likely to disagree that it was quick and easy to find what they were looking for.

Overall

• Clients with disabilities were LESS likely to agree that they were able to move smoothly through all of the steps related to their EI claim. [Recall that this is an important driver of clients’ satisfaction.]
• Compared to non-vulnerable clients, clients with lower education were MORE likely to agree that they got bounced from person to person.
• Clients with lower education and who rarely or never use online services were LESS likely to say that completing the steps online made the process easier for them.

Only differences relative to the total EI population that are statistically significant or indicative of a pattern are presented.

Considerations for Service Improvement for Vulnerable Clients

Clients with disabilities

• Less likely to say they moved smoothly through the entire process

Improvements for these clients...

• make it easier to determine eligibility
• simplify the application process and required documents
• improve communications during follow-up stage

Clients with less than a high school diploma

• Less likely to say that completing the steps online made the process easier for them

Low use of online services

• Less likely to say that completing the steps online made the process easier for them

The online process was harder for these clients, across all stages of the client journey. They required more assistance, in particular through an SCC.

Conclusions

• Satisfaction with overall quality of service is high, and the availability of online services make the service experience better for most clients.
• Levels of dissatisfaction is not high, but significant; at least 1 in 5 had difficulty with various steps and stages of the service delivery process.
• Key areas for improvement for all EI clients and for vulnerable clients in particular are:
  • Ease of finding and understanding online information regarding eligibility, process
  • Access to assistance, i.e. quicker access to a call centre agent and ability to communicate online
  • Clarity of steps and of what to do if a problem is encountered, particularly during the follow-up stage
  • Wait time for a decision
• Smooth movement through steps of the client journey has a sizable impact on client satisfaction.
• Clients who currently complete the steps in what they consider to be a reasonable amount of time do so using multiple channels at various stages in the client journey. 
• The current impediment to smooth movement through the process for clients includes having to repeatedly follow up to get information, as reflected in the difficulty experienced by clients in contacting an agent by phone in a reasonable amount of time and number of attempts.

Appendix 1 Survey Questionnaire

Service Quality Review: EI Claimant Survey

FINAL: 2016-09-08

Introduction

Hello, may I speak to [Insert: your name](INSERT CLAIMANT’S NAME FROM LIST)

• IF PERSON IS AVAILABLE, CONTINUE.
• IF NOT AVAILABLE, ARRANGE CALL-BACK OR GET NEW NUMBER WHERE PERSON CAN BE REACHED.

IF NECESSARY, EXPLAIN PURPOSE OF PHONE CALL. SAY “I’M CALLING ON BEHALF OF THE GOVERNMENT OF CANADA TO ASK SOME QUESTIONS ABOUT THE QUALITY OF SERVICE PROVIDED BY THE GOVERNMENT”.

IF ASKED HOW WE GOT THE PERSON’S NAME, INFORM THEM IT WAS DRAWN FROM A RANDOM SAMPLE OF CANADIANS WHO MAY HAVE RECEIVED SERVICE FROM THE GOVERNMENT OF CANADA.

Would you prefer to continue in English or French? / Préférez-vous continuer en anglais ou en français? [SEE PROGRAM NOTE ON NEXT PAGE]

Hello, my name is [Insert: your name]. I’m calling on behalf of Phoenix, a Canadian public opinion research company. We’re conducting a survey for the Government of Canada to ask questions about the quality of service it provides to Canadians. This survey averages 15 minutes in length, and is registered with the national survey registration system.

Your participation in the research is completely voluntary and your decision on whether or not to participate will not affect any dealings you may have with the Government of Canada. By answering these questions, you will help the government improve the service it provides to Canadians. The information you provide will be used for research purposes only and will be administered in accordance with applicable privacy laws. Would you be willing to take part in this survey?

• Yes [CONTINUE]
• No [TRY TO SCHEDULE CALLBACK]

S1. During the past 7-8 months, have you been in contact with the Government of Canada to get information or service related to  Employment Insurance?

• Yes [GO TO Q1]
• No [CONTINUE]
• Refused [THANK AND DISCONTINUE]

S2. Did someone else contact the Government of Canada during the last 7-8 months on your behalf about Employment Insurance?

• Yes [INVITE PROXY RESPONDENT]
• No [THANK AND DISCONTINUE]
• Refused [THANK AND DISCONTINUE]

Interviewer/Programming Notes

SURVERY REGISTRATION SYSTEM: IF RESPONDENT ASKS ABOUT NATIONAL SURVEY REGISTRATION SYSTEM, SAY:

The registration system has been created by the survey research industry to allow the public to verify that a survey is legitimate, get information about the survey industry or register a complaint. The registration system’s toll-free phone number is 1-800-554-9996.

PROXY RESPONDENTS: IF RESPONDENT IS NOT COMFORTABLE PARTICIPATING IN THE SURVEY FOR WHATEVER REASON (OR HAS TROUBLE UNDERSTANDING WHAT THE INTERVIEWER IS SAYING), INFORM THE RESPONDENT THAT HE/SHE CAN IDENTIFY SOMEONE ELSE TO ANSWER QUESTIONS ON HIS/HER BEHALF IF THAT PERSON HAS DEALT WITH THE GOVERNMENT OF CANADA ON HIS/HER BEHALF. IF SO, TAKE NAME/PHONE NUMBER OF PROXY CONTACT AND ASK RESPONDENT TO CONTACT THAT PERSON RIGHT AWAY TO INFORM THEM THAT THEY WILL BE CALLED (AND HAVE BEEN AUTHORIZED TO ANSWER ON THE RESPONDENT’S BEHALF). THIS IS MOST RELEVANT FOR DISABLED CANADIANS AND NEW CANADIANS, BUT NOT ONLY.

LANGUAGE OF SURVEY: IF THE RESPONDENT PREFERS TO RESPOND IN FRENCH [IF INTERVIEW IS STARTED IN ENGLISH], THEN THE INTERVIEWER MUST BE ABLE TO EITHER PROCEED WITH THE INTERVIEW IN FRENCH OR READ THE FOLLOWING STATEMENT: "JE VOUS REMERCIE. QUELQU'UN VOUS RAPPELLERA BIENTÔT POUR MENER LE SONDAGE EN FRANÇAIS." THE REVERSE IS TRUE FOR INTERVIEWS COMMENCING IN FRENCH.

HOW WE OBTAINED RESPONDENT’S NAME: IF RESPONDENT ASKS HOW WE GOT HIS/HER NAME, INFORM RESPONDENT OF THE FOLLOWING:

• HIS/HER NAME WAS PROVIDED BY THE GOVERNMENT OF CANADA AS SOMEONE WHO MAY HAVE RECEIVED SERVICE FROM THE GOVERNMENT. IT WAS PART OF A LIST OF CLIENTS WHO HAVE HAD A SERVICE INTERACTION REGARDING EMPLOYMENT INSURANCE.

AUTHORITY TO COLLECT INFORMATION: IF RESPONDENT ASKS UNDER WHAT AUTHORITY THIS INFORMATION IS BEING COLLECTED, INFORM HIM/HER IT IS BEING COLLECTED UNDER THE AUTHORITY OF THE DEPARTMENT OF EMPLOYMENT AND SOCIAL DEVELOPMENT ACT, SECTION 6(a).

EMPLOYMENT INSURANCE/EI PROGRAM: THE TERM ‘EI’ IS OFTEN USED TO DENOTE THE EMPLOYMENT INSURANCE PROGRAM. ENSURE THAT RESPONDENTS UNDERSTAND THIS – THAT THE ACRONYM ‘EI’ REFERS TO EMPLOYMENT INSURANCE.

ASSESSMENTS OF SERVICE CHANNELS: ALL RESPONDENTS WHO HAVE CALLED AN EMPLOYMENT INSURANCE SPECIALIZED CALL CENTRE (I.E. USED TELEPHONE SERVICE) WILL BE ASKED THE QUESTIONS IN THE PHONE SERVICE MODULE. ASSESSING THE OTHER TWO SERVICE CHANNALS [IN-PERSON OFFICE AND ONLINE] WOULD BE DONE RANDOMLY IF SOMEONE HAS USED BOTH THOSE CHANNELS. THE FILTER QUESTIONS TO DETERMINE CHANNEL USAGE ARE Qs 1, 11 & 20.

SCALE INSTRUCTIONS: RESPONSE CATEGORIES/INSTRUCTIONS FOR SOME OF THE SCALE QUESTIONS ARE REPETITIVE. ADJUST THE FREQUENCY OF REPEATING THE INSTRUCTIONS TO ENSURE CLARITY BUT AVOID TEDIUM.

HEADINGS: HEADINGS ARE USED IN THE QUESTIONNAIRE TO STRUCTURE THE CONTENT. THEY ARE NOT TO BE PROGRAMMED OR READ TO RESPONDENTS.

Pre-application—Information Gathering about EI Benefits

I’d like to ask you some questions about your experience getting information about Employment Insurance prior to submitting an application for benefits.

[INTERVIEWER NOTE: Ensure that the respondent is clear that these next questions deal with the steps he/she took before applying for EI.]

1. How did you find out how to apply for Employment Insurance? [READ LIST; RECORD YES/NO FOR EACH ITEM. MULTIPLE RESPONSES ACCEPTED] [USE TOP 4 ITEMS, EXCEPT 1-800 O’CANADA, AS FILTER FOR SERVICE CHANNEL MODULES] [CODING DEFAULT IS EI CALL CENTRE IF RESPONDENT IS UNSURE WHO THEY CALLED]

Visit the Government of Canada website
Call an Employment Insurance Specialized Call Centre
Call 1-800 O’Canada
Visit a Service Canada office

[INTERVIEWER NOTE: ONLY THE RESPONDENT’S EXPERIENCE RELATED TO EI IS RELEVANT NOT THEIR EXPERIENCE WITH OTHER GOVERNMENT PROGRAMS SUCH AS OLD AGE PENSION, CANADA PENSION PLAN, ETC.]

• VOLUNTEERED [DO NOT READ]:
• Do a general online or Google search
• Talk with family or friends
• Previous experience => PROBE TO DETERMINE WHETHER ANY OF THE 4 “READ LIST” CHANNELS WERE
• USED.
• Other  (Specify):[ ]

 [ASK IF MORE THAN 1 OF THE TOP 4 ITEMS SELECTED AT Q1]

2. Thinking about the government information sources you used, which did you use first? [PROMPT RESPONDENT WITH ITEMS FROM Q1 AS NEEDED]

[POPULATE WITH ITEMS FROM Q1]

[ASK NEXT TWO QUESTIONS IF WEBSITE SELECTED AT Q2]

3. When using the website, did you need any help or assistance?

• Yes
• No

4. Was the assistance helpful in getting you what you needed?

• Yes
• No

5. Did you get the information you needed in a reasonable amount of time?

• Yes
• No

[ASK IF OPTIONS 1-4 SELECTED AT Q1; EVERYONE ELSE GO TO NEXT SECTION]

6. Overall, please tell me how easy or difficult each of the following were when you were getting information about Employment Insurance. Would you say it was very difficult, somewhat difficult, somewhat easy, or very easy to…? [DO NOT ROTATE LIST]

1. Find the information you were looking for.
2. Determine if you were eligible for EI benefits.
3. Determine the steps in the application process.
4. Know what documents you needed to apply for EI benefits.

[ASK IF VERY DIFFICULT/DIFFICULT FOR Q6A]

7. You said it was hard to find the information you were looking for. Why was this difficult? [DO NOT READ LIST; ACCEPT 2 RESPONSES]

• Waited on hold too long to speak to telephone agent
• Couldn’t reach the call centre/all lines were busy
• Eligibility information was not clear
• Received conflicting information
• Staff were rude/not sympathetic
• Website was difficult to navigate
• Office was difficult to locate
• Long line at the office
• Other [SPECIFY]: [ ]

8. Thinking about your experience looking for information about Employment Insurance before you applied for benefits… which of the following changes, if any, would have had the biggest positive impact on this part of the process for you? [READ LIST; ACCEPT ONE RESPONSE; ROTATE OPTIONS]

• Information that is easier to find online
• Online information that is easier to understand
• Quicker to get through to an agent on the phone
• Shorter wait times in a Service Canada Centre
• [VOLUNTEERED: None of these]

Application Process—Applying for EI Benefits

Thinking back to when you actually applied for Employment Insurance benefits,

9. Where did you complete your online application? [READ LIST]

• At home
• In a Service Canada Centre, or
• Somewhere else?
• (VOLUNTEERED: Did not apply online)

10. Did you require any help or assistance to complete your online application?

• Yes
• No

[ASK NEXT TWO QUESTIONS IF ‘YES’ AT Q10]

11. How did you get that assistance? Anywhere else? [DO NOT READ LIST; ACCEPT MULTIPLE RESPONSES] [USE TOP 4 ITEMS, EXCEPT 1-800 O’CANADA, AS FILTER FOR SERVICE CHANNEL MODULES]

• Government of Canada website
• Calling an Employment Insurance Specialized Call Centre
• Calling 1-800 O’Canada
• Visiting a Service Canada office
• Doing a general online or Google search
• Talking with family or friends 
• Other [SPECIFY]: [ ]
  

12. Was the assistance helpful in getting you what you needed?

• Yes
• No

13. Were you able to complete the application in a reasonable amount of time?

• Yes
• No

14. Please tell me how easy or difficult each of the following were when you were applying for Employment Insurance. Would you say it was very difficult, somewhat difficult, somewhat easy, or very easy to…? [DO NOT ROTATE LIST]

1. Understand the requirements of the application.
2. Put together the information you needed to apply for EI.
3. Complete the online application form.

[ASK IF VERY DIFFICULT/DIFFICULT FOR Q14C]

15. You said it was hard to complete the online application form. Why was this difficult? [DO NOT READ LIST; ACCEPT 2 RESPONSES]

• Terminology/words used in application was not clear
• Website was hard to find
• Application was hard to find
• Didn’t have the right information to complete the application
• Other [SPECIFY]: [ ]
  

16. How long did it take for you to be informed about whether you would receive benefits? [DO NOT READ LIST]

• Two weeks or less
• Three to four weeks
• Five weeks or more
• Did not hear back [SKIP NEXT QUESTION]

17. Was this a reasonable amount of time to wait? [INTERVIEWER NOTE: THE QUESTION IS ABOUT THE DECISION REGARDING THEIR EI APPLICATION. WE DO NOT WANT RESPONDENTS TO ANSWER BASED ON THE LENGTH OF TIME IT TOOK TO RECEIVE THEIR FIRST PAYMENT. CLARIFY IF THIS IS NOT CLEAR BASED ON THE RESPONDENT’S RESPONSE.]

• Yes
• No

18. Which of the following changes, if any, would have had the biggest positive impact on your experience applying for EI benefits? [READ LIST; ACCEPT ONE RESPONSE; ROTATE OPTIONS]

• Quicker to get through to an agent on the phone
• Online, real-time support
• Shorter wait times in a Service Canada Centre
• Being able to upload documents, if required
• Have an app to apply through a mobile device
• [VOLUNTEERED: None of these]

Post-application—Follow-Up

19. After you submitted your EI application, did you…? [READ LIST; RECORD YES/NO FOR EACH ITEM. ACCEPT MULTIPLE RESPONSES]

• Receive a letter* with information about your application
• Need to provide missing information regarding your application
• Check the status of your application
• [VOLUNTEERED: None of these] => SKIP THIS SECTION

[*INTERVIEWER NOTE: IF NEEDED, PLEASE TELL THE RESPONDENT THAT THIS IS THE LETTER THEY WERE SENT THAT INCLUDED INFORMATION ABOUT THEIR CLAIM, LIKE HOW TO SET UP THEIR SERVICE CANADA ACCOUNT.]

20. Please tell me how easy or difficult each of the following were in terms of your communications with the government about Employment Insurance. Would you say it was very difficult, somewhat difficult, somewhat easy, or very easy to…? [DO NOT ROTATE LIST]

IF Q19 received a letter

• a. Understand the information in the letter you received.
• b. Understand the next steps.

IF Q19 needed to provide missing information

• c. Understand what information was missing.
• d. Submit the missing information to Service Canada.
• e. Know what to do if you had a problem in submitting the information

IF Q19 checked status of application

• f. Get information on the status of your application.

21. In which of the following ways did you contact the government to follow up on your EI application? [READ LIST; RECORD YES/NO. MULTIPLE RESPONSES ACCEPTED][USE TOP 4 ITEMS, EXCEPT MAIL, AS FILTER FOR SERVICE CHANNEL MODULES]

• By mail
• Visiting the Government of Canada website*
• Calling an Employment Insurance Specialized Call Centre*
• Visiting a Service Canada office
• [VOLUNTEERED: None of these] => SKIP THIS SECTION

[*INTERVIEWER NOTE: THIS QUESTION IS ASKING ABOUT FOLLOW-UP RELATED SPECIFICALLY TO THEIR EI APPLICATION. CONTACT DOES NOT INCLUDE THEIR WEEKLY REPORTING TO SERVICE CANADA.]

[ASK IF MULTIPLE ITEMS SELECTED AT Q21]

22. Which method did you use first? [PROMPT RESPONDENT WITH ITEMS FROM Q20 AS NEEDED]

[POPULATE WITH ITEMS FROM Q21]

23. Which of the following changes, if any, would have had the biggest positive impact for you when communicating with the government during your follow-up? [READ LIST; ACCEPT ONE RESPONSE; ROTATE OPTIONS]

• Letters that are easier to understand
• The ability to communicate online with Service Canada
• The ability to get a call-back from a call centre agent within a given time
• Staff in Service Canada Centres able to make decisions on claims
• Shorter in-person wait
• Shorter telephone wait
• [VOLUNTEERED: None of these]

Service Channel Assessments

QUESTIONS 1, 11 & 20 TO BE USED AS FILTERS TO DETERMINE PARTICIPATION IN SERVICE CHANNEL ASSESSMENTS.

Call Centre

[ASK THOSE WHO CONTACTED AN EMPLOYMENT INSURANCE SPECIALIZED CALL CENTRE. NO ONE WHO ONLY CALLED 1-800 O’CANADA ANSWERS THIS SECTION]

Earlier you mentioned that you had called an Employment Insurance Specialized Call Centre to obtain information or service related to EI.

24. On average, how many times did you call before you were able to reach an agent? [DO NOT READ LIST]

• Once
• 2-5 times
• More than 5 times
• [VOLUNTEERED: I never reached an Agent] => SKIP TO NEXT SERVICE CHANNEL

25. How long did you have to wait, on average, to speak to an agent? [DO NOT READ LIST]

• Less than 5 minutes
• 5 – 10 minutes
• More than 10 minutes

26. Was this a reasonable amount of time to wait?

• Yes
  No

27. Please tell me how much you agree or disagree with the following statements about your most recent experience calling an Employment Insurance Specialized Call Centre. Use a 5-point scale where ‘1’ means strongly disagree, and ‘5’ means strongly agree. (ROTATE LIST; REPEAT INSTRUCTIONS AS NEEDED; RECORD SEPARATELY ‘DOES NOT APPLY’ AND ‘DON’T KNOW’)

1. Your questions were answered completely.
2. The information you received was easy to understand.
3. Staff were helpful.
4. You were treated respectfully.
5. You received conflicting information from different agents.

28. How satisfied were you with the overall quality of service you received during your phone call(s) to Service Canada about EI in the last 7-8 months? Please use a 5-point scale, where ‘1’ means very dissatisfied, and ‘5’ means very satisfied.

Service Canada Office

[ASK THOSE WHO USED AN IN-PERSON OFFICE]

Earlier you mentioned that you had visited a Service Canada office to obtain information or service related to EI.

29. What was the purpose of your most recent visit? [READ LIST; ACCEPT 1 RESPONSE]

• To get information on EI benefits
• To provide Service Canada with required information for EI benefits
• To use a workstation to apply for EI benefits
• Other [SPECIFY]: [ ]

30. Thinking about your most recent visit to a Service Canada office, how long did you have to wait for service? [DO NOT READ LIST]

• Less than 10 minutes
• 10 – 25 minutes
• More than 25 minutes

31. Was this a reasonable amount of time to wait?

• Yes
• No

32. Please tell me how much you agree or disagree with the following statements about your most recent experience at a Service Canada office to obtain information or service related to EI. Use a 5-point scale where ‘1’ means strongly disagree, and ‘5’ means strongly agree. [READ ITEM; REPEAT INSTRUCTIONS AS NEEDED; RECORD SEPARATELY ‘DOES NOT APPLY’ AND ‘DON’T KNOW’]

1. Your questions were answered completely.
2. The information you received was easy to understand.
3. Staff were helpful.
4. You were treated respectfully.

33. How satisfied were you with the overall quality of service you received at the Service Canada office about EI in the last 7-8 months? Please use a 5-point scale, where ‘1’ means very dissatisfied, and ‘5’ means very satisfied.

34. I’d like you to tell me which of the following are most important to you when it comes to in-person service for EI…[PRESENT ITEMS IN PAIRS TO SUPPORT CONJOINT ANALYSIS]

[OPTIONS]

• Friendly service
• Short wait times
• Complete resolution in the one visit
• Computers available to complete the EI application with assistance

Online Services

[ASK THOSE WHO USED ONLINE SERVICE]

Earlier you mentioned that you visited the Government of Canada website to get information or service related to EI.

35. Please tell me how much you agree or disagree with the following statements about the government website related to EI using a 5-point scale where ‘1’ means strongly disagree, and ‘5’ means strongly agree. [ROTATE LIST; REPEAT INSTRUCTIONS AS NEEDED; RECORD SEPARATELY ‘DOES NOT APPLY’ AND ‘DON’T KNOW’]

1. You were quickly able to find what you were looking for.
2. The process to create a My Service Canada Account was straightforward.

36. How satisfied were you with the overall quality of service you received when you visited the government website you used about EI in the last 7-8 months? Please use a 5-point scale, where ‘1’ means very dissatisfied, ‘5’ means very satisfied,

Overall Satisfaction

I’m now going to ask you some questions about the overall quality of service you received related to Employment Insurance.

37. Thinking about the service you received during the past 7-8 months, please tell me how much you agree or disagree with the following statements, using a 5-point scale where ‘1’ means strongly disagree, and ‘5’ means strongly agree. [ROTATE LIST; REPEAT INSTRUCTIONS AS NEEDED]

1. You were able to move smoothly through all of the steps related to your EI claim.
2. You got bounced around from one person to another
3. You are confident that any personal information you provided will remain confidential. 
4. Being able to complete steps online made the process easier for you.
5. It was clear what to do if you had a problem or question.
6. You had to keep following up to get information.

38. Did you have any difficulty with your EI application because you are not comfortable in English or French?

• Yes
• No

39. How satisfied were you with the overall quality of service you received from Service Canada related to your EI benefits during the last 7-8 months. Please use a 5-point scale, where ‘1’ means very dissatisfied, ‘5’ means very satisfied,

40. If someone were to ask you, would you speak positively about the service you received? [READ LIST]

• Definitely
• Probably
• Probably not
• Definitely not

Demographic Profile

These last questions are for statistical purposes only. All responses will remain confidential.

41. In general, how often would you say you use online services provided by government, businesses and other organizations through their websites? [NOTE: E.G. ONLINE BANKING, ONLINE PURCHASING, ON-LINE ACCOUNTS, BILL PAYMENT]

• Routinely or all the time
• Only sometimes
• Rarely
• Never

42. What is the highest level of formal education that you have completed? [READ LIST]

• Grade 8 or less
• Some high school
• High School diploma or equivalent
• Registered Apprenticeship or other trades certificate or diploma 
• College, CEGEP or other non-university certificate or diploma 
• University certificate or diploma below bachelor's level 
• Bachelor's degree
• Post graduate degree above bachelor's level 
• [DO NOT READ] Prefer not to answer

43. Were you born in Canada?

• Yes
• No
• [DO NOT READ] Refused

If NO to Q43, ASK:

44. In what year did you come to Canada?

• Record year: [ ]
• [DO NOT READ] Refused

45. Are you an Aboriginal person, that is, First Nations (North American Indian), Métis or Inuk (Inuit)?

• First Nations
• Métis
• Inuk

46. Are you a person with a disability?

• Yes
• No

That completes the survey. Thank you very much. Your participation is appreciated.

READ ONLY IF RESPONDENT ASKS HOW TO ACCESS RESEARCH RESULTS:
Once the study is complete, your anonymous responses will be added to all responses gathered during the study, then aggregated, analyzed and published as public information. The survey results will be posted on the Library and Archives Canada website. 

Characteristics to be identified in the sample:

• Region
• Benefit Type
• Approved/Denied/Reconsidered
• Occupation (9 categories)
• Frequent/Non-frequent claimants
• Age (year of birth)
• Gender
• Language of application
• Rural/Urban (from postal code)

Appendix 2 Composition of Overall Ease and Effectiveness Variables

Composition of Variables: Overall Ease and Effectiveness

Ease and Effectiveness Account for 44% of Variance in Overall Satisfaction

EASE MEASURES

• The information you received was easy to understand.
• The information you received was easy to understand.
• It was clear what to do if you had a problem or question.
• Understand the information in the letter you received
• Understand the next steps.
• Understand what information was missing.
• Understand the requirements of the application.
• Put together the information you needed to apply for EI.
• Easy to find the information you were looking for.
• Easy to determine if you were eligible for EI benefits.

EFFECTIVENESS MEASURES

• Timeliness
• Your questions were answered completely.
• You received conflicting information from different agents
• You were able to move smoothly through all of the steps
• Submit the missing information to Service Canada.
• Know what to do if you had a problem in submitting the information
• Get information on the status of your application.
• Was the assistance helpful in getting you what you needed?
• Complete the online application form.
• Easy to determine the steps in the application process.
• Easy to know what documents you needed to apply for EI benefits.