POR Number: POR 032-16
Contract Number: G9292-177438-001-CY
Contract Award Date: 2016-08-17
Date of Submission: 2017-01-11
Prepared for: Employment and Social Development Canada
Email address: nc-por-rop-gd@hrsdc-rhdcc.gc.ca
February 2017
Ce rapport est aussi disponible en français.
Phoenix SPI is a Gold Seal Certified Corporate Member of the MRIA
Employment and Social Development Canada (ESDC) commissioned Phoenix SPI to conduct a service experience survey with Employment Insurance clients.
The Government of Canada committed to improve services for Canadians in Budget 2016. On May 12th, 2016, the Government launched a nationwide Employment Insurance (EI) Service Quality Review (SQR) consultation process to seek input from key stakeholders and the public on ways Service Canada can improve services to EI claimants. The EI SQR included national roundtables with stakeholders, an online consultation for Canadians, and the opportunity to offer written submissions. It focused on three key areas: streamlining applications, reducing wait times for service delivery, and reducing administrative burden for employers.
In addition to the consultations, representative feedback from clients was sought through a survey of EI clients. In line with the broader EI SQR consultation, the survey assessed overall client experience1 across the entire EI service delivery experience and across all channels, including telephone, internet and in-person. The survey was designed to address the following areas of investigation: 1) ease of access within and across all channels; 2) timeliness within and across all channels; and 3) overall service quality. The measurement approach drew on several sources, including the framework and questions of the Common Measurements Tool (CMT), owned and licensed by the Institute for Citizen-Centred Service (ICCS), and a client experience measurement model developed by the Treasury Board Secretariat of the Government of Quebec. The survey findings will be used for decision-making in accordance with the requirement to improve service delivery for clients and to base such decisions on complete information.
To address the research objectives, a 15-minute telephone survey of 1,528 EI clients was conducted between September 2nd and 16th, 2016. Interviewing was conducted by Elemental Data Collection (EDCI) using Computer Aided Telephone Interviewing (CATI) technology.
Respondents were randomly selected from the EI database of clients who had a service interaction between April 1st and June 30th, 2016. This included clients who received a decision about their EI application, clients who began receiving benefits, and clients who submitted a partial application that was never completed. The results of the survey can be extrapolated to the entire EI population. The margin of error for this sample is ±2.5%, at a 95% confidence level (adjusted for a finite population).
The sample was proportionate to the EI population in terms of service delivery region (Atlantic Canada, Quebec, Ontario, and Western Canada, including the territories), claim status (approved or denied), and type of benefit (regular or special benefits). Survey data were weighted by these same characteristics (i.e., service delivery region, claim status, and type of benefit) to ensure results that are representative of the EI population. Population figures from the EI administrative database were used to construct the weights.2
The table below presents information about the final call dispositions for this survey, as well as the associated response rate (using the Marketing Research and Intelligence Association [MRIA] formula), which was 14.5%.
| Total | |
|---|---|
| Total Numbers Attempted | 13,432 |
| Out-of-scope - Invalid | 3,814 |
| Unresolved (U) | 3,478 |
| No answer/Answering machine | 3,478 |
| In-scope - Non-responding (IS) | 6,140 |
| Household language barrier3 | 97 |
| Incapable of completing (ill/deceased) | 47 |
| Callback (Respondent not available) | 2,941 |
| Refusal | 2,968 |
| Termination | 87 |
| In-scope - Responding units (R) | 1,637 |
| Completed Interview | 1,527 |
| Ineligible (no contact with EI) | 110 |
| Response Rate4 | 14.5% |
The potential for non-response bias was assessed by comparing the characteristics of respondents through unweighted and weighted data. The characteristics evaluated were region, benefit type and claim status. The table below presents a profile of the survey sample, unweighted and then weighted to match the EI population.
| Unweighted | Weighted | |
|---|---|---|
| Region | ||
| Atlantic | 13% | 13% |
| Quebec | 23% | 23% |
| West and the Territories | 35% | 35% |
| Benefit Type | ||
| Regular | 77% | 64% |
| Special | 23% | 36% |
| Claim Status | ||
| Approved | 71% | 71% |
| Denied | 29% | 29% |
The unweighted region and claim status sample proportions were identical to the weighted sample proportions. The only differences evident related to benefit type. Those who applied for regular benefits are over-represented in the unweighted sample, while those who applied for special benefits are under-represented. This was corrected through weighting.
To assess whether this discrepancy by benefit type introduced bias in the survey sample, the unweighted and weighted results for the two primary satisfaction indicators were compared for statistically significant differences. No statistically significant differences were found, which suggests that the over-representation of clients who applied for regular benefits and the under-representation of clients who applied for special benefits had no biasing effect on the sample.
In addition, it should be noted that efforts were made to ensure that clients who speak neither French nor English were included in the sample. Specifically, proxy respondents were permitted as a way for those who do not speak an official language to participate in the survey. Only seven clients contacted to complete an interview referred interviewers to a proxy respondent and none of these referrals ended up completing an interview. To the extent that official language and non-official language speakers have different service experiences, the under-representation of non-official language speakers could have a biasing effect on the survey sample depending on the size of this sub-population. Clients who speak neither French nor English comprise 1% of the EI population.5 This means we would expect to have 15 clients who do not speak one of the official languages in the survey sample (which was not the case). On a sample of 1,528 clients, 15 clients would have a very limited (if any) biasing effect on the overall results.
Most clients had a generally positive service experience, in particular: they were able to move smoothly through all steps (74%) and were clear about what to do if they had a problem or question (70%). Underscoring their positive experience, 90% of clients said they would speak positively about their service experience and 78% expressed satisfaction with the overall quality of service they received from Service Canada related to their EI benefits during the last 7-8 months (preceding the data collection). Moreover, when clients required assistance, almost all found staff to be respectful (91% for service by phone, and 90% for in-person service) and helpful (82% for service by phone, and 87% for in-person service).
However, about one in five clients experienced difficulty with some aspects of the process, such as finding the information they were looking about the EI program, figuring out whether they were eligible and the steps to follow to apply, and using the online channel in general. These clients were more likely to be seniors, to have less than high school education, to be unfamiliar with online services, or to be applying for special benefits, such as the EI Sickness Benefit.
In terms of channel use, the majority of clients attempted to self-serve online at each step of the journey, but only 12% said they went through all stages online without assistance from Service Canada. More than half visit an in-person office at some point in their client journey, and clients who tried to access assistance were less likely to be satisfied with the overall quality of service received in relation to their EI benefits than those clients who self-served throughout.
Most found it easy to access information about EI benefits, but nearly one-quarter found it difficult to determine whether or not they were eligible for benefits.
Of the Government of Canada information sources used by clients, two-thirds (66%) used the online channel first to find information about EI benefits. Following this, just over one-quarter (27%) said they visited an office as the first step before applying for EI benefits. Few used other Government of Canada service channels: 5% called an EI specialized call centre and 2% called 1-800 O’Canada. When using the website to gather information about EI benefits, 22% said they needed help or assistance. The majority of those who needed assistance (77%) said it was helpful.
When asked to consider their overall experience gathering information about EI benefits, most clients (88%) said the process took a reasonable amount of time. Furthermore, three-quarters or more of clients who used a Government of Canada service channel found the process of gathering information to be somewhat or very easy: 86% found it easy to find the documents needed to apply for EI, 80% found it easy to determine the steps in the application process, 78% found it easy to find the information they were looking for on the website, and 75% found it easy to determine eligibility for EI benefits.
Aspects that caused some difficulty for at least one in five clients included: determining eligibility (24%), finding the information they were looking for (21%), and determining the steps in the application process (19%). These clients (n=280) were most likely to attribute the difficulty they experienced to website navigation (37%) and perceived lack of clarity of the eligibility requirements (27%).
Turning to potential improvements to improve access to information about EI, quicker access to an agent by phone was viewed as having the biggest positive impact on this part of the process. In total, 39% of surveyed clients who used a Government of Canada service channel said that this would have improved their experience looking for information about EI.
Most found the process of applying for benefits to be easy, but some clients said they had at least some difficulty.
Nearly three-quarters of clients (72%) completed their application for EI at home, while 23% did so in a Service Canada Centre (SCC). Almost all clients (96%) felt they were able to complete the application in a reasonable amount of time. That said, one-third of clients (32%) needed help or assistance to complete their online application. The vast majority of those who needed assistance (92%) said the assistance was helpful.
Most clients found the process of applying for benefits to be somewhat or very easy. Specifically, 86% said it was easy to put together the information needed to apply, 85% said it was easy to understand the application requirements, and 80% said it was easy to complete the application online. Twelve percent of clients had at least some difficulty completing the online application form. Difficulties were most often attributed to unclear terminology (39%) and lack of clarity of some questions in the application (31%). In terms of improvement, clients expressed a preference for changes to the phone channel: quicker access to an agent (35%), followed by online, real-time support (22%).
During the stage between submitting an application and receiving a decision, about 1 in 5 clients who followed up about their application encountered difficulty and were often not sure what to do if they had a problem.
Most clients contacted (or had contact with) Service Canada prior to receiving a decision about their EI application: 78% received a letter with information about their application, 70% checked the status of their application, and 19% needed to provide missing information regarding their application. When following up after submitting their application, 52% used the online channel first to contact Service Canada. Following this, one-quarter called the EI specialized call centre and 15% visited an SCC.
Among clients who received a letter (n=1,225), most found it easy to understand the information (94%) or the next steps (89%). In addition, 79% of clients who checked their application status (n=1,050) did so easily. Difficulties were most likely to be encountered by those who had to provide missing information (n=285): 29% said they found it somewhat or very difficult to know what to do if they had a problem, 27% found it difficult to submit missing information to Service Canada, and 23% found it difficult to understand what information was missing. In terms of improvement, clients expressed a preference for shorter wait times when calling Service Canada (29%), followed by the ability to communicate online with Service Canada (25%).
Almost half the clients surveyed reported receiving a decision within two weeks, but one-third waited for what they felt to be an unreasonable amount of time.
When asked how long it took to receive a decision about their EI application, 46% said they waited two weeks or less, 31% three to four weeks, 17% five weeks or more, and 2% had not heard back (at the time they were interviewed). Most (86%) of those who waited up to two weeks said they was a reasonable wait time. In addition, 58% considered the three to four weeks they waited to be reasonable. Fewer (18%) thought it was reasonable to have had to wait five weeks or more to receive a decision.
Telephone
Most clients who used the phone service channel said they reached an agent in five or fewer call attempts: 24% said they called once before reaching an agent and 48% said they called two to five times. Conversely, 17% said they had to call more than five times before they were able to reach an agent and 9% never reached an agent. Once in queue to speak with an agent, the majority of clients (69%) had to wait more than 10 minutes. Ten percent waited less than five minutes and 18% waited five to 10 minutes before they were able to speak with an agent.
Most had a positive service experience—in particular, clients found the agents they spoke with to be respectful (91%) and helpful (82%). In addition, 80% said the information they received was easy to understand, 79% said their questions were fully answered, and 58% said the information they received from different agents was consistent. Overall, almost three-quarters (72%) of client who used the phone channel were satisfied with the quality of the service they received.
In-person
When asked what the purpose of their most recent visit to an SSC was, clients who visited an office said they did so to get information about EI benefits (37%) and to provide required information (34%). Current wait times in an SCC were considered reasonable, with 61% of clients who sought service in-person estimating that they received service in under 10 minutes. One-quarter (26%) waited 10 to 25 minutes, and one in ten (11%) waited more than 25 minutes.
The vast majority of clients who visited an SCC had a positive service experience. Staff were considered respectful (92%) and helpful (87%), were able to answer questions completely (84%) and provided information that was easy to understand (84%). The most important aspect of in-person service was complete resolution in one visit, followed closely by friendly service. Overall, 82% of clients who used the in-person channel were satisfied with the quality of the service they received (59% were very satisfied).
Online
Most clients (84%) felt that being able to complete steps online made the overall service experience easier for them. Among those who used the online channel, 71% agreed that the process to create an My Service Canada Account (MSCA) was straightforward and two-thirds (66%) said they were quickly able to find what they were looking for when using the website. Overall, 76% of clients who used the online channel were satisfied with the quality of the channel.
Contract Amount: $77,975.65 (HST included). The research instruments and tabulated data are available under separate covers.
I hereby certify as a Senior Officer of Phoenix Strategic Perspectives that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not contain any reference to electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leader.
Alethea Woods
President, Phoenix Strategic Perspectives Inc.
1 The client journey includes five phases: 1. client seeks general information regarding the EI program; 2. client submits application; 3. client exchanges further information with Service Canada; and 4. Client receives service outcomes.
2 Weighting is a statistical adjustment that redistributes the survey responses to match the population surveyed—in the case of this survey, the results are “weighted” to reflect the distribution of the EI population by region of the country, claim status, and claim type.
3 This refers to people who answered the phone, and not necessarily to the EI clients themselves.
4 The MRIA response rate formula is as follows: [R=R/(U+IS+R)]. This means that the response rate is calculated as the number of responding units [R] divided by the number of unresolved [U] numbers plus in-scope [IS] non-responding households and individuals plus responding units [R].
5 Statistics Canada. 2011. National Household Survey.