1. Executive Summary
1.1. Research Purpose and Objectives
The Citizen Service Branch (CSB) has developed a Client Experience (CX) Survey Measurement Model to understand how clients perceive their experience with Service Canada. This model draws on a Quebec Government model, the Common Measurements Tool from the Institute for Citizen-Centered Service (ICCS), and elements of Forrester’s Client Experience Measurement approach. The draft model has been developed in consultation with stakeholders within Service Canada and was piloted through the Employment Insurance Service Quality Review Survey before being finalized following qualitative research conducted with clients. The model is now being used as the backbone for CX work across Employment and Social Development Canada (ESDC). It was used to design ESDC’s Annual CX Survey and is also being used to facilitate the design of the questionnaire for the regular collection of data by channel that will be piloted in the fourth quarter of Fiscal Year 2017-18.
In keeping with the Treasury Board Policy on Service, and the ESDC Service Strategy, which call for collecting client feedback to support client-centric service management, and with the Client-Centric Feedback solution within the Service Transformation Plan, CSB is leading a pilot project to generate client experience data that is not currently available for corporate performance monitoring and reporting. Regularly reported client experience data for each channel will complement the annual data collected by the CX Survey.
The annual CX survey examines the service experience delivered by all channels interacting together for results tracking, while collecting data by channel will provide a pulse check on how well each channel is working for clients for course correction during the year.
CSB is implementing a pilot project to determine the most appropriate mechanism to collect data from clients on their service experience in each channel on an ongoing basis. In proceeding with the pilot, CSB is following through on the third of four client feedback priority activities that were approved by the Service Management Committee in 2016; the regular collection of client experience data that is aligned and automated across service delivery channels.
The pilot project is an initial step to automating and aligning data collection from clients in each channel to allow the Department to monitor the client experience for service management purposes, beginning in 2018/19.
The results of the pilot project will position ESDC to determine which automated mechanisms for collecting ongoing client experience data in the phone and in-person channels can generate data that is sufficiently representative and cost-effective for Service Canada to use in monitoring the performance of its service delivery system. The portion of the pilot addressed in this report includes:
1) generating the control data for the in-person channel by administering intercept surveys to clients as they leave offices after a service interaction; 2) transferring phone channel clients to a set of client experience questions administered through an interactive voice response (IVR) system after a service interaction.
Service Canada will assess the data collected through the set of mechanisms it is testing in-house in the in-person channel against the data collected by Kantar TNS from the same offices to determine the coverage and reliability of results generated by the in-house mechanisms. It will use the data collected by Kantar TNS on the service experience delivered by the phone channels to assess the effectiveness of the IVR mechanism of collecting data.
The set of questions used to collect the client experience data conform with Service Canada's Client Experience Survey Measurement Model, and gather data on the ease and effectiveness of, and satisfaction with the service experience, and on the program and service task involved in the service experience.
The specific research objectives are to:
- collect data on the service experience of clients who have completed a service interaction in selected in-person offices, including:
- Service Canada Centres (SCC)
- Passport Offices
- collect data on the service experience of clients who have completed a service interaction with the phone channel, consisting of:
- 1 800 O-Canada
- Employment Insurance and Pensions Specialized Call Centres
1.2. Summary of Findings
The following results describe the period of March 19th to 29th, 2018 in selected offices, and from April 10th to May 4th, 2018 in the phone channels. They will be used by Service Canada to assess the response rate and coverage or comparability of results of a range of automated client experience data collection mechanisms tested in each channel.
In-Person Channel
The results of the in-person analyses found no significant differences between the findings for Passport Offices and Service Canada Centres except for education level, where it was found that those who accessed a Service Canada Centre were more likely to have only completed high school or less and those who accesses a Passport Office were more likely to have completed university or above. For this reason, in-person intercept findings are discussed at a total level only and not broken out by Passport Office and Service Canada Centre.
In almost all instances, staff were regarded as helpful and respectful, contributing to most respondents finding their task easy to accomplish. Most respondents were equipped with the information they needed to understand their next step in the process with only a small minority leaving without understanding what they needed to do next. With such high levels of success in achieving what visitors set out to do, it is not surprising that 9 out of 10 respondents were satisfied with their visit to the Passport Office or SCC.
Four-in-five people visiting the Passport Office or SCC wanted to receive service in English and the majority agreed that services were provided in the official language of their choice.
Phone Channel
The IVR results of the analyses found no significant differences between the findings for the different services accessed except for ease of service, where 1 800 O-Canada was perceived as being easier to use. On the flip side, compared with the two other services assessed, respondents accessing 1 800 O-Canada were less likely to accomplish the task they set out to do. Due to the lack of significant differences between the services, IVR findings will be discussed at a total level only and not broken out by services accessed.
The main reason for calling 1 800 O-Canada was to obtain general information about a program or service. Calls were made to the EI line mainly to obtain an application status or to get assistance with a two-week report. There were multiple main reasons reported for accessing the pension service, including: obtaining general information about benefits, application status and changing an address or direct deposit.
As was the case with the in-person findings, in almost all instances, staff were considered helpful and respectful and most respondents found their task easy to accomplish. While most respondents accessing services via telephone could accomplish the task they set out to do, those accessing 1 800 O-Canada were less likely to do so than those who used the other two services offered.
More than 9-in-10 respondents were given the information they needed to understand their next step in the process, again resulting in very high satisfaction.
1.3. Methodology
Two methods of data collection were used. In-person intercept surveys were conducted among 1421 Canadians who made use of a Service Canada Centre or Passport Canada Office. In addition to this, 2741 IVR phone surveys were conducted from referrals from both EI and Pension specialized call centres and 1 800 O‑Canada call centres.
For the in-person intercept survey, a pre-test consisting of 77 interviews was completed on March 15 and regular fieldwork took place on 19-29 March 2018. No changes were made to the survey after the pre-test and these results were included in the final dataset.
For the IVR phone surveys, a pre-test consisting of 24 English interviews and 10 French interviews was completed on April 9th, 2018 and fieldwork took place between April 10 and May 4, 2018. No changes were made to the pre-test survey and the results were included in the final dataset.
As this was a non-probability sample, margin of error does not apply and conclusions from these results cannot be generalized to any population. Surveying was conducted in the respondent’s official language of choice and took an average of 3.15 minutes for the in-person intercept and 3.52 for the IVR to complete. A detailed methodology can be found in Chapter 4.
1.4. Contract Value
The total contract value for this project was $121,038.14 including HST.
1.5. Statement of Political Neutrality
I hereby certify as Senior Research Director & Public-Sector Practice Lead of Kantar TNS that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Policy on Communications and Federal Identity and the Directive on the Management of Communications. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences and standings with the electorate or ratings of the performance of a political party or its leaders.
Tanya Whitehead
Kantar TNS
Senior Director & Public-Sector Practice Lead