It is also available upon request in multiple formats (large print, MP3, Braille, audio CD, e-text CD, DAISY or accessible PDF), by contacting 1 800 O-Canada (1-800-622-6232).
Rétroaction sur l’expérience client concernant un nouveau mode de prestation téléphonique : le Centre d’appui des services mobiles de Service Canada (Centre d’appui des services mobiles ) - Rapport méthodologique
Vous pouvez télécharger cette publication en ligne sur le site canada.ca/publicentre-ESDC.
Ce document est aussi offert sur demande en médias substituts (gros caractères, MP3, braille, audio sur DC, fichiers de texte sur DC, DAISY, ou accessible PDF) auprès du 1 800 O-Canada (1-800-622-6232).
Si vous utilisez un téléscripteur (ATS), composez le 1-800-926-9105.
The Citizen Service Branch (CSB) has developed a Client Experience (CX) Survey Measurement Model to understand how clients perceive their experience with Service Canada. The CX Survey Measurement Model informs exit survey design to allow clients to provide feedback at the point of service. Client feedback immediately after a service interaction provides Service Canada with a pulse check on how well each channel is working and allows for course correction throughout the year.
This project collected data from clients experiencing a new person-to-person service. This new toll-free telephone service, The Service Canada Outreach Support Center (OSC), offers service and additional support for Indigenous communities and vulnerable clients facing barriers to accessing Employment and Social Development Canada’s (ESDC) programs and services. Research was conducted with clients who call into the OSC system using Integrated Voice Response (IVR) survey methodology.
The overall objectives for the research are:
Monitor service satisfaction, ease, and effectiveness of the OSC channel; and
Identify opportunities for improvement (pain points) that occur within the OSC channel.
The questionnaire was designed by Service Canada and updated by Kantar to reflect an IVR appropriate approach and to ensure it complied with all industry and Public Opinion Research Directorate (PORD) requirements including those of the Government of Canada Standards for the Conduct of Public Opinion Research – Quantitative Research (Telephone and Online). The survey was designed to have a maximum estimated length of 5 minutes and was available in English or French.
All respondents were routed from existing OSC call centres to the IVR survey between December 2021 and May 2022. Respondents were recruited by OSC outreach workers by asking if clients would be willing to complete a feedback questionnaire about their service experience. Clients who consented, were transferred to the IVR software platform. The IVR component was conducted using Kantar’s partner Logit.
A pre-test was undertaken on December 13-20, 2021 obtaining 20 completions, no changes were required and as such the results of the 20 completes were included in the final data set.
Fieldwork was conducted using IVR technology from December 13th, 2021, to May 31st, 2022, and the total number of interviews included the results of the pre-test. In total, 155 surveys were completed. The average survey length was 4:17 minutes with the shortest being 2:58 minutes and the longest being 7:10 minutes. No weighting was applied to the final data.
1.2. Contract Value
The total contract value for this project was $71,936.94 including HST.
1.3. Statement of Political Neutrality
I hereby certify as Senior Research Director & Public-Sector Practice Lead of Kantar TNS that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Policy on Communications and Federal Identity and the Directive on the Management of Communications. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences and standings with the electorate or ratings of the performance of a political party or its leaders.
Tanya Whitehead Kantar TNS Vice President & Public-Sector Practice Lead
2. Methodology
2.1. Background and Objectives
A fundamental part of the Employment and Social Development Canada’s (EDSC) mandate is service delivery, which contributes to policy and program results. To this end, the Citizen Service Branch (CSB) has developed a Client Experience (CX) Survey Measurement Model to understand how clients perceive their experience with Service Canada.
The annual CX survey examines the service experience delivered by all channels interacting together for results tracking. In addition to the annual CX survey, the CSB collects ongoing feedback from clients on the service experience via service channels. These shorter questionnaires allow clients to provide feedback at the point of service, and provides a pulse check on how well each channel is working for clients and for course correction during the year. Currently, feedback questionnaires are in place in the web channel and for My Service Canada Account (MSCA), eServiceCanada and 1800 O-Canada. Feedback mechanisms are also in development for the in-person network.
The CX Feedback on New Service Channel Involving Phone: The Outreach Support Center (OSC) project collected data from clients experiencing a new person-to-person service. This was previously delivered via in-person service delivery and is currently delivered by phone due to the pandemic measures including COVID-19 social distancing measures. This new toll-free telephone service, The Service Canada Outreach Support Center (OSC), offers service and additional support for Indigenous communities and vulnerable clients facing barriers to accessing ESDC’s programs and services. Research was conducted with clients who call into the OSC system using Integrated Voice Response (IVR) survey methodology.
The overall objectives for the research are:
Monitor service satisfaction, ease, and effectiveness of the OSC channel; and
Identify opportunities for improvement (pain points) that occur within the OSC channel.
2.2. Survey Design
The questionnaire was designed by Service Canada, consulting other versions of feedback questionnaires drafted for use by Service Canada in consultation with the OSC team. The questionnaire was then updated by Kantar to reflect an IVR appropriate approach and to ensure it complied with all industry and Public Opinion Research Directorate (PORD) requirements including those of the Government of Canada Standards for the Conduct of Public Opinion Research – Quantitative Research (Telephone and Online).
The survey was designed to have an average maximum estimated the length of 5 minutes. The questionnaire consisted of exclusively closed ended and scale questions, with no coding required.
Once an approved final English version of the survey was ready, it was professionally translated to French.
Final questionnaires used for this research can be found in the Appendix of this report.
2.3. Programming
Once the questionnaires were finalized, they were programmed into the IVR system. This was done by experienced programmers. After programming, the questionnaires underwent extensive testing to ensure that everything was working as intended.
2.4. Pretest
A pre-test was undertaken on December 13-20, 2021, obtaining 20 completions. The results were reviewed to ensure the survey was working as expected and that the questions were being interpreted as expected. Based on the results of the pre-test, no changes were required and as such the results of the 20 completes were included in the final data set.
2.5. Sampling
All respondents were routed from existing OSC call centres to the IVR survey between December 2021 and May 2022. Respondents were recruited by OSC outreach workers by asking if they would be willing to complete a feedback questionnaire about their service experience. Clients who consented, were transferred to the IVR software platform. The IVR component was conducted using our partner Logit.
2.6. Data Collection
Fieldwork was conducted using IVR technology. This technology allows a computer to interact with humans using voice and keypad input tones. This ensured that the interview flowed as it should with pre-programmed skip patterns where necessary. It also controlled responses to ensure appropriate ranges and data validity.
As discussed previously, outreach workers recruited respondents during their service call. Clients who agreed to participate were forwarded to the IVR survey where they were offered the opportunity to respond to the survey in the official (English or French) language of their choice.
All participants were informed of the general purpose of the research, they were informed of the sponsor and the supplier and that all their responses would be confidential.
The IVR surveys were conducted from December 13th, 2021 to May 31st, 2002 and the total number of interviews included the results of the pre-test that took place December 13-20th, 2021. In total, 155 surveys were completed. The average survey length was 4:17 minutes with the shortest being 2:58 minutes and the longest being 7:10 minutes. Another 75 surveys were partially completed by respondents.
2.7. Completed Interviews
The following outlines the number of interviews completed by language.
Table 2.7: IVR completed interviews
Total
English
French
Total connections
230
189
41
Completions
155
141
14
Partial completions
75
48
27
2.8. Non-Response Bias
As with all samples, there is a possibility of non-response bias. For example, persons who were too busy or declined to participate in the survey were not included in the survey. In addition, some groups within the population are systematically less likely to answer surveys.
2.9. 2.9. Weighting
Population level data about clients who contact OSC outreach workers was not available at the time of tabulation and as such no weighting was applied to the final data.
2.10. Tabulated Data
Detailed tables are included under separate cover.
3. Appendix: IVR Survey Instrument
3.1. English Survey
OSC Feedback Questionnaire
S1A - [ROC number] Would you like to continue in English [press1] or French? [press2] Préférez-vous continuer en français [press2] ou en anglais[press1]?
S1B - [QUEBEC number] Préférez-vous continuer en français [press2] ou en anglais[press1]? Would you like to continue in English [press1] or French? [press2]?
S2A - “Kantar is conducting research on behalf of the Government of Canada on your experience today with Service Canada. Participation is voluntary and confidential. Your responses will remain anonymous and the information you provide will be administered according to the requirements of the Privacy Act, the Access to Information Act, and any other pertinent legislation. It will take no more than 5 minutes to complete.”
Should you wish to verify the legitimacy of this survey you may contact Carole at carole.adam@kantartns.com.
S2B - « Kantar effectue actuellement un sondage au nom du gouvernement du Canada au sujet de votre expérience aujourd’hui auprès de Service Canada. La participation au sondage est volontaire et confidentielle. Vos réponses demeureront anonymes et les renseignements que vous fournirez seront protégés conformément aux exigences de la Loi sur la protection des renseignements personnels, de la Loi sur l’accès à l’information et de toute autre loi pertinente. Le sondage ne devrait pas prendre plus de 5 minutes de votre temps. »
Si vous souhaitez vérifier la légitimité de ce sondage, vous pouvez communiquer avec Carole à carole.adam@kantartns.com.
Attribute from CX Model
Scope of Assessment
Proposed Questions (DRAFT)
Overall Satisfaction
Multi-channel Service Design
Q1 - Overall, how satisfied were you with your experience on the phone today? Press a number from 1 to 5 closest to your level of satisfaction, where 1 is very dissatisfied and 5 is very satisfied. Please press 9 for don’t know.
1 - Very dissatisfied
2 - Dissatisfied
3 - Neither satisfied nor dissatisfied
4 - Satisfied
5 - Very satisfied
9 - Don’t know
General - purpose
Multi-channel Service Design
Q2 - When you called today, which of the following was the main reason for your call?
Press 1 for Employment Insurance
Press 2 for Canada Pension Plan or Old Age Security
Press 3 for Social Insurance Number
Press 4 for Canada Revenue Agency
Or Press 5 for Other
Objective
Q3 - What specifically were you trying to do?
Press 1 for Get help with my application
Press 2 for Get general information
Press 3 for Update my information
Press 4 for Get the status of my application
Or Press 5 for Other
Effectiveness
Multi-channel Service Design
Q4 - Were you able to do what you needed to do during your call?
Press 1 for Yes
Press 2 for No
Press 3 for Partially
Or Press 9 for Don’t know
Ranking
Q5 - Which of the following would have most improved your experience?
Press 1 for More helpful service
Press 2 for Clearer information provided during the call
Press 3 for The Service Canada employee could do more with my request while they were on the phone with me
Press 4 for No selection. The service was great as is
Press 5 for Other
Demographic
Q6_Trans For each of the following items please indicate if they apply to you. Press 1 for yes, 2 for no and 9 for prefer not to say
Q6_1 - You are uncomfortable in English and French
Press 1 for yes
Press 2 for No
Or press 9 for prefer not to say
Q6_2 - You are aged 18 to 30 (DO NOT READ - programming note: Selecting this age category precludes selection of the other age category)
Press 1 for yes
Press 2 for No
Or press 9 for prefer not to say
Q6_3 - [PN: If yes at Q6_2 skip Q6_3] You are aged 60 or over (DO NOT READ - programming note: Selecting this age category precludes selection of the other age category) – [IF QX=1, DO NOT ASK QXB]
Press 1 for yes
Press 2 for No
Or press 9 for prefer not to say
Q6_4 – You identify as Indigenous () (DO NOT READ - programming note: IF YES, ASK Q8)
Press 1 for yes
Press 2 for No
Or press 9 for prefer not to say
Q6_5 - You have access to the Internet
Press 1 for yes
Press 2 for No
Or press 9 for prefer not to say
Q6_6 - You live in a northern or remote area
Press 1 for yes
Press 2 for No
Or press 9 for prefer not to say
Q6_7 - You live on a First Nations reserve
Press 1 for yes
Press 2 for No
Or press 9 for prefer not to say
Q6_8 - You have a disability [IF YES, ASK Q7 QNEXT]
Press 1 for yes
Press 2 for No
Or press 9 for prefer not to say
Type of disability
Q7 - What type of disability?
Press 1 for Hearing/Communicating
Press 2 for Seeing
Press 3 for Mobility
Press 4 for Cognitive
Press 5 for Mental health-related
Press 6 for Other
Or press 9 for prefer not to say
Indigenous
Q8 – Do you identify as First Nations, Métis or Inuit? (DO NOT READ - programming note: ASK Q8 IF SAID YES AT Q6_4)
Press 1 for First Nations
Press 2 for Métis
Press 3 for Inuit
Or press 9 for prefer not to say
Q8A - “Thank-you for your participation. Your feedback is important to the Government of Canada and will be used to help improve service. Goodbye.”
Q8B - « Nous vous remercions de votre participation. Vos commentaires sont importants pour le gouvernement du Canada, et ils serviront à améliorer les services. Au revoir. »
3.2. French Survey
Centre d’appui des services mobiles – questionnaire de rétroaction
S1B - [Numéro pour le QUÉBEC] Préférez-vous continuer en français [appuyez sur le 2] ou en anglais [appuyez sur le 1]? Would you like to continue in English [press1] or French? [press2]?
S1A - [Numéro pour l’ensemble du Canada, sauf le Québec] Would you like to continue in English [press1] or French? [press2] Préférez-vous continuer en français [appuyez sur le 2] ou en anglais [appuyez sur le 1]?
S2B « Kantar effectue actuellement un sondage au nom du gouvernement du Canada au sujet de votre expérience aujourd’hui auprès de Service Canada. La participation au sondage est volontaire et confidentielle. Vos réponses demeureront anonymes et les renseignements que vous fournirez seront protégés conformément aux exigences de la Loi sur la protection des renseignements personnels, de la Loi sur l’accès à l’information et de toute autre loi pertinente. Le sondage ne devrait pas prendre plus de 5 minutes de votre temps. »
Si vous souhaitez vérifier la légitimité de ce sondage, vous pouvez communiquer avec Carole à carole.adam@kantartns.com.
Caractéristique du modèle sur l’expérience client
Portée de l’évaluation
Questions proposées (VERSION PROVISOIRE)
Satisfaction globale
Conception de services multimodes
Q1 - Dans l’ensemble, dans quelle mesure avez-vous été satisfait(e) de votre expérience au téléphone aujourd’hui? Appuyez sur un chiffre de 1 à 5 pour indiquer votre niveau de satisfaction, où 1 signifie Tout à fait insatisfait(e) et 5 signifie Tout à fait satisfait(e). Appuyez sur le 9 pour « Je ne sais pas ».
1 - Très insatisfait ou insatisfaite
2 - Insatisfait ou insatisfaite
3 - Ni satisfait ou satisfaite, ni insatisfait ou insatisfaite
4 - Satisfait ou satisfaite
5 - Très satisfait ou satisfaite
9 - Je ne sais pas
General - purpose
Conception de services multimodes
Q2 - Quelle était la principale raison de votre appel aujourd’hui
Appuyez sur 1 pour Assurance emploi
Appuyez sur 2 pour Régime de pensions du Canada ou Sécurité de la vieillesse
Appuyez sur 3 pour Numéro d’assurance sociale
Appuyez sur 4 pour Agence du revenu du Canada
Ou Appuyez sur 5 pour Autre
Objectif
Q3 - Que vouliez vous faire, exactement?
Appuyez sur 1 pour Obtenir de l’aide concernant ma demande.
Appuyez sur 2 pour Obtenir des renseignements généraux.
Appuyez sur 3 pour Mettre à jour mes renseignements.
Appuyez sur 4 pour Connaître l’état de ma demande.
Ou Appuyez sur 5 pour Autre
Efficacité
Conception de services multimodes
Q4 - Avez vous réussi à faire ce que vous vouliez faire pendant l’appel?
Appuyez sur 1 pour Oui
Appuyez sur 2 pour Non
Appuyez sur 3 pour En partie
Ou Appuyez sur 9 pour Je ne sais pas
Classement
Q5 - Lequel des éléments suivants aurait le plus amélioré votre expérience? [Les réponses apparaissent dans un ordre aléatoire]
Appuyez sur 1 pour Un service plus utile
Appuyez sur 2 pour Renseignements plus clairs fournis pendant l’appel
Appuyez sur 3 pour La possibilité pour l’employé(e) de Service Canada d’en faire davantage avec ma demande pendant qu’il ou elle est au téléphone avec moi
Appuyez sur 4 pour Aucune de ces réponses – le service était excellent tel quel
Ou Appuyez sur 5 pour Autre
Données démographiques
Q6_1 Pour chacun des énoncés suivants, veuillez indiquer s’il s’applique à vous. Appuyez sur 1 pour Oui, sur 2 pour Non et sur 9 pour « Je préfère ne pas répondre ».
Q6_1 Vous n’êtes pas à l’aise en français ni en anglais.
Appuyez sur 1 pour oui
Appuyez sur 2 pour Non
Ou appuyez sur 9 pour ne pas dire
Ou appuyez sur 9 pour ne pas dire
Q6_2 - Vous avez entre 18 et 30 ans (NE PAS LIRE – note de programmation : la sélection de cette catégorie d’âge empêche la sélection de l’autre catégorie d’âge).
Appuyez sur 1 pour oui
Appuyez sur 2 pour Non
Ou appuyez sur 9 pour ne pas dire
Q6_3 - Vous avez plus de 60 ans (NE PAS LIRE – note de programmation : la sélection de cette catégorie d’âge empêche la sélection de l’autre catégorie d’âge) – [Si la réponse à la Q1 est oui, ne pas poser l’autre question]
Appuyez sur 1 pour oui
Appuyez sur 2 pour Non
Ou appuyez sur 9 pour ne pas dire
Q6_4 –Vous vous identifiez comme Autochtone. (NE PAS LIRE – note de programmation : Si la réponse est oui, poser la question 8)
Appuyez sur 1 pour oui
Appuyez sur 2 pour Non
Ou appuyez sur 9 pour ne pas dire
Q6_5 - Vous avez accès à Internet.
Appuyez sur 1 pour oui
Appuyez sur 2 pour Non
Ou appuyez sur 9 pour ne pas dire
Q6_6 - Vous habitez dans le Nord ou dans une région éloignée.
Appuyez sur 1 pour oui
Appuyez sur 2 pour Non
Ou appuyez sur 9 pour ne pas dire
Q6_7 - Vous habitez dans une réserve des Premières Nations.
Appuyez sur 1 pour oui
Appuyez sur 2 pour Non
Ou appuyez sur 9 pour ne pas dire
Q6_8 - Vous avez une incapacité [Si la réponse est oui, passez à la question suivante].
Appuyez sur 1 pour oui
Appuyez sur 2 pour Non
Ou appuyez sur 9 pour ne pas dire
Type d’incapacité
Q7 - Quel est le type d’incapacité?
Appuyez sur 1 pour Trouble de l’audition ou de la communication
Appuyez sur 2 pour Trouble de la vision
Appuyez sur 3 pour Trouble lié à la mobilité
Appuyez sur 4 pour Troubles cognitifs
Appuyez sur 5 pour Trouble de santé mentale
Appuyez sur 6 pour Autre
Ou appuyez sur 9 pour ne pas dire
Type d’incapacité
Q8 - Est-ce que vous vous identifiez comme un/une membre des Premières Nations, Métis ou Inuit? (DO NOT READ - programming note: ASK Q8 IF SAID YES AT Q6_4)
Appuyez sur 1 pour Premières Nations
Appuyez sur 2 pour Métis
Appuyez sur 3 pour Inuit
Ou appuyez sur 9 pour ne pas dire
Q8A - “Thank-you for your participation. Your feedback is important to the Government of Canada and will be used to help improve service. Goodbye.”
Q8B - « Nous vous remercions de votre participation. Vos commentaires sont importants pour le gouvernement du Canada, et ils serviront à améliorer les services. Au revoir. »
