Qualitative assessment of the Pay Insight Concept and Track MyCASE update—Executive Summary

Summer 2018

Prepared for the Department of Public Works and Governmnet Services Canada

Supplier Name: Léger
Contract Number: EP961-191339/001/CY
Contract Value: $39,550.00 (Harmonized sales tax (HST) included)
Award Date: 2018-08-23
Delivery Date: 2018-10-12

Related publications (registration number: POR  038-18)

For more information on this report, please contact the Department of Public Works and Governmnet Services Canada at:
porcoordcomm-ropcoordcomm.pwgsc@tpsgc-pwgsc.gc.ca

1. Executive Summary

Léger is pleased to present Public Services & Procurement Canada (PSPC) with this report on findings from a series of focus groups dedicated to assessing self-service online tools dedicated to pay issues.

This report was prepared by Léger who was commissioned by PWGSC (contract number EP961-191339/001/CY awarded August 23, 2018).

1.1 Background and Objectives

PSPC commissioned Léger to conduct a qualitative assessment study of two online tools. PSPC’s teams are currently developing two online tools that will allow Government of Canada employees to obtain information on their pay file and file tracking. These two applications, "Pay Insight" and “Track MyCASE”, are still in the design stage. The intent of these online tools is to display information in using a structure and logic that will be easily understood by users, while providing employees with enhanced visibility into their pay file and possibly even contributing to the prevention and resolution of problems, resulting in an increased level of control and satisfaction.

The objective of the study was to assess the likely communications effectiveness of both tools among the intended target audience: public servants. Key elements of the study are to evaluate clarity, user-friendliness, ease of understanding, user expectations about both online solutions (prototype stage) and to identify elements and/or features that generate the most positive or negative reactions.

Copies of the moderator’s guide used in the focus group sessions can be found in Appendix.

1.2 Qualitative Methodology

1.2.1 Focus groups

A total of five (5) focus groups were held: three (3) in person and two (2) online. The in-person focus groups were held in two (2) different locations: Ottawa and Montréal. Two (2) separate groups were conducted in Ottawa in English and one (1) group in Montréal in French. In addition to the in-person groups, Léger conducted a series of two (2) online via the ITracks video chat platform with participants from different regions of Canada.

Groups consisted of Canadian public servants from different departments with a good mix of genders and experience working for the federal government. The reader is directed to Appendix for a more detailed methodology and for a copy of the recruitment screener.

Recruitment

All participants were recruited using the Government Electronic Directory Services list of employees. Public servants were separated in two (2) different groups: 1) public servant working in Departments and Agencies that are served by Pay Centre; and 2) public servant working in Departments and Agencies that are not served by Pay Centre. Public servants working in Departments and Agencies that are not served by Pay Centre did not have to evaluate the “Track MyCASE” online tool concept as part of this study.

Léger adhered to all MRIA and Government of Canada standards for conducting qualitative research. To thank them for their time and to show our appreciation for their participation, participants received a financial incentive of $150 each. The following table shows the distribution of participants in this study for each group:

1.3 Overview of Qualitative Findings

It should be noted that the presentation of the screenshots of the Pay Insight and Track my Case application is happening within a broader context of fairly low confidence, sometimes even outspoken cynicism regarding pay and compensation issues at the Government of Canada. At many times, comments expressed about the actual content of the App tested tended to be about the expected outcome of whatever action the civil servant may undertake as opposed to an evaluation of the value of the content presented in the App. While the overall perspective of participants is rather positive, many of the negative comments heard tended to be about broader considerations vis-à-vis pay issues rather than the prototype applications tested.

Current Usage of Compensation Web Applications

Almost all participants visit Compensation Web Applications (CWA) just about every pay cycle. They mainly use CWA to track their pay check for accuracy or changes over time. Although they use CWA for many other Human resources (HR) and pay related information, the ‘pay stub’ remains the key driver of most visits. While they remain fairly positive about the user-friendliness of CWA, one main critique remains the “Track MyCASE” application, mainly for its perceived lack of transparency and the pace of change in the cases participants are tracking.

Presentation of "Pay Insight" Concept (screenshots)

When presented with the initial screenshots of "Pay Insight", focus group participants tended to have similar reactions:

• Reactions to the look-and-feel tended to be positive and the idea of having all their key information at-a-glance was seen as a step forward from the current tools used. The “current” look and nice design were spontaneously mentioned by many.
• Participants struggled to understand the purpose of "Pay Insight". The one-stop-shop element of the solution was barely mentioned, the fact that the Concept would be more transparent and add key information they do not have or cannot find right now was spontaneously mentioned by participants. Some immediately jumped to the conclusion that the App would be a highly interactive self-serve tool, while others “only” saw it a tool centralizing existing information.
• While participants rate themselves fairly high on digital skills overall, they often struggled with what would be clickable, how words relate to icons or indications of drop-down menus, and so on. When faced with a screenshot, they did not necessarily know what they would or could do on the page.
• Language and precision of information were comments often made, such as the difference between and definition of a case, activity or event. We did not feel that there is a common language used by participants to refer to pay and HR issues. Participants frown at the usage of codes or reference numbers if they cannot expand the view of hover over to find what the reference code or number actually means (i.e. PCCM04262431 in the “Recent Activities” section of the Homepage). On most pages, participants wanted to add information and to some extent to clutter the interface. Many times they commented when information was last updated, saying that this information is key and that dates should allow them to track progress. The Profile, Employment and Benefits pages tended to generate positive comments initially but also generated fairly high expectations as to what they could do (interact, modify, change) with the information but also about functionalities (Benefits page in particular).
• The Support Pages generated some confusion and some negative comments. The difference between Help and Take Action and Contact Us were not clear, or they did not know what the best course of action was if they found some information about them that is missing or incorrect. The current design of the support pages also generated some confusion, but this may be due to the fact that they could not interact with the pages themselves. Much of the negative comments on the support pages were related to some frustration regarding the quality of the “backend” support they would get, which is not tied to the application tested, but rather about the general context of Pay and HR issues at the Government of Canada.

The Demo

One interesting finding in sessions is that initial reactions to the screenshots tended to be positive, comments on each page when drilling down were more negative and that overall reactions to the App after the demo was shown tended to be positive as well. This may be due to the fact that the demo was clearer on the intended purpose of the App and its value added compared to what the actual screens of the App delivered. As well, when taking a step back after the demo, participants tended to be better at separating comments about the App itself from pay and HR issues in general.

Track MyCASE

The design of ”Track MyCASE” tended to generate positive comments about the added layers of information the App would provide. The added precision and the change is some language used compared to what they know now, were often identified as positive (i.e. In the Status section “In Review” and “Awaiting Approval”). However, information provided by the hover bubbles is seen as redundant with the status. Several comments were again tied to the language used to describe cases and their progress. Again many of the additions suggested were tied to their perceptions of the current process in resolving cases more than about the App itself (total time to resolve a case, wanting a service standards or “resolved by” date, etc.). In this regards, participants would expect the information to be more detailed in the pop-up ticket.

1.4 Note on Interpretation of Research Findings

This report was compiled by Léger, based on the research conducted specifically for this project. The analysis presented represents what Léger believes were the most salient points during the focus group sessions.

Findings from this qualitative research (i.e. focus groups) should be considered directional only and results should not be projected as representative of all federal employees and who match the profile of the group participants. It is intended to provide deeper insights into the underlying reasons for opinions or lack thereof.

1.5 Report

This report includes a detailed analysis of the focus group findings in section two (2). Details of the research methodology, the recruitment guide and moderator’s guide are included in the appendices.

1.6 Intended Use of the Results

The findings of this research will help officials of PSPC and the Government of Canada in making a decision on the self-service online tool concepts to be designed and implemented.

1.7 Political Neutrality Statement and Contact Information

I hereby certify as Senior Officer of Léger that the deliverables fully comply with the Government of Canada’s political neutrality requirements outlined in the Directive on the Management of Communications. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leaders.

Christian Bourque
Executive Vice-President and Associate
Léger

1.8 Contact Information

Christian Bourque
Executive Vice-President and Associate
Léger
cbourque@leger360.com
514-982-2464

Additional information

Supplier name:

Léger

Léger:

PBN 103038444PG0001

Contract Number:

EP961-191339/001/CY

Contract Award Date:

2018-08-23

Study Cost:

$39,550.00 (HST included)

For more information, contact: porcoordcomm-ropcoordcomm.pwgsc@tpsgc-pwgsc.gc.ca.