PUBLIC OPINION AND UX RESEARCH FOR CANADA
BORDER SERVICES AGENCY
Final
Report
Prepared for Canada Border Services Agency
Supplier name: Leger marketing inc.
Contract
number: 47419-195001/001/CY
Contract
value: $122,328.15
Awarded
date: 2019-01-14
Delivery
date: August 14, 2019
Registration
number: POR-106-18
For more
information on this report, please contact Canada Border Services Agency at: Erika-Kirsten.Easton@cbsa-asfc.gc.ca
Ce rapport est aussi disponible en
français.
This report presents the
results of a UX study conducted with employees of the Canada Border Services
Agency (CBSA) and a user study conducted with visitors to the “Travellers
section” of the CBSA's website conducted by Leger Marketing Inc. on behalf of Canada
Border Service Agency. The research was conducted between January, 2019 and March, 2019.
Cette publication est aussi disponible en
français sous le titre : Recherche sur l’opinion publique et tests
utilisateurs pour l'Agence des services frontaliers du Canada.
Catalogue Number:
PS38-98/2019E-PDF
International Standard Book Number (ISBN):
ISBN 978-0-660-30420-5
Related publications (registration number: POR
101-18):
Catalogue Number: PS38-98/2019F-PDF (Final Report,
French)
ISBN 978-0-660-30421-2
© Her Majesty the Queen in Right of Canada,
as represented by the Canada Border Service Agency, 2019.
Léger is pleased to
present to the Canada Border Services Agency (CBSA) this report on the results
of qualitative and quantitative studies conducted with CBSA employees and
quantitative and qualitative studies conducted with visitors to the Travellers
section of the CBSA's website.
This report was
prepared by Leger who was contracted by Canada Border Services Agency (contract
number 47419-195001/001/CY awarded January 14, 2018).
This public opinion
research project is divided into two main sections: a section dedicated to the
Agency's internal intranet (accessible only to employees) and a section
dedicated to the Agency's external website (accessible to the general public).
Intranet
CBSA’s Intranet
(Atlas) is a major communication tool to inform employees (about 14,000) of the
day-to-day business of the Agency as well as a resource for policies, guidance
and direction. The content on Atlas is structured according to the organization
rather than its functions, and therefore does not best meet the needs of the
user.
Currently, Atlas
content is built upon a division of branches and regions. This structure does
not take users’ requirements into consideration and creates repetition or an
overlap of content across sections. As such, Atlas has become very difficult to
navigate and search and often leads users to the wrong information,
unsearchable data or a duplication of information. These factors result in a
tremendous loss of time for employees.
The Intranet Public
Opinion Research (POR) aspect of this project focuses on CBSA employees with
the intent of gathering the information required to assist in rebuilding Atlas.
The new site must be organized by audience and user tasks and not according to
the organizational model.
Objectives of
the Intranet Section:
• Asses satisfaction towards the Intranet;
• Learn more about expectations towards the
Intranet;
• Know perceived usefulness of content in the
Intranet;
• Qualify content types (topics) as useful,
desired, proposed, or mandatory;
• Propose a modified information architecture
that better reflects employees’ understanding of the Intranet content.
Internet – Traveller
Section of the CBSA website
Visitor data from the
Travellers pages of the CBSA website shows that there are approximately 237,000
unique visitors per month. Most of the visitors access the website from Canada
but many other visitors come from other countries. People visit the Travellers section to gather
information on different subjects (what they can bring across the border, wait
times, prohibited materials, etc.), and based on the available website data, it
is known that the Nexus pages are among the most popular.
Objectives of
the Internet Section:
·
Asses visitors’
satisfaction with the traveller’s section of the CBSA websites;
·
Identify the
reasons for visiting the website
·
Assess the
clarity of the information on the site
·
Identify
sources of misunderstanding
·
Identify
navigation problems
·
Evaluate how
the Nexus section is used by visitors
·
Identify
potential areas for improvement of the traveller’s section of the CBSA’s
website.
To
achieve the study objectives, a research plan based on a hybrid method,
qualitative and quantitative, was developed.
1.2.1 Intranet Section
To achieve the
objectives set for the Intranet portion of the study, we used a four-step
methodology: 1) focus groups, 2) individual interviews, 3) persona creation,
and 4) a tree structure test (reverse card sorting)
First, a qualitative
methodology consisting of focus groups with the CBSA’s employees was set up. It
was followed by a series of one-on-one interviews and card-sorting exercise.
These two first phases were the foundation of persona creation. Based on phases
1 and 2, a new information architecture was proposed for Atlas renewal. This
new architecture was further tested with employees using a tree testing
methodology, also known as reverse card sorting.
1.2.1.1 Six
Focus Group
Leger conducted a
series of six focus groups with CBSA employees. There were groups of employees
recruited from (1) employees in the field, (2) employees from the regional HQs
and (3) employees for the national HQ in Ottawa. All sessions were held in CBSA
locations. Participants were recruited by CBSA. Each group session lasted
approximately 120 minutes. Every session was recorded for analysis purposes. Leger was responsible for preparing the moderation guides and moderating
the groups. The guide was
developed in consultation with CBSA’s project authority. The groups were
moderated in both French and English according to the needs of the employees.
The guides and tools were available in both languages.
Participants were informed of all their rights under
Canada’s Privacy Act and the
Standards for the Conduct of Government of Canada Public Opinion Research. Specifically,
their confidentiality was guaranteed, and their participation was voluntary. CBSA was responsible for ensuring the participation
and availability of its employees for scheduled interview dates and times.
There was no financial incentive to ensure the participation of CBSA employees.
Locations and dates
Groups were
held in the following cities on the dates specified.
Table 1. Detailed Recruitment
|
City |
Recruits |
Participants |
Target |
Language |
Date |
|
Ottawa |
10 |
8 |
HQ employees |
EN/FR |
January 28, 2019 |
|
Ottawa |
10 |
7 |
HQ employees |
EN/FR |
January 28, 2019 |
|
Mississauga |
10 |
9 |
HQ employees |
EN |
February 5, 2019 |
|
Toronto Pearson Int’ Airport |
10 |
11 |
BSO |
EN |
February 5, 2019 |
|
Vancouver |
10 |
4 |
BSO |
EN |
February 6, 2019 |
|
Vancouver |
10 |
11 |
BSO |
EN |
February 6, 2019 |
|
Total |
60 |
50 |
|
|
|
1.2.1.2 One-on-one
interviews
Leger conducted
one-on-one interviews with participants of each main profile (employees in the
field, employees from a regional HQ and employees from national HQ). Each interview
lasted approximately 45 minutes. Every session was recorded for analysis
purposes.
Leger was responsible for preparing the interview
guide, preparing the card sorting exercise and conducting the interviews in
English and French. The recruitment guide and card sorting exercise were developed
in consultation with CBSA’s project authority.
Participants were informed of all their rights under
Canada’s Privacy Act and the
Standards for the Conduct of Government of Canada Public Opinion Research. Specifically,
their confidentiality was guaranteed, and their participation was considered
voluntary. CBSA was be responsible for ensuring the participation and
availability of its employees for scheduled interview dates and times. There was
no financial incentive to ensure the participation of CBSA employees.
Table 2. Detailed Recruitment
|
City |
Recruits |
Participants |
Target |
Language |
Date |
|
Montreal |
5 |
5 |
HQ employees
and BSO |
EN/FR |
February 6 and February 8, 2019 |
|
|
5 |
5 |
HQ employees
and BSO |
EN |
February 12, 2019 |
|
Ottawa |
5 |
5 |
HQ employees |
EN |
February 15, 2019 |
|
Total |
15 |
15 |
|
|
|
1.2.1.3 Persona
Creation
Based on the
information collected in phases 1 and 2, we have created as set of six personas.
The final number of personas has been dictated by the outcome of data analysis.
The main objectives of personas are to efficiently present and share
information related to intranet users.
Personas document
several dimensions, such as:
• Employee role
• Technological profile
• Daily challenges
• Main frustrations with Atlas
• Short bio to give life to the persona
1.2.1.4 Tree
testing
In this phase of the research, we
evaluated an information architecture with Atlas users. The architecture that
was evaluated was designed using the results of the focus groups and the card
sorting done during the individual interviews. To perform this tree structure
test, we used the Treejack
tool of the OptimalWorkshop
platform. CBSA employees who visited Atlas between March 12 and March 22 were
invited to participate in the test via an open link. The tree testing consisted
of ten tasks that employees were required to perform in the proposed
information structure. The test was available in both French and English at the
respondent's preference.
A total of 1,164 respondents took the test
in English and a total of 200 in French. 434 users were frontline employees and
930 were not.
For each task, we measured several indicators, such as:
·
Success rate and failure
·
The paths
·
The final destination
The detailed
methodology is presented in Appendix.
1.2.2 Internet Section
Quantitative
Methodology
In order to meet the
Internet-related objectives, a methodology in two phases was followed. First, a
quantitative methodology consisting of an open-link survey on CBSA’s website was
set up. It was followed by a series of online focus groups with CBSA’s website visitors.
1.2.2.1
Open-link Survey on the CBSA Website
This quantitative research was
conducted through an online survey, using a Computer Aided Web Interviewing
(CAWI) technology. The public consultation was launched by means of an
open-link survey questionnaire available on the CBSA website. Any individual
who visited the traveller’s section of the site between February 4, 2019 and
March 4, 2019 was invited to answer the questionnaire by clicking on the link.
This part of the public consultation
generated a significant volume of responses. A total of 2,729 respondents were
gathered via the open-link. We should remind the reader that the results of
this part of the public consultation should not be interpreted as representing
the opinions or attitudes of the Canadian public at large nor representative of
the visitors of CBSA’s website. It is a collection of respondents who volunteered
to answer the questionnaire. No statistical weighting was performed on this
sample.
Since this is a sample of
volunteers, no margin of error can be calculated for this portion of the study.
Nor can we comment on the participation rate, as we do not know the traffic and
the exact volume of visitors to the Agency's website during the period when the
open link was active on the web page.
The online survey has
given us the opportunity to recruit participants to conduct focus groups with
users of the Agency's website. We therefore asked all survey participants if
they were open to participate in a second phase of the study. Those who agreed
were invited to leave their names and contact information so that we could
contact them to complete the recruitment process.
Leger adheres to the most stringent guidelines for quantitative
research. The survey instrument was compliant with the Standards of Conduct of
Government of Canada Public Opinion Research – Series E – Qualitative and
Quantitative Research. The questionnaire was developed by Léger in
collaboration with the CBSA research project leaders.
The
details of the methodology and more information on Leger’s quality control
mechanisms are presented in Appendix.
The survey
questionnaire is available in Appendix.
Leger conducted a
series of three (3) online focus groups with visitors of the CBSA Website. Their
were all recruited from the online survey from the previous research phase of
the study. All three sessions were held online via the ITracks
video chat platform with participants from different regions of Canada or the
US. The following table is a summary of the locations, date, profile and number of participants for all the
discussion groups.
|
GROUP |
Group profile |
Language |
Recruited |
Participants |
Dates and |
Time (Eastern time) |
Type |
|
GR01 |
CBSA’s website visitors |
FR |
10 |
8 |
March 7, 2019 |
4:30 PM |
Online |
|
GR02 |
CBSA’s website visitors |
EN |
10 |
8 |
March 7, 2019 |
6:00 PM |
Online |
|
GR03 |
CBSA’s website visitors |
EN |
10 |
7 |
March 7, 2019 |
8:00 PM |
Online |
Participants were informed of all their rights under
Canada’s Privacy Act and the
Standards for the Conduct of Government of Canada Public Opinion Research.
Specifically, their confidentiality was guaranteed, and their participation was
voluntary. Léger was responsible for the recruitment of the
participants and of the moderation of the online focus groups.
Léger's professional recruiters ensured the
availability and participation of recruits. Léger was responsible for
organizing the sessions on Itracks' video chat
platform. A financial incentive of $100 per participant was given to all group
participants to thank them for taking the time to participate.
1.3 Overview
of the Intranet Study Findings
Given that ‘Atlas’ has created some frustration
and that many employees have written it off as an effective work tool, we
believe that the Intranet needs a full re-launch both to signal a significant
change in direction and usability and a clear indication that the new tool was
in large part “designed” by users. That
full re-launch would require a new name.
If brand equity in Atlas is low, change the brand. We suggest a name along the line of “MyBorder” to give it a relevant name and a clear indication
that it is ‘their’ work tool, ‘their’ home.
In order to achieve this goal, we strongly
recommend that the Atlas modernization team create a small number of working
groups to co-design the new architecture.
This will drive support for the initiative and improve usability. We readily see the need for a communication
group (including the regions), a HQ group (finance, procurement and HR
functions in particular), a BSO supervisor group (with representation for the
regions).
Building engagement should be at the core of the
new design. All employees expressed the
need for a common “News Headlines” section that would represent what the Agency
does and how it serves its mission.
While not limited to frontline activities, the core of that section
should be about protecting and serving Canadians at the border. Frontline staff want to hear more about themselves
and HQ employees want to hear about the front line. Initiatives like Border Update (if captioned)
can serve this purpose, but regional staff expressed the need to hear about
news from all regions and share initiatives, successes, stories and kudos in a
format that is more “bottom-up” rather than “top-down”.
Atlas should be first conceived as a “work
tool”. It is there to support employees in
carrying out their duties, tasks and responsibilities. What leads employees to Atlas today is
task-based, not “let’s see what’s new or going on”. Users will look for “softer” information only
if they believe Atlas will allow them to get their job done.
The new design should aim for
oversimplification. A design with fewer
menus versus more levels should be considered.
This will be tested in the tree testing validation stage.
The wiki in its current form may be harmful for
the Agency. Participants in the research
who use their regional wiki only go there for a very limited number of pages,
while they have strong doubts about the quality, accuracy and validity of the
information found in the wiki. The
working groups should determine what are the key functions served by the wiki
now and integrate this into, potentially, a collaboration zone within Atlas.
Moving towards Apollo as the document repository
should continue. If some still resist
the change because of a login process that is seen as not optimal and despite
slowdowns and downtime, many employees feel a ‘leaner’ Atlas efficiently
directing users to the relevant Apollo documents can work.
The proposed information architecture for Atlas
could be as follows:
Homepage
Employee
Awards
and Recognition
Employee
Assistance Program
Employee
Orientation / Onboarding / New Hires
Employment
Equity and Diversity
Integrity,
Values and Ethics
Jobs
and Career Development
Occupational
Health and Safety
Pay
and Benefits
Training
and Learning
Well-being
Security
Tools
Information Management
Apollo
ATIP
Forms and Template Library
Immigration
Commerce
Traveller
Enforcement of the Law
Other Themes
Frontline Bulletins and
National Document Centre Publications
Immigration
Commerce
Traveller
Enforcement of the Law
Other Themes
Guides / Manuals and Standard Operating Procedures
Immigration
Commerce
Traveller
Enforcement of the Law
Other Themes
Policy Library
Immigration
Commerce
Traveller
Enforcement of the Law
Other Themes
Finance Volume
Procurement and Contracting
Accommodations and Facilities
The
Organization / About Us
Agency Organization Chart / About Branches, Directorates and Divisions
Executive Offices
Human
Resources
Labour
Relations
Performance
Management
Staffing
IT
Portal
Helpdesk
IT Portal
ACROSS
ICES
CAS
ESS
Daily
News
Agency and Branch Initiatives Priorities
Border Updates (Video
Series)
CBSA Gives / GCWCC
(Charitable Campaign
Event Calendar
Messages from
Executives
News and Photo Galleries
There are some
important lessons learned in developing the final information architecture for
the renewal of Atlas:
- The
“Frontline Bulletins and National Document Centre Publications” should be
classified under the main menu "Tools" and not under the menu
"Daily News".
- Some
items, as currently identified in Atlas, should be renamed to clarify
their content.
a. The "Employment and
Professional Development" tab can easily be confused with the
"Training and Learning" tab.
b. The tabs "Agency Organization
Chart / About Branches, Directorates and Divisions" and "Executive
Offices" do not adequately allow users to know what content they will find
under either one.
c. The information found under the
"Human Resources" tab should be reformulated. For the moment, some of
the information found under "Human Resources" is also searched under
the "Employee" menu. Work should be done at this level to clarify and
reorganize that information.
d. Identify more clearly where to find
the content of the weekly video series in the "Daily News" tab to
make it easier for users to find it.
- The
results of the tree test indicate that the main menu created by employees
during card sorting works properly to guide users in their task on the
intranet.
- The
accuracy of libraries of forms, guides, manuals, manuals, policies and
newsletters on different topics, such as immigration, commerce, travel,
etc., helps users to guide and refine their search for information. In a
real work environment, we believe this approach is promising.
- The tree
test does not pretend to have evaluated all categories and items of the
submenus. The effort to clarify category wordings must be systematically
carried out by the CBSA for every item.
1.4 Overview of the Internet Study Findings
When it comes to the Internet research with travellers, both the
quantitative and qualitative research phases confirmed that visitors tend to be
satisfied. Indeed, they “got what they
were looking for”. Beyond the usual
critique concerning the look & feel of government-type websites, and the
feeling that pages were “too wordy” or “too busy”, most participants were able
to get the answers they needed. Some
processes were more laborious, however.
It is namely the case of those seeking to fill out a Nexus application
for the first time. For them, the
process is not seamless and fully transparent and could be reviewed. The language of the menus makes them sound
like “action items” or “transactional items” while they are not and many were
surprised to the taken out of the GoC environment
later in the process.
Other changes were suggested by participants. First, menus should be changed from the
current language to simple questions (i.e. How to apply for Nexus? What can I
bring back to Canada?). Participants
feel they access the website to answer a question they have and that if the
website mirrored these questions, it would provide for a smoother
navigation. They feel that the current
language can be ambiguous or make some of the menu items not mutually exclusive.
Second, participants believe CBSA could improve the transparency and
clarity of the Nexus pages (e.g. forms cannot be filled in the GoC environment, redirection to a US government site is
“normal”). The current funnel leaves the
impression for many first-timers that they will apply and complete the process
right there.
Third, many participants felt they had not noticed the top blue menu
items and went to the menus at the bottom of the home page or Nexus page. They felt the blue bar at the top blends into
the CBSA logo making the menu difficult to spot
And finally, those coming to renew Nexus (yet did not know the address
of the US Government site) felt they should have an obvious bottom they could
click on a get re-directed right away.
They struggled to find their way on the CBSA website and find the needed
link. Some said they left CBSA, back to
Google to make a different query.
Qualitative research provides
insight into the opinions of a population or a group, rather than providing a
measure in percent of the opinions held, as would be measured in a quantitative
study. The results of this type of research should be viewed as directional
only.
1.6 Notes on Interpretation of Research
Findings
The views and observations
expressed in this document do not reflect those of Canada Border Services
Agency. This report was compiled by Leger based on the research conducted
specifically for this project.
1.7 Political Neutrality Statement and Contact
Information
I hereby certify as Senior Officer of Leger
that the
deliverables fully comply with the Government of Canada political neutrality
requirements outlined in the Policy on
Communications and Federal Identity and the Directive
on the Management of Communications- Appendix C (Appendix C:
Mandatory Procedures for Public Opinion Research).
Specifically, the deliverables do not include
information on electoral voting intentions, political party preferences,
standings with the electorate, or ratings of the performance of a political
party or its leaders.
Signed:
Christian Bourque
Executive Vice-President and Associate
Leger
507 Place d’Armes, bur 700
Montréal, Québec
H2Y 2W8