PUBLIC OPINION AND UX RESEARCH FOR CANADA BORDER SERVICES AGENCY

Final Report

Prepared for Canada Border Services Agency

Supplier name: Leger marketing inc.

Contract number: 47419-195001/001/CY

Contract value: $122,328.15

Awarded date: 2019-01-14

Delivery date: August 14, 2019

Registration number: POR-106-18

For more information on this report, please contact Canada Border Services Agency at: Erika-Kirsten.Easton@cbsa-asfc.gc.ca

Ce rapport est aussi disponible en français.

Copyright

This report presents the results of a UX study conducted with employees of the Canada Border Services Agency (CBSA) and a user study conducted with visitors to the “Travellers section” of the CBSA's website conducted by Leger Marketing Inc. on behalf of Canada Border Service Agency. The research was conducted between January, 2019 and March, 2019.

Cette publication est aussi disponible en français sous le titre : Recherche sur l’opinion publique et tests utilisateurs pour l'Agence des services frontaliers du Canada.

Catalogue Number:

PS38-98/2019E-PDF

International Standard Book Number (ISBN):
ISBN 978-0-660-30420-5

Related publications (registration number: POR 101-18):

Catalogue Number: PS38-98/2019F-PDF (Final Report, French)
ISBN 978-0-660-30421-2

© Her Majesty the Queen in Right of Canada, as represented by the Canada Border Service Agency, 2019.

Table of Contents

Copyright 2

1. Executive Summary 5

1.1 Background and Objectives 5

1.2 Methodology 6

1.2.1 Intranet Section 6

1.2.1.1 Six Focus Group 6

1.2.1.2 One-on-one interviews 7

1.2.1.3 Persona Creation 8

1.2.1.4 Tree testing 8

1.2.2 Internet Section 9

1.2.2.1 Open-link Survey on the CBSA Website 9

1.2.2.2 Online Focus Group 10

1.3 Overview of the Intranet Study Findings 10

1.4 Overview of the Internet Study Findings 13

1.5 Statement of Limitations 14

1.6 Notes on Interpretation of Research Findings 14

1.7 Political Neutrality Statement and Contact Information 15

2. Detailed Results 16

2.1 Intranet 16

2.1.1 The Focus Groups and One-on-One Interviews Results 16

2.1.2 Card Sorting Exercise 22

2.1.3 Persona Creation 32

2.1.4 Tree Testing 38

2.2 Internet 49

2.2.1 Online Survey 49

2.2.2 Online Focus Groups 81

Appendix A – Detailed Research Methodology 86

A1 Intranet Section 86

A1.2 Six Focus Group 86

A1.3 One-on-one interviews 87

A1.4 Persona Creation 89

A1.5 Tree testing 89

A2 Internet Section 90

A2.1 Open-link Survey on the CBSA Website 90

A2.2 Online Focus Group 92

Appendix B – Moderation Guide – Focus Groups 94

Appendix C – Interview Guide 100

Appendix D – Exercises – Focus Groups 106

Appendix E – Tree Testing Questionnaire 112

Appendix F – Survey Questionnaire 114

Appendix G – Online Focus Groups Guide 134

Appendix H – Detailed Tables 141

1. Executive Summary

Léger is pleased to present to the Canada Border Services Agency (CBSA) this report on the results of qualitative and quantitative studies conducted with CBSA employees and quantitative and qualitative studies conducted with visitors to the Travellers section of the CBSA's website.

This report was prepared by Leger who was contracted by Canada Border Services Agency (contract number 47419-195001/001/CY awarded January 14, 2018).

1.1 Background and Objectives

This public opinion research project is divided into two main sections: a section dedicated to the Agency's internal intranet (accessible only to employees) and a section dedicated to the Agency's external website (accessible to the general public).

Intranet

CBSA’s Intranet (Atlas) is a major communication tool to inform employees (about 14,000) of the day-to-day business of the Agency as well as a resource for policies, guidance and direction. The content on Atlas is structured according to the organization rather than its functions, and therefore does not best meet the needs of the user.

Currently, Atlas content is built upon a division of branches and regions. This structure does not take users’ requirements into consideration and creates repetition or an overlap of content across sections. As such, Atlas has become very difficult to navigate and search and often leads users to the wrong information, unsearchable data or a duplication of information. These factors result in a tremendous loss of time for employees.

The Intranet Public Opinion Research (POR) aspect of this project focuses on CBSA employees with the intent of gathering the information required to assist in rebuilding Atlas. The new site must be organized by audience and user tasks and not according to the organizational model.

Objectives of the Intranet Section:

• Asses satisfaction towards the Intranet;

• Learn more about expectations towards the Intranet;

• Know perceived usefulness of content in the Intranet;

• Qualify content types (topics) as useful, desired, proposed, or mandatory;

• Propose a modified information architecture that better reflects employees’ understanding of the Intranet content.

Internet – Traveller Section of the CBSA website

Visitor data from the Travellers pages of the CBSA website shows that there are approximately 237,000 unique visitors per month. Most of the visitors access the website from Canada but many other visitors come from other countries. People visit the Travellers section to gather information on different subjects (what they can bring across the border, wait times, prohibited materials, etc.), and based on the available website data, it is known that the Nexus pages are among the most popular.

Objectives of the Internet Section:

· Asses visitors’ satisfaction with the traveller’s section of the CBSA websites;

· Identify the reasons for visiting the website

· Assess the clarity of the information on the site

· Identify sources of misunderstanding

· Identify navigation problems

· Evaluate how the Nexus section is used by visitors

· Identify potential areas for improvement of the traveller’s section of the CBSA’s website.

1.2 Methodology

To achieve the study objectives, a research plan based on a hybrid method, qualitative and quantitative, was developed.

1.2.1 Intranet Section

To achieve the objectives set for the Intranet portion of the study, we used a four-step methodology: 1) focus groups, 2) individual interviews, 3) persona creation, and 4) a tree structure test (reverse card sorting)

First, a qualitative methodology consisting of focus groups with the CBSA’s employees was set up. It was followed by a series of one-on-one interviews and card-sorting exercise. These two first phases were the foundation of persona creation. Based on phases 1 and 2, a new information architecture was proposed for Atlas renewal. This new architecture was further tested with employees using a tree testing methodology, also known as reverse card sorting.

1.2.1.1 Six Focus Group

Leger conducted a series of six focus groups with CBSA employees. There were groups of employees recruited from (1) employees in the field, (2) employees from the regional HQs and (3) employees for the national HQ in Ottawa. All sessions were held in CBSA locations. Participants were recruited by CBSA. Each group session lasted approximately 120 minutes. Every session was recorded for analysis purposes. Leger was responsible for preparing the moderation guides and moderating the groups. The guide was developed in consultation with CBSA’s project authority. The groups were moderated in both French and English according to the needs of the employees. The guides and tools were available in both languages.

Participants were informed of all their rights under Canada’s Privacy Act and the Standards for the Conduct of Government of Canada Public Opinion Research. Specifically, their confidentiality was guaranteed, and their participation was voluntary. CBSA was responsible for ensuring the participation and availability of its employees for scheduled interview dates and times. There was no financial incentive to ensure the participation of CBSA employees.

Locations and dates

Groups were held in the following cities on the dates specified.

Table 1. Detailed Recruitment

City	Recruits	Participants	Target	Language	Date
Ottawa	10	8	HQ employees	EN/FR	January 28, 2019
Ottawa	10	7	HQ employees	EN/FR	January 28, 2019
Mississauga	10	9	HQ employees	EN	February 5, 2019
Toronto Pearson Int’ Airport	10	11	BSO	EN	February 5, 2019
Vancouver	10	4	BSO	EN	February 6, 2019
Vancouver	10	11	BSO	EN	February 6, 2019
Total	60	50

1.2.1.2 One-on-one interviews

Leger conducted one-on-one interviews with participants of each main profile (employees in the field, employees from a regional HQ and employees from national HQ). Each interview lasted approximately 45 minutes. Every session was recorded for analysis purposes.

Leger was responsible for preparing the interview guide, preparing the card sorting exercise and conducting the interviews in English and French. The recruitment guide and card sorting exercise were developed in consultation with CBSA’s project authority.

Participants were informed of all their rights under Canada’s Privacy Act and the Standards for the Conduct of Government of Canada Public Opinion Research. Specifically, their confidentiality was guaranteed, and their participation was considered voluntary. CBSA was be responsible for ensuring the participation and availability of its employees for scheduled interview dates and times. There was no financial incentive to ensure the participation of CBSA employees.

Table 2. Detailed Recruitment

City	Recruits	Participants	Target	Language	Date
Montreal	5	5	HQ employees and BSO	EN/FR	February 6 and February 8, 2019
	5	5	HQ employees and BSO	EN	February 12, 2019
Ottawa	5	5	HQ employees	EN	February 15, 2019
Total	15	15

1.2.1.3 Persona Creation

Based on the information collected in phases 1 and 2, we have created as set of six personas. The final number of personas has been dictated by the outcome of data analysis. The main objectives of personas are to efficiently present and share information related to intranet users.

Personas document several dimensions, such as:

• Employee role

• Technological profile

• Daily challenges

• Main frustrations with Atlas

• Short bio to give life to the persona

1.2.1.4 Tree testing

In this phase of the research, we evaluated an information architecture with Atlas users. The architecture that was evaluated was designed using the results of the focus groups and the card sorting done during the individual interviews. To perform this tree structure test, we used the Treejack tool of the OptimalWorkshop platform. CBSA employees who visited Atlas between March 12 and March 22 were invited to participate in the test via an open link. The tree testing consisted of ten tasks that employees were required to perform in the proposed information structure. The test was available in both French and English at the respondent's preference.

A total of 1,164 respondents took the test in English and a total of 200 in French. 434 users were frontline employees and 930 were not.

For each task, we measured several indicators, such as:

· Success rate and failure

· The paths

· The final destination

The detailed methodology is presented in Appendix.

1.2.2 Internet Section

Quantitative Methodology

In order to meet the Internet-related objectives, a methodology in two phases was followed. First, a quantitative methodology consisting of an open-link survey on CBSA’s website was set up. It was followed by a series of online focus groups with CBSA’s website visitors.

1.2.2.1 Open-link Survey on the CBSA Website

This quantitative research was conducted through an online survey, using a Computer Aided Web Interviewing (CAWI) technology. The public consultation was launched by means of an open-link survey questionnaire available on the CBSA website. Any individual who visited the traveller’s section of the site between February 4, 2019 and March 4, 2019 was invited to answer the questionnaire by clicking on the link.

This part of the public consultation generated a significant volume of responses. A total of 2,729 respondents were gathered via the open-link. We should remind the reader that the results of this part of the public consultation should not be interpreted as representing the opinions or attitudes of the Canadian public at large nor representative of the visitors of CBSA’s website. It is a collection of respondents who volunteered to answer the questionnaire. No statistical weighting was performed on this sample.

Since this is a sample of volunteers, no margin of error can be calculated for this portion of the study. Nor can we comment on the participation rate, as we do not know the traffic and the exact volume of visitors to the Agency's website during the period when the open link was active on the web page.

The online survey has given us the opportunity to recruit participants to conduct focus groups with users of the Agency's website. We therefore asked all survey participants if they were open to participate in a second phase of the study. Those who agreed were invited to leave their names and contact information so that we could contact them to complete the recruitment process.

Leger adheres to the most stringent guidelines for quantitative research. The survey instrument was compliant with the Standards of Conduct of Government of Canada Public Opinion Research – Series E – Qualitative and Quantitative Research. The questionnaire was developed by Léger in collaboration with the CBSA research project leaders.

The details of the methodology and more information on Leger’s quality control mechanisms are presented in Appendix.

The survey questionnaire is available in Appendix.

1.2.2.2 Online Focus Group

Leger conducted a series of three (3) online focus groups with visitors of the CBSA Website. Their were all recruited from the online survey from the previous research phase of the study. All three sessions were held online via the ITracks video chat platform with participants from different regions of Canada or the US. The following table is a summary of the locations, date, profile and number of participants for all the discussion groups.

GROUP	Group profile	Language	Recruited	Participants	Dates and	Time (Eastern time)	Type
GR01	CBSA’s website visitors	FR	10	8	March 7, 2019	4:30 PM	Online
GR02	CBSA’s website visitors	EN	10	8	March 7, 2019	6:00 PM	Online
GR03	CBSA’s website visitors	EN	10	7	March 7, 2019	8:00 PM	Online

Participants were informed of all their rights under Canada’s Privacy Act and the Standards for the Conduct of Government of Canada Public Opinion Research. Specifically, their confidentiality was guaranteed, and their participation was voluntary. Léger was responsible for the recruitment of the participants and of the moderation of the online focus groups.

Léger's professional recruiters ensured the availability and participation of recruits. Léger was responsible for organizing the sessions on Itracks' video chat platform. A financial incentive of $100 per participant was given to all group participants to thank them for taking the time to participate.

1.3 Overview of the Intranet Study Findings

Given that ‘Atlas’ has created some frustration and that many employees have written it off as an effective work tool, we believe that the Intranet needs a full re-launch both to signal a significant change in direction and usability and a clear indication that the new tool was in large part “designed” by users. That full re-launch would require a new name. If brand equity in Atlas is low, change the brand. We suggest a name along the line of “MyBorder” to give it a relevant name and a clear indication that it is ‘their’ work tool, ‘their’ home.

In order to achieve this goal, we strongly recommend that the Atlas modernization team create a small number of working groups to co-design the new architecture. This will drive support for the initiative and improve usability. We readily see the need for a communication group (including the regions), a HQ group (finance, procurement and HR functions in particular), a BSO supervisor group (with representation for the regions).

Building engagement should be at the core of the new design. All employees expressed the need for a common “News Headlines” section that would represent what the Agency does and how it serves its mission. While not limited to frontline activities, the core of that section should be about protecting and serving Canadians at the border. Frontline staff want to hear more about themselves and HQ employees want to hear about the front line. Initiatives like Border Update (if captioned) can serve this purpose, but regional staff expressed the need to hear about news from all regions and share initiatives, successes, stories and kudos in a format that is more “bottom-up” rather than “top-down”.

Atlas should be first conceived as a “work tool”. It is there to support employees in carrying out their duties, tasks and responsibilities. What leads employees to Atlas today is task-based, not “let’s see what’s new or going on”. Users will look for “softer” information only if they believe Atlas will allow them to get their job done.

The new design should aim for oversimplification. A design with fewer menus versus more levels should be considered. This will be tested in the tree testing validation stage.

The wiki in its current form may be harmful for the Agency. Participants in the research who use their regional wiki only go there for a very limited number of pages, while they have strong doubts about the quality, accuracy and validity of the information found in the wiki. The working groups should determine what are the key functions served by the wiki now and integrate this into, potentially, a collaboration zone within Atlas.

Moving towards Apollo as the document repository should continue. If some still resist the change because of a login process that is seen as not optimal and despite slowdowns and downtime, many employees feel a ‘leaner’ Atlas efficiently directing users to the relevant Apollo documents can work.

The proposed information architecture for Atlas could be as follows:

Homepage

Employee

Awards and Recognition

Employee Assistance Program

Employee Orientation / Onboarding / New Hires

Employment Equity and Diversity

Integrity, Values and Ethics

Jobs and Career Development

Occupational Health and Safety

Pay and Benefits

Training and Learning

Well-being

Security

Tools

Information Management

Apollo

ATIP

Forms and Template Library

Immigration

Commerce

Traveller

Enforcement of the Law

Other Themes

Frontline Bulletins and National Document Centre Publications

Immigration

Commerce

Traveller

Enforcement of the Law

Other Themes

Guides / Manuals and Standard Operating Procedures

Immigration

Commerce

Traveller

Enforcement of the Law

Other Themes

Policy Library

Immigration

Commerce

Traveller

Enforcement of the Law

Other Themes

Finance Volume

Procurement and Contracting

Accommodations and Facilities

The Organization / About Us

Agency Organization Chart / About Branches, Directorates and Divisions

Executive Offices

Human Resources

Labour Relations

Performance Management

Staffing

IT Portal

Helpdesk

IT Portal

ACROSS

ICES

CAS

ESS

Daily News

Agency and Branch Initiatives Priorities

Border Updates (Video Series)

CBSA Gives / GCWCC (Charitable Campaign

Event Calendar

Messages from Executives

News and Photo Galleries

There are some important lessons learned in developing the final information architecture for the renewal of Atlas:

1. The “Frontline Bulletins and National Document Centre Publications” should be classified under the main menu "Tools" and not under the menu "Daily News".
2. Some items, as currently identified in Atlas, should be renamed to clarify their content.

a. The "Employment and Professional Development" tab can easily be confused with the "Training and Learning" tab.

b. The tabs "Agency Organization Chart / About Branches, Directorates and Divisions" and "Executive Offices" do not adequately allow users to know what content they will find under either one.

c. The information found under the "Human Resources" tab should be reformulated. For the moment, some of the information found under "Human Resources" is also searched under the "Employee" menu. Work should be done at this level to clarify and reorganize that information.

d. Identify more clearly where to find the content of the weekly video series in the "Daily News" tab to make it easier for users to find it.

3. The results of the tree test indicate that the main menu created by employees during card sorting works properly to guide users in their task on the intranet.
4. The accuracy of libraries of forms, guides, manuals, manuals, policies and newsletters on different topics, such as immigration, commerce, travel, etc., helps users to guide and refine their search for information. In a real work environment, we believe this approach is promising.
5. The tree test does not pretend to have evaluated all categories and items of the submenus. The effort to clarify category wordings must be systematically carried out by the CBSA for every item.

1.4 Overview of the Internet Study Findings

When it comes to the Internet research with travellers, both the quantitative and qualitative research phases confirmed that visitors tend to be satisfied. Indeed, they “got what they were looking for”. Beyond the usual critique concerning the look & feel of government-type websites, and the feeling that pages were “too wordy” or “too busy”, most participants were able to get the answers they needed. Some processes were more laborious, however. It is namely the case of those seeking to fill out a Nexus application for the first time. For them, the process is not seamless and fully transparent and could be reviewed. The language of the menus makes them sound like “action items” or “transactional items” while they are not and many were surprised to the taken out of the GoC environment later in the process.

Other changes were suggested by participants. First, menus should be changed from the current language to simple questions (i.e. How to apply for Nexus? What can I bring back to Canada?). Participants feel they access the website to answer a question they have and that if the website mirrored these questions, it would provide for a smoother navigation. They feel that the current language can be ambiguous or make some of the menu items not mutually exclusive.

Second, participants believe CBSA could improve the transparency and clarity of the Nexus pages (e.g. forms cannot be filled in the GoC environment, redirection to a US government site is “normal”). The current funnel leaves the impression for many first-timers that they will apply and complete the process right there.

Third, many participants felt they had not noticed the top blue menu items and went to the menus at the bottom of the home page or Nexus page. They felt the blue bar at the top blends into the CBSA logo making the menu difficult to spot

And finally, those coming to renew Nexus (yet did not know the address of the US Government site) felt they should have an obvious bottom they could click on a get re-directed right away. They struggled to find their way on the CBSA website and find the needed link. Some said they left CBSA, back to Google to make a different query.

1.5 Statement of Limitations

Qualitative research provides insight into the opinions of a population or a group, rather than providing a measure in percent of the opinions held, as would be measured in a quantitative study. The results of this type of research should be viewed as directional only.

1.6 Notes on Interpretation of Research Findings

The views and observations expressed in this document do not reflect those of Canada Border Services Agency. This report was compiled by Leger based on the research conducted specifically for this project.

1.7 Political Neutrality Statement and Contact Information

I hereby certify as Senior Officer of Leger that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Policy on Communications and Federal Identity and the Directive on the Management of Communications- Appendix C (Appendix C: Mandatory Procedures for Public Opinion Research).

Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leaders.

Signed: 

Christian Bourque

Executive Vice-President and Associate

Leger

507 Place d’Armes, bur 700

Montréal, Québec

H2Y 2W8

cbourque@leger360.com

2. Detailed Results

2.1 Intranet

2.1.1 The Focus Groups and One-on-One Interviews Results

The results of the focus groups and individual interviews are presented in the section below.

2.1.1.1 Motivation to Visit Atlas

Besides the fact that Atlas remains the home page of employees when logging in at the start of their work day, very few use this moment as a reason to visit. One key finding remains the fact that employees’ main motivation to visit Atlas is to perform a work-related task, which they hope to do efficiently and in a very limited time window. Visiting Atlas is undertaken with the objective of performing a specific action, not with the general objective of “finding out what’s going on”, “seeing what’s new” or “browsing to see what has changed” on Atlas. It is not a “social” page, it is a work tool.

While employees wish that Atlas helps foster a sense of belonging and promote employee engagement by bringing regions, branches and all levels of staff together, this should be the outcome of first making Atlas an efficient work tool designed to improve staff’s capacity to do their work better. The efficiency of finding what one needs quickly and with a high degree of confidence about the accuracy of the information will “drive traffic” to Atlas, not its social function. The latter function will depend of Atlas’ ease-of-use.

2.1.1.2 Satisfaction with Atlas Is Generally Low

Overall satisfaction with Atlas is fairly low but will vary across a number of factors. First, satisfaction with Atlas is highest among headquarter staff and at its lowest with BSOs. Some BSOs interviewed rarely visit Atlas, one not even remembering going to Atlas except to access their Self-Service Portal. In general, employees who use Atlas the least often are the ones who are more dissatisfied. Their dissatisfaction with the intranet is the reason why they rarely use Atlas. Among the Frontline staff, a lot of effort as gone in to finding ways to work outside of Atlas to improve work performance. Common drives (G drives) become the repository of forms, manuals and D memos, operational bulletins, with little or no external control over the accuracy of the documents on these drives. The wiki also serves a similar function of enabling staff to recover documents WITHOUT having to search Atlas. Some still archive emails in Outlook if they contained operational bulletins that they believe they would need to consult later.

Frontline employees also use Atlas in a unique context, most often facing their screen and a client at the same time or in a context where colleagues are dealing with clients in the next booth or with clients waiting in the room. Their capacity to quickly find what they are looking for is crucial. This work environment also explains why they rarely “browse” Atlas, as they are not given any time during their shift to do so and why they miss out on key employee engagement initiatives like Border Update … because watching with the sound on is not an option.

Non-frontline employees who use Atlas daily are the most satisfied with the tool. However, their overall level of satisfaction is still generally low. For them the poor quality of the search engine, too many “Archived” pages, slowdowns with Apollo and other factors explain the reason why Atlas rarely scores over 7 out of 10.

2.1.1.3 Key Perceived Weaknesses

Among the features of Atlas that employees had to rate during the qualitative stage, “ease of finding the information they are looking for”, followed by “ease of navigation” garnered the lowest satisfaction scores. Information found on Atlas is generally perceived as clear … once employees get there. Reasons why information retrieval is often perceived by employees as “hit-or-miss” in the current version of Atlas will be outlined later. While employees tended to be negative about Atlas in general, they do fundamentally believe that an Intranet tool is needed and could be extremely useful.

Another element where Atlas falls short in employees’ minds is fostering a sense of belonging to the Agency and representing them as employees. While they believe Atlas has improved recently in this area, they wish Atlas could do more. Frontline staff and BSOs do not believe Atlas represent them and what they do. They see Atlas as being too “HQ-driven”, “too corporate” or a “top-down” tool. They would like to know more about what is going on in the other regions, news about accomplishments, innovation, tricks-of-the-trade they could use locally, and many others. They want Atlas to clearly convey what they believe to be the mission of the agency: to serve and protect Canadians. While HQ did not get this same sense of not being represented on Atlas, they believe that sense of belonging would be heightened if they learned more and saw more about what the frontline is doing across regions. They seemed to wish for a very similar tone and feel to what the frontline staff were seeking.

2.1.1.4 Navigation and Search Time

The difficulty of navigating and finding the right information in Atlas is a concern for all employees, but particularly for BSOs. Information on procedures, rules and policies is often perceived as fragmented in several places and employees often struggle with the logic of where a piece of information “should be.” They often referred to information on the same subject not grouped together, while they do not have a clear understanding as to why this occurs.

Employees feel they lose too much time searching for information on Atlas which leads them to want to “cheat” and save versions on local drives, which they may not update appropriately across time. Some BSOs even consider that Atlas creates a security issue when they struggle to find the appropriate information or form with an agitated client in front of them. As one put it: “It is hard to always have an eye on a client’s eyes and hands, while looking for an immigration policy document for several minutes.”

Many employees, again mostly frontline, say they have “given-up” on Atlas and will ask a colleague or search Google, before they will consider Atlas.

2.1.1.5 Duplication of Information and Information Dissemination

Dissemination of information also appears problematic. Many believe that they will get the “important” information they need via the daily email and will not visit Atlas to seek that same information. Sometimes their supervisor’s morning email will copy information found in the daily email. This leads to confusion when it comes to where this information in the email actually found or stored and how they can retrieve it later.

After reading the email, most will delete the email but may not be able to quickly and effectively go back to the key elements of content found in the email. Some archive the emails but may struggle to find the right one if they are confronted with a situation that requires searching “old” emails. Many employees feel that Atlas should be used to disseminate information more than emails, which are seen as too long and not really fun to look at or read.

Emails should be short and quick with references (hyperlinks) to corresponding Atlas pages.

2.1.1.6 Home Page Clutter and Need to Scroll Down

Many comments point in the direction that there is an overabundance of information on the Atlas homepage and/or that the same information can be found in a variety of places. Most employees struggle with understanding the structure of the page or the logic as to why something would be found here and/or there.

Given the size of the home page and the very high number of clickable items on home page, many employees rarely scroll down the home page. Most research participants, when confronted with a static version of the home page, said that they had “never seen” some of the menus before the research (while actually seeing them on screen on a daily basis). The tabs/headings that appear at the bottom of the Atlas home page are rarely used by employees.

2.1.1.7 The Search Engine on Atlas

The Search Engine on Atlas is key reason for low satisfaction with Atlas overall and for employees avoiding Atlas to find information. Many times they believe the current search engine will give a list of “more or less relevant” links where ‘”What you are actually looking for will not be in the first 100 results shown.”

Being an older generation search engine, the exact word search limitation is a key element of frustration for employees. Prior to searching in Atlas, most will search elsewhere for the exact words to be used before searching on Atlas. If the employee believes that the document, template or form may be referenced on a pubic website, they will first use Google to find the right name for the form and then copy and paste the appropriate acronym, form number or name in the Atlas search engine to find the latest/correct version of the document.

Employees also believe the search tool should allow the use of advanced search functionality.

2.1.1.8 The Homepage Menus

The menus on Atlas use a lexicon that is not clear to many employees. For example, what is under “Headlines” is not what they expect (“news”) and employees do not know how that would be different from “CBSA News” further down.

It is difficult to know what each tab means when reading the words. Some menus, such as the blue menu bar in the middle of the page, go completely unnoticed or remain largely unused. Furthermore, that blue menu bar contains icons that some employees would not guess were clickable.

2.1.1.9 Archived Pages

When asked directly, almost all employees have ended up on pages with the yellow “Archived” banner. Except for a couple of employees (who were also content creators on Atlas), CBSA employees interpret the Archived banner as a warning that the content is either incorrect, no longer valid, inaccurate, and that it should not be used by employees. The main reason why the Archived banner creates frustration at the moment is because employees believe they were led into a dead end as the information found is no longer usable.

Employees tended to believe that the information they were seeking must have been replaced by other, more up-to-date version “somewhere else” on Atlas.

2.1.1.10 The Org Chart and the Directory

Employees commented on the difficulty they have in finding someone specific within the agency. They tend to use Outlook to serve the purposes of a directory, even if Outlook is limited in this capacity (you need to know who you are looking for by name).

The information in the employee directory is not up to date. While employees fully understand the challenges involved in having an up-to-date org chart, they would at least have the capacity to leave a demand for contact in general mailboxes that are identified by function.

Some employees also commented that the information available in the directory does not allow us to know exactly what the individual’s role and responsibilities are.

2.1.1.11 The Preferences of Frontline Employees Differ from those of Management/Admin Employees

Frontline employees want to see Frontline bulletins, regional news and news from other regions, updates on region and agency initiatives, Border Update (but captioned since they can rarely use sound) and to have access to the most requested tasks from Frontline staff.

Other employees want to see Updates on agency initiatives, frontline bulletins, upcoming events, messages from executives and to have access to the most requested tasks. The homepage had to be adjusted according to the role in the organization: 1) Frontline or 2) Other employee. While both “portals” would have shared content, their unique menu structure would convey a strong message of customization.

2.1.1.12 Atlas' interface and look and feel

Although the look and feel of the interface is not the major irritant, several employees mentioned that this aspect could be improved with the modernization of Atlas. They believe that compared to other GoC Intranets, Atlas is doing “OK”, but that it is not up to par with “private sector” interactive websites they deal with in their personal lives.

Several employees stressed the value of having the ability to customize Atlas according to their preferences and needs. Giving employees the opportunity to organize the information on their home page would be well received by employees. Using a widget system of at least a “favourites” page would be a welcomed change.

The use of icons would also be well received. One criticism of Atlas is that there are too many words, that it's too wordy on the home page. The use of clickable icons, such as those found on smart phones, would be an ideal alternative to modernize the intranet and do away with the clutter.

Many wish Atlas would move to a structure that has fewer menu options, even if it means more levels. Just a few buttons on one page would be preferred.

2.1.1.13 The Forms, Guides and Manuals

The current organization of forms in alphabetical order and not by theme or topic does not correspond to the way employees search for information. Frontline staff would prefer that the organisation be first by type (i.e. immigration, commercial, traveler), followed by type of port (i.e. highway, marine, airport).

Key information on the same topic/theme should be grouped together. Currently, information on the same topic/theme is disseminated in different places in Atlas. It is extremely difficult for Frontline staff—more so than HQ staff—to easily find all of the information and tools with the way classification is currently done.

2.1.1.14 Current Main Menu Structure

The current top menu is often used by employees. First, many believe it is not clear what one would find under the headings (what is under branch versus region or under employee versus frontline staff). Except for the “Managers” tab which was clear that it would include management tools and information, the other headings created confusion.

Regardless of the region, employees do not access their regional page through the Atlas menu. As such, many employees do not see their regional news, unless it is sent to them via an email from their field supervisor. There is reason to believe that some employee might have set the regional homepages as the browser homepage. In fact, Frontline staff demanded to see more news from their region as well as other regions.

2.1.1.15 No Expressed Need for Mobility

Employees do not see the need for a mobile app for Atlas. BSOs cannot use their mobile while on duty and government issued Blackberry smartphones would not make a mobile version of Atlas appealing or useful. Furthermore, since employees use Atlas to retrieve information or as a repository, documents sought need to be printed, sorted or filled out, which is not a function that employees believe their current mobile phones would allow.

2.1.1.16 Wiki

Popularity of the wiki remains fairly low. Those who use it in the regions tend to go to a few limited pages which they trust. However, these same employees have strong doubts about the accuracy and validity of the information found on the wiki and would rather trust an “official” source such as Atlas… if they felt they would easily find what they are looking for.

Only a few examples of “good” information not found on Atlas but only on the wiki were provided. This was a quick reference guide built in the PAC region which allows a quick search by make and model of “hiding spots” on cars coming through highway customs. Since no equivalent exists on Atlas, then the wiki becomes the source.

Most employees believe the wiki is a “jungle” or “chaos” or a “free-for-all” they tend not to trust.

2.1.1.17 Apollo

Opinions regarding Apollo are rather divided. Those who have some experience with it tend to appreciate what it does and how it allows the content to remain up-to-date. While some complaints about speed and downtime, Apollo always makes up for it in the ability to update and keep information accurate.

The double login aspect is somewhat of an annoyance, but employees generally agree that Atlas should be the “hub” where they are kept abreast of changes and priorities at CBSA but that the actual document repository could be Apollo (mainly for pdf versions).

2.1.2 Card Sorting Exercise

A card sorting exercise was carried out with Atlas users to draw the outlines of a new information architecture as part of the Atlas modernization effort. The card sorting method is a necessity when it comes to building an information architecture that is centered on the experience of platform users. In short, card sorting consists of asking users to group information on a set of cards and to give a name to each of these groups. It should be noted that the information appearing on cards consists of information currently existing on Atlas.

The card sorting technique was carried out in two steps: an open card sorting and a closed card sorting.

2.1.2.1 Open Card Sorting and Closed Card Sorting

First, participants were asked to create general categories (or headings) for the Atlas main menu. The objective of this open phase of card sorting was to create large categories without classification constraints. This step provided a deeper understanding of how employees conceptualize Atlas information based on their use of the intranet. As this portion of the exercise was completely open, it allowed us to capture the exact terms/words used by participants to create the Atlas main menu. This information will allow us to draw meaningful conclusions about the words that should be used in the categories of Atlas renewal.

Secondly, participants were asked to classify the cards (with categories of information that currently exist in Atlas) under the headings they created in the open card sorting exercise. This exercise, which could be described as closed card sorting, is particularly suitable for redesigning an existing information architecture, whose information structure cannot be completely modified or disrupted. During this portion of the interview, participants were asked to classify 32 items into the menu headings they had created during the open card sorting. This exercise was done using cardboard cards, no software was used in this portion of the interview.

Below is the content of the 32 cards used for this exercise:

1. Pay and Benefits
2. Awards and Recognition
3. Occupational Health and Safety
4. Security
5. Performance Management
6. CBSA Gives / GCWCC (Government of Canada Workplace Charitable Campaign)
7. Well-being
8. Finance Volume
9. Procurement and Contracting
10. Staffing
11. Messages from Executives
12. Frontline Bulletins and National Document Centre publications
13. Agency Org Chart / About Branches, Directorates, Divisions
14. Labour Relations
15. Training and Learning
16. Integrity, Values and Ethics
17. News and Photo Galleries
18. Event Calendar
19. Employee Orientation / Onboarding / New Hires
20. Executive Offices
21. Employment Equity and Diversity
22. Form and Template Library
23. Employee Assistance Program
24. Agency and Branch Initiatives and Priorities
25. Accommodations and Facilities (Evacuation Procedures, Boardrooms)
26. Information Management (Apollo, ATIP)
27. Jobs and Career Development
28. Helpdesks
29. Policy Library
30. Border Update (Video Series)
31. Guide, Manual, and Standard Operating Procedure Library
32. IT Systems (ACROSS, ICES, CAS, ESS)

During the card sorting exercise, the interviewers gave no guidance to the participants on how to classify the cards. If necessary, after a first attempt to classify all the cards, the interviewer could suggest to the participants to create new headings or modify those they had created in order to classify all 32 cards. As a result, menu categories could be created, deleted or redefined according to the needs of the participants. This process was repeated until all cards were sorted/classified in the menu categories.

2.1.2.2 The Steps of Card Sorting Analysis

The analysis of the card sorting results is performed in two steps. First, the open card sorting is analyzed by qualitatively grouping the categories. In fact, it is not uncommon for participants in an open card sorting process to use a different vocabulary to name the headings but, in the end, they have the same meaning. The objective of this qualitative grouping is to assign a generic name to each of these sets, in order to create the basis for the Atlas main information architecture menu.

In a second step, a table or histogram is created to determine, for each card, the classification that has been most frequent. This is to determine the percentage of classification that is common between categories (for example, card 1 has been classified 60% of the time in category A, 20% in category B and 20% in category C). The analysis of the results of this classification table makes it possible to design the information architecture of the intranet.

2.1.2.3 The Analysis

Qualitative grouping of categories

Interview participants were asked to create a main menu for Atlas. There was no limit on the number of headings imposed on participants. They were free to create as many headings as they wanted according to their need and use of Atlas. We therefore observed participants who created menus as undeveloped as three headings and others that were very elaborate with more than six headings.

Overall, a total of 38 categories were created by participants:

1. About Us
2. Branch and Region
3. Compensation
4. Corporate
5. Current Events
6. Employee
7. ESS
8. Events
9. Forms
10. Forms and Templates
11. Frequent Tasks
12. Frontline
13. GCMS
14. Guides
15. Guides / Forms and Manuals
16. Human Resources
17. HR
18. Information
19. IT
20. IT Portal
21. Manager
22. News
23. Daily News
24. Organization
25. The Org.
26. Org.
27. Priorities
28. Procurement
29. Programs
30. Programs and Initiatives
31. Reference Guides
32. Security
33. Service
34. Service Portal
35. Self-Service Portal
36. Staff
37. Tools
38. Other

It should be noted that several of these categories/items were named spontaneously by several participants, which gives confidence in their usefulness for the future Atlas information architecture. For example, the words "tools", "employee", "news" and "IT Portal" are terms that have been chosen by many participants to classify information. This volume of responses leads us to believe that these terms carry meaning for a large proportion of employees and should be used in the next information architecture in the upcoming version of Atlas. These words also have the potential to allow the classification of other elements; this is not the case for all the headings that were mentioned by participants.

In fact, some of the words used by participants to make their main menu are too similar to the content of the category itself, leaving little room for other elements to be classified. This is the case for the words "security", "guides", "procurement", and "compensation". However, these words were mentioned by a minority of participants. These topics therefore offer little benefit to be used in the intranet information architecture as part of the main menu.

Using a qualitative content analysis, we grouped all the categories created by the participants into 7 main themes. Headings that had a lower classification potential have been grouped into more general categories that provide an increased opportunity to classify a large number of cards. Thus, the terms "frontline" and "manager" have been grouped under the more general category "employee". Following the same logic, the terms "forms", "templates", "library" have been grouped together in the more general category "tools".

The results of the qualitative grouping of categories are as follows.

Grouped categories	Headings created by participants
Employee	Compensation Employee Frontline Manager Priorities Procurement Programs Programs and Initiatives Security Service Staff
Tools	Forms Forms and Templates Frequent Tasks Guides Guides / Forms and Manuals Library Reference Guides Tools
The Organization / About us	About Us Branch and Region Organization The Org. Org.
Human Resources	Human Resources HR
It Portal	Information IT IT Portal Service Portal Self-Service Portal
Daily News	Current Events Daily News Events Programs and Initiatives News
Other	Corporate ESS GCMS Info Procurement and contracting Other

Once the categories were grouped, we retained seven broad headings that offer the greatest potential for classifying information on the intranet: 1) Employee, 2) Tools, 3) The Organization / About us, 4) Human Resources, 5) IT Portal 6) Daily News and 7) Other.

Card sorting analysis

The next step is to determine how the 32 cards were classified under the seven headings created in the previous step. The following table shows the distribution of the optimal classification of the 32 cards under the six headings that have been developed in the open card sorting. The results will allow us to define the optimal classification of the cards in the menu.

	Employee	Tools	The organization / About Us	Human Resources	IT Portal	Daily News	Other
Security	43%	29%					29%
Procurement and Contracting	57%						43%
Guide / Manual, and Standard Operating Procedures	30%	60%					10%
Employee Assistance Program	62%	15%		23%
Helpdesk	25%	38%			38%
Agency Org Chart / About Branches, Directorates, Divisions	25%	13%	63%				13%
Executive Offices	22%	22%	44%				12%
Finance Volume	29%	29%					42%
Integrity, Values and Ethics	44%	12%	22%	22%
Occupational Health and Safety	33%	25%		25%			17%
Well-being	50%	10%	10%	30%
Staffing	33%	11%		44%			12%
Pay and Benefits	64%	11%		25%
Form and Template Library		90%				10%
Accommodations and Facilities (Evacuation Procedures, Boardrooms)		50%					50%
Policy Library		80%				10%	10%
IT Systems (ACROSS, ICES, CAS, ESS)		29%			58%		13%
Labour Relations	33%	22%		45%
Training and Learning	64%			27%		9%
Employee Orientation / Onboarding / New Hires	50%	17%		33%
Jobs and Career Development	54%			23%		23%
Performance Management	33%	25%		33%			9%
Awards and Recognition	55%	11%				11%	23%
Information Management (Apollo, ATIP)	25%	38%			12%		25%
Employment Equity and Diversity	50%	10%	10%	30%
CBSA Gives/GCWCC (Charitable Campaign)	43%					43%	14%
Agency and Branch Initiatives and Priorities	23%		15%			46%	16%
Border Update (Video Series)	14%					86%
Event Calendar	12%					88%
Frontline Bulletins and National Document Centre publications	45%					55%
News and Photo Galleries	12%					88%
Messages from Executives	14%		29%			43%	14%

The analysis of the classification table is very simple. The purpose is to identify under which heading the participants most regularly classified each of the cards. This operation makes it possible to design Atlas' information architecture based on the experience and logic of intranet users. In cases where a map may be classified under two headings with identical results, a decision based on a similarity analysis between maps is made. The similarity test makes it possible, as its name suggests, to determine at which level different cards have been classified in a similar way by the participants. Thus, determining the similarity between cards can help resolve less clear-cut cases.

Here are the results of the classification table. We present the cases that require a similarity test following this list:

• Security: Employee
• Procurement and Contracting: Employee
• Guide / Manual, and Standard Operating Procedures: Tools
• Employee Assistance Program: Employee
• Agency Org Chart / About Branches, Directorates, Divisions: The organization / About us
• Executive Offices: The Organization / About Us
• Integrity, Values and Ethics: Employee
• Occupational Health and Safety: Employee
• Well-being: Employee
• Staffing: Human Resources
• Pay and Benefits: Employee
• Form and Template Library: Tools
• Policy Library: Tools
• IT Systems (ACROSS, ICES, CAS, ESS): IT Portal
• Labour Relations: Human Resources
• Training and Learning: Employee
• Employee Orientation / Onboarding / New Hires: Employee
• Jobs and Career Development: Employee
• Awards and Recognition: Employee
• Information Management (Apollo, ATIP): Tools
• Employment Equity and Diversity: Employee
• Agency and Branch Initiatives and Priorities: Daily News
• Border Update (Video Series): Daily News
• Event Calendar: Daily News
• Frontline Bulletins and National Document Centre Publications: Daily News
• News and Photo Galleries: Daily News
• Messages from Executives: Daily News

Four cards are not clearly categorize-able by Atlas users. There is no consensus or a clear majority that allows us to put them under one heading. The Helpdesk card has been classified under the heading "Tools" and under "IT Service" in the same proportion. The "Accommodations and Facilities" card has been classified under the headings "Tools" and under "Other" in equal proportions. Similarly, the "Performance Management" card has been classified under "Employee" and under "Human Resources". Finally, the "CBSA Gives/GCWCC" card was classified under both "Employee" and "News of the Day". For its part, the "Finance Volume" card is the only card that has been clearly classified under the heading "Other".

We therefore conducted a similarity analysis for these cards. When analyzing the similarity for the helpdesk card, the one it is most similar to is IT Systems (ACROSS, ICES, CAS, ESS). Their similarity score is 47% while the similarity of "Helpdesk" with other cards does not exceed 40%. We will therefore classify "Helpdesk" with the "IT Systems" card under "IT Portal".

Since there are only two cards that would be classified under the "Other" heading, namely "Accommodations and Facilities (Evacuation Procedures, Boardrooms)" and "Finance Volume", we recommend that the "Accommodations and Facilities (Evacuation Procedures, Boardrooms)" card be classified under "Tools" and that the "Finance Volume" card be classified according to its similarity with the other cards. This has the advantage of eliminating the "Other" heading from the main menu. Analysis of the similarity of the "Finance Volume" card with the other cards shows that this card is the closest to the "Accommodations and Facilities (Evacuation Procedures, Boardrooms)" card. We will therefore classify the two items under the heading "Tools".

The analysis of the similarity results of the "Performance Management" card indicates that the content of this card is like the "Staffing", "Labour Relations" and "Training and Learning" cards. Two of the three cards to which "Performance Management" is similar to are classified under the heading "Human Resources" while the third is classified under "Employee". We suggest classifying "Performance Management" under "Human Resources".

The similarity analysis of the "CBSA Gives / GCWCC (Charitable Campaign)" card does not allow a clear conclusion to be drawn as to where to place this card. In fact, the similarity results are very low. It may be possible to arrange it under almost all headings. We have decided to place it under "News of the Day". The tree test (phase 4) will determine whether the selected heading is appropriate.

Several participants also mentioned that they would like to see the information in some categories of the Atlas subdivided into thematic categories such as, immigration, commerce, travellers, law enforcement and other themes. This request also echoes what we heard in the focus groups. Given the lack of organization of information in Atlas and the difficulty employees have in finding the information they are looking for, we believe that a classification of themes could help in finding information. The categories of information entitled "procedures", "forms", "policies" and "bulletins" have been subdivided according to these themes in our tree proposal.

2.1.2.4 Proposal of an information architecture

The information architecture we end up with following the card sorting exercise is as follows:

Homepage

Employee

Awards and Recognition

Employee Assistance Program

Employee Orientation / Onboarding / New Hires

Employment Equity and Diversity

Integrity, Values and Ethics

Jobs and Career Development

Occupational Health and Safety

Pay and Benefits

Training and Learning

Well-being

Security

Tools

Information Management

Apollo

ATIP

Forms and Template Library

Immigration

Commerce

Traveller

Enforcement of the Law

Other Themes

Guides / Manuals and Standard Operating Procedures

Immigration

Commerce

Traveller

Enforcement of the Law

Other Themes

Policy Library

Immigration

Commerce

Traveller

Enforcement of the Law

Other Themes

Finance Volume

Procurement and Contracting

Accommodations and Facilities

The Organization / About Us

Agency Organization Chart / About Branches, Directorates and Divisions

Executive Offices

Human Resources

Labour Relations

Performance Management

Staffing

IT Portal

Helpdesk

IT Portal

ACROSS

ICES

CAS

ESS

Daily News

Agency and Branch Initiatives Priorities

Border Updates (Video Series)

CBSA Gives / GCWCC (Charitable Campaign

Event Calendar

Frontline Bulletins and National Document Centre Publications

Immigration

Commerce

Traveller

Enforcement of the Law

Other Themes

Messages from Executives

News and Photo Galleries

This information architecture is evaluated as part of a tree test (see phase 4). The test of the tree structure makes it possible to evaluate the performance of this information architecture with real users of the intranet.

The participants who will evaluate this architecture will have to perform ten tasks (find specific information). They will have to navigate through the architecture to identify the location (the category of information under which they believe they can find the requested information).

2.1.3 Persona Creation

In the focus groups and in the one-on-one interviews, digital skills were broadly defined as the ability to successfully use available technologies in everyday life, both at work and in their private lives. In general, CBSA employees are quite confident in their current level of digital skills, but there are significant variations between employee profiles. This confidence is crucial in the context of Atlas renewal, as it affects the individual's willingness to integrate new digital tools in their routine or to cope with future changes to digital technologies used in their work. Those who are less confident, or feel less competent, are less inclined to adopt digital tools and will depend more on the support of their colleagues or more formal support.

Roles and responsibilities vary considerably from one employee to another and therefore their use of technology is also different. The most common task that all employees must do on Atlas is search for information, whether it is newsletters, policies, procedures, contacts, etc. But it is also the search for information that makes Atlas a thorn in the side of most Atlas users.

A cursory analysis of the results of the focus groups and individual interviews reveals fundamental differences between the two main types of CBSA employees: front-line and other employees.

Employees working in regional offices generally consider themselves more competent and skilled with digital technologies than their front-line colleagues. They are also more likely to enjoy working with digital tools than front-line employees. They are also slightly more frequent users of the CBSA external website than front-line employees.

These two employee profiles largely agree on their dissatisfaction with the digital tools made available to them by the Government of Canada. They generally find them more complicated to use than the digital tools they use in their private lives. For the most part, with a few exceptions, they also agree on their dissatisfaction with Atlas. Both headquarters and front-line employees find Atlas difficult to navigate and information difficult to find. These are the main criticisms.

Front-line employees claimed that they did not have as much time to consult Atlas as their colleagues at headquarters. The frequency with which they use the intranet is also less intense in this regard. This is probably also one of the reasons why front-line employees feel that Atlas does not keep them adequately up to date with new information, procedures or policies.

Beyond the few differences between these two major groups of employees, it is possible to distinguish more employee profiles. Indeed, these two major groups are not completely homogeneous. For example, there are front-line employees who use Atlas regularly and who consider themselves good with digital tools. Following the same logic, there are some headquarters employees who rarely use the Intranet and who consider themselves to have low digital literacy.

The Personae - Method

In the following section, we present a series of six personae. A persona is a fictional character who represents a specific segment within a specific group or population. It is a tool that is commonly used in marketing research. The use of personae makes it possible to represent a target group in a more lively and emotional way in order to develop adapted services and/or products. Having too similar personae is actually counterproductive - personae are used to highlight differences.

Our segmentation analysis is based on the results of focus groups and individual interviews with CBSA employees. The purpose of this analysis is to assemble groups of Atlas users so that each group is as homogeneous as possible, and also as different as possible from the other groups. This means that we want to bring together segments of the CBSA employee community that have a similar behaviour and relationship with technology.

It should also be kept in mind that each employee is actually a combination of several people. For example, we have met employees who have faced the challenges of each of these characters. Since the objective is to understand how Atlas employees use Atlas, the analysis is based on key criteria that influence Intranet use: the frequency of Atlas use, digital literacy, sources of frustration with the Intranet and topics important to them and, of course, their role in the Agency. We have identified six distinct profiles.

Frontline Employees

1. Persona one: “Phone-a-friend”. Most likely to be a long-time serving BSO.

"When I started working, everything was done on paper. Now it's mainly technology. It's a learning curve. I don't really use Atlas, it's very rare. It's too complicated. They expect us to be able to use Atlas to search for information and do all kinds of research, but it's not easy to navigate the Intranet and do searches when there's a line of customers in front of us. Moreover, Atlas almost never returns the right information, it's a waste of time. I prefer to ask my colleagues if they have the information I'm looking for, it's faster. The most important thing is the newsletter, but it is also received by email."

The use of digital tools in his work is a huge challenge. His low digital literacy and the inherent navigation and search difficulties of Atlas ensure that he avoids using Atlas as much as possible. He relies heavily on his colleagues to get the information he is looking for. However, by doing so, he does not improve his literacy or digital skills. He also avoids Apollo as much as possible and never uses the Wiki. He saves all the documents he uses frequently in a file to access them if necessary.

- Digital skills and digital literacy: medium or low

- Appreciation of using digital tools at work: moderate to low

- Use of government digital tools: struggling to use

Frequency of Atlas use: monthly frequency of use or less frequent (rare)

Satisfaction with Atlas: very dissatisfied

Daily challenges:

· Having to use digital tools and become familiar with new digital tools such as Apollo.

· Lack of time to improve skills. There is no time for consulting the Intranet on shifts.

· Finding the right information for customers when time is a major issue.

Most important features: Most Frequent Tasks, Front-line Newsletters, News from Other Regions, Upcoming Events, and Border Updates (videos).

2. Persona 2: “Where is that form?”. A BSO in a hurry.

"Atlas is an essential tool in our work as a front-line agent. But Atlas must absolutely be improved since there are huge gaps with this tool, which also appears to be outdated compared to tools external to the GC. Atlas' search engine follows incomprehensible rules. If you don't know the exact words, it's impossible to find the document you're looking for, and since documents often change to different names, then we can't find anything. It should be possible to use advanced search criteria and documents should be sorted by date. For now, it is better to use Google. It's faster, more efficient, and you always find what you're looking for. Atlas does everything, but very badly."

Very comfortable with the use of digital technologies, he enjoys working with digital tools as part of his work. However, he is extremely critical of the tools at his disposal. He only uses Apollo and the Wiki occasionally because he doesn't know if he can trust them. He finds Apollo useful, but too slow to use at work. He is also irritated by the navigation between Atlas, Apollo and the Wiki which is not fluid.

- Digital skills and digital literacy: moderate to high

- Appreciation of using digital tools at work: moderate

- Use of government digital tools: challenge in handling them

Frequency of Atlas use: weekly

Satisfaction with Atlas: rather dissatisfied

Daily challenges:

· Finding procedures, policies, and forms.

· The information is regularly archived, leaving no possibility of knowing whether the information is still accurate, current or outdated.

· The duplication of information and the dissemination of information in different locations makes it difficult to easily find all the information on a subject.

· Broken links

Frustrations with Atlas:

· Offset between HQ and first line requirements: Atlas is built according to a HQ perspective. Atlas does not adequately represent the operational side of the Agency.

· Does not feel represented as an employee on the Intranet: he considers that the focus of the Intranet is very corporate and focused on the employees at headquarters.

Most important features: Updates on Regional Initiatives, Regional News, Frontline bulletins, Most Frequent Tasks, Upcoming Activities.

3) Persona 3: “The Go-to guy or gal”. This BSO is the local reference for the others

"I use Atlas every day in my work. But all of the information is poorly organized on the Intranet; information should be classified under headings such as: immigration, trade, travellers, etc. In addition, the information is duplicated in Atlas and is received by email. Not to mention that it is almost impossible to find reliable contact numbers in Atlas. If I want to call Calgary Airport, I can't find the phone number in Atlas. The Intranet should be used to centralize information, but it should be reorganized and classified so that it can be found. A lot of things about Atlas are made for people in an office, we at the airport, don't have the luxury of taking the time to look at everything.”

Uses Atlas frequently to access forms and templates, operational bulletins, memos, timesheets and border updates (video) that he regularly views. Although dissatisfied with the experience Atlas provides, he believes that the Intranet is useful and essential to his work. He always ends up finding the information he is looking for, but the experience is frustrating. His desire is for Atlas to be a purely operational tool.

- Digital skills and digital literacy: moderate

- Appreciation of using digital tools at work: moderate

- Use of government digital tools: difficult to use

Frequency of Atlas use: daily

Satisfaction with Atlas: low

Daily challenges:

· The speed to finding information.

· The clarity of information.

Most important features: Frontline bulletins, More Requested Tasks, Regional News, Updates on Chapter Initiatives, Updates on Regional Initiatives, News from Other Regions, Border Updates (videos), and Congratulations.

National Headquarters and Regional Offices

1) Persona 4: “Just got here”. New to the Agency, this person needs to find their way.

"I'm new to the Agency. For me, it is very difficult to use the digital tools that are at my disposal. There is no training on how to use them. It is a real challenge to find the right information in Atlas. I've gotten into the habit of calling colleagues who can help me find what I'm looking for.”

Experiences great difficulty in finding forms, procedures and policies on a daily basis. He considers that the Intranet is not a tool that helps him to accomplish his tasks. Links and menus are difficult to understand. He prefers to read the information he receives by email. He would like to obtain training to improve his skills with the Agency's digital tools: Atlas and Apollo.

- Digital skills and digital literacy: moderate to high

- Appreciation of using digital tools at work: low

- Use of government digital tools: very hard to use

Frequency of Atlas use: monthly or rare

Satisfaction with Atlas: very low

Daily challenges:

· Finding the information that you are looking for.

· Understanding the information presented in the Atlas menus and links.

Most important features: Most Requested Tasks, Update on Regional Initiatives, Regional News and Executive Messages.

2) Persona 5: “the Information Junky”. This HQ policy officer want more

"Everything in Atlas is constantly moving and changing. Week after week, the forms change names or numbers so they can no longer be found. It's very frustrating and it's a huge waste of time. The information is also not very clear. The menus are repetitive, and you don't know what to expect once you go beyond the Atlas home page.”

He works daily with information and must have access to a wide range of information sources to do his job. He finds that Atlas limits him in his work more than it helps him. He would like to have access to much more information than he currently has access to on Atlas. Among other things, he would like to see international news that may have an impact on the Agency's work. He also wants employees to have access to social media so they can share and see the Agency's work across the country.

- Digital skills and digital literacy: moderate

- Appreciation of working with digital tools at work: low

- Use of government digital tools: very difficult to use

Frequency of Atlas use: weekly to daily

Satisfaction with Atlas: low

Daily challenges:

· Finding the information that you are looking for.

· Understanding the information presented in the Atlas menus and links.

Most important features: Frontline bulletins, Most Frequent Tasks, Updates on Agency Initiatives, Updates on Regional Initiatives, Updates on Branch Initiatives, Executive Messages, and Regional News.

3) Persona 6: “The Power user”. Great HQ resource to suggest improvements!

"I use Atlas a lot and I have to do a lot of research on it. Atlas is an essential and useful tool, but it is very difficult to work with. There are many things that should be improved in the Intranet but the most important is the search engine. If I type a word into the search engine, I will get millions of pages of results that are not classified correctly. They should be organized by date, but this is not the case. I love using the Wiki for this reason. It is more dynamic than Atlas and we know exactly when the information was last modified. It's very useful. And there is a lot of information on the Wiki that is not on Atlas."

He makes extensive use of the digital tools made available to him by the Agency. Atlas, Wiki and Apollo are among the tools regularly used for work. However, he considers that several of his colleagues misuse them, which can cause problems. He believes that there should be training to ensure that everyone uses the tools correctly, especially Apollo. In many cases, this persona is already a content creator on Atlas and/or Apollo.

- Digital skills and digital literacy: moderate to high

- Appreciation of working with digital tools at work: high

- Use of government digital tools: hard to use

Frequency of Atlas use: daily

Satisfaction with Atlas: average

Daily challenges:

· Searching with the Atlas search engine.

· Navigating between Apollo, Wiki and Atlas applications and the need to re-identify each time.

Most important features: Updates on Agency initiatives, frontline bulletins, upcoming events, updates on branch initiatives, updates on regional initiatives and regional news.

2.1.4 Tree Testing

In this phase of the study, we performed a reverse card test, also called a tree test. The main objective of this test is to evaluate the performance of an information architecture with users of a website, Intranet or an application. The tree test consists of a series of tasks that users must perform. In a tree test, the information architecture is evaluated without artifice, i.e. in a refined environment, without color, without theme, without design elements to avoid biasing the results. The results make it possible to validate or invalidate an information architecture or to make any necessary adjustments.

As part of our project, we tested the navigation tree developed in Phase 2 (interviews and card sorting). The tree structure was evaluated by real users of the Atlas Intranet. The invitation to participate in the test was made via an open link placed on Atlas for one week. A total of 1194 users tested the English tree structure while 200 tested the French tree structure.

Ten tasks were proposed to users. For each task, respondents were asked to indicate in which menu branches, and under which leaves of the tree structure, they thought they could find the information they were looking for. The menu items that were selected to be part of this test were randomly selected. Once the menu items were chosen, we developed corresponding tasks.

The analysis of the tree test results was done, calculating the success rate of each task, the menu items that were visited first to complete each task and the users' final destinations for each task.

2.1.4.1 The tasks

The ten tasks that have been created to evaluate the tree structure are as follows:

1. You would like to get information on the public-service health care system.
2. You want to learn new ways of doing things and develop your skills and abilities to improve yourself in your professional field.
3. You wish to review the passenger and crew screening procedures for marine vessels.
4. You are looking for the biography and professional information on your branch Vice President.
5. You are looking to issue a contract for an external supplier.
6. You are seeking information about the employer's accommodations for people with disabilities.
7. You want to find out the telephone number of the National IT Service Desk
8. You would like to view the weekly news series video.
9. You want to access Apollo.
10. You want to find the Operational Bulletin PRG-2019-09 -- New Commercial Allegation in the Integrated Customs Enforcement System (ICES) for Providing Inaccurate Information.

Each task is associated with a specific element of the tree structure, i.e. an item in the menus that corresponds to the expected destination of the users. The success rate of a task is determined by the proportion of respondents who correctly identified the location in the menus where the requested information was located. That is, they have reached the right destination. Here are the menu destinations that correspond to the correct answers for each task:

Task 1: Pay and Benefits under the "Employee" heading in the main menu

Task 2: Training and Learning under the "Employee" heading in the main menu

Task 3: Immigration under the "Guides / Manuals and Standard Operating Procedures" heading in the "Tools" sub-menu

Task 4: Executive Offices under the "The organization / About us" menu

Task 5: Procurement and Contracting under the "Tools" menu

Task 6: Labour Relations under the "Human Resources" menu

Task 7: The Helpdesk under the "IT Portal" menu

Task 8: Border Update under the "Daily News" menu

Task 9: Apollo under the "Information Management" heading in the "Tools" menu

Task10: Commerce under the "Frontline Bulletins and National Document Centre Publications" heading under the "Daily news" menu

The following table shows the success rates for each of the tasks according to the language of completion of the questionnaire and the tree structure tested. A success rate of 50% or more is considered good while a success rate below 50% is a low success rate that is worth investigating.

Task	Destination corresponding to success rate	French success rate	English success rate
You would like to get information on the public-service health care system.	Pay and Benefits under the "Employee" heading in the main menu	75%	68%
You want to learn new ways of doing things and develop your skills and abilities to improve yourself in your professional field.	Training and Learning under the "Employee" heading in the main menu	24%	57%
You wish to review the passenger and crew screening procedures for marine vessels.	Immigration under the "Guides / Manuals and Standard Operating Procedures" heading in the "Tools" sub-menu	9%	5%
You are looking for the biography and professional information of your branch Vice President.	Executive offices under the "The organization / About us" menu	22%	53%
You are looking to issue a contract for an external supplier.	Procurement and Contracting under the "Tools" menu	65%	76%
You are seeking information about the employer's accommodations for people with disabilities.	Labour Relations under the "Human Resources" menu	29%	32%
You want to find out the telephone number of the National IT Service Desk	The Helpdesk under the "IT Portal" menu	89%	94%
You would like to view the weekly news series video.	Border Update under the "Daily News" menu	67%	40%
You want to access Apollo.	Apollo under the "Information Management" heading in the "Tools" menu	77%	69%
You want to find the Operational Bulletin PRG-2019-09 -- New Commercial Allegation in the Integrated Customs Enforcement System (ICES) for Providing Inaccurate Information	Commerce under the "Frontline Bulletins and National Document Centre Publications" heading under the "Daily news" menu	14%	14%

2.1.4.2 Results

First, let's analyze the success rate of each task to get an idea of which tasks were successfully completed and which ones were less successful.

Success rate of tasks

Tasks 1-5-7-9 were successfully completed by more than 50% of respondents in both the English and French tree tests. These results confirm that the tree structure and labels associated with these menus and that have been used in these tasks are adequate and allow users to correctly perform the tasks they are required to perform.

Tasks 2-4-8 were successfully completed by more than half of the respondents in one or the other language, which leads us to believe that in these particular cases, a problem with menu formulations and/or tasks to be performed comes into play in one or the other of the two languages. These cases require further analysis.

Tasks 3-6-10 did not achieve a success rate of more than 50% in either official language. These results indicate a fairly serious problem in the performance of its tasks. These cases will be investigated more thoroughly to determine the cause of the high failure rates.

Analysis of the first visits

The analysis of the first visits shows the first place in the main menu where users clicked to find the information and complete the requested task. This is an important clue as to whether the cause of the failure is directly in the main menu.

The expected answer to Task 2 was found in the "Training and Learning" tab under the "Employee" menu of the main menu. The following table shows that 65% of respondents in French and 60% of respondents in English actually tried to accomplish the task under the right heading in the main menu. It is therefore in a lower level of the tree structure that the failure occurred.

Task 2	First visit French	First visit English
Employee	65%	60%
Tools	17%	19%
The Organization / About Us		1%
Human Resources	18%	18%
IT Portal	1%	2%
Daily News		1%

Once in the "Employee" menu, many users tried to complete the task using the "Employment and Professional Development" sub-menu rather than the "Training and Learning" sub-menu. This error has been more consistent in French than in English. This is probably an effect induced by the vocabulary used for these items. We believe that it would be wise to review the wording of these two items to make them more distinctive in the eyes of users and thus avoid any confusion.

Main Destinations – Task 2	FR	EN
Awards and recognition	3
Employee Assistance Program	1	4
Employee Orientation / Onboarding / New Hires	5	2
Employment Equity and Diversity		1
Integrity, Values and Ethics
Jobs and Career Development	115	280
Occupational Health and Safety
Pay and Benefits		3
Training and Learning	47	673
Well-being		2
Security		2

Task 3 has the lowest success rate of all the tasks that were tested in the tree structure test. The correct answer to the task was found under the "Immigration" heading in the "Library of Guides, Manuals and Standard Operating Procedures" sub-menu and under the "Tools" item in the main menu.

The analysis of the first visit shows that three-quarters of the users actually went under the right menu item: 76% of French users and 75% of English users first clicked on the "Tools" item in the main menu. The classification of "Procedures" under the "Tools" heading in the main menu is a good approach for users. This indicates that users have failed this task in the lower level of the tree structure.

Task 3	First visit French	First visit English
Employee	16%	13%
Tools	76%	75%
The Organization / About Us	4%	7%
Human Resources	1%	1%
IT Portal	3%	2%
Daily News	2%	2%

Once in the "Tools" sub-menu, the vast majority of users clicked on "Library of Guides, Manuals and Standard Operating Procedures". This is correct given the destination to be reached for this task. This is a good indication of the appropriate classification of "Procedures" in the tree structure under the "Library of Guides, Manuals and Standard Operating Procedures" heading.

To finalize the task of "Reviewing Passenger and Ship Crew Screening Procedures", users clicked on the "Travellers" item rather than under the "Immigration" item. It is at the last level of the tree structure that the users' failure occurred. According to our analysis, this is probably a problem related to the fact that the tree test exercise is done out of context. We believe that in a work context, users would probably have clicked on "Immigration" and not "Travellers" depending on the context of the situation. We can therefore consider that this item is properly classified.

Main Destinations – Task 3		FR	EN
Information Management
	Apollo	2	4
	ATIP
Forms and Template Library
	Immigration
	Commercial
	Traveller	1	1
	Enforcement of the law
	Others
Guides / Manuals and Standard Operating Procedures
	Immigration	17	58
	Commercial	9	192
	Traveller	111	503
	Enforcement of the law	12	63
	Others	10	151
Policy Library
	Immigration		3
	Commercial	1	15
	Traveller	9	25
	Enforcement of the law		2
	Others		8
Finance Volume		1
Procurement and Contracting		1	3
Accomodations and Facilities			2

To successfully complete Task 4, which is to search for your branch Vice-President’s resume, users had to click on "Executive Offices" in the "Organization / About Us" section of the main menu. The following table shows that the first visit was made correctly to the destination: 72% of French users and 87% of English users clicked on the right item. This indicates that the main menu seems to be adequate to guide users to the right destination.

Task 4	First visit French	First visit English
Employee	11%	4%
Tools	1%
The Organization / About Us	72%	87%
Human Resources	17%	7%
IT Portal	1%
Daily News	1%	1%

However, a majority of users chose the "Agency Organization Chart / About Branches, Directorates and Divisions" tab rather than "Executive Offices" to accomplish the task. English-speaking users did not make this mistake as regularly as French users. We believe that the wording of the items found under the "The Organization / About Us" heading in the main menu should be reviewed to clarify their content and clearly identify the content of the items.

Main Destinations- Task 4	FR	EN
Agency Organization Chart / About Branches, Directorates and Divisions	125	475
Executive Offices	44	622

Task 6 asked users to obtain information on the employer's arrangements for persons with disabilities. The destination of this task was under the "Labour Relations" heading which was under "Human Resources" in the main menu. This task had a low success rate in both the French and English tree tests.

Task 6	First visit French	First visit English
Employee	39%	39%
Tools	4%	4%
The Organization / About Us	5%	1%
Human Resources	52%	55%
IT Portal
Daily News	1%	1%

The analysis shows that just over half of the users went directly in the main menu under the "Human Resources" heading but most did not think they would find the right answer to this task in "Labour Relations", which would have been the right destination to accomplish the task. They preferred to go back through the menu to the "Employee" menu to find an item that would better match the task requested. For most users, they selected the "Employment Equity and Diversity" tab.

We can conclude that the content of the "Human Resources" menu is probably not clear enough and the terms used are not precise enough to allow users to find what they are looking for. We therefore recommend that the wording of the items placed in the "Human Resources" sub-menu be reviewed by creating clearer categories and using distinctive language.

Main Destination - Task 6	FR	EN
Labour Relations	58	376
Performance Management	1	6
Staffing	7	96

Task 8 was to find the right tab to view the weekly news series video. The destination was under "CBSA in Brief" under "Daily News" in the main menu. This task was less successful in English than in French. The analysis of the first visit shows that eight out of ten users clicked on the right item in the main menu to find the video. It is therefore possible to conclude that this item is placed in the right section of the main menu.

Task 8	First visit French	First visit English
Employee	8%	3%
Tools	2%	1%
The Organization / About Us	8%	7%
Human Resources	1%	1%
IT Portal	2%	1%
Daily News	81%	88%

However, once in the "Daily News" sub-menu, most users thought they could find the weekly news series video by clicking on "News and Photo Galleries". This error was made more regularly among users in the English tree structure than in the French one. The wording of the weekly news series video should probably be revised to ensure that it is easily identifiable by Atlas users. However, it is positioned under the right heading.

Main Destinations – Task 8		FR	EN
Agency and Branch Initiative Priorities		3	2
Border Update		133	474
CBSA Gives!		1	7
Event Calendar		1	4
Frontline Bulletins and National Document Centre Publications
	Immigration
	Commercial		2
	Traveller	1
	Enforcement of the law		2
	Others	1	6
Messages from the Executives		2	7
News and Photo Galleries		46	619

Task 10 is one of the tasks with the lowest success rate among users. Task 10 was to find Operational Bulletin PRG-2019-09 -- New Trade Claim in the Integrated Customs Enforcement System for providing incorrect information. The destination of this task was under "Commerce" in the "Front Line Bulletins and Publications of the National Documentation Centre" section and under "Daily News" in the main menu.

Task 10	First visit French	First visit English
Employee	14%	12%
Tools	66%	63%
The Organization / About Us	4%	5%
Human Resources	1%	1%
IT Portal	3%	4%
Daily News	13%	14%

Only a little more than one in ten users correctly searched for information under "Daily News" while the majority went directly in the "Tools" tab of the main menu. This is a clear indicator that first-line operational bulletins are not classified in the right place in the tree structure. We strongly recommend classifying them under "Tools" in the proposed Atlas tree structure.

Most users indicated that they thought they could find the information they were looking for in this task in the "Business" tab of the "Library of Guides and Manuals and Standard Operating Procedures". We are in a position to believe that if the first-line operational bulletins had been placed under the "Tools" heading of the main menu, the success rate of this task would have been higher.

Main Destinations – Task 10		FR	EN
Information Management
	Apollo	1	8
	ATIP		3
Forms and Template Library
	Immigration	1	1
	Commercial	4	10
	Traveller
	Enforcement of the law	2	10
	Others		2
Guides / Manuals and Standard Operating Procedures
	Immigration	93	437
	Commercial	2	15
	Traveller	17	140
	Enforcement of the law	5	58
	Others	93	437
Policy Library
	Immigration		1
	Commercial	15	86
	Traveller	1	6
	Enforcement of the law	2	23
	Others		5
Finance Volume			2
Procurement and Contracting		2	1
Accomodations and Facilities			1

2.1.4.3 Final recommendations on Atlas information architecture

The analysis of the first visits and the main destinations for each of the tasks correspond, in the majority, adequately to the users' expectations. However, there are some important lessons to be learned in developing the final information architecture for the renewal of Atlas:

6. The “Frontline Bulletins and National Document Centre Publications” should be classified under the main menu "Tools" and not under the menu "Daily News".
7. Some items, as currently identified in Atlas, should be renamed to clarify their content.

a. The "Employment and Professional Development" tab can easily be confused with the "Training and Learning" tab.

b. The tabs "Agency Organization Chart / About Branches, Directorates and Divisions" and "Executive Offices" do not adequately allow users to know what content they will find under either one.

c. The information found under the "Human Resources" tab should be reformulated. For the moment, some of the information found under "Human Resources" is also searched under the "Employee" menu. A work should be done at this level to clarify and reorganize that information.

d. Identify more clearly where to find the content of the weekly video series in the "Daily News" tab to make it easier for users to find it.

8. The results of the tree test indicate that the main menu created by employees during card sorting works properly to guide users in their task on the intranet.
9. The accuracy of libraries of forms, guides, manuals, manuals, policies and newsletters on different topics, such as immigration, commerce, travel, etc., helps users to guide and refine their search for information. In a real work environment, we believe this approach is promising.
10. The tree test does not pretend to have evaluated all categories and items of the submenus. The effort to clarify category wordings must be systematically carried out by the CBSA for every item.

2.1.4.4 The Proposed Tree Structure

Homepage

Employee

Awards and Recognition

Employee Assistance Program

Employee Orientation / Onboarding / New Hires

Employment Equity and Diversity

Integrity, Values and Ethics

Jobs and Career Development

Occupational Health and Safety

Pay and Benefits

Training and Learning

Well-being

Security

Tools

Information Management

Apollo

ATIP

Forms and Template Library

Immigration

Commerce

Traveller

Enforcement of the Law

Other Themes

Frontline Bulletins and National Document Centre Publications

Immigration

Commerce

Traveller

Enforcement of the Law

Other Themes

Guides / Manuals and Standard Operating Procedures

Immigration

Commerce

Traveller

Enforcement of the Law

Other Themes

Policy Library

Immigration

Commerce

Traveller

Enforcement of the Law

Other Themes

Finance Volume

Procurement and Contracting

Accommodations and Facilities

The Organization / About Us

Agency Organization Chart / About Branches, Directorates and Divisions

Executive Offices

Human Resources

Labour Relations

Performance Management

Staffing

IT Portal

Helpdesk

IT Portal

ACROSS

ICES

CAS

ESS

Daily News

Agency and Branch Initiatives Priorities

Border Updates (Video Series)

CBSA Gives / GCWCC (Charitable Campaign

Event Calendar

Messages from Executives

News and Photo Galleries

2.2 Internet

2.2.1 Online Survey

As mentioned before, this portion of the study is interested in people visiting the Traveller Section of the Canada Border Services Agency website.

2.2.1.1 Profile

Three quarters of the CBSA website’s visitors (75%) are Canadians or permanent residents of Canada, more specifically seven out of ten (70%) are Canadian citizens and 5% are permanent residents of Canada. The other quarter (25%) are citizens of another country.

Figure 5.1. Answer to question D1: Are you a Canadian citizen, a permanent resident of Canada or a citizen of another country? Base: All respondents (n=2,729)

The greater part of Canadian citizen who visited the CBSA website are Ontarians (40%), about one out of five of them (22%) are from British-Columbia and about one out of ten are Quebecers (13%) or Albertans (11%). They are less numerous to live in the Atlantic provinces (6%) or in the Prairies (5%). Only a minority (2%) currently live in another country.

Figure 5.2. Answer to question D2: In which province or territory do you live? Base: Canadians or permanent residents (n=2,042)

Half of the CBSA visitors that are non-Canadians are Americans (52%). Other visitors live in 20 other countries. (For more detailed results, please refer to the Appendix H)

Figure 5.3. Answer to question D3: In which province or territory do you live? SPONTANEOUS MENTIONS Base: Non-Canadians (n=687)

Three-quarter of the website visitors (75%) are English and about one out of ten are French (12%) or allophones (12%).

Figure 5.4. Answer to question D5: What is the language you first learned at home during your childhood and that you still understand? SPONTANEOUS MENTIONS Base: All respondents (n=2,729)

Two-thirds of the visitors (68%) accessed the CBSA website from within Canada and three out of ten (31%) from outside the country.

Figure 5.5. Answer to question D4: Did you access the Canada Border Services Agency website from within Canada or from outside Canada? SPONTANEOUS MENTIONS Base: All respondents (n=2, 729)

Eight out of ten (78%) visitors to the CBSA website are currently planning to travel somewhere and 17% are not. Non-Canadians (88%) are more likely to be currently planning to travel.

Figure 5.6. Answer to question B7: Are you currently planning to travel somewhere? Base: All respondents (n=2,729)

Table 5.1. Currently Planning to Travel Somewhere According to Visitors Country of Origin

B7. Are you currently planning to travel somewhere? Base : All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
Yes	78%	75%-	88%+
No	17%	21%+	7%-
DNK / Refusal	5%	5%	5%

Three quarters of respondents planning to travel (73%) intend to do it by air, four out of ten (38%) intend to travel by land and 8% plan on traveling by sea.

Figure 5.7. Answer to question B8: How do you intend to travel? SEVERAL MENTIONS ALLOWED* Base: Respondents who are currently planning to travel (n=2,131)

*Because respondents were able to give multiple answers, total mentions may exceed 100%.

Canadians and permanent residents (79%) are more likely to intend to travel by air and non-Canadians are more likely to intend to do so by land (47%) or by sea (13%).

Table 5.2. Modes of Transport for Travelling According to Visitors Country of Origin

*Because respondents were able to give multiple answers, total mentions may exceed 100%.

B8. How do you intend to travel? SEVERAL MENTIONS ALLOWED* Base: Respondents who are currently planning to travel	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,131	1,526	605
Air (by plane or helicopter)	73%	79%+	56%-
Land (car, bus, motorcycle, other motor vehicle, train)	38%	34%-	47%+
Sea (boat, cruise ship)	8%	5%-	13%+
DNK	1%	1%	1%

Half of the CBSA website visitors (51%) cross the borders a few times a year and 15% do it less often, mainly every month (11%). Two out of ten rarely do it (21%) and 11% never did.

Figure 5.8. Answer to question B10: How often do you cross the Canadian borders? Base: All respondents (n=2,729)

Canadians and permanent residents are more likely to cross the Canadian borders at least one a year (83%). On their part, non-Canadians are more likely to do it rarely (43%) or to have never done so (38%).

Table 5.3. Frequency of Crossing the Canadian Borders According to Visitors Country of Origin

B10. How often do you cross the Canadian borders? Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
Almost every day	1%	1%	1%
Every week	3%	4%+	1%-
Every month	11%	14%+	2%-
A few times a year	51%	64%+	13%-
Rarely	21%	14%-	43%+
I have never crossed Canadian borders	11%	2%-	38%+
DNK	2%	1%-	3%+

2.2.1.2 Behaviour of Website Visitors

Most visitors to the CBSA website (59%) arrived by searching for the subject they were looking for using an external search provider, and almost a quarter (23%) by searching for the agency by name 'Canada Border Services Agency' using an external search provider. Visitors were less numerous to access the website by using a search engine on another Government of Canada website (7%), by navigating directly to the website by typing the address into the address bar (4%) or through another website (3%).

Figure 5.9. Answer to question A1: How did you arrive at the Canada Border Services Agency website? Base: All respondents (n=2,729)

Canadians and permanent residents (64%) are more likely to have arrived on the website by searching for the subject they were looking for using an external search provider. Non-Canadians are more likely to have arrived by using a search engine on another Government of Canada website (13%) or through another website (9%).

Table 5.4. Ways to Arrive at the Website According to Visitors Country of Origin

A1. How did you arrive at the Canada Border Services Agency website? Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
I searched for the subject I was looking for (for example: Nexus program, wait time at the border, consumer goods allowed at borders, and so on.), using an external search provider (Google, Bing, and so on.).	59%	64%+	46%-
I searched for the agency by name 'Canada Border Services Agency' using an external search provider (Google, Bing, and so on.).	23%	23%	22%
I used a search engine on another Government of Canada website.	7%	5%-	13%+
I navigated directly to the website by typing the address into the address bar.	4%	4%	5%
Through another website	3%	2%-	9%+
Another person/agency gave me the link	2%	1%-	5%+

Six out of ten (63%) visitors to the CBSA website rarely or exceptionally visit it. A quarter (26%) visit it once or twice a year and about 10% visit it more regularly than once or twice a year.

Figure 5.10. Answer to question A2: How often do you visit the Canada Border Services Agency website? Base: All respondents (n=2,729)

Canadians and permanent residents are more likely to visit the CBSA website once or twice a year (30%) or monthly (5%). On their part, non-Canadians are more likely to visit it rarely or exceptionally (75%).

Table 5.5. Frequency of Visits of the CBSA Website According to Visitors Country of Origin

A2. How often do you visit the Canada Border Services Agency website? Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
On a daily basis	2%	1%	2%
On a weekly basis	3%	3%	4%
On a monthly basis	5%	5%+	3%-
Once or twice a year	26%	30%+	12%-
Rarely or exceptionally	63%	58%-	75%+
DNK	2%	2%-	5%+

One out of five (22%) visitors to the CBSA website were visiting the website for the first time. Fourteen percent (14%) had visited it the week before, about two out of ten (19%) had visited it in the last three months and a similar proportion (19%) had visited it between 3 and 12 months ago. Finally, one out five (20%) had visited it more than a year ago.

Figure 5.11. Answer to question A3: When was the last time you visited the Canada Border Services Agency website? Base: All respondents (n=2,729)

Non-Canadians are more likely to have never visited the website before (48%) or to have visited it in the past week (16%). Conversely, Canadians and permanent residents are more likely to have visited it between a week and 12 months ago (45%) or more than a year ago (23%).

Table 5.6. Last Visit of the CBSA Website According to Visitors Country of Origin

A3. When was the last time you visited the Canada Border Services Agency website? Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
In the past week	14%	13%-	16%+
Within the past month	10%	12%+	5%-
Between 1 and 3 months ago	9%	11%+	5%-
Between 3 and 6 months ago	7%	8%+	4%-
Between 6 and 12 months ago	12%	14%+	7%-
More than 12 months ago	20%	23%+	11%-
Never	22%	13%-	48%+
DNK	6%	7%+	4%-

Four out of ten (38%) visitors used their laptop to navigate the CBSA website and a quarter (27%) used a desktop computer. A quarter (24%) instead used a smartphone, such as an iPhone (14%) or other mobile device (10%). Finally, one out of ten (9%) used an iPad tablet and 2% an Android tablet.

Figure 5.12. Answer to question A4: What device do you use to navigate on the Canada Border Services Agency website? Base: All respondents (n=2,729)

Canadians and permanent residents (10%) are more likely to have navigated the website using their iPad tablet, whereas non-Canadians are more likely to have use another type of smartphone that was not an iPhone (15%).

Table 5.7. Device Used to Navigate the CBSA Website According to Visitors Country of Origin

A4. What device do you use to navigate on the Canada Border Services Agency website? Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
Laptop	38%	39%	36%
Desktop computer	27%	27%	27%
iPhone	14%	14%	14%
Other smartphone	10%	8%-	15%+
iPad Tablet	9%	10%+	6%-
Android Tablet	2%	2%	2%

Only a minority of respondents who had previously visited the CBSA website (6%) ever had problems accessing it. There are no differences among Canadians and permanent residents or non-Canadians.

Figure 5.13. Answer to question A5: Have you had, or have you ever had any problems accessing the Canada Border Services Agency website? Base: Respondents who had previously visited the CBSA website (n=1,961)

A third of respondents who ever had a problem accessing the CBSA website mentioned various programming issues (34%), a quarter (24%) mentioned it was difficult to navigate, and 15% mentioned information issues. Also, one out of ten could not connect (10%). Others mentioned the links to third party agencies (7%) and only few respondents mentioned other problems, such as the fact they could not access Nexus renewal link on mobile (2%) or they could not process until they did the survey (2%). (For more detailed results, please refer to Table A.1 of the Appendix C)

Figure 5.14. Answer to question 5A: What was the nature of the problem? SPONTANEOUS MENTIONS Base: Respondents who ever had a problem accessing the CBSA website (n=123)

A third of the CBSA website visitors (36%) spent 5 minutes or less on it, about a quarter (23%) spent between 6 and 10 minutes on it and another third (33%) spent more than 10 minutes navigating on it.

Figure 5.15. Answer to question A6: How much time did you spend on the Canada Border Services Agency website during your visit? Base: All respondents (n=2,729)

Non-Canadians visitors (37%) are more likely to have spent more than 10 minutes navigating the CBSA website.

Table 5.10. Time Spent on the CBSA Website According to Visitors Country of Origin

A6. How much time did you spend on the Canada Border Services Agency website during your visit? Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
Net 5 minutes or less	36%	37%	33%
Less than 2 minutes	11%	12%+	8%-
2 to 5 minutes	25%	25%	25%
6 to 10 minutes	23%	24%	22%
Net More than 10 minutes	33%	31%-	37%+
11 to 15 minutes	12%	12%	12%
More than 15 minutes	21%	19%-	25%+
DNK	8%	8%	8%

Two-thirds of the visitors of the CBSA website (67%) found the information they were looking for on the landing page, but 19% did not.

Figure 5.16. Answer to question A7: Did the page you landed on contain the information that you were looking for? Base: All respondents (n=2,729)

A majority of the visitors (90%) mentioned it was easy to find the information on the page they landed on. Conversely, 8% said it was not.

Figure 5.17. Answer to question A8: Was it easy to find information on the page you landed on? Base: Respondents who did not find the information they were looking for on the landing page (n=1,829)

The main difficulties encountered in finding the information on the landing page are caused by design problems (32%). Issues or bugs are also mentioned by one in five (18%) respondents who said it was not easy to find the searched information. One in ten also mentioned issues with links (10%) and confusing, vague or conflicting information (10%). Finally, 5% mentioned that the quantity of information provided was an issue for them. (For more detailed results, please refer to Table A.2 of the Appendix C)

Figure 5.18. Answer to question A8A: What was difficult about finding information on the page? SPONTANEOUS MENTIONS Base: Respondents who said it was not easy to find the searched information (n=146)

Non-Canadians are more likely to complain that the information is confusing, vague or conflicting (32%).

Table 5.13. Difficulty About Finding the Searched Information According to Visitors Country of Origin

A8A. What was difficult about finding information on the page? SPONTANEOUS MENTIONS Base: Respondents who said it was not easy to find the searched information	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	146	115	31
NET DESIGN PROBLEMS	32%	36%	19%
NET ISSUES OR BUGS	18%	17%	19%
NET LINKS	10%	11%	6%
NET CONFUSING, VAGUE, OR CONFLICTING INFORMATION	10%	4%-	32%+
NET INFORMATION QUANTITY	5%	5%	6%
Other	5%	4%	6%
No difficulty	2%	3%	0%
Don't know / refusal	18%	20%	13%

Slightly less than a third of respondents who did not find the information they were looking for on the page they landed on (31%) ended up finding it, but 40% did not. Also, three out of ten of them (29%) aren’t sure if they did.

Figure 5.19. Answer to question A10: You mentioned that you did not find the information you were looking for on the page you landed on. Did you end up finding the information you were looking for on other pages of the Canada Border Services Agency website? Base: Respondents who did not find the information they were looking for on the page they landed on (n=900)

Canadians and permanent residents (34%) are more likely to say they ended up finding what they were looking for than non-Canadians.

Table 5.14. Ending Up Finding the Searched Information According to Visitors Country of Origin

A10. You mentioned that you did not find the information you were looking for on the page you landed on. Did you end up finding the information you were looking for on other pages of the Canada Border Services Agency website? Base : Respondents who did not find the information they were looking for on the page they landed on	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	900	672	228
Yes	31%	34%+	23%-
No	40%	39%	42%
DNK	29%	27%-	35%+

A quarter of the visitors who could not find the information they were looking for (27%) were searching information about the NEXUS program. About one out of ten could not find information about declaration or importations and their fees (11%), about items allowed at the border (9%) or about requirements (9%). Some other information was missing but in a lower proportion. (For more detailed results, please refer to Table A.3 of the Appendix C)

Figure 5.20. Answer to question A10A: What information were you looking for that you didn't find? Base: Respondents who ended up not finding the information they were looking for (n=619)

Canadians and permanent residents are more likely to have complained about missing information concerning NEXUS (36%) or declaration, importations and their fees (14%). Non-Canadians were more likely not to have found information about requirements (26%) and other information (45%).

Table 5.15. Time Spent on the CBSA Website According to Visitors Country of Origin

A10A. What information were you looking for that you didn't find? Base: Respondents who ended up not finding the information they were looking for	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	619	444	175
NET NEXUS PROGRAM	27%	36%+	3%-
NET DECLARATION/IMPORTATION AND FEES	11%	14%+	3%-
NET ITEMS ALLOWS AT THE BORDER	9%	8%	10%
NET REQUIREMENTS	9%	2%-	26%+
NET OTHER	35%	31%-	45%+
Don't know / refusal	12%	11%	15%

Slightly less than one third of visitors (29%) used the search bar during their visit to the CBSA website, whereas six out of ten (63%) did not.

Figure 5.21. Answer to question A9: During your visit to the Canada Border Services Agency website, did you use the search bar on the website? Base: All respondents (n=2,729)

2.2.1.3 Traveller Section Visitors Assessment and Satisfaction

The main reason to visit the CBSA website concerns NEXUS and Canpass, more precisely half the visitors (52%) visit the travel Section of the CBSA for that reason. The other main reasons are to help plan upcoming travel (25%), to get information regarding consumer good allowed at the border (18%) or to know requirements for non-Canadians (18%). Others visit to get information regarding border wait times (10%) or office and contact information (8%). (For more detailed results, please refer to Table A.4 of the Appendix C)

Figure 5.22. Answer to question B1: Please indicate, from the list below, the reasons why you have visited the Travellers Section of the Canada Border Services Agency website. SEVERAL MENTIONS ALLOWED* Base: All respondents (n=2,729)

*Because respondents were able to give multiple answers, total mentions may exceed 100%.

Canadians and permanent residents are more interested in everything regarding the NEXUS and the Canpass program (66%) or in getting information regarding consumers goods allowed at the border (19%). On their part, non-Canadians are more likely to have visited the CBSA website to get information on requirements (57%), to plan for upcoming travel (51%) or to find office contact and/or information (11%).

Table 5.17. Reasons to Visit the Travellers Section of the CBSA Website According to Visitors Country of Origin

B1. Please indicate, from the list below, the reasons why you have visited the Travellers Section of the Canada Border Services Agency website. SEVERAL MENTIONS ALLOWED* Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
NET NEXUS & CANPASS	52%	66%+	12%-
To help plan for upcoming travel	25%	15%-	51%+
To get information regarding consumer goods allowed at borders	18%	19%+	15%-
NET REQUIREMENTS	18%	4%-	57%+
To get information regarding border wait times	10%	10%	9%
NET OFFICE CONTACT & INFORMATION	8%	7%-	11%+
To get declaration/import information	2%	2%	1%
Passport or other documents concerns	1%	1%	2%
Other reason	1%	1%	1%
Do not recall	3%	3%	2%

*Because respondents were able to give multiple answers, total mentions may exceed 100%.

Overall, seven out of ten (70%) visitors to the CBSA website are satisfied with it, approximately one out of ten (11%) is neither satisfied nor dissatisfied and a similar proportion (10%) is dissatisfied.

Figure 5.23. Answer to question B2: Please rate your overall level of satisfaction with the Travellers Section of the Canada Border Services Agency website. Base: All respondents (n=2,729)

Non-Canadians (44%) are more likely to be very satisfied with the website. To be noted that they are also more likely to be very satisfied with most of the aspects of the CBSA website.

Table 5.18. Overall Satisfaction with the Travellers Section of the CBSA Website According to Visitors Country of Origin

B2. Please rate your overall level of satisfaction with the Travellers Section of the Canada Border Services Agency website. Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
Net Satisfied	70%	70%	72%
Very Satisfied	41%	40%-	44%+
Somewhat Satisfied	30%	30%	28%
Neither satisfied nor dissatisfied	11%	12%	9%
Net Dissatisfied	10%	10%	11%
Somewhat Dissatisfied	6%	5%	7%
Very dissatisfied	4%	4%	4%
DNK	9%	9%	8%

About seven out of ten visitors to the CBSA website agree that the information available in the Travellers Section is easy to understand (72%), meets their needs (69%) and the amount of time it generally takes to find it is reasonable (71%). About two thirds of them also agree that it is easy to find what they are looking for (65%) and that the information is accurate (67%), up to date (65%) and complete (63%).

Figure 5.24. Answer to question B3: Please indicate the extent to which you agree or disagree with the following statements regarding the Travellers Section of the Canada Border Services Agency website. Base: All respondents (n=2,729)

All in all, most visitors (69%) do not know what could improve the Travellers Section of the CBSA and 15% don’t have any suggestions. Only a few have suggestions concerning information (5%), NEXUS (3%), the website’s design (3%). Other suggestions are to enable contact (2%), to add a FAQ section (1%), to fix non-functional links (1%) or to improve the search engine (1%). (For more detailed results, please refer to Table A.5 of the Appendix C)

Figure 5.25. Answer to question B4: Do you have any suggestions for improving the Travellers Section of the Canada Border Services Agency website? SPONTANEOUS MENTIONS Base: All respondents (n=2,729)

Canadians and permanent residents are more likely to suggest improvements related to NEXUS (4%) or to update the current border wait times more often (1%). Non-Canadians are more likely to have suggestions concerning information (8%).

Table.5.20. Suggestion for Improvements for the Travellers Section of the CBSA Website According to Visitors Country of Origin

B4. Do you have any suggestions for improving the Travellers Section of the Canada Border Services Agency website? SPONTANEOUS MENTIONS Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
NET INFORMATION	5%	4%-	8%+
NET NEXUS	3%	4%+	1%-
NET DESIGN	3%	3%	3%
NET ENABLE CONTACT	2%	2%	2%
Add FAQ section	1%	1%	1%
Update more often the current border wait times / more detailed border waiting time	1%	1%+	0%-
Fix the non-functional links / fix the bugs (website didn't work)	1%	1%	1%
Improve the search engine	1%	1%	1%
Other	1%	1%	1%
No suggestions	15%	15%	16%
Don't know / refusal	69%	69%	68%

Three quarters of the visitors to the website (76%) stated that all the language used on the agency’s website was clear and about one out of ten (11%) mentioned that some words or terms used were not clear to them. Only a minority (2%) admitted that many of the words or terms on the website were not clear.

Figure 5.26. Answer to question B5: Thinking about your visit to the agency's website, which of the following situations best describes your experience? Base: All respondents (n=2,729)

There are no significant differences among Canadians and permanent residents or non-Canadians.

Table 5.21. Understanding of the Language on the Agency’s Website According to Visitors Country of Origin

B5. Thinking about your visit to the agency's website, which of the following situations best describes your experience? Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
All the language used on the agency's website was clear	76%	75%	78%
Some words or terms used on the website were not clear	11%	11%	11%
Many of the words or terms used on the website were not clear	2%	2%	2%
DNK	10%	11%+	8%-

Almost two thirds of respondents for whom many of the words or terms used on the website were not clear (63%) don’t know nor remember which ones were more problematic for them. About one out of ten of them only repeated that definitions or terms were not clear or too technical (13%). Others specify that some sections were not clear, such as the one about what travellers can bring with them (4%) or the NEXUS renewal process (2%).

Figure 5.27. Answer to question B6: You mentioned that you had some problems with terms and words used on the website, mention which ones. SPONTANEOUS MENTIONS Base: Respondents for whom many of the words or terms used on the website were not clear (n=367)

Six respondents out of ten visit media related to travel (62%) to inform themselves about travel. Four out of ten consult their family, friends or colleagues (41%) and government websites (37%). About two out of ten go on social media (22%) or refer to travel suppliers (22%). Other sources of information are less popular, such as newspapers (8%) or search engines (1%).

Figure 5.28. Answer to question B9: Apart from the Travellers Section of the Canada Border Services Agency website, where else do you go for travel information? SEVERAL MENTIONS ALLOWED* Base: All respondents (n=2,729)

*Because respondents were able to give multiple answers, total mentions may exceed 100%.

Canadians and permanent residents are more likely to inform themselves on various media related to travel (63%), on social media (23%) and through travel suppliers (22%). Non-Canadians, on the other hand, are less likely to use these same sources of information.

Table 5.23. Other Sources of Travel Information According to Visitors Country of Origin

B9. Apart from the Travellers Section of the Canada Border Services Agency website, where else do you go for travel information? SEVERAL MENTIONS ALLOWED* Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
NET TRAVEL MEDIA	62%	63%+	58%-
Family/friends/colleagues	41%	42%	38%
NET GOVERNMENT WEBSITES	37%	37%	39%
Social media	22%	23%+	19%-
NET TRAVEL SUPPLIERS	22%	23%+	18%-
Newspapers	8%	8%	7%
Search engines	1%	1%	2%
Other	1%	0%-	2%+
Nothing	1%	1%	1%
Do not know	11%	11%	11%

*Because respondents were able to give multiple answers, total mentions may exceed 100%.

2.2.1.4 Frequency of Internet Use and Preferences

The great majority of the CBSA websites visitors (91%) surf the Internet on a daily basis, 6% do it weekly and only a few (2%) do so less often.

Figure 5.29. Answer to question C1: How often do you surf the Internet? Base: All respondents (n=2,729)

Canadians and permanent residents are more likely to surf the Internet daily (92%) and non-Canadians are more likely to do so weekly (7%) or rarely or exceptionally (2%).

Table 5.24. Frequency of Surfing the Internet According to Visitors Country of Origin

C1. How often do you surf the Internet? Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
On a daily basis	91%	92%+	87%-
On a weekly basis	6%	5%-	7%+
On a monthly basis	1%	1%	1%
Once or twice a year	0%	0%	0%
Rarely or exceptionally	1%	0%-	2%+
DNK	1%	1%	2%

When they visit website, six respondents out of ten (57%) like infographics and half of them (47%) like videos.

Figure 5.30. Answer to question C4: Please indicate how much you like or dislike having the following elements on websites you visit. Base: All respondents (n=2,729)

2.2.2 Online Focus Groups

2.2.2.1 Introduction and Methodology Recap

This document is the summary report of the CBSA online qualitative research with travellers who recently visited the CBSA external website. It is only intended to complement the quantitative visitor evaluation of the CBSA website. Results of the quantitative survey are presented in full in another section of the full report.

This summary report is based on a total of three focus groups held online, using Itracks technology, with one group help in French and two in English. Given they are held online, the geographic representation of participants allowed for seven provinces to the represented, from the Atlantic to BC, but also allowed four American citizens from New Jersey to California to participate in the discussions.

Participants were recruited from the quantitative research (potential of over 2700 respondents), with those who answered that they would volunteer to participate in a follow-up focus group phase. The main criteria to be eligible to participate in the focus group was tied to the fact participants had recently visited the CBSA website, regardless of the reason for their visit.

2.2.2.2 Reason for Last Visit and Arrival on Landing Page

The online focus groups were dominated by two profiles of travellers. The first, and largest, group come to the website for Nexus renewal or application. The second group come to the site to figure out what they can bring back across the border at the end of their stay abroad. As for the focus group participants who are USA residents, the main reason for visiting was regarding rules and regulations about what can be brought into Canada or any limitations on who can enter Canada (one example was the spouse of a person with a DUI charge). Other reasons for visit were about customs charges on items bought or ordered in a foreign country. Finally, one visited the website to consult the GoC safety warnings on a specific country at the time of planning a trip to a country currently in conflict.

It should be noted that while both Nexus first time applicants and renewals were represented in the groups and did reflect on very different experiences during their visit. Even renewal visitors remembered that their first visit at the time of application had not been simple at all.

Most participants arrived on the site via a search engine search. A small group first started with the Canada.ca general website and entered a query there, while a very limited number of more frequent visitors already had bookmarked the CBSA website (that person was a Canadian citizen working on a visa in the United-States. Among those who used Google or another search engine, most entered a direct question rather than a search for the agency name. they tended to be generic questions such as “Where do you apply for Nexus?”, “Can you bring food back into Canada?”, “What is the limit of alcohol I can bring into the country?”

When shown, participants remembered the Traveller landing page, as only some remembered that their search engine query took them one layer down, directly to where they needed to go. One participant looking for Nexus information said they left the Traveller page right away to try and access the American Nexus website directly as he felt the CBSA Traveller page did not make it evident he was “in the right place”.

2.2.2.3 Overall Satisfaction with Last Visit

In general, participants were rather satisfied with their visit to the CBSA website. This “good but not great” rating stems from the fact that government websites in general tend to fall below their expectations when it comes to look and feel, but all said they eventually “got what they were looking for”. In the end, the website delivered the desired result, even if some believed the process was not always easy or that they felt the visit took too much time.

When commenting in general on navigation, look and feel, participants comments remained positive. Their expectations of government websites are that they will be more “wordy” and contain more official language as opposed to plain language, but say that is “understandable”. Topics covered, and the need to be explicit, in part, explains why they believe government websites should not be expected to compete with retail or other commercial websites. As such, they felt the CBSA website was above par on look and feel, and, while some language did not appear intuitive to them, participants agreed they were able to muddle through easily. One American participant said: “Compared to the US government website I visit, this was much better!”

Those who experienced more difficulty remain the first time Nexus applicants. They believed the process was not so easy, as many had to go back or out of the website to try again. As well, they felt the menus were not clear and expected that they would be able to apply and pay directly through the website. Most first-timers did not clearly know the program and the fact they need to apply through a USA government website.

2.2.2.4 General Perceptions of the Website: Navigation and Language

When shown the home page once more, most said they went directly to the bottom menu rather than click “travellers” at the top. Some confusion arose from the home page as most of the page was taken up by the warnings. While they did not specifically read the warnings, they believed it was taken their attention away from their intended task.

Many commented that the language is not quite clear. For example, when asked where they would click to apply for Nexus (among those who had not recently visited for that purpose), some said they did not know while others said: “I guess I would click on save time at the border, right?” Participants wondered why the language was not more explicit and direct. “It should say click here to apply for Nexus or click here to find out more about Nexus.”

Some also felt that the categories could be confusing or did not appear mutually exclusive. Some said that “I would not know if I should click what to expect at the border or bring goods across the border, if I want to know if I can bring fine cheese from France back to Canada”. Many said the language brings about a “best guess” response from the visitor. Several suggested that menus should be “more like what you find on travel websites, in the form of questions: What to do? Where to go?” Some suggested that this approach would simplify their visit because they are coming to the website to answer a question they have: “How much can I bring back? “Can I bring this into Canada?” “How do I apply for Nexus?” “How to renew my Nexus Card?” While some commented that they expect to find “bureaucratic language” on government websites for legal reasons, they also believe that navigation could be done using plain language.

The person who wanted to know the GoC official position on travelling to war-torn country, said she would have likely clicked on “Travel tips” to find the information she was looking for.

American participants said they browsed around the website before seeing the “non-Canadians” clickable link on the homepage. Once they did click this hyperlink, they then felt they could easily find out what they were looking for.

2.2.2.5 Visitors to Canada Perceptions

The American citizens who visited the CBSA website did so mainly to obtain information on what they could expect at the border. Some wanted to know what they could bring into Canada, others wanted to see if a person would be denied entry or not and one was not sure what documents they would need coming into Canada.

Like their Canadian counterparts, they arrived on the site following a search engine query, which took them directly to the home page.

All US citizens clicked on the non-Canadians box on the homepage to initiate their search. All found the information they were looking for during their visit. The menus at the bottom of the “Non-Canadians” page was said to be clear and efficient. All of them had clicked on “Visitors to Canada” to continue their visit and ultimately got the information they wanted.

In general, non-Canadians in the groups were quite satisfied with their visit. Although they felt pages tended to be “too wordy” and “too busy”, they felt their visit was better than expected, compared to US Government sites they have visited in the past.

2.2.2.6 Nexus Application and Renewal

Several issues were raised when it comes to the Nexus application, among those who came to either seek out information about the program or to fill out the application. While those coming to renew felt the process was quite simple, they did recall that their first-time application had also been problematic.

First timers were not sure what to click on the home page to get to the Nexus application page. Some went directly to the search tool at the top, while others said they “guessed” it could be “save time at the border”. When commenting on the home page access, participants wondered why Nexus was not more prominent or why there would not be a menu item for it.

Once on the Nexus page itself, participants hesitated as to where they would click to fill out the application. Their expectations are that this should be a transactional page where they would fill out their application online and that the process would be all done within the GoC environment. While some said they knew that Nexus was a “joint” program with the United-States Government, they had no prior knowledge that at any time during the process they would need to access a US Government site. That did cause problems for several. Some said the process was not transparent or clear and that they “backed out” and went back to Google to find out more about the application process itself. Some said they went back and started the process again. Others stopped and feared accessing the US Government site not knowing that it was “legit” and felt they arrived on that site without knowing that it was normal to land there. As well, some remembered being prompted with third party websites (“private” sites who said they help them fill out the application).

On the Nexus page, participants felt the “renewal” tab was “buried” in a long list and they had missed it when coming to the page. Some then went directly to the US Government website through other means. As such, they believed their visit to the CBSA website was not successful.

While most said they would click on the “join nexus” green button, but some also noticed the Nexus Application link in the top right-hand side and wondered what the difference was between the “two types of applications”. Not a single participant had watched the video, mainly because of its length (3:50) which they believe is “too long”.

Participants believed that at the top of the page, there should be clear “warnings” or “statements” that Canadians would be able to fill out the form online, that if they pursued, they would be redirected to a legitimate US Government website and that the process could not be fully completed online. While they said this information is mostly present, it was either not obvious enough or not visually evident.

Once on the “Join Nexus” page, many first timers still feel that the website is not transparent regarding the process. They feel that the step-by-step menu at the bottom suggest that they can complete the application, submit documents, pay for the fee and schedule their interview right from the CBSA website. Again, some believe that the page should more obviously state something along the lines of that CBSA can only provide information and useful links but that all forms and payment requires that they be redirected to a US Government website.

2.2.2.7 Traveller Information Pages

When asked to find the page related to the legal amounts of alcohol and tobacco a Canadian citizen can bring back across the border, participants said the information should be found if they had clicked on “Bring goods across the border” menu from homepage. But to these participants this choice was still somewhat of a guess. Some felt it could be under “travel tips” or under “what to expect at the border”. Some believed the menu should be spelled in the form of a question: “What can I bring back to Canada?” This question represents what they have in mind when coming to the CBSA website.

Whilst they feel the “Alcohol and Tobacco Limits” page is somewhat wordy and busy, they feel it remains fairly clear. Some suggested that the table on the left-hand side for alcoholic beverages should contain an “OR” between the different rows. For some it is not clear that it is 1.5 litres of wine OR 8.5 litres of beer OR 1.14 litres of liquor.

For most participants, the “I declare” link is something they had not noticed when visiting the website. They believe it is too small and isolated and that “Canadian Citizens” heading of the box does not suggest that this is where they would find information relative to the declaration.

Participants did however like the organisation of the “I Declare” page once they landed on it. The two-sides design with the clear statements “Know before you go” and “Returning to Canada” made it easy to understand where to look. While some commented that drop downs, buttons or tabs would look more modern that clickable links, most felt they could easily get the information they are looking for.

2.2.2.8 Participants’ Suggested Changes

Participants made several suggested changes during the focus group sessions, some of which were introduced in the previous sections. Here is list of suggested changes to the CBSA website:

1. Menus should be changed from the current language to simple questions (i.e. How to apply for Nexus? What can I bring back to Canada?). Participants feel they access the website to answer a question they have and that if the website mirrored these questions, it would provide for a smoother navigation. They feel that the current language can be ambiguous or make some of the menu items not mutually exclusive.
2. Improve the transparency and clarity of the Nexus pages (e.g. forms cannot be filled in the GoC environment, redirection to a US government site is “normal”). The current funnel leaves the impression for many first-timers that they will apply and complete the process right there.
3. Many participants felt they had not noticed the top blue menu items and went to the menus at the bottom of the home page or Nexus page. They felt the blue bar at the top blends into the CBSA logo making the menu difficult to spot
4. Those coming to renew Nexus (yet did not know the address of the US Government site) felt they should have an obvious bottom they could click on a get re-directed right away. They struggled to find their way on the CBSA website and find the needed link. Some said they left CBSA, back to Google to make a different query.

Appendix A – Detailed Research Methodology

A1 Intranet Section

A four-step methodology was developed for the Intranet portion of this public opinion research conducted with employees of the Canada Border Services Agency.

A series of six focus groups with employees with a wide variety of profiles were conducted in several regions across Canada. The objectives of these focus groups were to develop their digital profile, identify their use of Atlas, identify the challenges they faced in their work about the use of digital technologies, their expectations and their needs regarding Atlas's renewal.

The focus groups were followed by a series of fifteen individual interviews with employees from various backgrounds in different regions of Canada. The objective of these interviews was to deepen and validate the focus group results regarding the digital profile of employees, their satisfaction with Atlas and digital tools, their needs and expectations for Atlas renewal. In addition, these interviews were an opportunity to experiment with card sorting to identify an ideal information structure for Atlas based on employees' use of the intranet.

The results of the focus groups and individual interviews were combined to construct segments based on the numerical profile of CBSA employees. A series of six persona were created during the segmentation exercise.

Finally, a tree structure test (or an inverted card sorting) was performed to evaluate, adjust and validate the information structure that was developed during the individual interviews.

A1.2 Six Focus Group

Leger conducted a series of six focus groups with CBSA employees. There were groups of employees recruited from (1) employees in the field, (2) employees from the regional HQs and (3) employees for the national HQ in Ottawa. All sessions were held in CBSA locations. Participants were recruited by CBSA. Each group session lasted approximately 120 minutes. Every session was recorded for analysis purpose. Leger was responsible for preparing the moderation guides and moderating the groups. The guide was developed in consultation with CBSA’s project authority. The groups were moderated in both French and English according to the needs of the employees. The guides and tools were available in both languages.

Participants were informed of all their rights under Canada’s Privacy Act and the Standards for the Conduct of Government of Canada Public Opinion Research. Specifically, their confidentiality was guaranteed, and their participation was voluntary. CBSA was be responsible for ensuring the participation and availability of its employees for scheduled interview date and time. There was no financial incentive to ensure the participation of CBSA employees.

Locations and dates

Groups were held in the following cities on the dates specified.

Table 1. Detailed Recruitment

City	Recruits	Participants	Target	Language	Date
Ottawa	10	8	HQ employees	EN/FR	January 28, 2019
Ottawa	10	7	HQ employees	EN/FR	January 28, 2019
Mississauga	10	9	HQ employees	EN	February 5, 2019
Toronto Pearson Int’ Airport	10	11	BSO	EN	February 5, 2019
Vancouver	10	4	BSO	EN	February 6, 2019
Vancouver	10	11	BSO	EN	February 6, 2019
Total	60	50

The discussion guide is presented later in the Appendix section.

A1.3 One-on-one interviews

Leger conducted one-on-one interviews with participants of each main profile (employees in the field, employees from the regional HQ and employees for the national HQ). Each interview lasted approximately 45 minutes. Every session was recorded for analysis purpose.

Participants were informed of all their rights under Canada’s Privacy Act and the Standards for the Conduct of Government of Canada Public Opinion Research. Specifically, their confidentiality was guaranteed, and their participation was considered voluntary. CBSA was be responsible for ensuring the participation and availability of its employees for scheduled interview date and time. There was no financial incentive to ensure the participation of CBSA employees.

Table 2. Detailed Recruitment

City	Recruits	Participants	Target	Language	Date
Montreal	5	5	HQ employees and BSO	EN/FR	February 6 and February 8, 2019
	5	5	HQ employees and BSO	EN	February 12, 2019
Ottawa	5	5	HQ employees	EN	February 15, 2019
Total	15	15

Leger was responsible for preparing the interview guide, preparing the card sorting exercise, conducting the interviews in English and French. The recruitment guide and card sorting exercise were developed in consultation with CBSA’s project authority.

The card sorting exercise was carried out in two steps: an open card sorting was performed to create the main menu of the intranet and a closed card sorting was performed to organize the intranet content. In the closed card sorting process, we used 32 cards:

1. Pay and Benefits

2. Awards and Recognition

3. Occupational Health and Safety

4. Security

5. Performance Management

6. CBSA Gives / GCWCC (Government of Canada Workplace Charitable Campaign)

7. Well-being

8. Finance Volume

9. Procurement and contracting

10. Staffing

11. Messages from executives

12. Frontline Bulletins and National Document Centre publications

13. Agency Org Chart / About branches, directorates, divisions

14. Labour Relations

15. Training and Learning

16. Integrity, Values and Ethics

17. News and Photo Galleries

18. Event calendar

19. Employee Orientation / Onboarding / New Hires

20. Executive offices

21. Employment Equity and Diversity

22. Form and Template Library

23. Employee Assistance Program

24. Agency and Branch Initiatives and Priorities

25. Accommodations and Facilities (Evacuation Procedures, Boardrooms)

26. Information Management (Apollo, ATIP)

27. Jobs and Career Development

28. Helpdesks

29. Policy Library

30. Border Update (video series)

31. Guide, Manual, and Standard Operating Procedure Library

32. IT Systems (ACROSS, ICES, CAS, ESS)

A1.4 Persona Creation

The segmentation analysis is based on the results of focus groups and individual interviews with CBSA employees. The purpose of this analysis is to form Atlas user groups in such a way that each group is as homogeneous as possible, but also as different as possible from the other groups. This means that we want to bring together segments of the CBSA employee community that have a similar behaviour and relationship with technology.

It should also be kept in mind that each employee is actually a combination of several people. For example, we met with employees who are addressing the challenges identified in each persona. Since the objective is to understand how Atlas employees use Atlas, the analysis is based on key criteria that influence intranet use: the frequency of Atlas use, digital literacy, sources of frustration with the intranet and topics important to them and, of course, their role in the Agency. We have identified six distinct profiles.

Based on the information collected in phases 1 and 2, we have created as set of six personas. The final number of personas has been dictated by the data analysis. The main objectives of personas are to efficiently present and share information related to the intranet users.

Personas document several dimensions, such as:

• Employee role

• Technological profile

• Daily challenges

• Main frustrations with Atlas

• Short bio to give life to the persona

A1.5 Tree testing

In this phase of the research, we evaluated an information architecture with Atlas users. The architecture that was evaluated was designed using the results of the focus groups and the card sorting done during the individual interviews. To perform this tree structure test, we used the Treejack tool of the OptimalWorkshop platform. CBSA employees who visited Atlas between March 12 and March 22 were invited to participate to the test via an open link. The tree testing consisted of ten tasks that employees were required to perform in the proposed information structure. The test was available in both French and English at the respondent's preference.

A total of 1,164 respondents took the test in English and a total of 200 in French. 434 users were frontline employees and 930 were not.

For each task, we measured several indicators, such as:

· Success rate and failure

· The paths

· The final destination

The questionnaire is available in the Appendix section of this research report.

A2 Internet Section

Quantitative Methodology

In order to meet the Internet-related objectives, a methodology in two phases was followed. First, a quantitative methodology consisting of an open-link survey on CBSA’s website was set up. It was followed by a series of online focus groups with CBSA’s website visitors.

A2.1 Open-link Survey on the CBSA Website

This quantitative research was conducted through an online survey, using a Computer Aided Web Interviewing (CAWI) technology. The public consultation was launched by means of an open-link survey questionnaire available on the CBSA website. Any individual who visited the traveller’s section of the site between February 4, 2019 and February 21, 2019 was invited to answer the questionnaire by clicking on the link.

This part of the public consultation generated a significant volume of responses. A total of 2,729 respondents were gathered via the open-link. We should remind the reader that the results of this part of the public consultation should not be interpreted as representing the opinions or attitudes of the Canadian public at large nor representative of the visitors of CBSA’s website. It is a collection of respondents who volunteered to answer the questionnaire. No statistical weighting was performed on this sample.

Since this is a sample of volunteers, no margin of error can be calculated for this portion of the study. Nor can we comment on the participation rate, as we do not know the traffic and the exact volume of visitors to the Agency's website during the period when the open link was active on the web page.

The online survey has given us the opportunity to recruit participants to conduct focus groups with users of the Agency's website. We therefore asked all survey participants if they were open to participate in a second phase of the study. Those who agreed were invited to leave their names and contact information so that we could contact them to complete the recruitment process.

Leger adheres to the most stringent guidelines for quantitative research. The survey instrument was compliant with the Standards of Conduct of Government of Canada Public Opinion Research – Series E – Qualitative and Quantitative Research. The questionnaire was developed by Léger in collaboration with the CBSA research project leaders.

The survey questionnaire is available in Appendix.

The Validity of Responses

By answering an online survey, participants feel they represent an anonymous portion of a targeted group, providing reassurance regarding response confidentiality and validity. Also, the questionnaire was designed with a minimum number of clicks and a reasonable number of questions because experience has shown that after a certain amount of time, respondents lose interest and no longer concern themselves with answering properly.

A computer simulation was conducted prior to the launch of the survey on the website to catch any skip errors between questions and research analysts tested the navigation fluidity from one question to the next. A process of elimination is applied to avoid keeping a questionnaire in which question fatigue ratios are detected at the end of the questionnaire.

The Expertise of a Team Dedicated to Online Surveys

The LegerWeb team includes about ten professionals and technicians who specialize in information technology and e-marketing and who ensure follow-up from 8h00 AM to 7h00 PM every day of the week. Most members of our team come from the fields of telephone polling or face-to-face interviews, and draw on their mastery of the art of the interview.

With its customer support, the LegerWeb team provides panellists with all the technical information and assistance they may need when answering an online survey.

The LegerWeb team works in close collaboration with the research and statistics teams and shares all pertinent information concerning a polling project. Constant communication allows us to proceed very quickly, to detect errors or problems as soon as they arise and to resolve them in record time.

Software developed by the LegerWeb team to conduct online surveys was designed to allow maximum flexibility, efficiency and security when administering the questionnaire. Furthermore, the team performs continuous monitoring of each online survey.

Leger uses the most recent innovations in the field of online surveys and acts in full knowledge of the limits and possibilities of research performed through online surveys.

Finally, it should be specified that Leger strives to develop its expertise in online polling by implementing the same quality standards and criteria from its renowned telephone surveys.

Also, for all our quantitative research projects, Leger has established quality control measures that are identical to the ISO process, in which all stages are verified, allowing us to verify previous stages as well. In practical terms, the quality assurance process is based on the following elements:

· Designating a project manager responsible for final product quality to avoid diluting responsibility internally;

· Scrupulously verifying how well client objectives match the final questionnaire, making sure that each dimension is found in the questionnaire;

· Verifying how each question is formulated, from the perspective of simplicity of expression and the unequivocal meaning of the syntax according to the specific idea to be covered during the interview;

· Verifying the effects of contamination a priori, i.e., that the location of each question in the survey overall does not have undue effects on the following responses (generally by providing information indirectly to respondents, making the sample un-representative);

· Closely verifying the computerized version of the questionnaire with the reference questionnaire approved by the client;

· Before the pre-test, verifying programmed skips in the computerized system;

· Conducting a pre-test, which allows to verify comprehension of questions and concepts, possible ambiguities, and logical question skips, etc.;

· The highly vigilant LegerWeb team detects all questions with problems while on field;

· The use of software prevents data entry errors, non-established skips, etc. Logical validation is therefore done beforehand and not after the fact;

· Coding of open-ended questions is done with a first sample selection of responses on file and by setting up the codes, which are submitted to the client for approval.

We consider that the implementation of all these procedures is a guarantee of optimal quality when conducting online surveys.

Data Cleaning

Upon completion of data collection, Leger’s data analysts and data processing department cleaned the data thoroughly, ensuring that:

· all closed-ended questions were within the allowable or logical range (allowable ranges would be confirmed with the client under all circumstances, whether or not it is obvious from the questionnaire);

· outliers were verified and, if necessary, excluded from the data;

· all skip patterns had been followed correctly;

· the data was complete (except where it is intentional and within client expectations); and

· information was consistent and logical across questions, with no contradictions in the data.

The data was checked and cleaned after the first night of field and at project completion. During analysis, all numbers were double-checked, and any outliers are double-checked to ensure the data has been entered accurately in the first place

As with all research conducted by Leger, contact information was kept entirely confidential and all information that could allow for the identification of participants was removed from the data, in accordance with Canada’s Privacy Act.

A2.2 Online Focus Group

Leger conducted a series of three (3) online focus groups with visitors of the CBSA Website. Theyr were all recruited from the online survey from the previous research phase of the study. All three sessions were held online via the ITracks video chat platform with participants from different regions of Canada or the US. The following table is a summary of the locations, date, profile and number of participants for all the discussion groups.

GROUP	Group profile	Language	Recruited	Participants	Dates and	Time (Eastern time)	Type
GR01	CBSA’s website visitors	FR	10	8	March 7, 2019	4:30 PM	Online
GR02	CBSA’s website visitors	EN	10	8	March 7, 2019	6:00 PM	Online
GR03	CBSA’s website visitors	EN	10	7	March 7, 2019	8:00 PM	Online

Participants were informed of all their rights under Canada’s Privacy Act and the Standards for the Conduct of Government of Canada Public Opinion Research. Specifically, their confidentiality was guaranteed, and their participation was voluntary. Léger was responsible for the recruitment of the participants and of the moderation of the online focus groups.

Léger's professional recruiters ensured the availability and participation of recruits. Léger was responsible for organizing the sessions on Itracks' video chat platform. A financial incentive of $100 per participant was given to all group participants to thank them for taking the time to participate.

Appendix B – Moderation Guide – Focus Groups

Introduction: General Presentation

Duration: 10 minutes

Prior to the start of the session, participants are asked to fill out a quick self-evaluation questionnaire about their digital skills (Questionnaire 1)

PRESENTATION

• Welcome participants

• Moderator introduces himself

• Presentation of Léger

PURPOSE OF THE MEETING

• The purpose of this meeting is to share your insights and discuss the digital environment at CBSA and your needs to facilitate how you work.

• Definition of digital environment: information systems, digital technologies, platforms, software and services used at CBSA.

• The main focus of the discussion will be Atlas (the Intranet), the Wiki and Apollo

DISCUSSION RULES

• Discussion dynamics (length, discussion, going around the table)

• No right or wrong answers

• Importance of giving your personal, spontaneous and honest opinions

• Importance of reacting to the opinions of others

• Importance of taking your turn to speak

PRESENTATION OF FOCUS GROUP ROOM

• Audio recording for subsequent analysis

• Presence of an observer for note taking

• Information is collected only to inform the digital strategy process and improve services

CONFIDENTIAL RESULTS

• Our discussions today will remain confidential at all times.

• The report will make absolutely no mention of your name.

Do you have any question before we begin?

PRESENTATION OF PARTICIPANTS (GO AROUND THE TABLE)

• First name only

• Position

• Service, and/or department

• Number of years working at CBSA

• Full-time or part-time employee (time sheet)

PART 1: The Intranet at Work

Duration: 20 minutes

Warm-up

Let’s start with your general use of technology.

1.1 In that short self-evaluation questionnaire, how did you rate your overall proficiency with digital tools at question 1?

Follow-up questions to aid conversation:

• Devices (e.g., smartphone, tablet)? Which and why?

• Social Media? Which and why?

• Websites? Most visited and why?

• Mobile apps? Most used and why?

• Softwares and platforms (e.g., Google Docs, Office 365)? Which and why?

• Text messages? Why?

Now, let’s discuss how technology is used in your work.

1.2 Outside of Atlas (we will dive into that subject in a minute), can you describe what technologies are used in your work and how?

• Do they improve your work experience? If so, how? (Faster, more enjoyable, more independent)

• Do you think they are used to their full potential? How could they be better used?

• Can you think of other technologies that could be useful in your work?

1.3 Now, let’s focus on your use of technology on a normal weekday/workday. When and what type of technological tools do you daily?

Wait for spontaneous answers, and ask about:

• Devices (e.g., smartphone, tablet)? Which and why?

• Social Media? Which and why?

• CBSA website? How do you use it?

• Mobile apps? Do you use mobile apps for work? Which and why?

• Intranet Atlas

• Wiki

• Apollo

• Text messages? Why?

• Physical spaces (e.g., meeting spaces)? Which and why?

1.4 BSO specific questions:

· How do you get your shift/operational bulletins? Does a shift supervisor print them out? Do you print them out yourselves? Do you view them on Atlas?

· Do you have any time set aside during your shift to read Agency communications and/or visit Atlas?

· Would you access Atlas from a mobile device if you could?

1.5 Now let’s focus on Atlas. Let’s continue discussing a typical day in your life as a CBSA employee. It can be yesterday, today or even how you think tomorrow will be. Just picture your typical day. During this day:

• What are you trying to do? What are your main goals? Which tasks do you need to accomplish? (Access databases, share documents with other employees, get in touch with other employees)

• At what time of your work day do you go to Atlas? Why at that specific time?

• Please write on this second questionnaire, the top five things you need or want to accomplish using Atlas?

• Are some of those things you want to do difficult for you currently? Why?

PART 2: Satisfaction and Usage of Atlas (and other tools)

Duration: 40 minutes

2.1 First of all, how would you rate your familiarity and knowledge of Atlas in general? How would you explain that level of knowledge and familiarity?

2.2 What do you think of CBSA’s Atlas?

• If you had to rate it from 1 to 10, what rating would you give Atlas? Why is that?

• What do you like about Atlas? How does it help you in your work?

• Have you ever had a negative experience with Atlas? What happened?

• Do you always, most often, rarely or never find what you are looking for? What are examples of things you wanted to find on Atlas but could not?

• Any ideas on how Atlas could be improved? What would make it easier for you to find what you are looking for?

2.3 Here is the home page of Atlas. What do you look for first? What should be featured to make sense for you?

• What about the menu items at the top? Are they the right ones? Is it clear what you will find in each menu item if you clicked on it?

• Do you feel that the home page represents you and your role in the Agency?

• Do you think Atlas helps create a sense of belonging to CBSA? Why or why not?

• Do you want to know what is happening in the other regions or branches of the Agency? Why or why not?

• Should Atlas include more info that is not directly related to your job or task? Social events, news stories, kudos for example.

• What about the org structure? Do you refer to it? Useful?

• What do you think the “Archived” banner stands for? What are your assumptions about the contents found on a page with “Archived” at the top?

2.3.1 Here is a list of topics that could be on the Atlas home page (Questionnaire 2). Can you please pick out the top 5 for you?

(GO THROUGH LIST AND OBTAIN SHOW OF HANDS AND ASK WHY IT WAS SELECTED)

2.4 What do you think of the Wiki Tool?

• How often do you use it and what do you use it for? If we look at the total page views the Wiki is very popular. Why is that?

• Do you always trust the information found on the Wiki (always accurate)?

• What do you and do you not like about it?

• How could Atlas be improved so that you would not need to go the Wiki as often?

2.5 What do you think of Apollo?

• How often do you use it to download or print documents?

• What do you and do you knot like like about it?

• Several documents referred to on Atlas need to be downloaded using Apollo. Do you mind doing that? Is that a problem for you?

• When downloading documents from Apollo, do you prefer pdf versions or original document format?

2.6 What do you think of CBSA’s social media? (Skip if time is running out.)

• Which social media do you follow? Why do you follow/not follow them?

• How often do you use them and what do you use them for?

• What do you like and not like about them?

• How could they be improved?

PART 3: Possible Avenues for Change

Duration: 20 minutes

3.1 Would you like it if Atlas had a more social aspect to it? You could add your profile, include bio, pictures, etc.

• What about moderated content like blog posts about programs or changes in legislation? Holiday messages?

3.2 What if the org structure could be modified? What would make it useful as a reference?

3.3 Should newsworthy information across the Agency be made more prominent?

• What about the following ideas for news stories? Would you “like” them? Share them?

o Notable enforcement actions

o Agency participation in volunteer activities

o New technology innovations at CBSA

o Awards and recognition

o OITP ceremonies

o Etc.

3.4 Here is a suggestion for a new look and feel (show project home page). Would this be an improvement from the current Atlas? How so? Beyond look and feel, what’s different/better about it?

3.5 If it could be done very simply, should you be allowed to flag information you know to be incorrect or out of date?

3.6 Would you like to be able to use a feature that would allow you to bookmark certain pages or content for later reference?

3.7 Would a mobile version be helpful? Would you use it? When and why?

3.8 Are images, videos and other forms of visual content important in an Intranet like Atlas?

3.9 What about an alert system that pops up in case of an emergency (or storm specific alert)?

3.10 What about an “Admin desk” where you would find all procedures\forms\templates and other help features all in one place?

3.11 Should we feature summaries of some official policies as quick reference guides? (like the Code of Conduct)

PART 4: Conclusion

Duration: 10 minutes

4.1 Before I let you go, are there any other suggestions or recommendations you have to improve Atlas?

THANK YOU VERY MUCH FOR YOUR PARTICIPATION, THIS INPUT IS VERY IMPORTANT TO THE AGENCY!

END OF GROUP

Appendix C – Interview Guide

Introduction: General Presentation

Duration: 10 minutes

Prior to the start of the session, participants are asked to fill out a quick self-evaluation questionnaire about their digital skills (Questionnaire 1)

PRESENTATION

• Welcome participant

• Moderator introduces himself

• Presentation of Léger

PURPOSE OF THE MEETING

• The purpose of this meeting is to share your insights and discuss the digital environment at CBSA and your needs to facilitate how you work.

• Definition of digital environment: information systems, digital technologies, platforms, software and services used at CBSA.

• The main focus of the discussion will be Atlas (the Intranet), the Wiki and Apollo

DISCUSSION RULES

• Discussion dynamics (length, discussion 45 minutes)

• No right or wrong answers

• Importance of giving your personal, spontaneous and honest opinions

PRESENTATION OF THE DISCUSSION

• Audio recording for subsequent analysis

• Presence of an observer for note taking

• Information is collected only to inform the digital strategy process and improve services

CONFIDENTIAL RESULTS

• Our discussions today will remain confidential at all times.

• The report will make absolutely no mention of your name.

Do you have any question before we begin?

PRESENTATION OF PARTICIPANT

• First name only

• Position

• Service, and/or department

• Number of years working at CBSA

• Full-time or part-time employee (time sheet)

PART 1: The Intranet at Work

Duration: 7 minutes

Warm-up

Let’s discuss how technology is used in your work.

1.1 Outside of Atlas (we will dive into that subject in a minute), can you describe what technologies are used in your work and how?

• Do they improve your work experience? If so, how? (Faster, more enjoyable, more independent)

• Do you think they are used to their full potential? How could they be better used?

• Can you think of other technologies that could be useful in your work?

1.2 Now, let’s focus on your use of technology on a normal weekday/workday. When and what type of technological tools do you daily?

Wait for spontaneous answers, and ask about:

• Devices (e.g., smartphone, tablet)? Which and why?

• Social Media? Which and why?

• CBSA website? How do you use it?

• Mobile apps? Do you use mobile apps for work? Which and why?

• Intranet Atlas

• Wiki

• Apollo

• Text messages? Why?

• Physical spaces (e.g., meeting spaces)? Which and why?

1.3 BSO specific questions:

• How do you get your shift/operational bulletins? Does a shift supervisor print them out? Do you print them out yourselves? Do you view them on Atlas?
• Do you have any time set aside during your shift to read Agency communications and/or visit Atlas?
• Would you access Atlas from a mobile device if you could?

1.4 Now let’s focus on Atlas. Let’s continue discussing a typical day in your life as a CBSA employee. It can be yesterday, today or even how you think tomorrow will be. Just picture your typical day. During this day:

• What are you trying to do? What are your main goals? Which tasks do you need to accomplish? (Access databases, share documents with other employees, get in touch with other employees)

• At what time of your work day do you go to Atlas? Why at that specific time?

• Please write on this second questionnaire, the top five things you need or want to accomplish using Atlas?

• Are some of those things you want to do difficult for you currently? Why?

1.5 What do you think of the Wiki Tool?

• How often do you use it and what do you use it for? If we look at the total page views the Wiki is very popular. Why is that?

• Do you always trust the information found on the Wiki (always accurate)?

• What do you and do you not like about it?

• How could Atlas be improved so that you would not need to go the Wiki as often?

1.6 What do you think of Apollo?

• How often do you use it to download or print documents?

• What do you and do you not like about it?

• Several documents referred to on Atlas need to be downloaded using Apollo. Do you mind doing that? Is that a problem for you?

• When downloading documents from Apollo, do you prefer pdf versions or original document format?

1.7 What do you think of CBSA’s social media? (Skip if time is running out.)

• Which social media do you follow? Why do you follow/not follow them?

• How often do you use them and what do you use them for?

• What do you like and not like about them?

• How could they be improved?

PART 2: Satisfaction and Usage of Atlas (and other tools)

Duration: 8 minutes

2.1 First of all, how would you rate your familiarity and knowledge of Atlas in general? How would you explain that level of knowledge and familiarity? (Answer Questionnaire #2)

2.2 Here is the home page of Atlas (SHOW ATLAS HOMEPAGE). What do you think of CBSA’s Atlas?

• If you had to rate it from 1 to 10, what rating would you give Atlas? Why is that?

• What do you like about Atlas? How does it help you in your work?

• Have you ever had a negative experience with Atlas? What happened?

• Do you always, most often, rarely or never find what you are looking for? What are examples of things you wanted to find on Atlas but could not?

• Do you feel that the home page represents you and your role in the Agency?

• Do you think Atlas helps create a sense of belonging to CBSA? Why or why not?

• Do you want to know what is happening in the other regions or branches of the Agency? Why or why not?

• Should Atlas include more info that is not directly related to your job or task? Social events, news stories, kudos for example.

• What about the org structure? Do you refer to it? Useful?

• What do you think the “Archived” banner stands for? What are your assumptions about the contents found on a page with “Archived” at the top?

2.3 ON THE HOMEPAGE. What do you look for first? What should be featured to make sense for you?

• What about the menu items at the top? Are they the right ones? Is it clear what you will find in each menu item if you clicked on it?

• Any ideas on how Atlas could be improved? What would make it easier for you to find what you are looking for?

2.3.1 On this image of the Atlas homepage, indicate everything that is essential for you by circling it.

In the contrary, mark an X or a cross on everything that you find superficial or useless at work and that you never use.

2.3.2 Here is a list of topics that could be on the Atlas home page (Questionnaire 2). Can you please pick out the top 5 for you?

PART 3: Possible Avenues for Change

Duration: 5 minutes

3.1 What could be done to improve Atlas? How could your Atlas experience at work be improved?

How could we encourage you to go to Atlas more often?

• What about moderated content like blog posts about programs or changes in legislation? Holiday messages?

• Should newsworthy information across the Agency be made more prominent?

• What about the following ideas for news stories? Would you “like” them? Share them?

o Notable enforcement actions

o Agency participation in volunteer activities

o New technology innovations at CBSA

o Awards and recognition

o OITP ceremonies

o Etc.

If it could be done very simply, should you be allowed to flag information you know to be incorrect or out of date?

Would you like to be able to use a feature that would allow you to bookmark certain pages or content for later reference?

Would a mobile version be helpful? Would you use it? When and why?

Are images, videos and other forms of visual content important in an Intranet like Atlas?

What about an alert system that pops up in case of an emergency (or storm specific alert)?

What about an “Admin desk” where you would find all procedures\forms\templates and other help features all in one place?

Should we feature summaries of some official policies as quick reference guides? (like the Code of Conduct)

Ideally, what would you like to see on the Atlas home page.

PART 4: Card Sorting Exercice

Duration: 20 minutes

OPEN-ENDED CARDS SORTING EXPERIMENT

4.1 With the objective of improving Atlas, I would like you to complete exercise #3. Thinking about your daily or weekly use of Atlas, I would ask you to create Atlas menus according to your needs.

I would ask you to create the main and secondary menus of the Intranet. What would Atlas look like?

Note whenever a category could fall into one or more categories - follow up with the respondent on this subject why then?

HYBRID CARDS SORTING EXPERIMENT

4.2 I would like to do this exercise again, but with existing categories. Here are some cards with a lot of items/categories that could make up the Atlas menus. Please repeat the same exercise but with these categories. I also have some free cards (on which you could write). Please create the Atlas menu architecture with these categories. You can add categories (with free cards) and you can leave some aside if you think these categories are useless.

Note whenever a category could fall into one or more categories - follow up with the respondent on this subject why then?

Note whenever the participant states that there are missing categories - follow-up with the respondent, which ones?

Note whenever a participant mentions that the category wording is not adequate - follow-up with the respondent what should be the appropriate wording?

THANK YOU VERY MUCH FOR YOUR PARTICIPATION, THIS INPUT IS VERY IMPORTANT TO THE AGENCY!

END OF GROUP

Appendix D – Exercises – Focus Groups

QUESTIONNAIRE 1

Position : ___________________________________

Branch : ___________________________________

Region : ___________________________________

1. Overall, how would you rate yourself when it comes to your digital skills, using websites, online forms, mobile apps or other software, whether it is at work or for personal reasons?

Very low – I struggle							Very high – I am a Wizz
1	2	3	4	5	6	7	8	9	10
⃝	⃝	⃝	⃝	⃝	⃝	⃝	⃝	⃝	⃝

2. To what extent do you like or dislike having to use online tools AT WORK?

Dislike it a lot							Like it a lot
1	2	3	4	5	6	7	8	9	10
⃝	⃝	⃝	⃝	⃝	⃝	⃝	⃝	⃝	⃝

3. Do you find that Government of Canada (GC) online tools (in general) are easier to navigate or tougher to navigate compared to other online tools you use outside of GC tools?

Much harder							Much easier
1	2	3	4	5	6	7	8	9	10
⃝	⃝	⃝	⃝	⃝	⃝	⃝	⃝	⃝	⃝

4. How often do you use the following:

	Several times a day	Once a day	Couple of times per week	Once a week	Every month or so	Rarely	Never
Apps on your smart phone	⃝	⃝	⃝	⃝	⃝	⃝	⃝
Social media (i.e. Facebook, Instagram)	⃝	⃝	⃝	⃝	⃝	⃝	⃝
Text messaging on your phone	⃝	⃝	⃝	⃝	⃝	⃝	⃝
Transactional websites (i.e. your bank, Amazon)	⃝	⃝	⃝	⃝	⃝	⃝	⃝
The external CBSA website	⃝	⃝	⃝	⃝	⃝	⃝	⃝
Atlas Intranet	⃝	⃝	⃝	⃝	⃝	⃝	⃝
The Wiki on Atlas	⃝	⃝	⃝	⃝	⃝	⃝	⃝

QUESTIONNAIRE 2

1. What is your overall level of satisfaction with Atlas?

Very dissatisfied							Very satisfied
1	2	3	4	5	6	7	8	9	10
⃝	⃝	⃝	⃝	⃝	⃝	⃝	⃝	⃝	⃝

2. Please rate Atlas on each of the following criteria:

	NOT AT ALL	nOT VERY	SOMEWHAT	vERY
eASY TO NAVIGATE	⃝	⃝	⃝	⃝
eASY TO FIND WHAT YOU ARE LOOKING FOR	⃝	⃝	⃝	⃝
nICE LOOK AND FEEL	⃝	⃝	⃝	⃝
rEPRESENTS ME AS A cbsa EMPLOYEE	⃝	⃝	⃝	⃝
uSEFUL IN MY WORK	⃝	⃝	⃝	⃝
lEVEL OF COMFORT THAT IT KEEPS ME UP TO DATE	⃝	⃝	⃝	⃝
iNFORMATION FOUND IS CLEAR	⃝	⃝	⃝	⃝

3. Here is a list of topics that are or could be on the Atlas home page. Please select the top 5 items you feel need to be prominent on the home page for you personally. Check five boxes to indicate priority for you.

	Priority 1	Priority 2	Priority 3	Priority 4	Priority 5
Frontline bulletins
Most requested tasks
Updates on agency initiatives
Updates on branch initiatives
Updates on region initiatives
Messages from executives
Regional news
News from other regions
Upcoming events
Border update (weekly video)
Atlas asks (poll)
Website updates
Kudos
Suggestion:
Suggestion:
Suggestion:

Questionnaire 3

If you had to redesign the menu options on Atlas, what would it look like?

Appendix E – Tree Testing Questionnaire

Tree Testing Questionnaire

1. You will be asked to find a certain item and presented with a list of links.

2. Click through the list until you arrive at one that you think helps you complete the task.

3. If you take a wrong turn, you can go back by clicking one of the links above.

This is not a test of your ability, there are no right or wrong answers.

That's it, let's get started!

Screening

Would you prefer to complete the survey in English or French?
Préféreriez-vous répondre à ce questionnaire en anglais ou en français?

English

French

Profile

Are you a front-line employee or not?

Yes, I am a front-line employee

No, I am not a front-line employee

Task 1. Pay and benefits

You would like to get information on the public-service health care system.

Task2. Training and Learning

You want to learn new ways of doing things and develop your skills and abilities to improve yourself in your professional field.

Task3. Standard Procedures

You wish to review the passenger and crew screening procedures for marine vessels.

Task4. Executives Offices

You are looking for the bio and professional information of your Vice President of your branch.

Task5.Procurement

You are looking to issue a contract for an external supplier

Task6. Collective Agreement

You are seeking information about the employer's accommodations for people with disabilities

Task7. IT Service if travelling abroad

You want to find out the telephone number of the National IT Service Desk

Task8. Videos Series Boarder Update from Previous Weeks

You would like to view the weekly video news series

Task9. Apollo

You want to access Apollo

Task10. Frontline Bulletins

You want to find the Operational Bulletin PRG-2019-09 -- New Commercial Allegation in the Integrated Customs Enforcement System (ICES) for Providing Inaccurate Information

Appendix F – Survey Questionnaire

Canadian Border Services Agency

Website Visitor

Final Questionnaire

INT01 - INT01 - ASK ALL

Thank you for taking the time to participate in this study. Canada Border Services Agency is very interested in your comments as a visitor to our website. Your input will be very useful to the agency as it is constantly looking to improve its website.

Completing this questionnaire should not take more than 10 minutes of your time. The programming of this survey implies that you must complete the questionnaire in one go. You will not be able to pick up the questionnaire from where you left off at a later time. Once you press the "Next" button at the very end of the survey, your answers will have been recorded and you will have completed the survey.

"I agree to answer the following survey questions truthfully and thoughtfully."

Yes (1)

No (NE)

INFOSECTA - INFOSECTA - SHOW ALL

The following section focuses specifically on your visit to the Canada Border Services Agency website.

QA1 - QA1 - ASK ALL

How did you arrive at the Canada Border Services Agency website?

I searched for the agency by name 'Canada Border Services Agency' using an external search provider (Google, Bing, and so on.). (1)

I used a search engine on another Government of Canada website. (2)

I navigated directly to the website by typing the address into the address bar. (3)

I searched for the subject I was looking for (for example: Nexus program, wait time at the border, consumer goods allowed at borders, and so on.), using an external search provider (Google, Bing, and so on.). (4)

I have landed on the agency's website by mistake. It was not what I was looking for. (97)

Other, please specify (96)____________

QA2 - QA2 - ASK ALL

How often do you visit the Canada Border Services Agency website?

On a daily basis (1)

On a weekly basis (2)

On a monthly basis (3)

Once or twice a year (4)

Rarely or exceptionally (5)

Do not know (98)

QA3 - QA3 - ASK ALL

When was the last time you visited the Canada Border Services Agency website?

In the past week (1)

Within the past month (2)

Between 1 and 3 months ago (3)

Between 3 and 6 months ago (4)

Between 6 and 12 months ago (5)

More than 12 months ago (6)

Never (7)

Do not know (98)

QA4 - QA4 - ASK ALL

What device do you use to navigate on the Canada Border Services Agency website?

(A)

Desktop computer (1)

Laptop (2)

(B)

iPhone (3)

Blackberry (4)

Other smartphone (5)

(C)

iPad Tablet (6)

Android Tablet (7)

Other, please specify (96)____________

QA5 - QA5 - ASK ALL EXCEPT IF Q3= 7 ,8

Have you had, or have you ever had any problems accessing the Canada Border Services Agency website?

Yes (1)

No (2)

Do not know (8)

Q5A - Q5A - ASK IF Q5=1

What was the nature of the problem?

Please specify:

Do not know (98)

QA6 - QA6

How much time did you spend on the Canada Border Services Agency website during your visit?

Please do not include the time it takes to load or connect to the website.

Less than 2 minutes (1)

2 to 5 minutes (2)

6 to 10 minutes (3)

11 to 15 minutes (4)

More than 15 minutes (5)

Do not know (98)

QA7 - QA7 - ASK ALL

Did the page you landed on contain the information that you were looking for?

Yes (1)

No (2)

Do not know (8)

QA8 - QA8 - ASK IF QA7=1

Was it easy to find information on the page you landed on?

Yes (1)

No (2)

Do not know (8)

QA8A - QA8A - ASK IF Q8=2

What was difficult about finding information on the page?

Please specify:

Do not know (98)

QA10 - QA10 - ASK IF QA7=2,8

You mentioned that you did not find the information you were looking for on the page you landed on.

Did you end up finding the information you were looking for on other pages of the Canada Border Services Agency website?

Yes (1)

No (2)

Do not know (8)

QA10A - QA10A - ASK IF QA10=2, 8

What information were you looking for that you didn't find?

Please specify:

I prefer not to answer (98)

QA9 - QA9 - ASK ALL

During your visit to the Canada Border Services Agency website, did you use the search bar on the website?

Yes (1)

No (2)

Do not know (8)

INFOSECTB - INFOSECTB - SHOW ALL

The next section focuses specifically on the Travellers Section of the Canada Border Services Agency website.

QB1 - QB1 - ASK ALL

Please indicate, from the list below, the reasons why you have visited the Travellers Section of the Canada Border Services Agency website.

(A)

To get information regarding border wait times (1)

To get information regarding consumer goods allowed at borders (2)

To get information on entry requirement for non-Canadians (3)

(B)

To get information regarding the NEXUS program (4)

To register for the NEXUS program (5)

To get information regarding the CANPASS programs (6)

(C)

Find location of CBSA’s offices (7)

Find border and/or contact information (8)

(D)

To help plan for upcoming travel (9)

Other reason, please specify (96)____________

Do not recall (98)

QB2 - QB2 - ASK ALL

Please rate your overall level of satisfaction with the Travellers Section of the Canada Border Services Agency website .

Very Satisfied (5)

Somewhat Satisfied (4)

Neither satisfied nor dissatisfied (3)

Somewhat Dissatisfied (2)

Very dissatisfied (1)

Do not know (8)

QB3 - QB3

Please indicate the extent to which you agree or disagree with the following statements regarding the Travellers Section of the Canada Border Services Agency website .

Totally Agree (5) Agree (4) Neither Agree nor Disagree (3) Disagree (2) Totally Disagree (1) Do not know (8)

It is easy to find what I am looking for. (A)

The information is up to date. (B)

The information is accurate. (C)

The information is complete. (D)

The information available in the Travellers Section is easy to understand. (E)

The amount of time it generally takes to find information is reasonable. (F)

The website meets my needs. (G)

QB4 - QB4 - ASK

Do you have any suggestions for improving the Travellers Section of the Canada Border Services Agency website ?

Please specify:

Do not know (98)

QB5 - QB5

Thinking about your visit to the agency's website, which of the following situations best describes your experience?

All the language used on the agency's website was clear (1)

Some words or terms used on the website were not clear (2)

Many of the words or terms used on the website were not clear (3)

Do not know (8)

QB6 - QB6 - ASK IF QB5 = 2,3

You mentioned that you had some problems with terms and words used on the website, mention which ones.

You can mention several words or terms. Separate each of them with a /.

Please specify:

Do not know (98)

QB7 - QB7 - ASK ALL

Are you currently planning to travel somewhere?

Yes (1)

No (2)

Do not know (8)

QB8 - QB8 - ASK IF QB7 =1

How do you intend to travel?

Select all that apply

Land (car, bus, motorcycle, other motor vehicle, train) (1)

Air (by plane or helicopter) (2)

Sea (boat, cruise ship) (3)

Do not know (4)

QB9 - QB9 - ASK ALL

Apart from the Travellers Section of the Canada Border Services Agency website , where else do you go for travel information?

Select all that apply

Travel.gc.ca (1)

Government or customs websites in other countries (2)

Private websites related to travel (3)

Travel agencies (4)

Social media (5)

Mobile travel application(s) (6)

Tourist brochures/brochures (7)

Travel/vacation guides (8)

Specialized magazines (9)

Newspapers (10)

Family/friends/colleagues (11)

Other (please specify) (96)____________

Do not know (98)

QB10 - QB10 - ASK ALL

How often do you cross the Canadian borders?

Almost every day (1)

Every week (2)

Every month (3)

A few times a year (4)

Rarely (5)

I have never crossed Canadian borders (6)

Do not know (8)

INFOSECTC - INFOSECTC - SHOW ALL

The next section focuses specifically on your general Internet usage and preferences.

QC1 - QC1 - ASK ALL

How often do you surf the Internet?

On a daily basis (1)

On a weekly basis (2)

On a monthly basis (3)

Once or twice a year (4)

Rarely or exceptionally (5)

Do not know (8)

QC4 - QC4 - ASK ALL

Please indicate how much you like or dislike having the following elements on websites you visit.

Likes a lot (5) Likes a bit (4) Neither likes nor dislikes (3) Does not like much (2) Does not like at all (1) Do not know (8)

Videos (A)

Infographics (C)

INFOSECTD - INFOSECTD - SHOW ALL

The next section focuses on your profile and will be used to categorize your responses.

QD1 - QD1 - ASK ALL

Are you a Canadian citizen, a permanent resident of Canada or a citizen of another country?

Canadian citizen (1)

Permanent resident of Canada (2)

Citizen of another country (3)

QD2 - QD2 - ASK IF QD1 =1,2

In which province or territory do you live?

Newfoundland and Labrador (01)

Prince Edward Island (02)

Nova Scotia (03)

New Brunswick (04)

Quebec (05)

Ontario (06)

Manitoba (07)

Saskatchewan (08)

Alberta (09)

British Columbia (10)

Nunavut (11)

Northwest Territories (12)

Yukon (13)

I am currently living in another country (14)

QD3 - QD3 - ASK IF QD1 =3

In which country do you live?

Afghanistan (1)

South Africa (2)

Åland Islands (3)

Albania (4)

Algeria (5)

Germany (6)

Andorra (7)

Angola (8)

Anguilla (9)

Antarctica (10)

Antigua and Barbuda (11)

Saudi Arabia (12)

Argentina (13)

Armenia (14)

Aruba (15)

Australia (16)

Austria (17)

Azerbaijan (18)

Bahamas (19)

Bahrain (20)

Bangladesh (21)

Barbados (22)

Belarus (23)

Belgium (24)

Belize (25)

Benin (26)

Bermuda (27)

Bhutan (28)

Bolivia (29)

Bonaire, Sint Eustatius and Saba (30)

Bosnia and Herzegovina (31)

Botswana (32)

Bouvet Island (33)

Brazil (34)

Brunei Darussalam (35)

Bulgaria (36)

Burkina Faso (37)

Burundi (38)

Cayman Islands (39)

Cambodia (40)

Cameroon (41)

Canada (42)

Cape Verde (43)

Central African Republic (44)

Chile (45)

China (46)

Christmas Island (47)

Cyprus (48)

Cocos (Keeling) Islands (49)

Colombia (50)

Comoros (51)

Congo-Brazzaville (52)

Congo-Kinshasa (53)

Cook Islands (54)

South Korea (55)

North Korea (56)

Costa Rica (57)

Côte d'Ivoire (58)

Croatia (59)

Cuba (60)

Curaçao (61)

Denmark (62)

Djibouti (63)

Dominican Republic (64)

Dominica (65)

Egypt (66)

El Salvador (67)

United Arab Emirates (68)

Ecuador (69)

Eritrea (70)

Spain (71)

Estonia (72)

United States (73)

Ethiopia (74)

Falkland Islands (Malvinas) (75)

Faroe Islands (76)

Fiji (77)

Finland (78)

France (79)

Gabon (80)

Gambia (81)

Georgia (82)

South Georgia and the South Sandwich Islands (83)

Ghana (84)

Gibraltar (85)

Greece (86)

Grenada (87)

Greenland (88)

Guadeloupe (89)

Guam (90)

Guatemala (91)

Guernsey (92)

Guinea (93)

Equatorial Guinea (94)

Guinea-Bissau (95)

Guyana (96)

French Guiana (97)

Haiti (98)

Heard Island and McDonald Islands (99)

Honduras (100)

Hong Kong (101)

Hungary (102)

Isle of Man (103)

United States Minor Outlying Islands (104)

Pacific Islands (105)

British Virgin Islands (106)

U.S. Virgin Islands (107)

India (108)

Indonesia (109)

Iran (110)

Iraq (111)

Ireland (112)

Iceland (113)

Israel (114)

Italy (115)

Jamaica (116)

Japan (117)

Jersey (118)

Jordan (119)

Kazakhstan (120)

Kenya (121)

Kyrgyzstan (122)

Kiribati (123)

Kosovo (124)

Kuwait (125)

Laos (126)

Lesotho (127)

Latvia (128)

Lebanon (129)

Liberia (130)

Libya (131)

Liechtenstein (132)

Lithuania (133)

Luxembourg (134)

Macao (135)

Macedonia (FYROM) (136)

Madagascar (137)

Malaysia (138)

Malawi (139)

Maldives (140)

Mali (141)

Malta (142)

Northern Mariana Islands (143)

Morocco (144)

Marshall Islands (145)

Martinique (146)

Mauritius (147)

Mauritania (148)

Mayotte (149)

Mexico (150)

Federated States of Micronesia (151)

Moldova (152)

Monaco (153)

Mongolia (154)

Montenegro (155)

Montserrat (156)

Mozambique (157)

Myanmar (158)

Namibia (159)

Nauru (160)

Nepal (161)

Nicaragua (162)

Niger (163)

Nigeria (164)

Niue (165)

Norfolk Island (166)

Norway (167)

New Caledonia (168)

New Zealand (169)

British Indian Ocean Territory (170)

Oman (171)

Uganda (172)

Uzbekistan (173)

Pakistan (174)

Palau (175)

State of Palestine (176)

Panama (177)

Papua New Guinea (178)

Paraguay (179)

Netherlands (180)

Peru (181)

Philippines (182)

Pitcairn (183)

Poland (184)

French Polynesia (185)

Puerto Rico (186)

Portugal (187)

Qatar (188)

Réunion (189)

Romania (190)

United Kingdom (191)

Russia (192)

Rwanda (193)

Western Sahara (194)

Saint Barthélemy (195)

Saint Kitts and Nevis (196)

San Marino (197)

Saint Martin (French part) (198)

Sint Maarten (Dutch part) (199)

Saint Pierre and Miquelon (200)

Vatican City State (201)

Saint Vincent and the Grenadines (202)

Saint Helena, Ascension and Tristan da Cunha (203)

Saint Lucia (204)

Solomon Islands (205)

Samoa (206)

American Samoa (207)

Sao Tome and Principe (208)

Senegal (209)

Serbia (210)

Seychelles (211)

Sierra Leone (212)

Singapore (213)

Slovakia (214)

Slovenia (215)

Somalia (216)

Sudan (217)

South Sudan (218)

Sri Lanka (219)

Sweden (220)

Switzerland (221)

Suriname (222)

Svalbard and Jan Mayen (223)

Swaziland (224)

Syria (225)

Tajikistan (226)

Taiwan (227)

Tanzania (228)

Chad (229)

Czech Republic (230)

French Southern Territories (231)

Thailand (232)

Timor-Leste (233)

Togo (234)

Tokelau (235)

Tonga (236)

Trinidad and Tobago (237)

Tunisia (238)

Turkmenistan (239)

Turks and Caicos Islands (240)

Turkey (241)

Tuvalu (242)

Ukraine (243)

Uruguay (244)

Vanuatu (245)

Venezuela (246)

Viet Nam (247)

Wallis and Futuna (248)

Yemen (249)

Zambia (250)

Zimbabwe (251)

Other (996)

Please specify:

QD4 - QD4 - ASK ALL

Did you access the Canada Border Services Agency website from within Canada or from outside Canada?

Within Canada (1)

Outside Canada (2)

Do not know (8)

QD5 - QD5 - ASK ALL

What is the language you first learned at home during your childhood and that you still understand?

French (1)

English (2)

Other (3)

I prefer not to answer (9)

QD6 - QD6 - ASK ALL

In the coming weeks, a second phase of this study will be conducted by Léger, an independent Canadian research firm.

Would you like to participate in the second phase of this study?

Yes (1)

No (2)

Resp_infos - Resp_infos1

Please complete the following information to participate in the second phase of the study. An employee of Léger, the research firm which was commissioned by the Canada Border Services Agency to conduct this study, will contact you for this purpose.

Please verify your details:

Title (Mr, Mrs, Ms) (TITRE)	______________________________
First name (PRENOM)	______________________________
Last name (NOM)	______________________________
Telephone number (TELE1)	______________________________
Telephone extension number (POST)	______________________________
Email address (COUR)	______________________________
Email address confirmation (COUR2)	______________________________

QD6P - QD6P - ASK IF QD6 =1

Country of residence:

Afghanistan (1)

South Africa (2)

Åland Islands (3)

Albania (4)

Algeria (5)

Germany (6)

Andorra (7)

Angola (8)

Anguilla (9)

Antarctica (10)

Antigua and Barbuda (11)

Saudi Arabia (12)

Argentina (13)

Armenia (14)

Aruba (15)

Australia (16)

Austria (17)

Azerbaijan (18)

Bahamas (19)

Bahrain (20)

Bangladesh (21)

Barbados (22)

Belarus (23)

Belgium (24)

Belize (25)

Benin (26)

Bermuda (27)

Bhutan (28)

Bolivia (29)

Bonaire, Sint Eustatius and Saba (30)

Bosnia and Herzegovina (31)

Botswana (32)

Bouvet Island (33)

Brazil (34)

Brunei Darussalam (35)

Bulgaria (36)

Burkina Faso (37)

Burundi (38)

Cayman Islands (39)

Cambodia (40)

Cameroon (41)

Canada (42)

Cape Verde (43)

Central African Republic (44)

Chile (45)

China (46)

Christmas Island (47)

Cyprus (48)

Cocos (Keeling) Islands (49)

Colombia (50)

Comoros (51)

Congo-Brazzaville (52)

Congo-Kinshasa (53)

Cook Islands (54)

South Korea (55)

North Korea (56)

Costa Rica (57)

Côte d'Ivoire (58)

Croatia (59)

Cuba (60)

Curaçao (61)

Denmark (62)

Djibouti (63)

Dominican Republic (64)

Dominica (65)

Egypt (66)

El Salvador (67)

United Arab Emirates (68)

Ecuador (69)

Eritrea (70)

Spain (71)

Estonia (72)

United States (73)

Ethiopia (74)

Falkland Islands (Malvinas) (75)

Faroe Islands (76)

Fiji (77)

Finland (78)

France (79)

Gabon (80)

Gambia (81)

Georgia (82)

South Georgia and the South Sandwich Islands (83)

Ghana (84)

Gibraltar (85)

Greece (86)

Grenada (87)

Greenland (88)

Guadeloupe (89)

Guam (90)

Guatemala (91)

Guernsey (92)

Guinea (93)

Equatorial Guinea (94)

Guinea-Bissau (95)

Guyana (96)

French Guiana (97)

Haiti (98)

Heard Island and McDonald Islands (99)

Honduras (100)

Hong Kong (101)

Hungary (102)

Isle of Man (103)

United States Minor Outlying Islands (104)

Pacific Islands (105)

British Virgin Islands (106)

U.S. Virgin Islands (107)

India (108)

Indonesia (109)

Iran (110)

Iraq (111)

Ireland (112)

Iceland (113)

Israel (114)

Italy (115)

Jamaica (116)

Japan (117)

Jersey (118)

Jordan (119)

Kazakhstan (120)

Kenya (121)

Kyrgyzstan (122)

Kiribati (123)

Kosovo (124)

Kuwait (125)

Laos (126)

Lesotho (127)

Latvia (128)

Lebanon (129)

Liberia (130)

Libya (131)

Liechtenstein (132)

Lithuania (133)

Luxembourg (134)

Macao (135)

Macedonia (FYROM) (136)

Madagascar (137)

Malaysia (138)

Malawi (139)

Maldives (140)

Mali (141)

Malta (142)

Northern Mariana Islands (143)

Morocco (144)

Marshall Islands (145)

Martinique (146)

Mauritius (147)

Mauritania (148)

Mayotte (149)

Mexico (150)

Federated States of Micronesia (151)

Moldova (152)

Monaco (153)

Mongolia (154)

Montenegro (155)

Montserrat (156)

Mozambique (157)

Myanmar (158)

Namibia (159)

Nauru (160)

Nepal (161)

Nicaragua (162)

Niger (163)

Nigeria (164)

Niue (165)

Norfolk Island (166)

Norway (167)

New Caledonia (168)

New Zealand (169)

British Indian Ocean Territory (170)

Oman (171)

Uganda (172)

Uzbekistan (173)

Pakistan (174)

Palau (175)

State of Palestine (176)

Panama (177)

Papua New Guinea (178)

Paraguay (179)

Netherlands (180)

Peru (181)

Philippines (182)

Pitcairn (183)

Poland (184)

French Polynesia (185)

Puerto Rico (186)

Portugal (187)

Qatar (188)

Réunion (189)

Romania (190)

United Kingdom (191)

Russia (192)

Rwanda (193)

Western Sahara (194)

Saint Barthélemy (195)

Saint Kitts and Nevis (196)

San Marino (197)

Saint Martin (French part) (198)

Sint Maarten (Dutch part) (199)

Saint Pierre and Miquelon (200)

Vatican City State (201)

Saint Vincent and the Grenadines (202)

Saint Helena, Ascension and Tristan da Cunha (203)

Saint Lucia (204)

Solomon Islands (205)

Samoa (206)

American Samoa (207)

Sao Tome and Principe (208)

Senegal (209)

Serbia (210)

Seychelles (211)

Sierra Leone (212)

Singapore (213)

Slovakia (214)

Slovenia (215)

Somalia (216)

Sudan (217)

South Sudan (218)

Sri Lanka (219)

Sweden (220)

Switzerland (221)

Suriname (222)

Svalbard and Jan Mayen (223)

Swaziland (224)

Syria (225)

Tajikistan (226)

Taiwan (227)

Tanzania (228)

Chad (229)

Czech Republic (230)

French Southern Territories (231)

Thailand (232)

Timor-Leste (233)

Togo (234)

Tokelau (235)

Tonga (236)

Trinidad and Tobago (237)

Tunisia (238)

Turkmenistan (239)

Turks and Caicos Islands (240)

Turkey (241)

Tuvalu (242)

Ukraine (243)

Uruguay (244)

Vanuatu (245)

Venezuela (246)

Viet Nam (247)

Wallis and Futuna (248)

Yemen (249)

Zambia (250)

Zimbabwe (251)

Other (996)

Please specify:

MessComplete - MessComplete

This concludes the survey. Thank you very much for your participation.

Appendix G – Online Focus Groups Guide

Introduction: General Presentation

Duration: 10 minutes

· Introduce moderator and welcome participants to the focus group.

· As we indicated during the recruiting process, we are conducting focus group discussions on behalf of the Government of Canada (Canada Border Service Agency)).

· The focus of tonight’s discussion will be to obtain your opinion on CBSA’s website.

· Recently, you completed a survey on the Canada Border Services Agency website and volunteered to participate in this second portion of the study. So, you are here to provide feedback to the government on the website that will be redesigned.

· The discussion will last approximately 90 minutes.

Explanation

· Audio-and videotaping – The session is being audio videotaped for research purposes, in case we need to double-check the discussion against our notes. These videotapes remain in our possession and will not be released to anyone without written consent from all participants.

· Another thing that I would like to point out is one aspect of the online focus group. This aspect of the platform to allow those who are working on the project to hear what participants have to say, in a way that won’t disrupt a group discussion. So, there are some people who have worked on the project currently listening to the discussion. They are very interested in what you have to say.

· It is also important for you to know that your responses today will in no way affect your dealings with the Government of Canada.

· Confidentiality – Please note that anything you say during these groups will be held in the strictest confidence. We do not attribute comments to specific people. Our report summarizes the findings from the groups but does not mention anyone by name. The report will be available through Library and Archives Canada.

· The incentive will be sent to you at the end of the focus group.

Describe how a discussion group functions:

· Discussion groups are designed to encourage an open and honest discussion. My role as a moderator is to guide the discussion and encourage everyone to participate. Another function of the moderator is to ensure that the discussion stays on topic and on time.

· Your role is to answer questions and voice your opinions. We are looking for all opinions in a focus group, so don't hold back if you have a comment even if you feel your opinion may be different from others in the group. There may or may not be others who share your point of view. Everyone's opinion is important and should be respected.

· I would also like to stress that there are no wrong answers. We are simply looking for your opinions and attitudes. This is not a test of your knowledge. We did not expect you to do anything in preparation for this group.

· It is also important that you talk loud enough for everyone to hear and that you talk one at a time so I can follow the discussion.

Please note that I am not an employee of the Government of Canada and may not be able to answer all of your questions. I have no personal interest in the topic we are going to discuss – so feel free to speak your opinion.

· Moderator introduces herself/himself. Participants should introduce themselves, using their first names only.

· What TV show are you currently watching? Or what was the last great movie you watched?

PART 1: Digital Profile

Duration: 15 minutes

First, let’s discuss how you feel about your digital skills.

1.1 First, how would you rate your digital skills? Would you say they are poor, okay, good or excellent? By digital skills, I mean your ability to successfully and effectively use technology at your disposal.

• Where do you think your digital skills are most lacking?

• On the other hand, where are your digital skills the strongest?

1.2 How often do you surf the internet: many time per day, daily basis, weekly basis, less than a weekly basis?

1.3 Do you think your digital skills are sufficient to effectively use the technologies and interactive tools to accomplish everything you want to do on the Internet?

Now, let’s discuss the CBSA’s website.

1.4 The last time you visited the CBSA’s website

• Was it the first time you visited the website?

• How often have you visited the CBSA’s website?

• Did you find it easy to navigate?

• Did you encounter difficulties using them? Which ones? Explain.

• What were you looking for?

• Is this information easy to find on the webpages?

• If you had to think of one word to describe your visit on the CBSA’s webpage, what would it be?

PART 2: Assessment Traveller Section

Duration: 85 minutes

I will now show you some screenshots. These are screenshots of some pages of the website. We would like to hear your opinion on these.

A- The Traveller's Homepage

2.1 Travellers Homepage

The moderator introduces the Travellers Homepage. Participants are invited to look at the different elements/items of the pages.

• What stands out the most for you on the Traveller Homepage?

• Is the information displayed clearly enough? Is there anything on the page that is not clear?

• Do you understand all the information displayed or not? Which ones?

• Would you change anything to the homepage?

• Is there any piece of information that seems to be missing from the homepage?

• What could be done to improve it?

NEXUS

How many of you have come to look for information about Nexus or to apply/renew their Nexus?

For those who have not sought information / join or renew their Nexus. If you had to join or renew Nexus where would you click?

Why? Why?

What would you expect once you clicked?

The moderator introduces the NEXUS page. Participants are invited to look at the different elements/items of the pages.

Once you land on that NEXUS page, what would you do next?

Where would you click? Why?

What would you expect to happen?

For those who have been looking for information / joining or renewing their Nexus. How did you proceed? Was it simple? Was everything clear?

How would you describe the process on the website?

Would you improve things? What would you do differently?

• What stands out the most for you on the NEXUS page?

• Is the information displayed clearly enough? Is there anything on the page that is not clear?

• Do you understand all the information displayed or not? Which ones?

• Would you change anything to the NEXUS page?

• Is there any piece of information that seems to be missing from the NEXUS Homepage?

• What could be done to improve it?

• Do you like the way the information in this section is presented?

• Any of you have looked at the video?

Tobacco and Alcohol Limits

Did some of you come to the website to find out the limits of products that can be brought back to Canada, such as Alcohol and tobacco products?

For the others, if you had to search for information on alcohol and tobacco limits, under which tab would you click? Why? What would you expect?

For those who sought this information, was the information easy to find?

The moderator introduces the tobacco and alcohol exemption limit pages to the participants by showing the screenshot to the participants. Participants are invited to look at the different elements/items of the pages.

• What stands out the most for you on these pages?

• Is the information displayed clearly enough? Is there anything on these pages that is not clear?

• Do you understand all the information displayed or not? Which ones can cause a problem of understanding?

• Is there any piece of information that seems to be missing?

• Would you change anything to this page? What could be done to improve it?

RETURN TO THE TRAVELLER'S HOMEPAGE

Review the information on the homepage by major sections:

• The Notices (Cannabis, Surtax, African Swine Fever)
• Canadians and permanent residents and Non-Canadians
• Expand All, Collapse All
• Menu “Plan your trip across the border; Travel Tips, What to expect at the border, Save time at the border, Bring goods across the border.

2.2 For each section of the Travellers' Homepage?

• Is the information clear and easy to understand/find?

• Are there any elements that attract your attention?

• Do you like the way the information in this section is presented?

• Is there any piece of information that seems to be missing?

• How could it be improved?

• “What would you expect if you click on each one of them?”

• What information will be found under each topic.

B- I Declare Page

The moderator introduces the I declare page to the participants by showing the screenshot to the participants. Participants are invited to look at the different elements/items of the pages.

2.3 I declare, a guide for residents returning to Canada.

Please take a moment to look at the following page entitled: I declare, a guide for residents returning to Canada.

• What stands out the most for you on these pages?

• Is the information displayed clearly enough? Is there anything on these pages that is not clear?

• Is it useful?

• Do you understand all the information displayed or not? Which ones can cause a problem of understanding?

• Is there any piece of information that seems to be missing?

• Would you change anything to this page? What could be done to improve it?

C- The Non-Canadians page

2.4 Non-Canadians page

Please take a moment to look at the following page entitled: Non-Canadians

Some of you are not in Canada, but in the United States. Have you visited these pages?

What did you expect? What type of content were you looking for while visiting this page?

The moderator introduces the Non-Canadians page to the participants by showing the screenshot to the participants. Participants are invited to look at the different elements/items of the pages.

• What stands out the most for you on the support pages?

• Is the information displayed clearly enough? Is there anything on these pages that is not clear?

• Is it useful?

• Does the concept provide you with new information that is relevant to you? Please explain.

• Would you change anything to these pages or one of these pages? What could be done to improve them?

• Is there any piece of information that seems to be missing?

C- The Non-Canadians page

2.4 Non-Canadians page

Please take a moment to look at the following page entitled: Non-Canadians

Some of you are not in Canada, but in the United States. Have you visited these pages?

What did you expect from that page? What type of content were you looking for while visiting this page?

The moderator introduces the Non-Canadians page to the participants by showing the screenshot to the participants. Participants are invited to look at the different elements/items of the pages.

• What stands out the most for you on the support pages?

• Is the information displayed clearly enough? Is there anything on these pages that is not clear?

• Is it useful?

• Does the concept provide you with new information that is relevant to you? Please explain.

• Would you change anything to these pages or one of these pages? What could be done to improve them?

• Is there any piece of information that seems to be missing?

D- Visitors to Canada

2.4 Visitors to Canada pages

Please take a moment to look at the following page entitled: Visitors to Canada

Some of you are not in Canada, but in the United States. Have you visited these pages?

What did you expect from that page? What type of content were you looking for while visiting this page?

The moderator introduces the Visitors to Canada page to the participants by showing the screenshot to the participants. Participants are invited to look at the different elements/items of the pages.

• What stands out the most for you on the support pages?

• Is the information displayed clearly enough? Is there anything on these pages that is not clear?

• Is it useful?

• Does the concept provide you with new information that is relevant to you? Please explain.

• Would you change anything to these pages or one of these pages? What could be done to improve them?

• Is there any piece of information that seems to be missing?

PART 3: General perceptions

E- Final Impression about the Website

2.5 What is your overall impression of the website? What makes you say that?

• Is the website providing the right functionalities?

• Are there any features/functionalities that seem to be missing from the website?

• Is the website providing the right information for you?

• Does the website provide you with information that is relevant to you? Which ones? Please explain.

• Is there any piece of information that seems to be missing in what you saw from the website?

• Would you change anything to the website? What could be done to improve it?

• But what do you think about the look of the website? Do you like it or not? Why? What would you change about the look of the website?
• What do you think about the organization of information?
• What do you think of the menus?
• What do you think of the language used, the tone?

Conclusion

Duration: 5 minutes

4.1 Finally, if you had a chance to speak directly with the creator of this website, what would you tell him or her? What should be his number one priority to ensure the website adequately meets the needs of people like you?

4.2 Do you have any other remarks/suggestions or comments about this website?

THANK YOU VERY MUCH FOR YOUR VALUABLE PARTICIPATION!

Appendix H – Detailed Tables

Figure A.1. Answer to question D3: In which province or territory do you live? SPONTANEOUS MENTIONS Base: Non-Canadians (n=687)

Table A.1. Nature of the Problem According to Visitors Country of Origin

5A. What was the nature of the problem? SPONTANEOUS MENTIONS Base: Respondents who ever had a problem accessing the CBSA website	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	123	101	22*
NET PROGRAMMING ISSUES	34%	37%	23%
Links don't work (broken, dead)	15%	16%	14%
Glitchy website (stuck in loops, delay in loading, freezes, buttons don't work, etc.)	14%	15%	9%
Application was not working, had to send paper documents	2%	2%	0%
Difficult to navigate (not user friendly/can't find info)	24%	26%	18%
NET INFORMATION ISSUES	15%	16%	14%
Information is incorrect / outdated / incomplete / missing	11%	13%	0%
Very difficult to understand / needs simplified language	5%	3%	14%
Cannot connect (login, password, username, etc.)	10%	11%	5%
Links to third party agencies	7%	8%	0%
Cannot access Nexus renewal link on mobile	2%	3%	0%
Couldn't proceed until I did the survey	2%	2%	0%
Other	7%	5%	14%
None / Nothing	2%	0%	14%
Don't know / Refused	11%	10%	14%

*Given the small number of respondents (n<30) data are presented for illustrative purposes only.

Table A.2. Difficulty About Finding the Searched Information According to Visitors Country of Origin

A8A. What was difficult about finding information on the page? SPONTANEOUS MENTIONS Base: Respondents who said it was not easy to find the searched information	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	146	115	31
NET DESIGN PROBLEMS	32%	36%	19%
Couldn't find what I needed it	12%	11%	13%
Confusing / poor layout	12%	14%	3%
Difficult to navigate on the website (not user friendly)	4%	4%	3%
Too many different places to look for the needed link	3%	3%	0%
Didn't see the Apply or Join button	2%	3%	0%
The website is not intuitive	1%	2%	0%
NET ISSUES OR BUGS	18%	17%	19%
Renewal of NEXUS issues	10%	12%+	0%-
Search engine issues	5%	3%-	13%+
The website had bugs	1%	2%	0%
Couldn't access the forms	1%	2%	0%
Book appointment issues	1%	0%-	6%+
NET LINKS	10%	11%	6%
Too many hyperlinks directing to another page	9%	10%	6%
Non-functional link	4%	4%	3%
Couldn't progress to the next page / couldn't go back to the previous page	1%	2%	0%
NET CONFUSING, VAGUE, OR CONFLICTING INFORMATION	10%	4%-	32%+
Information about eTA and visa confusing/not precise	5%	1%-	19%+
Difficult to understand the content (complicated words)	4%	3%	6%
Conflicting information on different pages	1%	0%-	6%+
NET INFORMATION QUANTITY	5%	5%	6%
Too much information	3%	3%	3%
You have to read a lot of information to find where you need to go	3%	3%	3%
Other	5%	4%	6%
No difficulty	2%	3%	0%
Don't know / refusal	18%	20%	13%

Table A.3. Time Spent on the CBSA Website According to Visitors Country of Origin

A10A. What information were you looking for that you didn't find? Base: Respondents who ended up not finding the information they were looking for	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	619	444	175
NET NEXUS PROGRAM	27%	36%+	3%-
Information regarding the NEXUS program	9%	11%+	1%-
Information regarding NEXUS renewal	8%	11%+	2%-
Information regarding the NEXUS application	6%	8%+	1%-
Information regarding my status of NEXUS application	4%	5%+	0%-
Sign in the NEXUS page	1%	2%	0%
NET DECLARATION/IMPORTATION AND FEES	11%	14%+	3%-
Information regarding declaration/importation	6%	7%+	2%-
Information regarding duties, taxes and other clearance charges	5%	7%+	1%-
NET ITEMS ALLOWS AT THE BORDER	9%	8%	10%
Information regarding consumer goods allowed at borders	6%	5%	8%
Information regarding pet owner at the border	1%	2%	1%
Information regarding alcohol limits	1%	1%	0%
Information regarding items confiscated by CSBA	1%	1%	1%
NET REQUIREMENTS	9%	2%-	26%+
Information on visa / visa requirements	5%	1%-	16%+
Information regarding on entry/re-entry requirement for non-Canadians	2%	0%-	5%+
Information regarding requirements for US citizen on entry in Canada	2%	1%-	5%+
NET OTHER	35%	31%-	45%+
Information regarding individuals with DUI or other criminal convictions	3%	0%-	9%+
Information regarding border wait times	2%	2%	3%
Information regarding Electronic Travel Authorization (eTA)	2%	0%-	6%+
Information regarding passport or other travel documents (ex: passport for children)	2%	2%	4%
Information regarding border crossing	2%	1%-	5%+
Information regarding changing my personal information (ex: name, address, employment, etc.)	1%	2%	1%
Contact information	1%	1%	2%
Information regarding permanent resident	1%	1%	1%
Book an appointment	1%	1%	1%
Other	20%	22%	16%
Don't know / refusal	12%	11%	15%

Table A.4. Reasons to Visit the Travellers Section of the CBSA Website According to Visitors Country of Origin

B1. Please indicate, from the list below, the reasons why you have visited the Travellers Section of the Canada Border Services Agency website. SEVERAL MENTIONS ALLOWED* Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
NET NEXUS & CANPASS	52%	66%+	12%-
To get information regarding the NEXUS program	33%	42%+	7%-
To register for the NEXUS program	24%	31%+	6%-
To renew my Nexus pass	5%	7%+	1%-
To get information regarding the CANPASS programs	1%	1%	2%
To help plan for upcoming travel	25%	15%-	51%+
To get information regarding consumer goods allowed at borders	18%	19%+	15%-
NET REQUIREMENTS	18%	4%-	57%+
To get information on entry requirement for non-Canadians	17%	4%-	57%+
To get information on re-entry requirement for Canadians	0%	1%	0%
To get information regarding border wait times	10%	10%	9%
NET OFFICE CONTACT & INFORMATION	8%	7%-	11%+
Find border and/or contact information	5%	4%-	9%+
Find location of CBSA’s offices	4%	5%	4%
To get declaration/import information	2%	2%	1%
Passport or other documents concerns	1%	1%	2%
Other reason	1%	1%	1%
Do not recall	3%	3%	2%

*Because respondents were able to give multiple answers, total mentions may exceed 100%.

Table A.5. Suggestion for Improvements for the Travellers Section of the CBSA Website According to Visitors Country of Origin

B4. Do you have any suggestions for improving the Travellers Section of the Canada Border Services Agency website? SPONTANEOUS MENTIONS Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
NET INFORMATION	5%	4%-	8%+
More precise / detailed information (in general)	2%	2%-	4%+
Make the information easier to understand	1%	1%	1%
Update information on the website (outdated information)	1%	1%	1%
List more complete	1%	0%-	2%+
Make other languages available	0%	0%	0%
NET NEXUS	3%	4%+	1%-
Make access to Nexus applications easier (ex: create a link access)	1%	2%+	0%-
More assistance for the Nexus renew / More detailed information about Nexus	1%	2%+	1%-
Make the website/Nexus compatible for smartphones (ex: androids, iPhone, etc.)	0%	0%	0%
Fix the Nexus application	0%	0%	0%
NET DESIGN	3%	3%	3%
Make it easier to navigate (ex: create a link access)	2%	2%	3%
Better design (more modern)	0%	1%	0%
NET ENABLE CONTACT	2%	2%	2%
Being able to contact someone on the phone / to live chat	2%	1%	2%
Contact by email available	0%	0%	0%
Add FAQ section	1%	1%	1%
Update more often the current border wait times / more detailed border waiting time	1%	1%+	0%-
Fix the non-functional links / fix the bugs (website didn't work)	1%	1%	1%
Improve the search engine	1%	1%	1%
Other	1%	1%	1%
No suggestions	15%	15%	16%
Don't know / refusal	69%	69%	68%

Table A.6. Other Sources of Travel Information According to Visitors Country of Origin

B9. Apart from the Travellers Section of the Canada Border Services Agency website, where else do you go for travel information? SEVERAL MENTIONS ALLOWED* Base: All respondents	Total	Canadians and Perm. Residents	Non-Canadians
n= (unweighted)	2,729	2,042	687
NET TRAVEL MEDIA	62%	63%+	58%-
Family/friends/colleagues	41%	42%	38%
NET GOVERNMENT WEBSITES	37%	37%	39%
Social media	22%	23%+	19%-
NET TRAVEL SUPPLIERS	22%	23%+	18%-
Newspapers	8%	8%	7%
Search engines	1%	1%	2%
Other	1%	0%-	2%+
Nothing	1%	1%	1%
Do not know	11%	11%	11%

*Because respondents were able to give multiple answers, total mentions may exceed 100%.