Measuring Agency Strategic Performance Framework Outcomes (2022-2023)

Executive summary

Prepared for the Canada Revenue Agency

Supplier name: Ipsos Public Affairs
Contract Number: CW2269423
Contract value: $112,265.16 (including HST)
Award Date: January 23rd, 2023
Delivery Date: April 14th, 2023
Registration number: POR 121-22

For more information on this report, please contact Canada Revenue Agency at: cra-arc.media@cra-arc.gc.ca

Ce rapport est aussi disponible en français

Measuring Agency Strategic Performance Framework Outcomes (2022-2023)

Prepared for Canada Revenue Agency(CRA)
Supplier name: Ipsos Public Affairs
April 2023

This public opinion research report presents the methodology of the Measuring Agency Strategic Performance Framework Outcomes (2022-2023) online and telephone survey conducted by Ipsos on behalf of the Canada Revenue Agency (CRA).

The telephone survey was conducted with a sample of n=700 Canadians aged 18+ between February 22nd and March 31st, 2023, and an online survey was conducted with n=246 Tax intermediaries and n=400 Businesses between March 16th and March 23rd, 2023.

Cette publication est aussi disponible en français sous le titre : Mesurer les Résultats du Cadre Stratégique de Performance de l'Agence (2022-2023)

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Canada Revenue Agency. For more information on this report, please contact Canada Revenue Agency at:  cra-arc.media@cra-arc.gc.ca or at:
101 Colonel By Drive
Ottawa, Ontario   K1A 0K2
Canada

Catalogue Number: Rv4-154/2023E-PDF
International Standard Book Number (ISBN): 978-0-660-48639-0

Related publications (registration number: POR 121-22):
Catalogue Number: Rv4-154/2023F-PDF (Executive summary, French)
ISBN: 978-0-660-48640-6

Executive Summary

Ipsos Public Affairs
Contract Number: CW2269423
Registration number: POR 121-22
Award Date: January 23rd, 2023
Contract value: $112,265.16 (including HST)

Background and Objective

The CRA required public opinion research (POR) to develop indicators that will meet the need of measuring the Agency’s Strategic Performance Framework (ASPF) outcomes. These indicators included deterrence effectiveness, perceived fairness, channel usage, and transparency, and will address the existing measurement gaps.
The results of this POR will be used to:

1. develop meaningful performance indicators using robust scientific
   methodology, and
2. establish baselines and targets.

In addition, the POR results will generate knowledge in the field of tax administration and further strengthen the CRA’s position as a research-driven organization. This research project will also support and contribute to the following objectives outlined in the Minister’s Mandate Letter:

• Ensuring the Canada Revenue Agency (CRA) delivers client-centric service and intensifying efforts to reinforce the fairness of the tax system by ensuring that everyone pays their fair share.
• To continue working to modernize the CRA to provide a seamless, empathetic and client–centric experience, including by making information easier to find and understand, accelerating the use of digital tools, and enhancing the Community Volunteer Income Tax Program.

Target Audience

The research was conducted on the following three target audiences.

1. General population: This survey was conducted by telephone with a random sample of Canadians 18 years and older.
2. Businesses: This survey was conducted online with small and medium-sized businesses that employ fewer than 100 employees – including decision-makers or those involved in decisions related to tax matters, payroll, GST/HST preparation, or bookkeeping. Acceptable job titles include:
   • President/CEO/Owner
   • CFO/Comptroller
   • Accountant
   • Payroll Manager/Officer
   • Manager
   • Bookkeeper
   • Financial Officer
3. Tax Intermediaries: This survey was conducted online with tax intermediaries, who prepare taxes for individuals and/or business clients on tax-related or payroll matters.

Political Neutrality Statement

I hereby certify as Senior Officer of Ipsos that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate or ratings of the performance of a political party or its leaders.

Mike Colledge
President
Ipsos Public Affairs

Methodology

Telephone survey (National)

Ipsos conducted a 15-minute telephone survey among a nationwide sample of n=700 Canadian adults between February 22nd and March 31st, 2023. The sample is a probability sample generated through random digit dialing. To ensure we reach the more mobile segments of the population (namely younger Canadians), the sample included a minimum of 70% of cellphone users in addition to landline numbers (30%). The sample was also stratified by region. For respondents contacted by landline, respondents within households were selected at random, by using the "birthday method" of identifying and interviewing members of the household (aged 18+) who had their birthday last.

Respondents were contacted on a cellular phone and random digit dialed; they needed to be 18+ to participate. Wireless samples were selected on a provincial level (as it is not practical to accurately select by the market given the mobile nature of the technology) from a database containing all possible numbers in 1000 blocks of area codes and exchanges dedicated to wireless numbers.

Within the total sample of 700 Canadians for this survey, 201 respondents were contacted on their landlines, while the other 499 respondents were contacted on their cellphones. The margin of error for a telephone survey of 700 respondents is ±3.7%, using a confidence interval of 95% (19 times out of 20).

Telephone sample weighting

The tables below indicate the unweighted and weighted distributions of the telephone sample. The sample was stratified by region, with soft quotas also set for gender and age to ensure appropriate representation across categories. Weighting was applied to the sample to ensure that the final data reflects the adult population of Canada by region, age, and gender according to the 2021 Census. The range of the weighting factor for the telephone survey was between 0.853 and 1.421. No weighting was applied to the online data.

Please note that the totals below may not add up to 700 due to some respondents' refusal to provide socioeconomic information.

Statistics presented in the table above show minimal differences between the final unweighted and weighted samples. However, the youngest age group (18 to 34 years old) is underrepresented, resulting in a higher unweighted to weighted ratio of 1.42:1, which remains well within acceptable ranges for a survey of the general population and fares favorably compared to the gap observed in the telephone sample.

Call dispositions

The following table provides the call dispositions and response rate calculation, as per the former MRIA's empirical method of calculating response rates for telephone surveys.

The response rate, calculated as the number of in-scope – responding units divided by the sum of unresolved units, in-scope – non-responding units, and in-scope – responding units, was 6.96% for landline numbers, 7.71% for cellphone numbers, and 7.51% for all telephone numbers. The total response rate of 7.51% for a telephone survey of the Canadian general population with up to 8 call-backs per household is typical.

Non-response analysis

As with any probability sample, there exists within the current sample the possibility of non-response bias. In particular, this survey would not include members of the population who do not have access to a telephone (either landline or cell phone) or who are not capable of responding to a survey in either English or French. In addition, some groups within the population are systemically less likely to answer surveys.

The table below compares the unweighted sample to the 2021 Census results by region, age, and gender. The comparison between the two samples for the three variables used in the weighting scheme shows the underrepresentation of younger Canadians (18 to 34 years of age). However, this discrepancy is small enough that it can be corrected through weighting without affecting the quality of the final results. As the regional distribution was set through hard quotas, the weighting had virtually no impact on final numbers.

Differences among the variables used in the weighting scheme are minimal. The distribution in the sample is consistent, with only slight differences observed between the unweighted percentages and the 2021 Census data.

Online Sample

Respondents for the survey samples were drawn from a trusted partner panel vendor, Dynata. The survey was conducted with a sample of n=246 Tax intermediaries and n=400 Businesses. Interviews were conducted in the language of their choice, either English or French.

For Businesses, respondents selected were those who have small and medium-sized businesses including organizations that employs fewer than 100 employees – including decision-makers or those involved in decisions related to tax matters, payroll, GST/HST preparation, or bookkeeping.

For Tax intermediaries, respondents selected were those who prepare taxes for individuals and/or business clients on tax-related or payroll matters

A pre-test was launched on March 16th, 2023, which garnered 38 completes (27 English / 11 French). An open-ended question was asked at the end of the survey where any problems, questions, or unclear questions could be brought to our attention; no issues were flagged. The survey was fully launched and ran between March 17th and March 23rd, 2023.

The sampling methodology utilized email invitations and router technology to invite participants. Each participant received a unique URL link. This link could only be used once, with respondents being allowed to pause during completion and return to complete. The online data was not weighted.

Soft quotas were provided for the online survey for regions and industries to ensure all groups were as representative as possible. For online data, weighting was not applied.

* Asked only respondents who identify as businesses (n=400)

Participant Recruitment

Sample Source

Ipsos partnered with sub-contractors Canadian Viewpoint Inc. (CVI) and Dynata. Dynata has over 300,000 active panelists. Dynata's panels are continually refreshed and recruited through various channels including

1. loyalty program sourcing across travel, entertainment, and other sectors.
2. online banners, cable TV advertising, mailings, social media influencers, mobile app, etc.
3. integrated channels including access to online communities, social media platforms, publishers, and others (this last group has not opted into a panel), but each participant has a Dynata profile.

The online survey was conducted using Computer Assisted Web Interviewing (CAWI). Every panelist opted-in or chose to participate in research surveys. Panelists that meet the criteria we were looking for were randomly selected and sent an email invitation to complete the survey. Participants were offered a choice to complete the survey in either English or French.

The survey platform was Accessibility for Ontarians with Disabilities Act (AODA) compliant, according to Web Content Accessibility Guidelines (WCAG2.0AA).

Incentives and Quality Control Measures

Dynata's panel includes members who have opted in and receive a form of incentive. Incentives are based on a point-based system cashed in for rewards (electronic gift certificates, high street vouchers, charity donations, and long-term loyalty rewards), as well as sweepstakes. The amount of the incentives is based on the specific requirements of each survey, depending on the length and complexity of the survey, the subject matter of the study, and the time required to complete a minimum number of interviews.

The comprehensive background profiling data gathered when respondents join a panel allows for the targeting of respondents based on key criteria, such as region, age, gender, education, income level, intention to buy a home within 12 months, profession, and other characteristics.

The data excludes any duplicate respondents based on IP capture and excludes panelists who have completed another Government of Canada survey in the previous 30 days as a member of that panel.

Email Statistics

For this survey, a sample router was used. Therefore, a response rate cannot be calculated. However, the participation rate for the survey was 92%, which is calculated as follows: (qualified completes + over quota + terminates)/click-through).