Prepared for the Canada Revenue Agency
Prepared by Narrative Research
PSPC Contract Number: 46558-210915/001/CY
Contracted Value: $189,997.35
Award date: January 15, 2020
Delivery date: March 2020
Registration number: POR 061-19
For more information, please contact: media.relations@cra-arc.gc.ca.
Ce rapport est aussi disponible en français.
2019-2020 Annual Corporate Research (ACR)
Executive Summary
Prepared for Canada Revenue Agency
Supplier name: Narrative Research
March 2020
This report presents the methodological details for the 2019-2020 Annual Corporate Research quantitative study conducted by Narrative Research, on behalf of the Canada Revenue Agency. The quantitative study was conducted through two data collection phases, a telephone survey with the general public and an online survey with small and medium sized businesses as well as tax intermediaries. Both surveys were administered February 4 – March 2, 2020.
This report also presents the findings of the focus groups conducted from February 24th to March 4th, 2020 by Narrative Research on behalf of the Canada Revenue Agency. This component of the research entailed a total of 16 in-person focus groups, namely four groups in each of Halifax (NS) and Calgary (AB), and two groups in each of Kitchener (ON), Toronto (ON), Montreal (QC), and Sherbrooke (QC). Two business audiences were included in the study, namely tax intermediaries and small and medium sized enterprises. Two audiences within the general adult population were also included in the study, namely those considered as having a high tax compliance attitude, and those with low tax compliance attitude.
Cette publication est aussi disponible en français sous le titre:
Recherche d’entreprise annuelle de l’ARC de 2019 - 2020 - Constatations qualitatives et rapport méthodologique
This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Canada Revenue Agency. For more information on this report, please contact Canada Revenue Agency at: media.relations@cra-arc.gc.ca or at:
101 Colonel By Drive
Ottawa, Ontario K1A 0K2
Canada
Catalogue Number: Rv4-126/2-2020E-PDF
International Standard Book Number (ISBN): 978-0-660-35853-6
Related publications (registration number: POR-061-19):
Catalogue number: Rv4-126/1-2020E-PDF (final report, English)
ISBN: 978-0-660-35558-0
© Her Majesty the Queen in Right of Canada, as represented by the Minister of National Revenue, 2020
Narrative Research Inc.
Contract Number: 46558-210915/001/CY
POR Registration Number: 061-19
Contract Award Date: January 15, 2020
Contracted Cost: $189,997.35
In 2005, the Canada Revenue Agency (CRA) launched an annual survey that focuses on corporate-wide issues, with the aim to provide a public opinion context for strategic planning and reporting. A review of the methodology was conducted by the CRA in 2010 which led to the redesign of the Annual Corporate Research (ARC) to include a core survey each year along with annually rotating modules and focus groups, featuring expanded target audiences. The fiscal year of 2011-2012 saw the first iteration of the ACR's new cycle. In 2013, new questions were added to examine experiences with the CRA website. The 2013, 2016 and 2018 editions repeated the service-focused module and the 2017 iteration was the first compliance module. In 2019-2020, the CRA is back to a second compliance focus.
In this context, the CRA was interested in assessing public perceptions of taxpayers, small businesses, and professionals involved in assisting small businesses with their taxes, to gather background contextual information. This information will enable the CRA to gauge factors such as trust in, and satisfaction with, the CRA, contextualize other study information, inform strategic decisions, and provide information for reporting on engagement and reputation management.
Specific research objectives included to:
There were three target audiences:
There were three components to this research:
This report presents the findings from the qualitative portion of this study. Caution must be exercised when interpreting the results from this study, as qualitative research is directional only. Results cannot be attributed to the overall population under study, with any degree of confidence.
I hereby certify as a Senior Officer of Narrative Research that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not contain any reference to electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leader.
Signed
Peter MacIntosh
Chief Research Officer & Partner
Narrative Research
pmacintosh@narrativeresearch.ca
902-493-3832
The following provides key findings and conclusions from the focus groups with members from the general public:
The following provides key findings and conclusions from the focus groups with tax intermediaries (TIs) and small and medium size businesses (SMEs):
There were four target audiences:
Two surveys were administered including a telephone survey with individual taxpayers, and an online survey with small and medium-sized businesses (SMEs) as well as tax intermediaries:
The target SMEs were businesses with less than 100 employees and included decision-makers or those involved in decisions related to tax matters, payroll, GST/HST preparation, or bookkeeping. Acceptable job titles included:
The target tax intermediaries were those that worked with small business clients (<100 employees) on tax-related or payroll matters.
The questions utilized in this study were provided by Canada Revenue Agency. Narrative Research reviewed the questions, providing suggestions and prepared them in a format suitable for telephone or online methodology, as required.
As required by Government of Canada standards, English and French pre-test surveys were conducted.
Sampling
The individual taxpayers survey was designed to be administered to a random sample of 1,300 adult Canadians 18 years of age or older. The sample provided for dialing entailed contact records reflecting Random Digit Dialing (RDD), thus engendering a large number of Not In Service (NIS) telephone numbers. For landlines, records were drawn from a list of randomly-selected households compiled from telephone numbers in Canada, drawn from a database that is updated quarterly. Both listed and unlisted numbers were included in the sample.
Narrative Research utilized ASDE Inc.'s Canada Survey Sampler (CSS) sampling software to generate general population telephone landline samples within Canada, which has become a standard software for many companies in Canada. This software has been proven to provide a sample that is equivalent to RDD. Randomly generated cellular numbers were also used. As per the Statement of Work requirements, the sample included from the outset a minimum of 25% of cellphone only users, in addition to landline telephone numbers.
The sample was stratified by region in order to ensure regional representation as follows:
Region | Target Completions |
---|---|
Atlantic Canada | 125 |
Quebec | 300 |
Ontario/Nunavut | 425 |
Prairies/NWT | 300 |
BC/Yukon | 150 |
Total | 1,300 |
The target interviews plan entailed overall quotas by age and gender, as well as by region. The gender quotas were approximately 50/50, while the age quotas were broken into three groupings: 18-34, 35-54, and 55 years of age or older
Administration
As noted, the survey was conducted via telephone. The survey was programmed by Narrative Research in both English and French via Computer-Assisted Telephone Interviewing software (CATI). Respondents had the option to complete the survey questionnaire in the official language of their choice. Assistance was available from bilingual staff members as required, with ongoing bilingual supervision.
In terms of training, in addition to a thorough general screening and training process, supervisors and interviewers were provided with customized project-specific training, a review of the questionnaire including specific terminology, acronyms and pronunciations, and background information on the project goals and objectives.
In terms of supervision, as with all projects there was a ratio of one supervisor for every 15 interviewers working. This "floor supervisor" answers questions, handles escalations, ensures that technology is functioning properly, and blind monitors interviewers. In addition, a minimum of 10 percent of interviews were audited through the review of recorded interviews or live monitoring (twice the percentage required by industry guidelines).
The programmed survey was thoroughly tested to ensure question order and skip patterns were properly represented. In addition to this testing, a pre-test was conducted with 10 English and 10 French surveys. The overall purpose of the pre-test was to ensure that:
After the pre-test, the data was carefully reviewed to ensure accuracy and identify any aspects that needed to be modified. A pre-test report was prepared outlining the results.
In addition to the actual survey queries, a section was included at the end of the questionnaire to ascertain respondent comprehension and experience with the survey. No changes in the survey instrument were deemed to be required as a result of this endeavour.
Fieldwork was monitored on an ongoing basis, allowing supervisors to determine if there were any challenges via the call disposition/reasons for non-response information. The survey required a mean average of approximately 22 minutes for respondents to complete.
A dialing plan was implemented whereby there were a minimum of eight call-backs for landline telephone numbers, and a minimum of five call-backs for cellular telephone numbers. Calls and call-backs were varied throughout the day which includes both daytime and evening calling, limited to 9 p.m. in a given time zone.
Narrative Research and its data collection partner employed a number of techniques for keeping response rates as high as possible:
If an interviewer did not speak the official language requested by the respondent, the interview transferred to another interviewer or a call-back was arranged within 20 minutes (or at another time if requested by the respondent). When calling to a particular location with a predominant language (e.g., French in Quebec), initial calling was conducted by interviewers who spoke the predominant language, to minimize the number of transfers required.
The tables below for the telephone survey display regional, gender, and age data in terms of the actual distribution of adult Canadians, as catalogued in the 2016 Statistics Canada Census. As well, approximate regional, gender, and age quota targets are detailed (both in terms of the actual number of surveys completed, and the percentage of all surveys completed). The tables on the pages below present data with the weighted and unweighted number as well as percentage of surveys collected, for relevant demographic dimensions.
Data Tabulation: There were a total of 36 overlapping or interlocking statistical weighting cells created from the study design using the weighting factors of: Region (6: Atlantic, Quebec, Ontario/Nunavut, Manitoba/Saskatchewan, Alberta/Northwest Territories, and British Columbia/Yukon); Age group (3: 18–34 years of age, 35–54 years of age, and 55 years of age or older); and Gender (2: Male, Female). The 36 overlapping or interlocking statistical weighting cells thus were derived from Region (6) x Age (3) x Gender (2) dimensions = 36 unique statistical weighting cells. Population data for the 36 statistical weighting cells were obtained from the most recent (2016) Census of Canada, and can be found here:
2016 Census Data tablesIt should also be noted that a small number of individuals were not placed into one of the 36 weighting cells, as they identified as gender diverse or preferred not to provide a response to the gender question. For tabulation purposes, these individuals were given a weight value of 1.0.
Region | 2016 Census | Quota Targets (n=) |
Quota Targets (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
Surveys Completed (Weighted) (n=) |
Surveys Completed (Weighted) (%) |
---|---|---|---|---|---|---|---|
Atlantic Canada | 6.8% | 125 | 9.6% | 136 | 10.4% | 90 | 6.9% |
Quebec | 23.4% | 300 | 23.1% | 343 | 26.3% | 303 | 23.2% |
Ontario/Nunavut | 38.4% | 425 | 32.7% | 389 | 29.8% | 499 | 38.3% |
Prairies/NWT | 17.8% | 300 | 23.1% | 273 | 20.9% | 233 | 17.9% |
BC/Yukon | 13.6% | 150 | 11.5% | 163 | 12.5% | 180 | 13.8% |
Gender | 2016 Census | Quota Targets (n=) |
Quota Targets (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
Surveys Completed (Weighted) (n=) |
Surveys Completed (Weighted) (%) |
---|---|---|---|---|---|---|---|
Male | 48.7% | 633 | 48.7% | 680 | 52.1% | 622 | 47.7% |
Female | 51.4% | 667 | 51.3% | 602 | 46.2% | 660 | 50.6% |
Age | 2016 Census | Quota Targets (n=) |
Quota Targets (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
Surveys Completed (Weighted) (n=) |
Surveys Completed (Weighted) (%) |
---|---|---|---|---|---|---|---|
18-34 | 27.4% | 357 | 27.5% | 285 | 21.9% | 357 | 27.4% |
35-54 | 34.1% | 444 | 34.2% | 464 | 25.6% | 447 | 34.3% |
55+ | 38.6% | 499 | 38.4% | 555 | 42.6% | 500 | 38.3% |
Participation/Response Rate: The rate below was derived using the principal elements of the formula recommended by the Public Opinion Research Directorate of the Government of Canada:
Completion Results | Total |
---|---|
A. Total Numbers Attempted | 137,140 |
Total Invalid Numbers | 67,916 |
Discontinued Number/Not in Service | 67,090 |
Fax/Modem | 333 |
Business Number | 493 |
B. Total Unresolved Numbers (U) | 50,560 |
Busy | 3,045 |
Answering Machine | 13,830 |
No Answer | 30,955 |
Scheduled Call-Back | 2,730 |
C. In-scope Non-Responding Units (IS) | 16,806 |
Household refusal | 1,720 |
Respondent refusal | 14,344 |
Illness, Incapable | 0 |
Qualified Not Available | 0 |
Language Problem | 659 |
Selected Respondent Not Available | 0 |
Qualified Respondent Break-Off | 83 |
D. Responding Units (R) | 1,858 |
Quota Full | 234 |
Disqualify | 320 |
Completed Interviews | 1,308 |
Rounded Participation/Response Rate: R ÷ (U + IS + R) = 1,858 ÷ (50,560 + 16,806 +1,858) | 3% |
For the telephone survey, the margin of error provides a reflection of the sampling error and is presented in the table below:
Region | Survey Completions | Margin of Error(19 in 20 times) |
---|---|---|
Atlantic Canada | 136 | ± 8.4 percentage points |
Quebec | 343 | ± 5.3 percentage points |
Ontario/Nunavut | 389 | ± 5.0 percentage points |
Manitoba/Saskatchewan | 80 | ± 11.0 percentage points |
Alberta/NWT | 193 | ± 7.1 percentage points |
British Columbia/Yukon | 163 | ± 7.7 percentage points |
Total | 1,304 | ± 2.7 percentage points |
Sampling
Given the overlap in questions for these two audiences (i.e., SMEs and Tax Intermediaries), the questions were combined into one survey with streams for the two target audiences. The survey questionnaire was administered online. The sampling was designed to achieve 500 completed surveys with each target audience.
Narrative Research utilized the services of online panel records provider, Dynata, for this research. Panelists are recruited from a large number of sources to increase diversity and representation. This includes loyalty panels, organic, open enrollment and partnerships, and an affiliate network. Dynata uses robust panel management techniques to monitor the quality of data through various quality checks such as participation limits, screening questions, digital fingerprinting, random and illogical responding, capturing and removing flat-liners and speeders. Dynata also regularly measure participant satisfaction on elements such as frequency of invitations, value and diversity of incentives and redemption choices, their willingness to complete various lengths of surveys, and level of responsiveness to any questions or concerns they share with Dynata's Member Services team.
Regional targets were established for SMEs and tax intermediaries in order to ensure robust regional representation across the country.
Administration
The online survey utilized were programmed by Narrative Research in both English and French, using Voxco Acuity programming software. Respondents were formally invited to the survey in the official language of their choice. As well, at any point when completing the questionnaire, respondents had the option to change the questionnaire language to the other official language. Assistance in completing the survey was available from bilingual Narrative Research staff, as required. Respondents were able to verify the legitimacy of the survey via contacting representatives of Narrative Research and/or the Canada Revenue Agency.
Each programmed survey was tested to ensure question order and skip patterns were properly implemented. Testing included Narrative Research researchers receiving the invitation via email just as a respondent would, to ensure accuracy of delivery, text, links, and so on. Canada Revenue Agency staff were also provided with the pre-test link and thus client feedback also was incorporated prior to the launch of the survey.
A total of 25 English and 17 French pre-tests were completed across both audiences, specifically entailing eight SME respondents and 34 tax intermediary respondents. These pre-test survey completions were conducted via a survey "soft launch" whereby a small number of panel respondents were invited to participate in the survey. The pre-testing of the survey allowed the collected data to be reviewed to ensure accuracy and to identify any programming aspects that should be modified.
In addition to the actual survey queries, a section was included at the end of the questionnaire to ascertain respondent comprehension and experience with the survey. One small change in the survey instrument was made as a result of this endeavour.
Reminder notices were forwarded to sampled respondents until such time as the target number of survey completions had been achieved. This study was administered between February 11 and February 24, 2020. The survey invitation as well as reminder invitations were forwarded to panel members during the data collection period. Fieldwork was monitored and reviewed on an ongoing basis to ensure target quotas were being met. Given that single use unique survey links were distributed to prospective respondents, no individual was able to complete the survey questionnaire more than once.
A total of 1,048 surveys were submitted by respondents, and 1,002 were ultimately used in the final data set, (501 SMEs and 501 tax intermediaries). It is important to note that for various reasons, a small percentage of submitted online panel surveys is often removed from study data sets after submission. Such was indeed the case in this instance for both target respondent audiences, as Narrative Research's initial quota targets for each audience exceeded the overall final requirement of 500 questionnaires per audience. Thus, given the unavoidable possibility of having to remove surveys, post collection, Narrative Research as a precautionary measure collected more than the initially targeted number of surveys.
Reasons for removing surveys ultimately included respondent 'speedsters' who were deemed to have moved too quickly through the questionnaire, as well as consistently non-intelligible verbatim responses. Thus a small number (n=46) were removed for reasons of speeding, unintelligible verbatim responses, and so forth. The survey required a mean average of approximately 11.5 minutes for respondents to complete. A non-probability sample approach was implemented given that the study was designed to be conducted among respondents drawn from an online Canadian business panel. All such panels are inherently non-probability in nature, given that panelists self-select to become members of such panels, and not everyone in the target audience who is eligible to participate in the panel, indeed belongs to the panel.
Minimum quotas were established for regions, to ensure a robust representation of completed surveys from across the country, while also respecting the fact that the largest proportions of target organizations are from the most heavily populated provinces and regions of the country.
Region | Quota Targets (n=) |
Quota Targets (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
---|---|---|---|---|
Atlantic | 35 | 7% | 40 | 8.0% |
Quebec | 115 | 23% | 92 | 18.4% |
Ontario | 170 | 34% | 187 | 37.3% |
Western | 165 | 33% | 180 | 35.9% |
Territories/National/Not specified | 15 | 3% | 2 | 0.4% |
Region | Quota Targets (n=) |
Quota Targets (%) |
Surveys Completed (Unweighted) (n=) |
Surveys Completed (Unweighted) (%) |
---|---|---|---|---|
Atlantic | 35 | 7% | 39 | 7.8% |
Quebec | 115 | 23% | 128 | 25.5% |
Ontario | 170 | 34% | 162 | 32.3% |
Western | 165 | 33% | 164 | 32.7% |
Territories/National/ Not specified | 15 | 3% | 8 | 1.6% |
Participation Rate: The rate below was derived using the formula recommended by the Public Opinion Research Directorate of the Government of Canada:
Completion Results | Total |
---|---|
A. Total Email Addresses Used | 18,669 |
Invalid Cases | 1,548 |
Invitations mistakenly sent to people who did not qualify for the study | 1,548 |
Incomplete or missing email addresses | 0 |
B. Unresolved Units (U) | 14,365 |
Email invitations bounced back | 5 |
Email invitations unanswered | 14,360 |
C. In-scope Non-Responding Units (IS) | 523 |
Non-response from eligible respondents | 0 |
Respondent refusals | 103 |
Language problem | 0 |
Selected respondent not available (illness; leave of absence; vacation; other) | 0 |
Early break-offs | 420 |
D. Responding Units (R) | 2,233 |
Completed surveys disqualified -quota filled | 1,231 |
Completed surveys | 1,002 |
Participation Rate = R/(U + IS + R) = 2,233/(14,365 + 523 + 2,233) = | 13% |
Reminders were distributed to potential respondents who were invited to complete a survey, but who chose not to do so. Given that the online methodology utilized a non-probability sample, a margin of error cannot be applied to the results as per the Standards for the Conduct of Government of Canada Public Opinion Research for Online Surveys.
Non-Response Bias Analysis
Any survey that is conducted is potentially subject to bias or error. When a survey is conducted with a sample of the population, there are two general classes of bias or error: sampling error, which is quantifiable, and non-sampling error, which is typically not quantifiable. Sampling error arises from the fact that interviews are conducted with only a subset of the population, and thus is it possible that the results obtained from this group of respondents is not reflective of the population as a whole. In contrast, non-sampling error encompasses a number of different types of errors including coverage error, measurement error, non-response error, and processing error.
No measurement of sampling error can be attributed to the online study, given that the contact records utilized in the data collection process were derived from an online panel of the target audience, which is to say, a non-probability sample source. Having stated that, measures were taken in the implementation of the data collection to ensure sufficient completed surveys were obtained from both target segments. Targets were also set to ensure robust geographic representation across the country.
With respect to non-sampling error, a number of steps were taken to minimize bias. Both surveys utilized survey programming technology to ensure proper survey skip patterns were followed and to minimize errors due to data entry and data capture. The French and English survey instruments themselves were pre-tested with a small sample of respondents to ensure the survey material was easily understood by respondents, and that the resultant data were being captured properly. Interviewers were also trained and supervised for the telephone survey.
In terms of coverage, the telephone survey was conducted from a random sampling of a robust sample frame of landline and cellular numbers. Quotas were established for demographic groups traditionally regarded as central in quantitative survey research, such as gender, age, and region/province. The final data set was statistically weighted to closely match the true distribution of these dimensions as reflected in the 2016 Statistics Canada census. The statistical weights implemented were relatively small, given that the data collected already closely matched the actual distribution of adult Canadians along these demographic dimensions.
The online survey was conducted with an online panel of the target audience, based on a randomized sampling of panel records for the target audience drawn from a reputable commercially available online general public panel.
The qualitative study included a total of 16 in-person focus groups, specifically one group with each of the four audiences. The following provides a breakdown of groups based on location, date, language and audience:
Location | Date (2020) | Language | Adults – General Population; Low Compliance Attitude | Adults – General Population; High Compliance Attitude | Tax Intermediaries | Small and Medium Size Businesses | Total # of sessions |
---|---|---|---|---|---|---|---|
Kitchener, ON | Feb 24 | English | 1 | 1 | 2 | ||
Toronto, ON | Feb 25 | English | 1 | 1 | 2 | ||
Calgary, AB | Feb 26-27 | English | 1 | 1 | 1 | 1 | 4 |
Montreal, QC | Mar 2 | French | 1 | 1 | 2 | ||
Sherbrooke, QC | Mar 3 | French | 1 | 1 | 2 | ||
Halifax, NS | Mar 3-4 | English | 1 | 1 | 1 | 1 | 4 |
TOTAL # SESSIONS: | 4 | 4 | 4 | 4 | 16 |
The CRA designed the recruitment screeners. All participants were recruited per the recruitment specifications for the Government of Canada. Recruitment was conducted through qualitative panels stored on Canadian servers, with follow up calls to confirm the details provided and to ensure quotas were met. Those with current or past employment in sensitive occupations were excluded from the research, in addition to those living in the household. These sectors included marketing research, media, public relations, governments (whether federal or provincial), advertising and graphic design. In addition, member from the general public excluded the tax preparation and accounting sectors.
Participants from the general population were at least 20 years or older, had lived in their respective markets for at least two years, and were either the head or the co-head of the household. Those employed represented a good mix of industries. A mix of gender, household income and education levels were also recruited in each general population focus group. Individuals who had been to at least four qualitative sessions in the past five years and those who had attended a session in the past six months were excluded from the research.
Within the business/tax intermediaries focus groups, a mix of business sizes (in number of employees from 1 to 99) were represented. Participants included those who have sole or shared responsibilities in their organization for matters related to taxes, payroll, GST/HST preparation or bookkeeping. At least five per session personally dealt with the CRA. Tax intermediaries were defined as someone who works with small business clients on tax-related or payroll matters. Individuals who had been to at least five sessions in the past five year and those who had already been invited to take part in a focus group in the next few weeks were excluded from the research.
The CRA designed the discussion guides. Group discussions were held in English with the exception of those in Montreal and Sherbrooke which were conducted in French. Each session lasted approximately 2 hours with participants each receiving $100 (general population) or $150 (tax intermediaries or businesses) in appreciation of their time. A total of 144 participants were recruited across all 16 groups (specifically 9 respondents per session). Across all groups, 126 participants attended the discussions.
Qualitative discussions are intended as moderator-directed, informal, non-threatening discussions with participants whose characteristics, habits and attitudes are considered relevant to the topic of discussion. The primary benefits of individual or group qualitative discussions are that they allow for in-depth probing with qualifying participants on behavioural habits, usage patterns, perceptions and attitudes related to the subject matter. This type of discussion allows for flexibility in exploring other areas that may be pertinent to the investigation. Qualitative research allows for more complete understanding of the segment in that the thoughts or feelings are expressed in the participants' "own language" and at their "own levels of passion." Qualitative techniques are used in marketing research as a means of developing insight and direction, rather than collecting quantitatively precise data or absolute measures. As such, results are directional only and cannot be projected to the overall population under study.