Canada-U.S. Relations Survey

Methodological Report

Prepared for the Privy Council Office of Canada

Supplier name: Advanis Inc.
Contract Number: 35035-200902/001/CY
Contract Value: $129,853.54 (including HST)
Award Date: 2020-11-30
Delivery Date: 2021-02-12

Registration Number: POR 071-20

For more information on this report, please contact the Privy Council Office at:por-rop@pco-bcp.gc.ca

Ce rapport est aussi disponible en français

Canada-US Relations Survey

Methodological Report

Prepared for the Privy Council Office of Canada
Supplier name: Advanis Inc.
February 2021

This public opinion research report presents the methodology of a telephone survey conducted by Advanis Inc. on behalf of the Privy Council Office. The research was conducted with 2,000 adult Canadians, and 1,107 adult Americans between December 8th, 2020 and February 3rd, 2021.

Cette publication est aussi disponible en français sous le titre : Les relations Canado-Américaines – rapport méthodologique.

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from the Privy Council Office. For more information on this report, please contact the Privy Council Office at:

por-rop@pco-bcp.gc.ca
Privy Council Office
85 Sparks Street
Room 228
Ottawa, Ontario K1A 0A3

Catalogue number:
CP22-194/2021E-PDF

International Standard Book Number (ISBN):
978-0-660-37513-7

Related publications (registration number: POR 071-20):
Catalogue number CP22-194/2021F-PDF (Methodological Report, French)
International Standard Book Number (ISBN): 978-0-660-37514-4

© Her Majesty the Queen in Right of Canada, as represented by the Privy Council Office of Canada, 2021.

Table of contents

1.0 Summary

1.1 Background and objectives

The Privy Council Office (PCO) is the hub of non-partisan, public service support to the Prime Minister and Cabinet and its decision-making structures. Led by the Clerk of the Privy Council, PCO helps the Government implement its vision and respond effectively and quickly to issues facing the government and the country.

As an advisor to the Prime Minister, PCO brings together quality, objective policy advice and information to support the Prime Minister and Cabinet, including (1) information on the priorities of Canadians and (2) expertise on outcomes-based policy and program approaches.

The Communications and Consultation Secretariat within PCO supports the Prime Minister’s Office in coordinating government communications and setting broad government communications themes and messages, in accordance with government priorities, as determined by the Prime Minister, Cabinet, Cabinet committees and the Clerk of the Privy Council. The Secretariat also works with PCO policy secretariats to advise and support Cabinet and its committees.

Through the use of the Canada-US Relations Survey, PCO aimed to explore the perceptions of Canadians and Americans on various bilateral issues, such as trade, security and the environment. By carrying out this research, PCO will be able to ensure a better understanding of public perceptions and develop communications strategies and products that are effective.

The study was conducted, in both English and French, using a 12-minute computer-assisted telephone interviewing (CATI) survey. Numbers were generated using RDD (random digit dialing) systems using both landline and wireless phone numbers across Canada and the United States. Overall, 2,000 Canadians completed the survey between December 8th, 2020 and January 16th, 2021 with a response rate of 11.9%, and a margin of error of 2.2% (19 times out of 20 at a 95% confidence interval). 1107 Americans completed the survey between December 8th, 2020 and February 3, 2021 with a response rate of 2.7%, and a margin of error of 2.95% (19 times out of 20 at a 95% confidence interval). Given the timing of the research calling was suspended for the last 2 weeks of December. In the United States, calling was also paused after the January 6th, 2021 attack on the US Capitol building, and did not resume until after the January 20th inauguration. The data was weighted based on gender, age, region and country.

The total cost of this research was $129,853.54 (including HST).

1.2 Political Neutrality Requirement

Political neutrality certification

I hereby certify as Senior Officer of Advanis that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Policy on Communications and Federal Identity and the Directive on the Management of Communications.

Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leaders.

Signed: Nicolas Toutant, Vice-President, Research and Evaluation

2.0 Methodology

The survey was conducted via telephone, using computer-assisted telephone interviewing (CATI) technology, with a representative random sample of 2,000 Canadians and 1,107 Americans. The average survey length was estimated to be 12 minutes, with the average for Canada ended up at just over 11.5 minutes, and the United States at just over 13 minutes. Advanis’ approach to CATI is to apply the best technology, interviewers, and quality assurance to each project to achieve the highest quality data in the most efficient manner that meets the project and client needs.

An overlapping dual-frame sample was proposed in order to reach those in both the younger age categories, and those considered to be “cord-cutters”. “Cord-cutting”, or using only a wireless phone, is becoming more prevalent, particularly in the United States, so increasing the proportion of wireless sample was required during field.

Data collection in Canada versus the United States was very different in terms of participation and challenges. In Canada, a total of 29,025 phone numbers were dialed, and 2,000 people completed in the survey. Within Canada, in order to better represent the population age and gender distributions, an additional 758 people qualified for the survey but were unable to participate because enough data with their demographic profile had already been collected. Of the calls conducted in Canada, 16 percent were conducted in French and 84 percent were conducted in English. This resulted in 372 (19%) of the 2000 completes being French-speaking interviews. Canada achieved a response rate of 11.9%, and a completion rate of 7.9%. The response rate calculation is presented in Appendix A.

In the United States, a total of 52,962 phone numbers were dialed, and 1,107 people completed the survey. The response rate in the United States was only 2.4% (calculation is presented in Appendix A). Calling in the United States was difficult for a number of reasons, in particular, getting respondents to answer the phone was quite difficult. The amount of no answer/answering machine type calls was 20% higher in the US than in Canada. Among US numbers dialed, 30,500 were wireless numbers. When comparing landline to wireless although the refusal rate was higher on wireless (30% wireless vs. 21% landline), the proportion of those willing to answer the phone was also higher, resulting in a 3.2% response rate among wireless numbers, compared to 1.9% on landline.

In addition to lower response rates, data collection in the United States also encountered cases of individuals making unsolicited comments to interviewers during calls.

It should be noted that the timing of data collection for the study may not have been ideal. Response rates prior to the holiday season in December are typically lower in general. US calling was also suspended after the attack on the US capital on January 6th, 2021, and paused until after the inauguration on January 20th, 2021. The fact that data collection was done at a moment of high political turmoil in the US probably impacted the participation rate of certain segments of the population this could have introduced some bias.

A pre-test was conducted for the study on Tuesday December 8th, 2020. A total of 33 completes was collected, including 10 English and 10 French in Canada, and 13 English completes from the United States. During the pre-test the average survey length was 18 minutes for the United States and 16 minutes for Canada, so it was recommended that some questions be removed to decrease the survey length.

3.0 Sampling Plan

The target audience was Canadians and Americans who are 18 years and older. The sample size targeted was 2,000 adult Canadians and 1,107 Americans. The target for Canada was to collect 50% of completes from women, and 50% from men in each age group (18 to 34, 35 to 54,  55 and older), while also collecting with regional distribution for each of the following regions:

Similarly, the American sample target was for 50% female and 50% males, among the same three age categories, targeting a distribution across 4 regions: Northeast, Midwest, South and West.  

The final sample consisted of the following split.

Canada BC/YT AB/NT MB/SK/NU ON QC Atlantic Total
Female 18-34 25 25 24 45 44 23 186
Female 35 to 54 44 48 44 107 68 36 347
Female 55+ 70 47 48 162 87 50 464
Male 18-34 39 33 37 70 39 23 241
Male 35 to 54 45 58 38 101 75 33 350
Male 55+ 59 50 43 136 77 47 412
Total 282 261 234 621 390 212 2000

United States Midwest Northeast South West Total
Female 18-34 14 15 20 23 72
Female 35 to 54 31 21 41 24 117
Female 55+ 67 56 95 71 289
Male 18-34 20 23 38 35 116
Male 35 to 54 47 34 59 39 179
Male 55+ 83 69 117 65 334
Total 262 218 370 257 1107

4.0 Weighting

Overall, 3,107 telephone interviews were conducted during data collection. The weighting of the final file was based on four variables: country, region, age, and gender. Given the population difference between Canada and the United States, the US data was weighted down to the Canadian data. The population sizes are based on the latest Statistics Canada census results published in the 2016 Census. The United States were based on the most recent available census as well.

Segment ID (segid) Unweighted N Weight Weighted %
CA AB/NT Female 18-34 25 1.105 0.89%
CA AB/NT Female 35-54 48 0.659 1.02%
CA AB/NT Female 55+ 47 0.614 0.93%
CA AB/NT Male 18-34 33 0.86 0.91%
CA AB/NT Male 35-54 58 0.553 1.03%
CA AB/NT Male 55+ 50 0.54 0.87%
CA Atlantic Female 18-34 23 0.536 0.40%
CA Atlantic Female 35-54 36 0.505 0.59%
CA Atlantic Female 55+ 50 0.489 0.79%
CA Atlantic Male 18-34 23 0.532 0.39%
CA Atlantic Male 35-54 33 0.515 0.55%
CA Atlantic Male 55+ 47 0.462 0.70%
CA BC/YT Female 18-34 25 1.116 0.90%
CA BC/YT Female 35-54 44 0.827 1.17%
CA BC/YT Female 55+ 70 0.639 1.44%
CA BC/YT Male 18-34 39 0.725 0.91%
CA BC/YT Male 35-54 45 0.759 1.10%
CA BC/YT Male 55+ 59 0.682 1.30%
CA MB/SK/NU Female 18-34 24 0.631 0.49%
CA MB/SK/NU Female 35-54 44 0.388 0.55%
CA MB/SK/NU Female 55+ 48 0.415 0.64%
CA MB/SK/NU Male 18-34 37 0.419 0.50%
CA MB/SK/NU Male 35-54 38 0.444 0.54%
CA MB/SK/NU Male 55+ 43 0.414 0.57%
CA Ontario Female 18-34 45 1.814 2.63%
CA Ontario Female 35-54 107 0.99 3.41%
CA Ontario Female 55+ 162 0.745 3.88%
CA Ontario Male 18-34 70 1.17 2.64%
CA Ontario Male 35-54 101 0.981 3.19%
CA Ontario Male 55+ 136 0.774 3.39%
CA Quebec Female 18-34 44 1.053 1.49%
CA Quebec Female 35-54 68 0.893 1.95%
CA Quebec Female 55+ 87 0.912 2.55%
CA Quebec Male 18-34 39 1.198 1.50%
CA Quebec Male 35-54 75 0.81 1.95%
CA Quebec Male 55+ 77 0.905 2.24%
US Midwest Female 18-34 14 3.384 1.52%
US Midwest Female 35-54 31 1.789 1.79%
US Midwest Female 55+ 67 0.939 2.02%
US Midwest Male 18-34 20 2.458 1.58%
US Midwest Male 35-54 47 1.163 1.76%
US Midwest Male 55+ 83 0.649 1.73%
US Northeast Female 18-34 15 2.588 1.25%
US Northeast Female 35-54 21 2.164 1.46%
US Northeast Female 55+ 56 0.92 1.66%
US Northeast Male 18-34 23 1.751 1.30%
US Northeast Male 35-54 34 1.317 1.44%
US Northeast Male 55+ 69 0.639 1.42%
US South Female 18-34 20 4.354 2.80%
US South Female 35-54 41 2.486 3.28%
US South Female 55+ 95 1.217 3.72%
US South Male 18-34 38 2.377 2.91%
US South Male 35-54 59 1.702 3.23%
US South Male 55+ 117 0.846 3.19%
US West Female 18-34 23 2.362 1.75%
US West Female 35-54 24 2.65 2.05%
US West Female 55+ 71 1.016 2.32%
US West Male 18-34 35 1.61 1.81%
US West Male 35-54 39 1.607 2.02%
US West Male 55+ 65 0.95 1.99%

5.0 Quality Control

5.1 - Quality Control in Survey Programming

Advanis utilizes technology to maximize quality control in survey programming. Having developed a proprietary survey engine tool, Advanis professionals are able to design and program a survey in a browser-based environment, eliminating the need to involve a programmer who is less familiar with the survey subject matter. The survey was thoroughly pre-tested. to ensure skip patterns and survey questions were correctly programmed.

5.2 - CATI Methodology and Quality Control

The CATI survey script was programmed on Advanis’ proprietary CATI platform. Telephone surveys were conducted by experienced and professional interviewers. Regular monitoring system was in place to ensure the accuracy of the responses.

5.3 - Quality Control in Data Handling and Reporting

For all of the data collected, Advanis develops rigorous checks to ensure the data is accurate and error-free according to the questionnaire logic (skip patterns). Data cleaning involved computing a gender variable for weighting from the survey question and the combined interviewer observance if the respondent did not provide this information. Region variables were computed for both Canada and the United States, as well as a bucketed age category variable. Finally, a segment id for weighting was computed from the country, region, age and gender variables. The weight was not designed to allow cross-tabulation or statistical testing between the 2 countries.

6.0 Appendices

Appendix A. Response Rate

Canadian Response Rate

  Total %
GENERATED 29025 100.0%
USED 29025 100.0%
Not in service 2208 7.6%
Not residential 382 1.3%
Line problem 105 0.4%
Fax 156 0.5%
Wrong number 6 0.0%
A. NOT VALID[1] 2857 9.8%
VALID 26168 90.2%
Not eligible 1171 4.5%
Language barrier 267 1.0%
Age - Illness 95 0.4%
Other 8 0.0%
B. OUT OF SAMPLE[2] 1541 5.3%
SAMPLE[3] 25385 87.5%
C. Household refusal 5357 21.1%
C. Multiple household refusal 1 0.0%
D. Respondent refusal 4915 19.4%
D. Multiple respondent refusal 7 0.0%
D. FINAL refusal 325 1.3%
D. Prolonged absence 15 0.1%
D. Incomplete 123 0.5%
C. No answer 10278 40.5%
D. Appointments 1606 6.3%
Quota blocked 758 3.0%
     
Completed interviews 2000 7.9%
     
% REFUSAL[4] 41.8%  
% COMPLETED[5] 7.9%  
ELIGIBILITY RATE[6] 86.4%  
RESPONSE RATE[7] 11.9%  

American Response Rate

  Total %
GENERATED 52962 100.0%
USED 52962 100.0%
Not in service 3890 7.3%
Not residential 502 0.9%
Line problem 361 0.7%
Fax 645 1.2%
Wrong number 14 0.0%
A. NOT VALID[1] 5412 10.2%
VALID 47550 89.8%
Not eligible 481 1.0%
Language barrier 515 1.1%
Age - Illness 59 0.1%
Other 4 0.0%
B. OUT OF SAMPLE[2] 1059 2.0%
SAMPLE[3] 46484 87.8%
C. Household refusal 11971 25.8%
C. Multiple household refusal 1 0.0%
D. Respondent refusal 3878 8.3%
D. Multiple respondent refusal 10 0.0%
D. FINAL refusal 474 1.0%
D. Prolonged absence 7 0.0%
D. Incomplete 40 0.1%
C. No answer 28461 61.2%
D. Appointments 535 1.2%
Quota blocked 0 0%
     
Completed interviews 1107 2.4%
     
% REFUSAL[4] 35.1%  
% COMPLETED[5] 2.4%  
ELIGIBILITY RATE[6] 85.1%  
RESPONSE RATE[7] 2.7%  

Appendix B. Questionnaire

Intro2

(if is Canada and IsLandline) Hello/Bonjour, my name is _______________. I’m calling from Advanis. We are conducting a brief study for the Government of Canada on current issues of interest to Canadians. Your responses will be kept entirely confidential and anonymous.

May I please speak with the person in your household who is 18 years of age or older and who has had the most recent birthday? Would that be you?

If person is not available arrange callback

(if is Canada and IsCellphone) Hello/Bonjour, my name is _______________. I’m calling from Advanis. We are conducting a brief study for the Government of Canada on current issues of interest to Canadians. Your responses will be kept entirely confidential and anonymous.

May I confirm that you are 18 years of age or older?

(if is USA and IsLandline) Hello/Bonjour, my name is _______________. I’m calling from Advanis. We are conducting a brief study for the Government of Canada on current issues of interest to Americans. Your responses will be kept entirely confidential and anonymous.

May I please speak with the person in your household who is 18 years of age or older and who has had the most recent birthday? Would that be you?
If person is not available arrange callback
(if is USA and IsCellphone) Hello/Bonjour, my name is _______________. I’m calling from Advanis. We are conducting a brief study for the Government of Canada on current issues of interest to Americans. Your responses will be kept entirely confidential and anonymous.

May I confirm that you are 18 years of age or older?

IF NEEDED: The survey will take about 10 minutes to complete.
FOR VALIDATION: To validate the research for this study you may go to https://www.canadianresearchinsightscouncil.ca/rvs/home/ and reference project 20201204-AD398

WHERE DID YOU GET MY NUMBER: We are calling a random sample of numbers across both Canada and the United States.

1 Yes, continue
2 Yes, getting person STAY ON PAGE AND READ AGAIN
3 Yes, callback
4 No, refused

Term2 Show if InitialRefusal2

Refusal. Status Code: 1000

Callback2Show if RequestedCallback2

Callback. Status Code: 1001

Section Survey

D1. Before we begin, please note that this call may be monitored for quality assurance purposes.
What is your sex?

D12. Show if D1 Unknown or other. DO NOT READ RECORD SELF ASSESED GENDER

D2a. Which of the following age categories do you fall into?

D10a. Show if is Canada. And to better understand how results vary by region, can you tell me which province you live in?

D11. Show if is USA. And to better understand how results vary by region, can you tell me which state you live in?

T1. Show if Refusedp prov or state. We're sorry but for this study we need to know the area you live in.

Thank you for your time. Status Code: 501

T2. Show if RegionAgeGender QuotaBlock. We're sorry but we have already fille our quota of people like yourself.

Thank you for your time. Status Code: 570

Q1. (if is Canada) For each of the following we are interested in learning about your level of trust when it comes to Canada working with other countries on various issues.

For [READ COUNTRY] would you say you [READ SCALE]

(if is USA)For each of the following we are interested in learning about your level of trust when it comes to the United States working with other countries on various issues.

For [READ COUNTRY] would you say you [READ SCALE]

Levels marked with * are randomized

Scales

Q2. (if is Canada) Overall, is your opinion of the United States ...? (if is USA)Overall, is your opinion of Canada ...? READ LIST

Q3. (if is Canada) Compared to 5 years ago has your opinion of the United States ...? (if is USA)Compared to 5 years ago has your opinion of Canada ...? READ LIST

Q5. For each of the following, please indicate what kind of priority, if at all, you think it is. [READ ITEM] is it a [READ SCALE]: ONLY REPEAT SCALE IF NECESSARY. Levels marked with * are randomized.

Scales

Q6. (if is Canada) Which of the following best describes your opinion: Free trade between Canada and the United-States ...
(if is USA)Which of the following best describes your opinion: Free trade between the United States and Canada ... Levels marked with * are randomized.

Q6a. (if is Canada) Please indicate your level of support for the following. To create more jobs for Canadians, [READ FIRST ITEM]. Would you say you [READ SCALE]:
(if is USA)Please indicate your level of support for the following. To create more jobs for Americans, [READ FIRST ITEM]. Would you say you [READ SCALE]:

ONLY REPEAT SCALE IF NEEDED. Levels marked with * are randomized.

Scales

Q8. (if is Canada) When it comes to Canada being part of a free trade agreement that includes Canada, Australia, Brunei Darussalam, Chile, Japan, Malaysia, Mexico, New Zealand, Peru, Singapore, and Vietnam, would you say you [READ LIST] this?
(if is USA)When it comes to the United States joining a free trade agreement that includes Canada, Australia, Brunei Darussalam, Chile, Japan, Malaysia, Mexico, New Zealand, Peru, Singapore, and Vietnam, would you say you [READ LIST] this?

Scales

Q9. The Keystone XL pipeline would transport oil from Canada through the United States to refineries in Texas. Do you [READ LIST] the pipeline?

Q10. For each of the following, do you have a [READ SCALE] opinion: IF RESPONDENT IS UNSURE OF WHAT SOMETHING IS CODE AS "NEVER HEARD OF". Levels marked with * are randomized

Scales

Q11. Show if is Canada. Overall, do you think the Government of Canada is standing up for Canadian interests in the United States [READ LIST]?

Q12. Show if is USA. Which of the following best describes your opinion:

Temporary work visas that allow foreign workers in specialty occupations – such as tech workers - to work in the United States for a set period of time… Levels marked with * are randomized.

Q13. Show if is USA. Which of the following best describes your opinion:

International students who are studying in the United States… Levels marked with * are randomized.

Q14. Show if is USA. Which of the following best describes your opinion:

International organizations that promote free trade and settle trade disputes… Levels marked with * are randomized.

D3. Show if is Canada. What is the highest level of formal education that you have completed?

D3a. Show if is USA. What is the highest level of formal education that you have completed?

D4. Which of the following best describes your total household income last year, before taxes, from all sources for all household members?

D5. Which of the following categories best describes your current employment status? Are you…

Thank you very much for your time and assistance. The survey is now complete. Good bye.

[1] No possible contacts at these numbers

[2] Respondents were not eligible or were unable to answer

[3] Usable numbers

[4] Refusal/sample

[5] Completed/sample

[6] (D+Completed+Quota Blocked)/(B+D+Completed+Quota Blocked)

[7] (Completed+Quota Blocked)/(C*(E.R.)+D+Completed+Quota Blocked)