Connecting with Canadians: Quantitative Research on International Development: Methodological Report

This report presents the methodological details for the Connecting with Canadians: Quantitative Research on International Development study conducted by Narrative Research Inc. on behalf of Global Affairs Canada (GAC). The survey was administered between February 9 and March 9, 2022, to 1,200 adult Canadians (in the general public) aged 18 or older.

Ce rapport est aussi disponible en français sous le titre Communications avec les Canadiens et Canadiennes : recherche quantitative sur le développement international.

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Global Affairs Canada. For more information on this report, please contact Global Affairs Canada at POR-ROP@international.gc.ca.

Catalogue number: 978-0-660-42383-8

International Standard Book Number (ISBN): FR5-175/2022E-PDF

Catalogue number (Final Methodological Report, French): 978-0-660-42387-6

ISBN (French): FR5-175/2022F-PDF

Executive summary

Background

Baseline and annual research collected in February 2020 and February/March 2021 by the market research company Narrative Research, on behalf of Global Affairs Canada (GAC), revealed that “while most Canadians remain unaware of [Canadian international] development efforts, awareness has increased modestly. Canadians are proud of Canada’s international development activities and also believe they are important. However, the public perception is that the Government of Canada performs poorly when communicating about international development, and there is a lack of confidence in Government of Canada information. Finally, the research revealed that Canadians prefer TV, print newspapers and the Internet as sources of information about development more than social media.”

In February and March 2022, Narrative Research conducted a new survey to gauge Canadians’ current understanding and awareness of Canada’s international development activities.

Research objectives

The research objectives of the 2022 survey were to:

a catalogue the current level of understanding among adult Canadians concerning international development activities

b understand Canadian awareness of international development

c measure Canadian understanding of and support for the role the Government of Canada, and GAC in particular, plays in international development

Target population

The target audience was adult Canadians (general population) 18 years of age and older. A telephone survey of 1,200 people was conducted between February 9 and March 9, 2022. A sample of 1,200 respondents engenders an overall margin of error of +/- 2.8 percentage points, 19 times out of 20. As this is a probability sampling conducted via randomized data collection, the survey results are projectable to the overall adult Canadian population.

Research usage

The research will support governmental and departmental priorities. The department’s goal is to increase Canadians’ awareness of international development issues. The immediate need is to find out the best ways to connect with Canadians, as the Statement of Work (SOW) imparts that GAC continues to adjust how it frames messaging and the channels it uses.

The research will benefit Canadians as they will understand how and why Canada assists people in developing countries. The results of the research will be used in public outreach and social media messaging targeted at the Canadian public.

Expenditure

The survey entailed the expenditure of $49,986.68, including tax.

Consent

Narrative Research offers this written consent allowing the Librarian and Archivist of Canada to post, in both official languages, this Methodological Report.

Political neutrality statement

I hereby certify as Senior Officer of Narrative Research that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Policy on Communications and Federal Identity and the Directive on the Management of Communications. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leaders.

Peter MacIntosh

Chief Research Officer & Partner

Narrative Research

pmacintosh@narrativeresearch.ca

902-493-3832

Methodology

The survey was administered between February 9 and March 9, 2022, to 1,200 adult Canadians (in the general public) aged 18 or older.

Questionnaire design

Global Affairs Canada provided the survey questions used in the study. Narrative Research reviewed the questions, provided suggestions and prepared them in a format suitable for telephone administration.

As required by Government of Canada standards, English and French pre-test surveys were conducted.

Sampling

The survey was designed to be administered to a random sample of 1,200 adult Canadians 18 years of age or older. Of the 1,200 surveys collected, 434 were from what are believed to be landline telephone numbers while the remaining 766 were from what are believed to be cellular telephone numbers. The sample provided for dialing was generated by telephone numbers obtained through random digit dialing (RDD), thus engendering a large number of not in service (NIS) numbers. For landlines, records were drawn from a list of randomly selected households compiled from telephone numbers in Canada, selected from a database that is updated quarterly. Both listed and unlisted numbers were included in the sample.

Narrative Research utilized ASDE Survey Sampler’s sampling software—which has become a standard software for many companies in Canada—to generate general population telephone landline samples within Canada. This software has been proven to provide a sample that is equivalent to RDD. Randomly generated cellular numbers were also used. The sample was stratified by region in order to ensure regional representation, with approximate targets as follows:

Region	Target Completions	Margin of Error (19 in 20 times)
Atlantic Canada	109	± 9.4 percentage points
Quebec	277	± 5.9 percentage points
Ontario	472	± 4.5 percentage points
Western/Northern Canada	342	± 5.3 percentage points
Total	1,200	± 2.8 percentage points

The target interviews plan entailed overall quotas by age and gender, as well as by region. The gender quotas were approximately 50/50, while the age quotas were broken into 3 groupings: 18-34, 35-54, and 55 years of age or older.

Administration

As noted, the survey was conducted via telephone. The survey was programmed by Narrative Research in both English and French via computer-assisted telephone interviewing software (CATI). Respondents had the option to complete the survey questionnaire in the official language of their choice. Assistance was available from bilingual staff members as required, with ongoing bilingual supervision. In terms of training, in addition to a thorough general screening and training process, supervisors and interviewers were provided with extensive, customized, project-specific training; a review of the questionnaire, including specific terminology, acronyms and pronunciations; and background information on project goals and objectives.

In terms of supervision, as with all projects there was a ratio of 1 supervisor for every 15 interviewers working. This “floor supervisor” answers questions, handles escalations, ensures that technology is functioning properly, and blind-monitors interviewers. In addition, a minimum of 10% of interviews was audited through a review of recorded interviews or live monitoring (twice the percentage required by industry guidelines). The programmed survey was thoroughly tested to ensure question order and skip patterns were properly represented. In addition to this testing, a pre-test was conducted with 14 English and 10 French surveys. The overall purpose of the pre-test was to ensure that:

· the wording of questions was clearly understood and unambiguous

· the sequence of the questions was appropriate

· the necessary response categories had been included for each question

· neither specific questions nor the survey overall evoked a negative reaction or discomfort among respondents

In addition to the survey questions, pre-test respondents were invited to offer comments on the questionnaire document. When respondents were asked to identify questions that were problematic from a comprehension or other perspective, no substantive issues were identified. The pre-test data was carefully reviewed to ensure accuracy and identify any aspects that needed to be modified. The telephone programming was determined to function as intended and no substantive changes were made. No survey data was affected as an outcome of this procedure. A pre-test report was prepared outlining the results. Fieldwork was monitored to ensure quotas were being met and to determine if there were any challenges via the call disposition/reasons for non-response information.

The survey required a mean average of approximately 15 minutes for respondents to complete.

A minimum of 5 callbacks each for landline and cellular telephone numbers was instituted. Calls and callbacks were staggered throughout the day, during daytime and evening hours, with calls limited to no later than 9 p.m. in any given time zone.

Narrative Research and its data collection partner employed a number of techniques for keeping response rates as high as possible:

· training of all interviewers in telephone and interview techniques and thorough project briefings to guarantee professional and thorough data collection activities

· 100% supervision of all interviewing by experienced supervisors

· continuous on-line monitoring of interviews in progress by supervisors (10% monitored or called back for verification)

· pre-testing of all survey instruments at the design stage by senior field personnel to ensure they provided the best possible respondent experience

· in-house sample development through consultation between and among Narrative Research staff members to ensure the project’s final sample or contact records were created in a consistent manner

· call rules to keep response rates high and minimize non-response bias, including a minimum of 5 telephone attempts to a telephone number before classifying it as not available (i.e. the original attempt plus 4 callbacks), calling at different times of the day, and arranging callbacks

If an interviewer did not speak the requested interview official language, the interview was transferred to another interviewer or a callback was arranged within 20 minutes (or at another time if requested by the respondent). When calling to a particular location with a predominant language (e.g. French in Quebec), calling was done by interviewers who spoke the predominant language to minimize the number of transfers required. The survey tables in the following pages list regional, gender and age data in terms of the actual distribution of adult Canadians as catalogued in the 2016 Statistics Canada Census.

As well, the general regional, gender and age quota targets are detailed (both in terms of the actual number of surveys completed and the percentage of all surveys completed). The tables present data with the weighted and unweighted number as well as percentage of surveys collected, for relevant demographic dimensions.

Data tabulation: A total of 30 overlapping or interlocking statistical weighting cells were created from the study design using the weighting factors of:

a region (5): Atlantic, Quebec, Ontario, Prairies, BC/Territories

b age group (3): 18-34 years of age, 35-54 years of age, and 55 years of age or older

c gender (2): male, female

The 30 overlapping or interlocking statistical weighting cells thus were derived from region (5) x age (3) x gender (2) dimensions = 30 unique statistical weighting cells. Population data for the 30 statistical weighting cells were obtained from the 2016 Canadian census, (the census for which data was most recently available).

It is important to note that the regions used for the purposes of the data collection quotas differ in certain instances from the regions used for tabulation weighting purposes. This more “granular” approach was employed so as to maximize the regional representativeness of the data being tabulated.

It should also be noted that a small number of individuals were not able to be placed into one of the 30 weighting cells as they preferred not to provide a response to the gender question. For tabulation purposes, these individuals were given a weighting value of 1.0.

Telephone Survey (Percentages may not sum exactly to 100%, owing to rounding)
	2016 Census	Quota Targets		Surveys Completed (Unweighted)		Surveys Completed (Weighted)
	2016 Census	(n=)	(%)	(n=)	(%)	(n=)	(%)
Region
Atlantic	6.8%	109	9.1%	114	9.5%	84	7.0%
Quebec	23.4%	277	23.1%	270	22.5%	277	23.1%
Ontario	38.3%	472	39.3%	464	38.7%	459	38.3%
Western/Northern	31.5%	342	28.5%	352	29.3%	380	31.7%
Gender¹
Male	48.6%	632	52.7%	632	52.7%	567	47.3%
Female	51.4%	536	44.7%	536	44.7%	601	50.1%
Age
18-34	27.4%	284	23.7%	279	23.3%	331	27.6%
35-54	34.1%	372	31.0%	372	31.0%	411	34.3%
55+	38.6%	551	45.9%	549	45.8%	458	38.2%

1 A total of 32 respondents who identified as gender diverse or preferred not to provide their gender are not presented in the table, as no quota targets were set.

Participation/response rate: The rate indicated in the following table was derived from the principal elements of the formula recommended by the Government of Canada’s Public Opinion Research Directorate.

Completion Results
A. Total Numbers Attempted	175,794
Total Invalid Numbers	72,409
Discontinued Number/Not in Service	69,536
Dead Air	2,393
Fax/Modem	257
Business Number	223
B. Total Unresolved Numbers (U)	80,703
Busy	6,530
Answering Machine	32,080
No Answer	41,021
Scheduled Call-Back	1,072
C. In-scope Non-Responding Units (IS)	21,117
Household refusal	955
Respondent refusal	19,344
Illness, Incapable	106
Qualified Not Available	0
Language Problem	552
Qualified Respondent Break-Off	160
D. Responding Units (R)	1,565
Quota Full	170
Disqualify	195
Completed Interviews	1,200
Participation/Response Rate [R ÷ (U + IS + R)] = [1,565 ÷ (80,703 + 21,117 + 1,565)]	1.5%

Non-response bias analysis

All surveys are potentially subject to bias or error. When a survey is conducted with a sample of the population, there are 2 general classes of bias or error: sampling error, which is quantifiable, and non-sampling error, which is typically not quantifiable. Sampling error arises when interviews are conducted with only a subset of the population, and thus is it possible that results obtained from this group of respondents are not reflective of the population as a whole.

In contrast, non-sampling error encompasses a number of different types of errors, including coverage error, measurement error, non-response error and processing error.

For the current telephone survey, the actual achieved margins of error provide a reflection of the sampling error and are presented in the table below:

Region	Completions	Margin of Error (19 in 20 times)
Atlantic Canada	114	± 9.2 percentage points
Quebec	270	± 6.0 percentage points
Ontario	464	± 4.5 percentage points
Western/Northern Canada	352	± 5.2 percentage points
Total	1,200	± 2.8 percentage points

With respect to non-sampling error, several steps were taken to minimize bias. The survey utilized survey programming technology to ensure proper survey skip patterns were followed and to minimize errors due to data entry and data capture. The French and English survey instruments themselves were pre-tested with a small sample of respondents to ensure the survey material was easily understood by respondents and that the resultant data were being captured properly. Interviewers were also trained and supervised.

In terms of coverage, the telephone survey was conducted from a random sampling of a robust sample frame of landline and cellular numbers. Quotas were established for demographic groups traditionally regarded as central in quantitative survey research—gender, age and region/province. The final dataset was statistically weighted to closely match the true distribution of these dimensions as reflected in the 2016 Statistics Canada census. The statistical weights implemented were relatively small, given that the data collected already closely matched the actual distribution of adult Canadians along these demographic dimensions.

Appendix A: Questionnaire

Global Affairs Canada

2022 Canadian Attitudes toward Development Survey

Public Opinion Questionnaire

Section A : Introduction

Hello/Bonjour, My name is ___, representing Narrative Research. Our company is today conducting a survey on behalf of the Government of Canada about international development assistance. The survey takes about 15 minutes and is voluntary and completely confidential. Your answers will remain anonymous and your answers will not be attributed to you personally in any way. Please advise if you have a disability that makes it difficult to complete today’s survey by telephone, and we will be pleased to take steps to provide you with another survey method. Would you prefer that I continue in English or French? (IF NEEDED: Je vous remercie. Quelqu'un vous rappellera bientôt pour mener le sondage en français.)

IF NEEDED: Your decision to participate is voluntary and will in no way affect your relationship with the Government of Canada. This call may be monitored or recorded for quality control purposes only. The information provided will be administered according to the requirements of the Privacy Act.

IF NEEDED: If you have any questions regarding this survey or would like to verify the legitimacy of this research, please contact Public Opinion Research/Recherche sur l'opinion publique (LCBE) at POR-ROP@international.gc.ca.

A1. May I please speak to a member of the household who is 18 years of age or older? Would that be you? [IF THAT PERSON IS NOT AVAILABLE ARRANGE A CALLBACK] [CODE ONE ONLY]

1 Yes CONTINUE

2 No ASK TO SPEAK TO ELIGIBLE PERSON

3 Refused THANK/DISCONTINUE

A2. [REPEAT INTRODUCTION IF SPEAKING TO A NEW PERSON] Your participation in this survey is voluntary, but would be extremely helpful. Would you be willing to take part in this survey? We can do it now or at a time more convenient for you. [CODE ONE ONLY]

1 Yes, now CONTINUE

2 Yes, but call later SPECIFY DATE/TIME

3 Refused THANK/DISCONTINUE

A3. Have I reached you on your cellphone? [CODE ONE ONLY]

1 Yes CONTINUE

2 No SKIP TO A5

A4. [POSE A4 ONLY IF ‘YES’ IN A3] Are you in an environment that allows you to comfortably continue with this survey? [CODE ONE ONLY]

1 Yes CONTINUE

2 No RESCHEDULE

A5. Do you work in or for any of the following? [READ ITEMS IN ORDER - CODE ONE ONLY FOR EACH ITEM]

a The Government of Canada

b Advertising

c Marketing research

d The media

1 Yes THANK AND TERMINATE IF ‘YES’ TO ANY

2 No CONTINUE

A6. Are you …: [READ ALL FOUR RESPONSES, IN ORDER – CODE ONE ONLY] [ENSURE GENDER MIX IS MET]

1 Male

2 Female

3 Gender diverse, or would you

4 Prefer not to say

A7. In what year were you born? [ENSURE AGE TARGETS ARE MET] [MUST BE 2003 OR EARLIER]

_______________

98 ENTER 4-DIGIT YEAR

9999 Refused

A7a. [POSE ONLY IF ‘REFUSED’ IN Q.A7] Would you be willing to tell me in which of the following age categories you belong? Are you …: [READ RESPONSES IN ORDER]

1 18 to 24

2 25 to 34

3 35 to 44

4 45 to 54

5 55 to 64, or are you

6 65 or older?

VOLUNTEERED

7 Prefer not to answer THANK, TERMINATE, AND RECORD

A8. In which province or territory do you live? [DO NOT READ RESPONSES - CODE ONE ONLY]

1 British Columbia

2 Alberta

3 Saskatchewan

4 Manitoba

5 Ontario

6 Quebec

7 New Brunswick

8 Nova Scotia

9 Prince Edward Island

10 Newfoundland and Labrador

11 Yukon

12 North West Territories

13 Nunavut

Section B: Awareness and Knowledge of International Development

B1. What, if anything, have you read, heard or seen anything recently on Canada’s international development efforts? PROBE: Anything else? [RECORD VERBATIM]

RECORD VERBATIM: ________________________

97 Nothing

98 Don’t know/Refused

B3. Would you say you have a [READ RESPONSES IN ORDER] knowledge of international development? [CODE ONE ONLY]

1 Very good

2 Good

3 Poor, or

4 Very poor

VOLUNTEERED

8 Don’t know/Refused

B4. To the best of your knowledge, in what countries, if any, does Canada provide international assistance? PROBE: Any others? [DO NOT READ RESPONSES – CODE AS MANY AS APPLY]

1 Afghanistan

2 Bangladesh

3 Congo

4 Ethiopia

5 Ghana

6 Haiti

7 India

8 Indonesia

9 Iraq

10 Jordan

11 Kenya

12 Lebanon

13 Mali

14 Mozambique

15 Myanmar

16 Niger

17 Nigeria

18 Pakistan

19 Senegal

20 South Sudan

21 Syria

22 Tanzania

23 Uganda

24 Ukraine

25 Vietnam

26 Yemen

98 Don’t know/Not aware of which countries

97 Refused/No answer

99 Other (SPECIFY: ________________________________)

Section C: Support for International Assistance

C1. As you may or may not know, Canada has for many years provided development support to various countries around the world. On a scale of 1 to 10, where 1 means “strongly oppose” and 10 means “strongly support,” to what extent do you support or oppose Canada’s development efforts in other countries? [DO NOT READ RESPONSES - PROBE TO AVOID ACCEPTING A RANGE]

01 Strongly oppose

10 Strongly support

98 Don’t know/Refused

C3. Now a question about the importance of international development during the COVID-19/coronavirus pandemic. On a scale from 1 to 10, where 1 means you believe international development has become “much less important,” 10 means you believe international development has become “much more important,” and a score of 5 or 6 means its importance has not changed much, in your opinion to what extent has the importance of international development changed during the pandemic? [DO NOT READ RESPONSES - PROBE TO AVOID ACCEPTING A RANGE]

01 Much less important

10 Much more important

98 Don’t know/Refused

C4. When thinking about Canada helping people in developing countries, how does it make you feel? PROBE: Anything else? [Open Ended] [RECORD VERBATIM]

RECORD VERBATIM: _____________

98 Refused/No answer

C5. Please tell me how important you consider each of the following aspects of Canada’s international development efforts. Using a scale from 1 to 10, where 1 means “not at all important” and 10 means “extremely important,” how important would you say it is for Canada to …: [RANDOMIZE AND READ STATEMENTS – REPEAT SCALE AS NEEDED] [PROBE TO AVOID ACCEPTING A RANGE – CODE ONE ONLY PER STATEMENT]

a Speak up in support of human rights

b Help developing countries deal with climate change

c Promote gender equality in developing countries

d Assist women business owners in developing countries

e Help with maternal and newborn health in developing countries

f Assist migrants who have fled their homes because of war, violence or famine

g Provide access to education for children, and particularly girls, in developing countries

h Vaccinate people in developing countries against disease

i Respond to natural disasters in developing countries

01 Not at all important

10 Extremely important

VOLUNTEERED

98 Don’t know/Refused

C6. How would you rate Canada’s performance on each of the following aspects of Canadian international development? Using a scale from 1 to 10, where 1 means “poor” and 10 means “excellent,” how would you rate Canada’s performance in terms of …: [RANDOMIZE AND READ RESPONSES – REPEAT SCALE AS NEEDED] [PROBE TO AVOID ACCEPTING A RANGE – CODE ONE ONLY PER STATEMENT]

a Speaking up in support of human rights

b Helping developing countries deal with climate change

c Promoting gender equality in developing countries

d Assisting women business owners in developing countries

e Helping with maternal and newborn health in developing countries

f Assisting migrants who have fled their homes because of war, violence or famine

g Providing access to education for children, and particularly girls, in developing countries

h Vaccinating people in developing countries against disease

i Responding to natural disasters in developing countries

01 Poor

10 Excellent

VOLUNTEERED

98 Don’t know/Refused

Section D: Communications

D1. On a scale of 1 to 10, where 1 means “a very poor job” and 10 means “a very good job,” how would you rate the performance of the Government of Canada in terms of communicating with Canadians about international development? [PROBE TO AVOID ACCEPTING A RANGE – CODE ONE ONLY]

01 Very poor job

10 Very good job

VOLUNTEERED

98 Don’t know/Refused

D2. How much confidence do you have in Government of Canada information on the following issues? Using a scale from 1 to 10, where 1 indicates you have “no confidence at all,” and 10 indicates you are “extremely confident,” how much confidence do you have in Government of Canada information in the following areas? [RANDOMIZE AND READ STATEMENTS – REPEAT SCALE AS NEEDED] [PROBE TO AVOID ACCEPTING A RANGE – CODE ONE ONLY PER STATEMENT]

a Information on education in developing countries

b Travel information

c Information on promoting gender equality in developing countries

d Information on assisting entrepreneurs in developing countries

e Information on maternal and newborn health in developing countries

f Information on charitable organizations

01 No confidence at all

10 Extremely confident

VOLUNTEERED

98 Don’t know/Refused

D3. How do you prefer to receive information about international development? PROBE: Any others? [DO NOT READ RESPONSES - RECORD FIRST AND SUBSEQUENT MENTIONS]

1 Facebook

2 Instagram

3 Twitter

4 Print newspapers

5 TV

6 Internet

7 Radio

8 Charitable organizations

9 Other (please specify:______________)

96 Do not want to receive information about international development

97 No preference

98 Refused/No answer

D4. Where on the Internet do you go to receive information on international development? PROBE: Anywhere else? [Open Ended-Code - RECORD VERBATIM]

RECORD VERBATIM: _____________

98 Refused/No answer

Section F: Respondent Profile

To finish up, I would like to ask you a few questions about you and your household for statistical purposes only. Please be assured that all of your answers will remain completely confidential.

F1. Which is the highest level of education that you have completed? [DO NOT READ LIST - ACCEPT ONE RESPONSE ONLY]

1 Grade 8 or less

2 Some high school

3 High school diploma or equivalent

4 Registered Apprenticeship or other trades certificate or diploma

5 College, CEGEP or non-university certificate or diploma

6 University certificate or diploma below bachelor’s level

7 Bachelor’s degree

8 Post graduate degree above bachelor’s level

98 Prefer not to answer/Refused

F2. In what country were you born? [ACCEPT ONE RESPONSE ONLY]

1 Canada

99 Other (SPECIFY: ________________________)

VOLUNTEERED

98 Prefer not to say/Refused

F3. Are you a Canadian citizen? [CODE ONE ONLY – DO NOT READ RESPONSES]

1 Yes

2 No

8 Don’t know/Refused

F4. Please tell me which of the following categories best represents your 2021 total household income, before taxes? Please stop me when I read the appropriate category. Would it be …:

[READ RESPONSES IN ORDER - STOP ONCE RESPONDENT CONFIRMS CATEGORY - ACCEPT ONLY ONE RESPONSE]

1 Under $20,000

2 $20,000 to under $40,000

3 $40,000 to under $60,000

4 $60,000 to under $80,000

5 $80,000 to under $100,000

6 $100,000 to under $120,000, or would it be

7 $120,000 or over

VOLUNTEERED

8 Don’t know/Refused

F5. Do you identify as a visible minority? [CODE ONE ONLY – DO NOT READ RESPONSES]

1 Yes

2 No

7 Prefer not to say

F6. Could you please provide the first three digits of your postal code? [RECORD DIGITS]

RECORD DIGITS: ________ ________ ________

997 Don’t know/Refused

F7. Would you describe the area in which you live as being urban, suburban or rural? [DO NOT READ RESPONSES - CODE ONE ONLY]

1 Urban

2 Suburban

3 Rural

8 Don’t know/Refused

F8. And in closing, what language do you speak most often at home? [DO NOT READ RESPONSES - ACCEPT ALL THAT APPLY]

1 English

2 French

3 Other [Specify: _________________________]

7 Prefer not to answer

That concludes the survey. This survey was conducted on behalf of the Global Affairs Canada. In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to participate, it is greatly appreciated.