Connecting with Canadians: Quantitative Research on International
Development
Methodological Report
Submitted
to:
Global
Affairs Canada
For more
information on this report, please email:
Submitted
by:
Supplier Name: |
Narrative Research |
Supplier Address: |
5001-7071 Bayers Road, Halifax NS B3L 2C2 |
Contact Phone: |
902.493.3820 |
Contact Fax: |
902.493.3879 |
POR Number: |
POR 090-20 |
Contract Number: |
08873-200490/001/CY |
Contract Value: |
$47,967.65 |
Contract Award Date: |
December 18, 2020 |
Delivery Date: |
March 12, 2021 |
Ce rapport est aussi
disponible en français
Political Neutrality
Statement and Contact Information
Appendix B: Study Questionnaire
Connecting with Canadians: Quantitative Research on
International Development
Methodological Report
Prepared
for Global Affairs Canada
Supplier Name: Narrative Research Inc.
March 2021
This report presents the methodological
details for the Connecting with Canadians: Quantitative Research on International
Development conducted by Narrative Research Inc. on behalf of Global
Affairs Canada (GAC). The survey for was administered among 1,200 members of
the adult Canadian general public aged 18 or older, between January 13 and February
4, 2021.
Ce rapport est aussi
disponible en français sous le titre: Communications avec les
Canadiens et
Canadiennes: recherche quantitative sur le développement international
This publication may be reproduced for
non-commercial purposes only. Prior written permission must be obtained from Global
Affairs Canada. For more information on this report, please contact Global
Affairs Canada at: POR-ROP@international.gc.ca
Lester B. Pearson Building Tower B3
125 Sussex Drive
Ottawa, Ontario K1A 0G2
Canada
Catalogue Number: 978-0-660-37957-9
International
Standard Book Number (ISBN): FR5-175/1-2021E-PDF
Catalogue
Number (Final Methodological Report, French): 978-0-660-37958-6
ISBN
(French): FR5-175/1-2021F-PDF
© Her
Majesty the Queen in right of Canada, as represented by Global Affairs Canada, 2021
Global Affairs Canada indicates that the
baseline wave of the research for this tracking study, collected in February
2020 by Narrative Research on behalf of Global Affairs Canada (GAC), revealed
that “while most Canadians remain unaware
of development efforts, awareness has increased modestly. Canadians are proud
of Canada’s international development activities and also believe they are
important. However, the public perception is that the Government of Canada
performs poorly when communicating about international development and there is
a lack of confidence in Government of Canada information. Finally, the baseline study revealed that
youth prefer the Internet over social media to receive information on Canada’s
development activities, and older Canadians prefer T.V. and print newspapers.”
The research objectives of the current survey
were to:
·
Catalogue
the current level of understanding among adult Canadians concerning international
development activities;
·
Understand
Canadian awareness of international development; and
·
Measure
Canadian understanding of and support for the role the Government of Canada, and GAC particularly,
plays in international development.
The
target audience includes adult Canadians (general population) 18 years of age
and older. A telephone survey of 1,200 was conducted between January 13 and
February 4, 2021.
The
research will support government and departmental priorities. The goal of the
department is to increase the awareness of Canadians on international
development issues. The immediate need is to find out the best ways to connect
with Canadians, as the SOW imparts that the current messaging does not seem to
work. The research will benefits Canadians as they will understand how and why
Canada assists people in developing countries. The results of the research will
be used in public outreach and social media messaging targeted at the Canadian
public.
The survey entailed the
expenditure of $47,967.65, including tax.
Narrative Research offers this written consent
allowing the Librarian and Archivist of Canada to post, in both official
languages, this Methodological Report.
I hereby certify as
Senior Officer of Narrative Research that the deliverables fully comply with
the Government of Canada political neutrality requirements outlined in the
Policy on Communications and Federal Identity and the Directive on the
Management of Communications. Specifically, the deliverables do not include
information on electoral voting intentions, political party preferences,
standings with the electorate, or ratings of the performance of a political
party or its leaders.
Peter MacIntosh
Chief Research Officer & Partner
Narrative Research
pmacintosh@narrativeresearch.ca
902-493-3832
The
survey for was administered among 1,200 members of the adult Canadian general
public aged 18 or older, between January 13 and February 4, 2021.
The questions utilized in this study
were provided by Global Affairs Canada. Narrative Research reviewed the
questions, providing suggestions and prepared them in a format suitable for
telephone administration.
As
required by Government of Canada standards, English and French pre-test surveys
were conducted.
The survey was designed to be administered to a random sample of 1,200 adult Canadians 18 years of age or older. The sample provided for dialing entailed contact records reflecting Random Digit Dialing (RDD), thus engendering a large number of Not in Service (NIS) telephone numbers. For landlines, records were drawn from a list of randomly-selected households compiled from telephone numbers in Canada, selected from a database that is updated quarterly. Both listed and unlisted numbers were included in the sample.
Narrative Research utilized ASDE Inc.’s Canada Survey Sampler (CSS) sampling software to generate general population telephone landline samples within Canada, which has become a standard software for many companies in Canada. This software has been proven to provide a sample that is equivalent to RDD. Randomly generated cellular numbers were also used. The sample was stratified by region in order to ensure regional representation with approximate targets as follows:
Target Completions |
Margin of
Error (19
in 20 times) |
|
Atlantic
Canada |
100 |
± 9.8 percentage points |
Quebec |
280 |
± 5.9 percentage points |
Ontario |
450 |
± 4.6 percentage points |
Western/Northern
Canada |
370 |
± 5.1 percentage points |
Total |
1,200 |
± 2.8 percentage
points |
The target interviews plan entailed overall
quotas by age and gender, as well as by region. The gender quotas were
approximately 50/50, while the age quotas were broken into three groupings:
18-34, 35-54, and 55 years of age or older.
As noted, the survey was conducted via
telephone. The survey was programmed by Narrative Research in both English and
French via Computer-Assisted Telephone Interviewing software (CATI).
Respondents had the option to complete the survey questionnaire in the official
language of their choice. Assistance was available from bilingual staff members
as required, with ongoing bilingual supervision. In terms of training, in
addition to a thorough general screening and training process, supervisors and
interviewers were provided with extensive, customized project-specific
training, a review of the questionnaire including specific terminology,
acronyms and pronunciations, and background information on the project goals
and objectives.
In terms of supervision, as with all projects
there was a ratio of one supervisor for every 15 interviewers working. This ‘floor
supervisor’ answers questions, handles escalations,
ensures that technology is functioning properly, and blind monitors
interviewers. In addition, a minimum of 10 percent of interviews were audited
through the review of recorded interviews or live monitoring (twice the
percentage required by industry guidelines).
The programmed survey was thoroughly tested to
ensure question order and skip patterns were properly represented. In addition
to this testing, a pre-test was conducted with 10 English and 10 French
surveys. The overall purpose of
the pre-test was to ensure that:
·
The wording of the questions was clearly understood
and unambiguous;
·
The sequence of the questions was appropriate;
·
The necessary response categories had been included
for each question; and
·
Neither specific questions nor the survey overall
evoked a negative reaction or discomfort among respondents.
In addition to the survey questions, pre-test
respondents were invited to offer comments on the questionnaire document. When indeed
respondents were asked to identify any questions that were problematic from a
comprehension or other problematic perspective, no substantive issues were
identified. The pre-test data was carefully reviewed to ensure accuracy and
identify any aspects that needed to be modified. A
small telephone programming issue was detected and corrected upon examining the
pre-test data. No survey data was affected as an outcome of this procedure. A
pre-test report was prepared outlining the results. Fieldwork was monitored to
ensure quotas were being met, and to determine if there were any challenges via
the call disposition/reasons for non-response information.
The survey
required a mean average of approximately 15 minutes for respondents to
complete.
A minimum of five call-backs for landline
telephone numbers, and five call-backs for cellular telephone numbers was
instituted. Calls
and call-backs were varied
throughout the day which includes both daytime
and evening calling,
limited to 9 p.m. in a given time zone.
Narrative Research and its data collection partner employed a number of techniques for keeping response rates as high as possible:
·
Training of all interviewers in
telephone and interview techniques and thorough project briefings to guarantee
professional and thorough data collection activities;
·
100 percent supervision of all
interviewing by experienced supervisors;
·
Continuous on-line monitoring of
interviews in progress by supervisors (10% monitored or called back for
verification);
·
Pre-testing of all survey
instruments at the design stage by senior field personnel to ensure it provides
the best possible respondent experience;
·
In-house sample development through
consultation between and among Narrative Research staff members to ensure the
project’s final sample or contact records have been created in a consistent
manner; and
·
Call rules to keep response rates
high and minimize non-response bias including a minimum of five telephone
attempts to a telephone number before classifying as not available (i.e., the
original attempt plus four call-backs), calling at different times of the day,
and arranging call-backs.
If an interviewer did not speak the
requested interview official language, the interview transferred to another
interviewer or a call back arranged within 20 minutes (or at another time if
requested by the respondent). When calling to a particular location with a
predominant language (e.g., French in Quebec), calling was done by interviewers
who speak the predominant language to minimize the number of transfers
required. The tables below for the survey display
regional, gender, and age data in terms of the actual distribution of adult
Canadians as catalogued in the 2016 Statistics Canada Census. As well, the
general regional, gender, and age quota targets are detailed (both in terms of
the actual number of surveys
completed, and the percentage of all
surveys completed). The tables on the pages below present data with the
weighted and unweighted number as
well as percentage of surveys
collected, for relevant demographic dimensions.
Data Tabulation: There
were a total of 24 overlapping or interlocking statistical weighting cells
created from the study design using the weighting factors of: Region (4:
Atlantic, Quebec, Ontario, Western/ Northern); Age group (3: 18–34 years of
age, 35–54 years of age, and 55 years of age or older); and Gender (2: Male,
Female). The 24 overlapping or interlocking statistical weighting cells thus
were derived from Region (4) x Age (3) x Gender (2) dimensions = 24 unique
statistical weighting cells. Population data for the 24 statistical weighting
cells were obtained from the most recent (2016) Census of Canada, and can be
found here:
It should be noted that a small number of
individuals were not able to be placed into one of the 24 weighting cells as
they preferred not to provide a response to the gender question. For tabulation
purposes, these individuals were given a weight value of 1.0.
1 A total of 21 respondents who preferred not
to provide their gender, are not presented in the table, as no quota targets
were set.
Participation/Response
Rate: The rate below was
derived using the principal elements of the formula recommended by the Public
Opinion Research Directorate of the Government of Canada:
Completion Results |
|
A. Total Numbers
Attempted |
139,057 |
Total Invalid Numbers |
66,281 |
Discontinued Number/Not in
Service |
65,831 |
Fax/Modem |
205 |
Business Number |
245 |
B. Total Unresolved
Numbers (U) |
57,784 |
Busy |
3,546 |
Answering Machine |
21,320 |
No Answer |
30,126 |
Scheduled Call-Back |
2,792 |
C. In-scope
Non-Responding Units (IS) |
13,362 |
Household refusal |
965 |
Respondent refusal |
11,836 |
Illness, Incapable |
0 |
Qualified Not Available |
0 |
Language Problem |
446 |
Qualified Respondent
Break-Off |
115 |
D. Responding Units (R) |
1,630 |
Quota Full |
129 |
Disqualify |
301 |
Completed Interviews |
1,200 |
Participation/Response Rate [R ÷ (U + IS + R)]
= [1,630 ÷ (57,784 + 13,362 + 1,630)] |
2% |
Any
survey that is conducted is potentially subject to bias or error. When a survey
is conducted with a sample of the population, there are two general classes of
bias or error: sampling error, which is quantifiable, and non-sampling error,
which is typically not quantifiable. Sampling error arises from the fact that
interviews are conducted with only a subset of the population, and thus is it
possible that the results obtained from this group of respondents is not
reflective of the population as a whole.
In
contrast, non-sampling error encompasses a number of different types of errors
including coverage error, measurement error, non-response error, and processing
error.
For the current
telephone survey, the actual achieved margins of error provide a reflection of
the sampling error and are presented in the table below:
Region |
Completions |
Margin
of Error (19 in 20 times) |
Atlantic Canada |
113 |
± 9.2 percentage
points |
Quebec |
302 |
± 5.6
percentage points |
Ontario |
435 |
± 4.7
percentage points |
Western/Northern Canada |
350 |
± 5.2
percentage points |
Total |
1,200 |
± 2.8 percentage points |
With
respect to non-sampling error, a number of steps were taken to minimize bias.
The survey utilized survey programming technology to ensure proper survey skip patterns
were followed and to minimize errors due to data entry and data capture. The
French and English survey instruments themselves were pre-tested with a small
sample of respondents to ensure the survey material was easily understood by
respondents, and that the resultant data were being captured properly.
Interviewers were also trained and supervised.
In terms
of coverage, the telephone survey was conducted from a random sampling of a
robust sample frame of landline and cellular numbers. Quotas were established
for demographic groups traditionally regarded as central in quantitative survey
research, such as gender, age, and region/province. The final data set was
statistically weighted to closely match the true distribution of these
dimensions as reflected in the 2016 Statistics Canada census. The statistical
weights implemented were relatively small, given that the data collected
already closely matched the actual distribution of adult Canadians along these
demographic dimensions.
Global Affairs Canada
2021 Canadian Attitudes
toward Development Survey
Public Opinion Questionnaire
Hello/Bonjour, My name is ___, representing
Narrative Research. Our company is today conducting a survey on behalf of the Government of
Canada. The survey
takes about 16 minutes and is voluntary and completely confidential. Your
answers will remain anonymous and your answers will
not be attributed to you personally in any way. Would you prefer that I continue in English or
French? (IF NEEDED: Je vous remercie. Quelqu'un vous rappellera bientôt pour
mener le sondage en français.)
IF NEEDED: Your decision to
participate is voluntary and will in no way affect your relationship with the
Government of Canada. This call may be monitored or recorded for quality
control purposes only. The information provided will be administered according
to the requirements of the Privacy Act.
IF NEEDED: If
you have any questions regarding this survey or would like to verify the
legitimacy of this research, you can contact Kevin Chappell at 613-623-0377, or
you may visit canada.ca/por-cra
A1. May I please speak to a member of the
household who is 18 years of age or older? Would that be you? [IF
THAT PERSON IS NOT AVAILABLE ARRANGE A CALLBACK] [CODE ONE ONLY]
1 Yes CONTINUE
2 No ASK TO SPEAK TO ELIGIBLE PERSON
3 Refused
THANK/DISCONTINUE
A2. [REPEAT
INTRODUCTION IF SPEAKING TO A NEW PERSON] Your participation in this survey
is voluntary, but would be extremely helpful. Would
you be willing to take part in this survey? We can do it now or at a time more
convenient for you. [CODE ONE ONLY]
1
Yes, now CONTINUE
2
Yes, but call later SPECIFY DATE/TIME
3
Refused THANK/DISCONTINUE
A3.
Have I reached you on your
cellphone?
[CODE
ONE ONLY]
1
Yes CONTINUE
2
No SKIP TO A5
A4. [POSE
A4 ONLY IF ‘YES’ IN A3] Are you in an environment that allows you to
comfortably continue with this survey?
[CODE
ONE ONLY]
1
Yes CONTINUE
2
No RESCHEDULE
A5. Do you work in or for any of the following?
[READ
ITEMS IN ORDER - CODE ONE ONLY FOR EACH ITEM]
a) The
Government of Canada
b) Advertising
c) Marketing
research
d) The
media
1 Yes THANK AND TERMINATE IF ‘YES’ TO ANY
2 No CONTINUE
A6. Are
you …:
[READ
ALL FOUR RESPONSES, IN ORDER – CODE ONE ONLY] [ENSURE GENDER MIX IS MET]
01 Male
02 Female
03 Gender Diverse, or would you
04 Prefer not to say
A7. In what year were you born?
[ENSURE AGE TARGETS
ARE MET] [MUST BE 2002 OR EARLIER]
98 – ENTER 4-DIGIT YEAR
9999 – Refused
A8. In which
province or territory do you live?
[DO NOT READ
RESPONSES - CODE ONE ONLY]
01 British
Columbia
02 Alberta
03 Saskatchewan
04 Manitoba
05 Ontario
06 Quebec
07 New
Brunswick
08 Nova
Scotia
09 Prince
Edward Island
10 Newfoundland
and Labrador
11 Yukon
12 North
West Territories
13
Nunavut
Section B. Awareness
and Knowledge of International Development
B1. What, if anything, have you read, heard or seen anything recently on Canada’s international
development efforts? PROBE: Anything
else?
[RECORD VERBATIM]
RECORD VERBATIM:
________________________
97 Nothing
98 Don’t Know/Refused
B3. Would
you say you have a [READ RESPONSES IN ORDER] knowledge
of international development?
[CODE ONE ONLY]
1
Very good
2
Good
3
Poor, or
4
Very Poor
VOLUNTEERED
8 Don’t know/Refused
B4. To the best of your knowledge, in what countries, if any, does
Canada provide international assistance? PROBE: Any others? [DO NOT READ RESPONSES – CODE AS MANY AS
APPLY]
01 Afghanistan
02 Bangladesh
03 Congo
04 Ethiopia
05 Ghana
06 Haiti
07 India
08 Indonesia
09 Iraq
10 Jordan
11 Kenya
12 Lebanon
13 Mali
14 Mozambique
15 Myanmar
16 Niger
17 Nigeria
18 Pakistan
19 Senegal
20 South Sudan
21 Syria
22 Tanzania
23 Uganda
24 Ukraine
25 Vietnam
26 Yemen
98 Don’t
know/Not aware of which countries
97 Refused/No answer
99
Other (SPECIFY: ________________________________)
B5. Which phrase among the following best describes Canada’s
international development activities? [READ &
RANDOMIZE – CODE ONE ONLY]
1
International
Aid
2
International
Development
3
International
assistance
4
International
cooperation
VOLUNTEERED
8 Don’t know/Refused
Section C. Support for International
Assistance
C1. As you may or may not know,
Canada has for many years provided development support to various countries
around the world. On
a scale of 1 to 10, where 1 means “strongly oppose” and 10 means “strongly
support,” to what extent do you support or oppose Canada’s development efforts
in other countries? [DO NOT READ
RESPONSES - PROBE TO AVOID ACCEPTING A RANGE]
01
Strongly oppose
10 Strongly support
98 Don’t Know/Refused
C3.
The
COVID-19/coronavirus pandemic shows that to address global problems,
international development is more important than ever. On a scale of 1 to 10, where 1 means you
“strongly disagree” and 10 means you “strongly agree” to what extent do you
agree or disagree that international development is more important than ever? [DO NOT READ RESPONSES -
PROBE TO AVOID ACCEPTING A RANGE]
01
Strongly disagree
10 Strongly agree
99 Don’t Know/Refused
C4. When thinking about Canada helping people in
developing countries, how does it make you feel? PROBE: Anything else? [Open
Ended] [RECORD
VERBATIM]
RECORD VERBATIM: _____________
98 Refused/No answer
C5. Please tell me how important you consider each of the following
aspects of Canada’s international development efforts. Using a scale from 1 to
10, where 1 means “not at all important” and 10 means “extremely important,”
how important would you say it is for Canada to …: [RANDOMIZE AND READ STATEMENTS – REPEAT SCALE AS NEEDED] [PROBE TO
AVOID ACCEPTING A RANGE – CODE ONE ONLY PER STATEMENT]
a.
Speak
up in support of human rights
b.
Help
developing countries deal with climate change
c.
Promote
gender equality in developing countries
d.
Assist
women business owners in developing countries
e.
Help
with maternal and newborn health in developing countries
f.
Assist
migrants who have fled their homes because of war, violence
or famine
g.
Provide
access to education for children, and particularly girls, in developing
countries
h.
Vaccinate
people in developing countries against disease
i.
Respond
to natural disasters in developing countries
01 Not at all important
10 Extremely important
VOLUNTEERED
98 Don’t Know/Refused
C6. How would you rate Canada’s performance on each of the following
aspects of Canadian international development? Using a scale from 1 to 10,
where 1 means “poor” and 10 means “excellent,” how would you rate Canada’s
performance in terms of …:
[RANDOMIZE
AND READ RESPONSES – REPEAT SCALE AS NEEDED] [PROBE TO AVOID ACCEPTING A RANGE
– CODE ONE ONLY PER STATEMENT]
a.
Speaking
up in support of human rights
b.
Helping
developing countries deal with climate change
c.
Promoting
gender equality in developing countries
d.
Assisting
women business owners in developing countries
e.
Helping
with maternal and newborn health in developing countries
f.
Assisting
migrants who have fled their homes because of war, violence
or famine
g.
Providing
access to education for children, and particularly girls, in developing
countries
h.
Vaccinating
people in developing countries against disease
i.
Responding
to natural disasters in developing countries
01
Poor
10
Excellent
VOLUNTEERED
98 Don’t Know/Refused
Section D. Communications
D1. On a
scale of 1 to 10, where 1 means “a very poor job” and 10 means “a very good
job,” how would you rate the performance of the Government of Canada in terms
of communicating with Canadians about international development?
[PROBE TO AVOID ACCEPTING A
RANGE – CODE ONE ONLY]
01
Very poor job
10 Very good job
VOLUNTEERED
98 Don’t Know/Refused
D2. How much confidence do you have in Government of Canada
information on the following issues? Using a scale from 1 to 10, where 1
indicates you have “no confidence at all,” and 10 indicates you are “extremely
confident,” how much confidence do you have in Government of Canada information
in the following areas?
[RANDOMIZE
AND READ STATEMENTS – REPEAT SCALE AS NEEDED] [PROBE TO AVOID ACCEPTING A RANGE
– CODE ONE ONLY PER STATEMENT]
a.
Information
on education in developing countries
b.
Travel
information
c.
Information
on promoting gender equality in developing countries
d.
Information
on assisting entrepreneurs in developing countries
e.
Information
on maternal and newborn health in developing countries
f.
Information
on charitable organizations
01 No confidence at all
10
Extremely confident
VOLUNTEERED
98 Don’t Know/Refused
D3. How do you prefer to receive information about international development?
PROBE: Any others? [DO NOT READ
RESPONSES - RECORD FIRST AND SUBSEQUENT
MENTIONS]
01
Facebook
02
Instagram
03
Twitter
04
Print newspapers
05
TV
06
Internet
07
Radio
08
Charitable organizations
09
Other (please specify: ______________)
96 Do not want to receive information about
international development
97 No preference
98 Refused/No answer
D4. Where on the Internet do you go to receive
information on international development? PROBE:
Anywhere else? [Open Ended-Code - RECORD VERBATIM]
RECORD VERBATIM: _____________
98 Refused/No answer
Section F. Respondent Profile
To finish up, I would like to ask you a few questions about you
and your household for statistical
purposes only.
Please be assured that all of your answers will remain completely confidential.
F1. Which is the highest level
of education that you have completed?
[DO
NOT READ LIST - ACCEPT ONE RESPONSE ONLY]
01
Grade 8 or less
02 Some
high school
03 High
school diploma or equivalent
04
Registered Apprenticeship or other trades certificate or diploma
05
College, CEGEP or non-university certificate or diploma
06
University certificate or diploma below bachelor’s level
07
Bachelor’s degree
08 Post
graduate degree above bachelor’s level
98
Prefer not to answer/Refused
F2. In
what country were you born?
[ACCEPT ONE RESPONSE
ONLY]
01
Canada
98
Other (SPECIFY:
________________________)
VOLUNTEERED
98
Prefer not to say/Refused
F3. Are you a
Canadian citizen?
[CODE ONE ONLY – DO NOT READ
RESPONSES]
1 Yes
2 No
8 Don’t
Know/Refused
F4. Please tell me which of the following
categories best represents your 2020 total household income, before taxes?
Please stop me when I read the appropriate category. Would it be …:
[READ RESPONSES IN ORDER
- STOP ONCE RESPONDENT CONFIRMS CATEGORY - ACCEPT ONLY ONE RESPONSE]
1 Under $20,000
2 $20,000 to under $40,000
3 $40,000 to under $60,000
4 $60,000 to under $80,000
5 $80,000 to under $100,000
6 $100,000 to under $120,000,
or would it be
7 $120,000 or over
VOLUNTEERED
8 Don’t Know/Refused
F5. Do you identify as a visible minority?
[CODE ONE ONLY – DO NOT
READ RESPONSES]
1 Yes
2 No
7 Prefer
not to say
F6. Could you please provide the first three digits of your postal code?
[RECORD DIGITS]
RECORD DIGITS: ________ ________
________
997 Don’t Know/Refused
F7. Would you describe the area in which you live as being urban, suburban or rural?
[DO NOT READ RESPONSES - CODE ONE ONLY]
1 Urban
2 Suburban
3 Rural
8 Don’t Know/Refused
That concludes the survey. This survey was conducted on behalf of the Global Affairs Canada. In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to participate, it is greatly appreciated.