Connecting with Canadians: Quantitative Research on International Development

Methodological Report

Submitted to: Global Affairs Canada

For more information on this report, please email:
POR-ROP@international.gc.ca

Submitted by:

Table of contents

• Copyright
• Executive Summary
  • Background
  • Research Objectives
  • Target Population
  • Research Usage
  • Expenditure
  • Consent
  • Political Neutrality Statement and Contact Information
• Appendix A
  • Study Methodology
  • Questionnaire Design
  • Sampling
  • Administration
  • Non-Response Bias Analysis

Copyright

Connecting with Canadians: Quantitative Research on International Development

Methodological Report

Prepared for Global Affairs Canada
Supplier Name: Narrative Research Inc.
March 2020

This report presents the methodological details for the Connecting with Canadians: Quantitative Research on International Development conducted by Narrative Research Inc. on behalf of Global Affairs Canada (GAC). The survey for was administered among 1,203 members of the adult Canadian general public aged 18 or older, between February 20 and March 13, 2020.

Ce rapport est aussi disponible en français sous le titre: Tisser des liens avec les Canadiens : recherche quantitative sur le développement international

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Global Affairs Canada. For more information on this report, please contact Global Affairs Canada at:  POR-ROP@international.gc.ca

Lester B. Pearson Building Tower B3
125 Sussex Drive
Ottawa, Ontario   K1A 0G2
Canada

Catalogue Number: FR5-175/2020E-PDF

International Standard Book Number (ISBN): 978-0-660-36065-2

Related publications (registration number: POR 075-19): Connecting with Canadians: Qualitative Research on International Development (registration number: POR 031-19)

Catalogue Number (Final Methodological Report, French): FR5-175/2020F-PDF

ISBN (French): 978-0-660-36066-9

© Her Majesty the Queen in right of Canada, as represented by Global Affairs Canada, 2020

Executive Summary

Background

The project’s Statement of Work notes that survey questions placed on the Privy Council Office’s “Current Issues Survey” in 2018 revealed that only two percent of Canadians have an understanding about the Sustainable Development Goals (2030).  

Research Objectives

The research objectives of the current survey were to:

• Establish a baseline for Canadians’ understanding of international development activities;
• Understand Canadian awareness of international development; and
• Measure each year Canadian understanding of and support for the role the Government of Canada, and GAC particularly, plays in international development.

Target Population

As specified in the Statement of Work (SOW), the target audience includes adult Canadians (general population) 18 years of age and older. A telephone survey of 1,203 was conducted.

Research Usage

The research will support government and departmental priorities. The goal of the department is to increase the awareness of Canadians on international development issues. The immediate need is to find out the best ways to connect with Canadians, as current messaging does not seem to work.

The research will benefits Canadians as they will understand how and why Canada assists people in developing countries.

The results of the research will be used in public outreach and social media messaging targeted at the
Canadian public.

Expenditure

The survey entailed the expenditure of $47,459.15, including HST.

Consent

Narrative Research offers this written consent allowing the Librarian and Archivist of Canada to post, in both official languages, this Methodological Report.

Political Neutrality Statement and Contact Information

I hereby certify as a Senior Officer of Narrative Research that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not contain any reference to electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leader.

Peter MacIntosh
Chief Research Officer & Partner
Narrative Research
pmacintosh@narrativeresearch.ca
902-493-3832

Appendix A

Study Methodology

The survey for was administered among 1,203 members of the adult Canadian general public aged 18 or older, between February 20 and March 13, 2020.

Questionnaire Design

The questions utilized in this study were provided by Global Affairs Canada. Narrative Research reviewed the questions, providing suggestions and prepared them in a format suitable for telephone administration. 

As required by Government of Canada standards, English and French pre-test surveys were conducted.

Sampling

The survey was designed to be administered to a random sample of 1,200 adult Canadians 18 years of age or older. The sample provided for dialing entailed contact records reflecting Random Digit Dialing (RDD), thus engendering a large number of Not In Service (NIS) telephone numbers. For landlines, records were drawn from a list of randomly-selected households compiled from telephone numbers in Canada, drawn from a database that is updated quarterly. Both listed and unlisted numbers were included in the sample.

Narrative Research utilized ASDE Inc.’s Canada Survey Sampler (CSS) sampling software to generate general population telephone landline samples within Canada, which has become a standard software for many companies in Canada. This software has been proven to provide a sample that is equivalent to RDD. Randomly generated cellular numbers were also used.

The sample was stratified by region in order to ensure regional representation with targets as follows:

The target interviews plan entailed overall quotas by age and gender, as well as by region. The gender quotas were approximately 50/50, while the age quotas were broken into three groupings: 18-34, 35-54, and 55 years of age or older.  

Administration

As noted, the survey was conducted via telephone. The survey was programmed by Narrative Research in both English and French via Computer-Assisted Telephone Interviewing software (CATI). Respondents had the option to complete the survey questionnaire in the official language of their choice. Assistance was available from bilingual staff members as required, with ongoing bilingual supervision.

In terms of training, in addition to a thorough general screening and training process, supervisors and interviewers were provided with extensive, customized project-specific training, a review of the questionnaire including specific terminology, acronyms and pronunciations, and background information on the project goals and objectives.

In terms of supervision, as with all projects there was a ratio of one supervisor for every 15 interviewers working. This “floor supervisor” answers questions, handles escalations, ensures that technology is functioning properly, and blind monitors interviewers. In addition, a minimum of 10 percent of interviews were audited through the review of recorded interviews or live monitoring (twice the percentage required by industry guidelines).

The programmed survey was thoroughly tested to ensure question order and skip patterns were properly represented. In addition to this testing, a pre-test was conducted with 10 English and 10 French surveys. The overall purpose of the pre-test was to ensure that:

• The wording of the questions was clearly understood and unambiguous;
• The sequence of the questions was appropriate;
• The necessary response categories had been included for each question; and
• Neither specific questions nor the survey overall evoked a negative reaction or discomfort among respondents.

In addition to the survey questions, pre-test respondents were invited to offer comments on the questionnaire document. When respondents were asked to identify any questions that were problematic from a comprehension or other problematic perspective, no substantive issues were identified.

The pre-test data was carefully reviewed to ensure accuracy and identify any aspects that needed to be modified. A small telephone programming issue was detected and corrected upon examining the pre-test data. No survey data was affected as an outcome of this. A pre-test report was prepared outlining the results. Fieldwork was monitored to ensure quotas were being met, and to determine if there were any challenges via the call disposition/reasons for non-response information.  

The survey required a mean average of approximately 14 minutes for respondents to complete.

A minimum of eight call-backs for landline telephone numbers, and five call-backs for cellular telephone numbers was instituted. Calls and call-backs were varied throughout the day which includes both daytime and evening calling, limited to 9 p.m. in a given time zone.

Narrative Research and its data collection partner employed a number of techniques for keeping response rates as high as possible:

• Training of all interviewers in telephone and interview techniques and thorough project briefings to guarantee professional and thorough data collection activities;
• 100 percent supervision of all interviewing by experienced supervisors;
• Continuous on-line monitoring of interviews in progress by supervisors (10% monitored or called back for verification);
• Pre-testing of all survey instruments at the design stage by senior field personnel to ensure it provides the best possible respondent experience;
• In-house sample development through consultation between and among Narrative Research staff members to ensure the project’s final sample or contact records have been created in a consistent manner; and
• Call rules to keep response rates high and minimize non-response bias including a minimum of nine telephone attempts to a telephone number before classifying as not available (i.e., the original attempt plus eight call-backs), calling at different times of the day, and arranging call-backs.

If an interviewer did not speak the requested interview official language, the interview transferred to another interviewer or a call back arranged within 20 minutes (or at another time if requested by the respondent). When calling to a particular location with a predominant language (e.g., French in Quebec), calling was done by interviewers who speak the predominant language to minimize the number of transfers required.

The tables below for the survey display regional, gender, and age data in terms of the actual distribution of adult Canadians as catalogued in the 2016 Statistics Canada Census. As well, the general regional, gender, and age quota targets are detailed (both in terms of the actual number of surveys completed, and the percentage of all surveys completed). The tables on the pages below present data with the weighted and unweighted number as well as percentage of surveys collected, for relevant demographic dimensions.

Data Tabulation: There were a total of 24 overlapping or interlocking statistical weighting cells created from the study design using the weighting factors of: Region (4: Atlantic, Quebec, Ontario, Western/Northern); Age group (3: 18–34 years of age, 35–54 years of age, and 55 years of age or older); and Gender (2: Male, Female). The 24 overlapping or interlocking statistical weighting cells thus were derived from Region (4) x Age (3) x Gender (2) dimensions = 24 unique statistical weighting cells. Population data for the 24 statistical weighting cells were obtained from the most recent (2016) Census of Canada, and can be found here:
https://www12.statcan.gc.ca/census-recensement/2016/dp-pd/dt-td/Rp-eng.cfm?LANG=E&APATH=3&DETAIL=0&DIM=0&FL=A&FREE=0&GC=0&GID=0&GK=0&GRP=1&PID=109525&PRID=0&PTYPE=109445&S=0&SHOWALL=0&SUB=0&Temporal=2016&THEME=115&VID=0&VNAMEE=&VNAMEF=

It should be noted that a small number of individuals were not able to be placed into one of the 24 weighting cells as they identified as gender diverse, or they preferred not to provide a response to the gender question. For tabulation purposes, these individuals were given a weight value of 1.0.

Participation/Response Rate: The rate below was derived using the principal elements of the formula recommended by the Public Opinion Research Directorate of the Government of Canada:

Non-Response Bias Analysis

Any survey that is conducted is potentially subject to bias or error. When a survey is conducted with a sample of the population, there are two general classes of bias or error: sampling error, which is quantifiable, and non-sampling error, which is typically not quantifiable. Sampling error arises from the fact that interviews are conducted with only a subset of the population, and thus is it possible that the results obtained from this group of respondents is not reflective of the population as a whole. In contrast, non-sampling error encompasses a number of different types of errors including coverage error, measurement error, non-response error, and processing error.

For the current telephone survey, the actual achieved margins of error provide a reflection of the sampling error and are presented in the table below:

With respect to non-sampling error, a number of steps were taken to minimize bias. The survey utilized survey programming technology to ensure proper survey skip patterns were followed and to minimize errors due to data entry and data capture. The French and English survey instruments themselves were pre-tested with a small sample of respondents to ensure the survey material was easily understood by respondents, and that the resultant data were being captured properly. Interviewers were also trained and supervised.

In terms of coverage, the telephone survey was conducted from a random sampling of a robust sample frame of landline and cellular numbers. Quotas were established for demographic groups traditionally regarded as central in quantitative survey research, such as gender, age, and region/province. The final data set was statistically weighted to closely match the true distribution of these dimensions as reflected in the 2016 Statistics Canada census. The statistical weights implemented were relatively small, given that the data collected already closely matched the actual distribution of adult Canadians along these demographic dimensions.