Standards for the Conduct of Government of Canada Public Opinion Research - Telephone Surveys

Table of Contents

• IMPORTANT NOTES

  • A. APPLICABILITY OF THESE STANDARDS
  • B. APPLICABILITY OF OTHER STANDARDS
  • C. EXCEPTIONS
  • D. FOUR-YEAR REVIEW

• 1. PROPOSAL DOCUMENTATION

  • 1.1. Introduction
  • 1.2. Technical Specifications of the Research

• 2. QUESTIONNAIRE DESIGN

  • 2.1. Standards

• 3. PRE-TESTING

  • 3.1. Standards

• 4. SAMPLING PROCEDURES

  • 4.1. General
  • 4.2. Probability Sampling
  • 4.3. Non-Probability Sampling
  • 4.4. Quota Sampling
  • 4.5. Multi-Mode Surveys
  • 4.6. Census Surveys

• 5. RETAINING PUBLIC CONFIDENCE

  • 5.1. Respondent Rights
  • 5.2. Avoidance of Harassment
  • 5.3. Use of Interactive Voice Response

• 6. DATA COLLECTION FROM CHILDREN, YOUNG PEOPLE OR VULNERABLE RESPONDENTS

  • 6.1. General
  • 6.2. Consent of Responsible Adult
  • 6.3. Adult Accompaniment
  • 6.4. Collection of Personal Information

• 7. DATA COLLECTION

  • 7.1. Project Briefing
  • 7.2. Call-Backs
  • 7.3. Fieldwork Validation
  • 7.4. Data Storage
  • 7.5. Incentives/Honoraria

• 8. OUTCOME RATES

  • 8.1. General
  • 8.2. Outcome Rates Definitions
  • 8.3. Case Category Definitions
  • 8.4. Final Case Dispositions and Response or Participation Rate Formula
  • 8.5. Government of Canada's Mandatory Requirements

• 9. DATA ENTRY

• 10. CODING

  • 10.1. Developing Code Frames
  • 10.2. Code Frame Approval/Coding Procedures

• 11. DATA EDITING/IMPUTATION

  • 11.1. Standards

• 12. DATA ANALYSIS

  • 12.1. Inferences and Comparisons
  • 12.2. Analysis Records
  • 12.3. Data Analysis Verification

• 13. DELIVERY OF DATA TABLES

  • 13.1. Delivery of Stand-alone Hard or Soft Copy of Data Tables
  • 13.2. Electronic Data Delivery
  • 13.3. Respondent Confidentiality

• 14. DATA SECURITY

  • 14.1. Retention of Technical Data
  • 14.2. Protection of Data/Servers
  • 14.3. Temporary Storage of Data on Servers
  • 14.4. Transmission of Data Internationally
  • 14.5. In the Event of Any Data Breach

• 15. MANDATORY SURVEY REPORT REQUIREMENTS

  • 15.1. Library and Archives Canada
  • 15.2. General
  • 15.3. Background
  • 15.4. Sample
  • 15.5. Data Collection
  • 15.6. Quality Controls
  • 15.7. Results
  • 15.8. Appendices

• ENDNOTES

A. APPLICABILITY OF THESE STANDARDS

These standards apply to custom telephone public opinion research surveys conducted for the Government of Canada by contracted third parties, including questions added to omnibus/syndicated surveys. Custom studies are those in which ownership of the data rests with Canada. The standards do not apply to syndicated studies (those in which ownership of the data rests with the researcher) unless the Government of Canada contracts to place questions in addition to those already included in those studies. They also do not apply to surveys undertaken directly by departments within the Government of Canada itself. Nonetheless, PWGSC invites all individuals to view these standards as a resource in support of the planning and execution of their internal survey research projects.

To ensure that these standards are a primary consideration for researchers as they develop survey proposals, we recommend the following statement be included in all survey-related Statements of Work and Requests for Proposals issued by the Government of Canada:

The Standards for the Conduct of Government of Canada Public Opinion Research must be respected and applied in whole to all aspects of the conduct of the research. The project authority must be informed should the firm determine that certain elements of the standards cannot be accommodated in the course of the contract. Justification must be provided and permission received from the project authority in writing in these instances.

These standards are to be understood as in addition to relevant laws, regulations, and policies, e.g., the Personal Information Protection and Electronic Documents Act, the Privacy Act, the Federal Accountability Act, and the Communications Policy of the Government of Canada.

B. APPLICABILITY OF OTHER STANDARDS

Where no relevant Government of Canada Standards exist, researchers must meet or exceed industry standards including those of the Marketing Research and Intelligence Association (MRIA).

C. EXCEPTIONS

These standards were developed with "typical" studies in mind. It is recognized, however, that public opinion research surveys have individual characteristics, which can sometimes clash with a "one size fits all" approach. Therefore, while departments and agencies are generally encouraged to apply the standards, they can also choose to waive a standard when they believe the characteristics of a study warrant it.

This document notes some of the more common instances in which it would be appropriate to waive or modify standards.

As departments and agencies are responsible for the quality of the research they undertake, no related procedure is required. In keeping with due diligence however, they must discuss the intention to waive standards with the Public Opinion Research Directorate. They must also consider specifying which standard is being waived and the related rationale in Statements of Work and other documents such as researcher proposals and final reports.

D. FOUR-YEAR REVIEW

The following standards have been developed to help ensure uniform high quality in public opinion research surveys conducted for the Government of Canada. It is recognized that changing technologies and other factors may necessitate their revision over time. As such, the intention is to review the standards every four years.

1. PROPOSAL DOCUMENTATION

Public Opinion Research contracts are based on the accepted final supplier proposal; therefore it is important that the proposal include sufficient detail to ensure that all requirements are clearly documented.

1.1. Introduction

1.1.1. Purpose

1. Describe the researcher's understanding of the problem/issues to be investigated and how the Government of Canada will use this information.

1.1.2. Research Objectives

1. Detail the information needs/research questions the research will address.

1.2. Technical Specifications of the Research

1.2.1. Overview

1. Provide a brief statement summarizing:

   1. data collection method, including the method(s) used to obtain a sample (probability survey, non-probability survey or census) and rationale for proposed methodology;
   2. expected final sample size;
   3. target population.

2. The researcher must recommend a research design that they believe is cost efficient, will produce quality results, and is appropriate to meet the objectives of the research. The researcher must also provide a rationale for their recommendation.

   If the researcher's design is the same as that described in the Statement of Work the researcher must state that they believe the design in the Statement of Work is cost efficient, will produce quality results, and is appropriate to meet the objectives of the research.

3. The researcher must state that the survey will be registered with the Research Registration System maintained by the MRIA (www) and commit to informing the client when the registration has been completed.

4. The researcher must identify the individual(s) who will have supervising authority and accountability for the project. The proposal must state that these individual(s) will only be replaced in consultation with the client and following written agreement from PWGSC.

5. The researcher must identify all subcontractors used for field work and/or data collection.

6. The researcher must state that all field staff directly involved in data collection, including interviewers, will be located in Canada, and that survey data will be stored on servers and back-up servers located solely in Canada (as specified in detail in subsection 14.2.).

7. The researcher must state that a signed statement of political neutrality will be submitted as part of the final research report.

8. The researcher must agree that the executive summary and report may be posted on the Library and Archives Canada (LAC) Web site or on any other Government of Canada Web sites.

1.2.2. Sample/Sampling Details

1. Provide details related to target population:

   1. the definition of the target population in terms of its specific characteristics and geographic scope, including the assumed incidence of the population and any key sub-groups, and how the incidence was determined/obtained (e.g., supplied by the client);
   2. the total sample size and the sample sizes of any key sub-groups.

2. Describe the sampling procedures, including:

   1. the sample source;
   2. the sample frame;
   3. whether a sample survey or a census will be conducted and, if a sample, whether probability or non-probability sampling will be applied (see section 4.).

3. Explain respondent selection procedures.

4. Indicate the number of call-backs and explain call-back procedures.

5. Define respondent eligibility/screening criteria, including any quota controls.

1.2.3. Response Rate/Participation Rate and Error Rate

1. State the expected response rate/participation rate for the sample(s). State expected response rates/participation rates for each mode when more than one mode is proposed (i.e. separately for online and telephone components). Similarly, if a component of the research will recruit respondents via their cell phones, the expected response rate/participation rate for that component must be stated separately as well. The expected response rate/participation rate is an estimation based on various factors such as previous response rates/participation rates, trends etc. The proposal must include a brief discussion of the factors that might cause the actual response/participation rate to fall short of the stated target.

2. For probability samples, state the level of precision, including the margin of sampling error and confidence interval for the sample(s). (See subsection 4.2. for detailed requirements related to margin of sampling error.)

3. Describe specific procedures that will be applied to eliminate or minimize the impact of any sampling or non-sampling errors, including non-response bias, on the research findings.

1.2.4. Description of Data Collection

1. State the method of data collection (in this case telephone interviewing).

2. Provide details on respondents' incentives/honoraria where applicable, including rationale.

3. If telephone survey respondents will be recruited from panels, a description of the following must be provided, at a minimum (when multiple panels are used, the information must be provided for each):

   1. active panel size (provide the definition of "active");
   2. panel recruitment;
   3. panel monitoring;
   4. panel maintenance;
   5. panel refreshment.

4. Describe how language requirements will be addressed, if other than English and French.

5. Describe the planned fieldwork validation methods and procedures.

6. Describe how the rights of children, youth and vulnerable respondents will be respected, if applicable (see section 6.).

7. For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method. When it is proposed to recruit at least some respondents via their cell phones, provide a rationale for using this approach in addition to or in place of landline phones.

1.2.5. Questionnaire Design

1. Provide the approximate duration of the questionnaire in minutes.

2. Specify the maximum number of open-ends.

3. Describe how the questionnaire will be pre-tested, including:

   1. the objectives of the pre-test;
   2. the method for the pre-test, including the description of the types of interviewers/researchers that will conduct the pre-test interviews;
   3. the total number of pre-test interviews to be completed;
   4. when questionnaire effectiveness with specific sub-groups is of concern (e.g., due to language, age, level of education, etc.), the number in each such subgroup to be included in the pre-test;
   5. how the results of the pre-test will be documented and communicated to the client.

Note: If no pre-test is to be conducted, a rationale must be provided.

1.2.6. Description of Data Processing/Data Management

1. Where known in advance, describe any weighting required.

1.2.7. Data Analysis

1. Briefly describe how the data will be analyzed in order to address the objectives/research questions, including any special analyses (e.g., segmentation).

1.2.8. Deliverables

1. List all deliverables including their scope, format, means of delivery and number of copies, including at minimum:

   1. questionnaire(s), including pre-test, if relevant;
   2. data tabulation/processing;
   3. the report format(s), including the number of copies, language of report;
   4. the nature, location and number of presentations, including the language of presentations.

1.2.9. Project Schedule

1. Provide a detailed work plan with dates and identify responsibilities.

1.2.10. Project Cost

1. Cost information must be presented in the format designated by PWGSC.

2. QUESTIONNAIRE DESIGN

2.1. Standards

1. Survey questionnaires must be designed:

   1. to collect only the information essential to the objectives of the study;
   2. to minimize the burden placed on respondents while maximizing data quality;
   3. to be completed in a maximum duration of 20 minutes. Exceptions could include projects with specialized audiences and those with pre-arranged interviews when the respondent is aware the survey will take longer than 20 minutes. Average questionnaire durations of 15 minutes or less are strongly encouraged in order to minimize respondent burden.

2. The following are required elements of all Government of Canada telephone survey questionnaire introductions:

   1. follow the Official Languages Act and Policies; in particular, introductions must include an active offer of both official languages (English and French); the language that is used first will depend on the province in which the respondent resides (e.g., in Quebec "Bonjour/Hello"; in the Rest of Canada "Hello/Bonjour");
   2. identify the Government of Canada or the department/agency sponsoring the survey;
   3. inform respondents of the general subject and purpose of the study. The subject and purpose of the study may be expressed in very general terms so long as these terms cannot be construed as an attempt to misinform respondents. When the researcher makes a convincing case that the subject and purpose of the survey is such that stating it at the outset will affect respondents' willingness to participate, stating them may be deferred to the conclusion of the study;
   4. inform respondents of the expected length of the interview;
   5. identify the researcher and interviewer (a pseudonym may be used for the interviewer as long as the individual remains identifiable by management internally for quality control purposes);
   6. inform respondents that their participation in the study is voluntary and completely confidential;
   7. inform respondents that their responses remain anonymous. In the exceptional cases where the research objectives require that respondent identity be revealed, the informed consent of the respondent must be obtained;
   8. inform respondents that the survey is registered with the Research Registration System maintained by the MRIA and provide information sufficient for respondents to access the System's Research Verification Service (project registration number and service's Web site URL, e-mail or phone number), if requested.

   Examples of exceptions: The sponsor would not be mentioned until the end of the survey in particular cases (e.g., advertising post-tests); other languages would be mentioned when the survey is available in non-official languages; and information about the availability of reports could be provided to specialized respondents when the incentive to participate outweighs the additional length of the introduction.

   Note: If requested by the respondent, interviewers need to be prepared to repeat the instructions. Interviewers must have information readily available to inform respondents about how respondents were chosen, how privacy is protected, and where the survey results can be obtained.

   For example the following introduction could be used for telephone questionnaires:

   Hello/Bonjour (pause), the Government of Canada is conducting a research survey on SUBJECT. Would you prefer that I continue in English or French? Préférez-vous continuer en français ou en anglais?

   Note: if at this point the respondent prefers to respond in French then the interviewer must be able to either proceed with the interview in French or read the following statement: "Je vous remercie. Quelqu'un vous rappellera bientôt pour mener le sondage en français."

   My name is FIRST NAME of NAME OF FIRM, the company hired to do the survey. The survey takes about ## minutes to complete. It is registered with the Marketing Research and Intelligence Association. Your participation is voluntary and completely confidential. Your answers will remain anonymous. May I continue?

3. The standard set of demographic questions provided below must be included in all telephone surveys of the general Canadian population, unless a convincing argument is made that the research objectives are better served by excluding one or more of them.

   In the case of business-to-business research, these standard demographic questions must be included whenever the unit of analysis is the individual rather than the organization. For example, if a survey's objective is to characterize the management styles of Canadian manufacturing executives, the individual (an executive) is the unit of analysis, and the standard set of demographic questions must be included. If, on the other hand, the research objective is to measure the hiring intentions of Canadian manufacturers, the organization (a Canadian manufacturer) is the unit of analysis, and the standard set of demographic questions need not be included.

   The wording used for each question must be that provided below, unless a convincing argument is made that particular research objectives require alternative wording. Even in these exceptional cases, the terms used and/or categories applied (e.g., for household income) to capture responses must be those provided.

   The data from the age, education, and language questions (along with the recording of geographic location and sex) allows comparison with Statistics Canada census data for the purpose of non-response analysis. The data, along with that from the employment status and income questions, also facilitate the comparison of results between Government of Canada public opinion research studies. (See section 8. for further detail on non-response bias.)

   This does not preclude the use of additional demographic questions.

   Gender [Do not ask: record based on interviewer observation]

   Age

   In what year were you born? [Record year – XXXX]
   [IF PREFERS NOT TO PROVIDE A PRECISE BIRTH YEAR, ASK:]
   Would you be willing to tell me in which of the following age categories you belong?

   18 to 34
   35 to 49
   50 to 54
   55 to 64
   OR 65 or older?
   [DO NOT READ] Refused

   Education

   What is the highest level of formal education that you have completed? [READ LIST]

   Grade 8 or less
   Some high school
   High School diploma or equivalent
   Registered Apprenticeship or other trades certificate or diploma
   College, CEGEP or other non-university certificate or diploma
   University certificate or diploma below bachelor's level
   Bachelor's degree
   Post graduate degree above bachelor's level
   [DO NOT READ] Prefer not to answer

   Mother Tongue

   What is the language you first learned at home as a child and still understand? [READ LIST — ACCEPT ALL THAT APPLY]

   English
   French
   Other [SPECIFY OR DO NOT SPECIFY DEPENDING ON THE NEEDS OF THE STUDY]
   [DO NOT READ] Don't know/Refused

   and/or

   Language Spoken at Home

   What language do you speak most often at home? [READ LIST — ACCEPT ALL THAT APPLY]

   English
   French
   Other [SPECIFY OR DO NOT SPECIFY DEPENDING ON THE NEEDS OF THE STUDY]
   [DO NOT READ] Don't know/Refused

   Employment Status

   Which of the following categories best describes your current employment status? Are you… [READ LIST – ACCEPT ONE ANSWER ONLY]

   Working full-time, that is, 35 or more hours per week?
   Working part-time, that is, less than 35 hours per week?
   Self-employed?
   Unemployed, but looking for work?
   A student attending school full-time?
   Retired?
   Not in the workforce? [FULL-TIME HOMEMAKER, UNEMPLOYED, NOT LOOKING FOR WORK]
   [DO NOT READ] Other – [DO NOT SPECIFY]
   [DO NOT READ] Refused

   Household Income

   Which of the following categories best describes your total household income? That is, the total income of all persons in your household combined, before taxes [READ LIST]?

   Under $20,000
   $20,000 to just under $40,000
   $40,000 to just under $60,000
   $60,000 to just under $80,000
   $80,000 to just under $100,000
   $100,000 to just under $150,000
   $150,000 and above
   [DO NOT READ] Refused

3. PRE-TESTING

3.1. Standards

1. Pre-testing of all components of a new or revised survey questionnaire that may influence data quality, respondent behaviour and interviewer performance is required.

2. Pre-testing must include probing that invites participants recruited for this purpose to provide input about their comprehension of and reaction to the questions. For example, a short series of questions could be included at the end of the pre-test survey. Researchers and clients must agree in advance as to whether probing will take place during or after administering the survey. If requested by the client a full cognitive pre-test must be conducted.

3. To help ensure questionnaire effectiveness with subgroups where there is reason for concern (e.g., due to language, age, level of education, etc.), the socio-demographic characteristics of the targeted participants must be approved by the client before recruiting begins.

4. Provision is to be made for the client to monitor pre-tests conducted by telephone or in-person. Audio recordings of pre-test interviews may be used for this purpose if the client so chooses.

5. A minimum of 10 pre-test interviews are to be completed in each language in which the final survey will be fielded. An exception could be projects with small survey populations, such as a client-satisfaction survey of a small client base. In such cases the researcher must, in consultation with the client, take steps to ensure that the smaller number of pre-tests are sufficient to guarantee questionnaire quality. For example, a cognitive pre-test may be warranted.

6. Pre-test completions shall not be included in the final dataset. An exception could be projects with:

   1. hard-to-reach target groups; or
   2. when no changes are made to the questionnaire.

7. Documentation of the pre-test(s) must be provided to the client before the questionnaire is finalized. The documentation must include (at minimum):

   1. a description of the pre-test approach and number of interviews completed;
   2. findings and any resulting modifications;
   3. average survey completion time;
   4. a statement of whether or not pre-test cases will be retained in the final data set.

The final research report must include this same information.

4. SAMPLING PROCEDURES

4.1. General

1. It must be clearly identified whether probability-based sampling methods, non-probability sampling methods or a census will be used. Researchers must clearly state the target group (universe) definition for the research study (including whether or not individuals without a landline telephone are part of the definition).

2. In the case of probability-based samples, the method used to obtain a representative sample of this target group must be clearly described.

3. The use of non-probability/convenience sampling may not be described as a representative sample. If the cost of sampling low incidence target groups prohibits the use of the usual quota controls to ensure sample representativeness, then such sampling limitations must be clearly stated (as they should for any sampling methodology).

4.2. Probability Sampling

1. The list or sample source must be clearly stated, including any of its limitations/exclusions in representing the universe for the target sample and the potential for bias.

2. National random samples must not exclude any area codes, including the territories.

3. A full description of the sample design and selection procedures must be stated including:

   1. sample stratification variables (if any);
   2. any multi-stage sampling steps taken (e.g., enumeration areas, followed by households, followed by respondents);
   3. at each sampling stage, the method of achieving a probability sample (e.g., random selection) shall be explained, and any subsets of the universe that have been excluded or underrepresented shall be stated (e.g., cell phone only households), although whenever possible, an estimate of the percentage of the universe that has been excluded or underrepresented must be provided;
   4. the number of call-backs and the call-back procedure must be stated;
   5. respondent eligibility/screening criteria must be defined, including any quota controls used (e.g., gender).

4. Records of the disposition of the contact sample must be retained and provided in the final report in sufficient detail to allow secondary analysis of the outcome rates (see section 8.4.).

5. Assuming that proper probability sampling procedures have been followed, the sampling error must then be stated based upon a given sample size at a given confidence level, but researchers must take care to:

   1. adjust for the design effect due to weighting when calculating the margin of sampling error (see endnote S4N1) when subsets of the sample (e.g., regions, age groups, language groups) have been over or under-sampled;
   2. not to mislead clients into believing that a sampling error quoted on the total sample will be the same as that based upon a subset of the total sample;
   3. where possible, express sampling error for key subsets in a survey;
   4. state that sampling error, as measured by the margin of error, is but one source of potential error, and that best practices will be applied to minimize the influence of non-sampling error, including non-response error, on the survey results.

4.3. Non-Probability Sampling

4.3.1. Justification for Use of Non-probability Samples

1. When a choice is made to use a non-probability sample, that choice must be justified, in both the research proposal and the research report. The justification must take into account the statistical limitations in reporting on data from a non-probability sample, and limitations in generalizing the results to the target population.

4.3.2. Sampling for Non-probability Samples

1. As for probability sampling, the list or sample source must be stated, including its limitations in covering the universe for the target sample.

2. The precise quota control targets and screening criteria must also be stated including the source of such targets (e.g., census data or other data source).

3. Deviations from target achievement must be shown in the report (i.e., actual versus target).

4.3.3. Statistical Treatment of Non-probability Samples

1. There can be no statements made about margins of sampling error on population estimates when non-probability samples are used.

2. The survey report must contain a statement on why no margin of sampling error is reported, based on the following template: "Respondents for this survey were selected from among those who have [volunteered to participate/registered to participate] in telephone surveys. The results of such surveys cannot be described as statistically projectable to the target population. [If weighting was done, state the following sentence on weighting:] The data have been weighted to reflect the demographic composition of (target population). Because the sample is based on those who initially self-selected for participation [in the panel], no estimates of sampling error can be calculated."

   This statement must be prominently placed in descriptions of the methodology in the survey report, including the executive summary.

3. In general, for non-probability surveys it is not appropriate to use statistical significance tests. However, tests of significance with non-probability samples are appropriate when the objective is to establish the extent of the relationship among variables. If tests of significance are used with non-probability samples, it must be clearly noted that conclusions from these tests cannot be generalized to any population.

   Any use of descriptive statistics must clearly indicate that they are not formally generalizable to any group other than the sample studied, and there cannot be any formal statistical inferences about how the descriptive statistics for the sample represent any larger population.

   In the case of non-probability surveys that employ an experimental design in which respondents are randomly assigned to different cells, it is appropriate to use statistical significance tests to compare results from different cells.

4.4. Quota Sampling

Quota sampling techniques are typically used for panel surveys and personal intercept studies to achieve sample representativeness. Quotas may also be used to control representativeness on other data collection methodologies.

1. A full description of the regional, demographic or other classification variable controls used for balancing the sample to achieve representativeness must be described.

2. The precise quota control targets and screening criteria must also be stated, including the source of such targets (e.g., census data or other data source).

3. Deviations from target achievement must be shown in the report (i.e., actual versus target).

4.5. Multi-Mode Surveys

Note: Multi-mode surveys are ones where different methods of questionnaire administration are used. They will often involve a combination of online and telephone methods, although there are other possibilities (e.g., in-person, mail, fax).

When a survey is conducted using multiple modes of questionnaire administration:

1. The reasons for using a multi-mode rather than a single-mode method must be stated, both in the research proposal and the survey report.

2. When the plan is to combine data collected via different modes in the data analyses, then steps must be taken to ensure as much comparability as possible across the different survey modes in terms of question wording and presentation of response options.

3. Steps must be taken to ensure avoidance of duplicate respondents in different modes. The steps taken, and the results, must be documented.

4.6. Census Surveys

In a census survey, an attempt is made to collect data from every member of a population. For example, an organization might want to do a survey of all of its employees. In this case, the population is "all of the organization's employees", and this would qualify as a census survey if all employees are invited to participate in the survey.

The list whereby all members of the target population are to be contacted and invited to respond must be clearly described, including any of its limitations/exclusions in representing that target population. Whenever possible, an estimate of the percentage of the population that is excluded from the list must be provided and the potential impact of their exclusion on the research results considered. For example, a list might comprise employee email addresses to be used in a census of an organization where it is known that 5% of employees do not have access to email. In this situation, the census would be of "all employees with email addresses", with due consideration given to the impact on results of any systematic demographic or other differences (location, age, job function) likely to distinguish employees with emails from those without emails.

1. The number of attempted re-contacts and procedure for attempted re-contacts must be stated.

2. Do not state a margin of sampling error, as this does not apply to census surveys because no sample is drawn.

   The survey report must contain a statement on why no margin of sampling error is reported, based on the following template: "Since the entire population of [target population] was invited to participate in this study there is no margin of sampling error to be estimated or reported. The potential impact of non-sampling error due to non-response is discussed in the results section of the report. [If weighting was done, state the following sentence on weighting:] The data have been weighted to reflect the composition of [the target population (if known) or the sampling frame (e.g., client-supplied list)] on the main known characteristics."

3. There is no need to use inferential statistical tests since the results (frequencies, percentages) reported in a census survey describe the entire target population.

   However, it is acceptable to use statistical significance tests to measure differences between sub-groups within the target population.

   As with any surveys, be they sample or census, the impact on the results of non-sampling error due to non-response must be assessed to the extent possible, and appropriate caveats on the interpretation of the results must be clearly stated.

5. RETAINING PUBLIC CONFIDENCE

Researchers must make every reasonable effort to gain, retain and increase public confidence in research organizations and in public opinion research conducted for the Government of Canada.

5.1 Respondent Rights

Researchers must act in a manner respectful of survey respondents and ensure the protection of their rights as these relate to participation in survey research. Interviews must be conducted free of embarrassment and in an environment of trust and goodwill.

5.1.1. Informed Consent and Use of Information

1. In obtaining the necessary agreement from respondents to participate, the researcher must inform them of the study sponsor and of the general subject and purpose of the survey. Exceptions may apply. (See section 2.1., paragraph 2).

2. Survey questions must be limited to the gathering of information relevant to the stated research objectives. The researcher must ensure that the data collected will not be used for any other purpose unless the respondent provides explicit informed consent for that additional use (e.g., to help resolve a customer complaint).

3. The survey data must not under any circumstances be used for direct marketing or other sales approaches to the respondent or the respondent's household.

4. Researchers must answer any questions the respondent might have in a clear, honest and non-deceptive manner.

5.1.2. Right to Refuse

1. Researchers must respect the right of a respondent to refuse to participate in a survey or to respond to particular questions, and to terminate the interview at any point.

5.1.3. Identity of Interviewer, Research Firm

1. Interviewers must always clearly state their name (first name or unique identifying nickname is sufficient) and the name of the research firm. If a respondent requests it, interviewers must provide contact information sufficient to allow them to re-contact the researcher without difficulty.

5.1.4. Protection of Anonymity and Confidentiality

1. The anonymity of respondents must always be preserved unless they have given their informed and explicit consent to the contrary. If these respondents have given informed consent for data to be passed on in a form which allows them to be personally identified, the researcher must ensure that the information will be used for research purposes only, OR, if requested by the respondent, to resolve a customer complaint. The same holds true when respondents' answers are collated, with their informed consent, with pre-existing data that allows such identification (e.g., with an administrative data base). Moreover, such personally identifiable information must not be used for any purpose unrelated to the current study, such as direct marketing, list-building, credit rating, fund-raising, or any marketing activities directed at those individual respondents.

2. Under no circumstances are completed questionnaires or other material (e.g., digital media) containing respondent identity, or information that might allow respondents to be identified, to be released by researchers to clients or other third parties.

3. When verbatim comments are collected they must not be attributed to the respondent, directly or indirectly, without the respondent's explicit informed consent. Care must be taken when reporting verbatim comments that nothing in the comment, including content, vocabulary and/or style of speech, could serve to identify the individual respondent.

4. When recording or observation techniques are used, researchers must advise the respondent at the beginning of the interview of this fact. Respondents' confidentiality must not be jeopardized by the use of such methods.

5.1.5. Use of Respondent Lists

1. Where lists are used for sample selection, the source of the list must be disclosed to potential respondents. When the sample comprises individuals outside the Government of Canada (e.g., clients of a particular department, Canadians more generally), researchers must ensure that lists are permission-based for research purposes, including, when appropriate, obtaining written assurances from the list provider (who may be the client) that this is the case.

2. When the sample comprises employees within the Government of Canada, employee lists may be used even if no explicit permission to use that information for survey purposes has been obtained so long as the survey topic is relevant to the respondents' employment (e.g., an employee satisfaction survey). To the extent possible the researcher must ensure that the data contained in permission-based and employee lists are up-to-date.

3. Client-supplied lists provided for specific projects must not be used for any other projects or for adding names to the researcher's databases. If they have been updated by the researcher during the project (e.g., to correct errors), those updated lists must be returned to the client upon completion of the project. All client-supplied lists still in the researcher's possession at project completion, and after any updates have been forwarded to the client, must be destroyed.

5.2. Avoidance of Harassment

1. The researcher must take all reasonable steps to ensure that respondents are not in any way hindered or embarrassed by any interview, and that they are not in any way adversely affected as a result of it. Researchers must address sensitive subject matter in a way that will minimize the discomfort and apprehension of both respondents and interviewers.

2. Using the respondent's local time, telephone interviewing can only be conducted from 9:00 to 21:00 Monday to Friday, 10:00 to 21:00 Saturday, and 12:00 to 21:00 Sunday and statutory holidays except when a pre-arranged interview is requested by the respondent outside these hours.

3. There must be no or minimal (one second) pause before the interviewer acknowledges that a potential respondent has answered the telephone.

5.3. Use of Interactive Voice Response

1. Characteristics of Interactive Voice Response (IVR) surveys, including the impersonal style conveyed by automation, put that method in conflict with the manner in which the Government of Canada wishes to engage Canadians. IVR can therefore be used only when a convincing case is made that the specific information to be collected is essential for making important decisions and cannot be obtained through other means. For example, IVR may be judged acceptable when the opinions of a hard-to-reach (low incidence) group are critical to the issue at hand and the very high call volume made economical by IVR is likely to markedly increase participation from members of that group in the survey. (IVR may also be used whenever respondents have agreed beforehand to this method.)

2. When IVR is used, the same information required for interviews conducted by live interviewers (sponsor, researcher, participation is voluntary, assurance of confidentiality, etc.) must be included in the IVR survey introduction. Respondents must also be provided early in the introduction with an easy method to opt out of the survey (e.g., by pressing a specific key) so that the call is terminated gracefully and no more calls are made to that number. The same requirements for the time-of-day of calls and delay in acknowledging an answered call (see section 5.2.) apply to the use of IVR.

6. DATA COLLECTION FROM CHILDREN, YOUNG PEOPLE OR VULNERABLE RESPONDENTS

6.1. General

1. Researchers must take special care when interviewing children and young people and respondents who are otherwise vulnerable. (A "child" is defined as "under the age of 13" and a "young person" as "aged 13-15".) The welfare of the children, young people and vulnerable persons must be the researcher's primary consideration. These respondents must not be disturbed or harmed by the interview experience. Sensitive topics or issues that could upset children and young people (e.g., relationships with other children or with parents, sexual activities, use of drugs or alcohol) or vulnerable persons (e.g., social embarrassment related to the source of their vulnerability) must be dealt with using special care to minimize any discomfort or apprehension.

2. All explanations related to data protection, privacy policy, adult consent and other notices must be capable of being understood by children.

3. In addition to these standards, researchers must observe all relevant laws and national codes specifically related to children and young people.

6.2. Consent of Responsible Adult

1. When interviewing a child or young person under 16, interviewers must obtain the consent of the parent or responsible adult (guardian, etc.) before the child/young person is approached for an interview. Sufficient information must be provided to this responsible adult to enable an informed decision about giving such consent. In particular, the researcher must specify the nature of any potentially sensitive questions when seeking parental consent for their children's participation in the research. The name of the person giving the consent must be recorded but normally it is not necessary to obtain the permission in writing.

2. The adult consent only allows the interviewer to invite the child/young person to participate. The child/young person must be given an opportunity to agree or refuse to take part in the survey. To that end, the researcher must inform the child/young person of the purpose of the research in terms likely to be readily understood by them.

3. When conducting telephone interviews, it might be difficult to determine with certainty the age of the respondent and whether or not the consent of a responsible adult is required. Despite those difficulties, the researcher must make every reasonable effort to comply with the rules set out in these standards.

4. An exception is when it is not possible to obtain parental/guardian consent (e.g., street kids) and the benefits of the information are judged to outweigh the possible harm to respondents (e.g., youth engaged in risky behaviour).

6.3. Adult Accompaniment

1. It is recommended that some responsible adult (beside the interviewer) remains close–not necessarily in the same room–while the interview is carried out. The interviewer and researchers must be protected against any misunderstandings or possible allegations of misconduct arising from their dealings with the children or young people taking part in that project.

6.4. Collection of Personal Information

1. Personal information relating to other people (for example, parents) must not be collected from children. Consistent with the Privacy Act and Personal Information Protection and Electronic Document Act, "Personal information" here means information about an identifiable individual. The exception is the contact information of a parent or responsible adult for purposes of obtaining parental consent.

7. DATA COLLECTION

7.1. Project Briefing

1. Fieldworkers shall be given a briefing and/or instructions for each project (or the specifics for each wave of a project), even if they start to take part after it has already started.

   1. The briefing and/or instructions shall include more detail (than is already part of the Introduction for survey respondents) on the background to the project, e.g., why it is being done, by whom and how the information will be used.
   2. The briefing and/or instructions shall be the responsibility of a member of staff with a full understanding of requirements for the project.
   3. While it is preferable to have briefing and/or instructions delivered face-to-face, these may be delivered by phone, by tape (audio or video), in writing or otherwise but records shall be kept to show the coverage of the briefing and instructions and that all fieldworkers allocated to the project have been briefed/instructed. Note that every effort must be made to avoid briefing and/or instruction in writing only.

2. The briefing is to be of sufficient detail as to ensure that each interviewer is familiar with the questionnaire, sample and respondent selection, and potential problem areas in the administration of the survey. Interviewers must be familiar with the question wording before commencing with the interviews. Normal standards of practice include rehearsal interviews and pre-testing.

3. The briefing and/or instructions shall include, as appropriate, the following information:

   1. the fieldwork dates;
   2. the sampling procedures;
   3. quotas to be covered;
   4. methodology;
   5. other special requirements of the project;
   6. instructions/conditions for administration of the questionnaire.

7.2. Call-Backs

1. There will be a minimum of eight call-backs made before retiring a telephone number and substituting it with another number. The call backs must be made at varying days and times over a minimum seven-day period. An exception could be made when the field period is shorter as a result of the need to assess recall of particular events or activities.

2. Every effort must be made to ensure that the respondent is called back if an appointment has been arranged and that the date and time of that appointment are respected.

3. No attempt will be made to call-back refusals.

7.3. Fieldwork Validation

7.3.1. General

1. The purpose of validation is to establish that data collection by fieldworkers has been carried out to project instructions including following questionnaires and general requirements.

2. Validation shall be carried out as soon as practical after the fieldwork and before the resulting data are processed and/or reported to clients.

3. Clients must be given an opportunity to arrange "checks" on the quality/validity of fieldwork subject to any legal constraints that may apply.

4. If requested, clients must be provided with interim data tabulation within the first few days and throughout the fieldwork period. Appropriate care must be taken to ensure that no individual responses can be identified.

7.3.2. Validation Methods

1. Depending on the type of data collection concerned, validation shall be undertaken as specified below, by means of checking the data records produced (e.g., questionnaires, Computer-Assisted Telephone Interview [CATI] data files) and/or respondent re-contact (also termed back checking) and/or monitoring.

2. Checking of data records may include, as appropriate, completeness of the data records, keeping to samples/quotas, consistency of responses and comparison of responses against normal data or between fieldworkers. Checks may be made manually or by computer. This may be linked to the checking of associated records.

3. Respondent re-contact shall include confirmation that the interview or equivalent took place and that instructions were followed, the length of the interview and responses to key questions including demographics and other qualifying questions related to quotas, etc. Re-contact may be by any medium involving direct communication with the respondent (e.g., face-to-face, by phone, by post or by e-mail).

4. Monitoring shall be carried out on an ongoing basis, throughout the field period. Monitoring shall involve listening to interviews at the time they are being carried out, using appropriate equipment or listening to recordings of the interviews. Both interviewer and respondent shall be audible. Interviewers shall be aware that any interview may be monitored but not know whether a specific interview is being monitored. Where interviewing is multilingual, staff fluent in the relevant languages shall carry out monitoring. Monitoring is particularly suitable for phone interviews carried out from suitably equipped central locations (phone units).

7.3.3. Validation Levels

1. For all projects where data collection is by interview, the researcher shall validate the data collection by either monitoring or re-contact to the levels shown below, whether or not data records are also checked. Validation levels shall be calculated on the basis of achieved interviews and achieved validations.

   1. The minimum monitoring validation level shall be 10% of the interviews/cases with at least 75% of the whole interview monitored/listened to.
   2. The minimum re-contact validation level shall be 10% of the interviews/cases.

2. Every fieldworker working on a project must be validated or monitored.

3. In exceptional cases it can be organizationally impossible to carry out re-contact or monitoring to the required level, or at all, or it may be considered contrary to respondents' interest. In such cases project records shall explain why this is the case and what other steps (e.g., checking data records) have been taken to validate data collection.

7.4. Data Storage

1. Data must be stored according to the requirements described in section 14.

7.5. Incentives/Honoraria

1. The details of any incentives/honoraria to be used for an telephone survey must be provided in both the proposal and report documentation, including:

   1. the type of incentive/honoraria (e.g., monetary, non-monetary);
   2. the nature of the incentive (e.g., for monetary, prize, points, donations, direct payments);
   3. the estimated dollar value of the incentives to be disbursed.

8. OUTCOME RATES

8.1. General

The most often reported outcome rate for a survey is the response rate. Other common outcome rates are cooperation, refusal and contact rates. While outcome rates are often used to compare survey instrument and field work quality across survey research projects, the reality is that there are many different ways of calculating and reporting outcome rates, including the ubiquitous response rate. For this reason, it is important that researchers who conduct surveys for the Government of Canada clearly define how their reported response rates (or other outcome rates) have been calculated and provide final dispositions of cases in sufficient detail to allow secondary analysis of outcome rates.

To help researchers calculate and report outcome rates that will allow useful comparisons between telephone surveys, definitions of the most commonly used outcome rates and of the case categories entering into their calculations are provided in this section. Moreover, the Government of Canada's mandatory requirements for managing and reporting outcome rates when conducting telephone surveys are stipulated in subsection 8.5.

8.2. Outcome Rates Definitions

Below is a definition of response rate/participation rate, cooperation, refusal and contact rate. Reporting the response/participation rate (response rate for probability sample surveys and censuses, participation rate for non probability samples) is mandatory; this rate must be included in the final report submitted to Library and Archives Canada as per the POR Contract Regulations. The other rates included below are optional and can assist in better understanding the causes of a low response rate. For example, a researcher might want to point out that even though the response rate is low, it was not due to refusals (shown by a low refusal rate) or because only a few of the attempted contacts were successful (shown by a low contact rate, for instance many emails bounced back in an online survey). This additional information provides greater context and detail on the quality of the data.

Response Rate/Participation Rate:

Total number of responding units divided by the total number of potentially eligible cases (including those whose eligibility could not be determined).

It is important to note that the term "response rate" is reserved for probability sample surveys and censuses, while the term "participation rate" must be used for non-probability sample surveys.

Cooperation Rate:

Total number of responding units divided by the total number of known eligible cases (excluding those whose eligibility could not be determined).

Refusal Rate:

Total number of refusals (including break-offs) divided by the total number of potentially eligible cases (including those whose eligibility could not be determined).

Contact Rate:

Total number of contacts that were made successfully (regardless of whether or not a contact was made with the targeted respondent, for example when a family member who does not qualify intercepts the call) divided by the total number of potentially eligible cases (including those whose eligibility could not be determined).

8.3. Case Category Definitions

Cases involved in a telephone survey using probability samples can be broken down into four broad categories:

1. Invalid numbers – These can only include clearly invalid telephone numbers such as numbers not in service, fax/modem, business/non-residential when surveying households.

2. Unresolved (U) – These include all the cases where it cannot be established whether the call was made to an eligible or an ineligible respondent or unit (e.g., household), for example when it is always busy, there is no response, or an answering machine or voice mail is reached but it cannot be established that this is not a household.

3. In-scope non-responding (IS) – These include all refusals, either implicit or explicit; all non-contacts and early break-offs of known eligible cases; and other eligible non-respondents.

4. Responding units (R) – These include cases (e.g., households) who would have participated but who were disqualified by the interviewer (e.g., not of the language group targeted; no one 18 years old or older living at this address); it also includes all completed interviews or partially completed interviews that meet the criteria set by the researcher to be included in the analysis of the data.

Cases in categories 2 through 4 are all included in the broad category of "potentially eligible" cases. However, cases in category 1 are considered "invalid" and are not included in the calculation of outcome rates.

8.4. Final Case Dispositions and Response or Participation Rate Formula

The following record of final case disposition must be included in the survey final report, making sure to identify the case categories (i.e., Invalid; Unresolved; In-scope non-responding; Responding). The elements provided as examples within each category are for studies where the unit of analysis is the household. They may be adjusted for the type of study conducted (e.g., B2B, specialty groups).

Total telephone numbers used

• Invalid numbers

  • Not in service, fax/modem, business/non-residential

• Unresolved (U)

  • Busy
  • No answer, answering machine

• In-scope non-responding units (IS)

  • Household refusal (stated explicitly)
  • Respondent refusal
  • Language problem
  • Illness, incapable
  • Selected respondent not available
  • Qualified respondent break-off

• Responding units (R)

  • Language disqualify
  • No one 18+
  • Other disqualify
  • Completed interviews

Response Rate/Participation Rate = R/(U+IS+R)

8.5. Government of Canada's Mandatory Requirements

8.5.1. Outcome Rates Targets

1. The telephone survey must be designed to achieve the highest practical rates of success.

8.5.2. Monitoring Outcome Rates During Data Collection

1. Monitoring of call dispositions/reasons for non-response shall be carried out on an ongoing basis, throughout the entire field period.

2. This information must be provided to the client upon request.

8.5.3. Reporting Outcome Rates

1. Researchers providing public opinion survey research services to the Government of Canada must, at a minimum, report the obtained response rate (for probability sample surveys and censuses) or participation rate (for non-probability samples).

2. Other outcome rates may also, optionally, be reported.

3. The report must describe, to the extent possible, the degree to which characteristics of the respondents match or depart from known characteristics (e.g., demographics) of the target population or sample frame, and discuss the possible influence of any resulting non-response bias on the interpretation of survey results (see subsection 15.7.).

8.5.4. Final Dispositions of Cases

1. The report must include the final disposition of cases in sufficient detail to allow accurate secondary analysis of the response/participation rate and other outcome rates.

9. DATA ENTRY

1. It shall be the responsibility of the researcher to ensure that data entry specifications for CATI are correct as specified and accurate, based on the client-approved questionnaire.

2. Upon request, the researcher will provide the client with the CATI version of the questionnaire.

3. In those rare cases where data are captured in paper documents:

   1. The researcher shall document the level of verification to be carried out. A systematic method of verifying data entry shall be carried out on a project or stage/wave. The minimum total percentage verification per project shall be 10% of entries. Procedures shall ensure that there is a systematic method of verification of each operator's work and the verification shall be undertaken by a second person.
   2. If an individual operator's work contains frequent errors, that individual's work (on the project) shall be 100% verified/re-worked. If necessary, appropriate retraining shall be given to that operator until error rates are acceptable.
   3. The researcher shall define the meaning of frequent errors and document that definition.

10. CODING

10.1. Developing Code Frames

1. The initial code list/frame shall be developed based on a systematic review of a minimum of 10% of open-ended responses and 50% of partial open-ended responses, where a frame does not already exist. The initial code list/frame must be provided to the client for approval upon request.

2. The researcher shall ensure that coders working on the project are provided with instructions and training that shall include, as a minimum:

   1. an overview of the project;
   2. identification of questions or variables to be coded;
   3. the minimum proportion or number of a sample (and its make-up) used to produce code frames;
   4. where necessary or appropriate, specific subgroups required to develop code frames (e.g., by region, user or non-user);
   5. guidelines for the inclusion of codes in the code frame (e.g., decisions or rules regarding what must be included or excluded from a given code);
   6. any use to be made of code frames from a previous project or stage;
   7. any other requirements or special instructions specific to the project.

10.2. Code Frame Approval/Coding Procedures

1. Where "don't know" and "no response" have been used, these shall be distinguishable from each other.

2. The researcher shall have clear rules or guidelines for the treatment of responses in "other" or catch-all categories; if the "other" or catch-all category exceeds 10% of responses to be coded, the responses must be reviewed with a view to reducing the size of the group.

3. Researchers shall ensure that there is a systematic method of verifying a minimum of 10% of questionnaires coded per project and the verification shall be undertaken by a second person.

11. DATA EDITING/IMPUTATION

11.1. Standards

1. An accurate record of any changes made to the original data set shall be kept. Comparison to the original data source shall be the first step in the process. Any imputation processes, including the logic of the imputation method(s) used shall be documented and provided to the client. All edit specifications shall be documented.

2. Where forced editing is used, the logic of the forcing shall be documented and test runs carried out, with the results documented to show that the forcing has the desired effect.

3. Data editing/imputation must be used cautiously. The degree and impact of imputation must be considered when analyzing the data, as the imputation methods used may have a significant impact on distributions of data and the variance of estimates.

4. The researcher shall include documentation of any imputation/forced editing, both in a technical appendix and in the final report.

5. In those rare cases where data are captured in paper documents and they need to be hand edited prior to data entry:

   1. It must be possible to distinguish the original answers of the respondent or interviewer from the codes or answers allocated by the person(s) carrying out the editing.
   2. When this type of editing is used, the logic and rules being applied must be documented and any staff working on this element of the project must be briefed as to the types of checks and corrections they may carry out.

12. DATA ANALYSIS

12.1. Inferences and Comparisons

1. Researchers must base statements of comparisons and other statistical conclusions derived from survey data on generally accepted statistical practice.

12.2. Analysis Records

1. The researcher shall keep accurate and descriptive records of the analysis process for a minimum of three years, to ensure that any analysis undertaken can be replicated at a later date.

12.3. Data Analysis Verification

1. The researcher shall have in place procedures to ensure the tabulations and other outputs have been checked.

2. As a minimum, these checks shall verify:

   1. completeness, i.e., that all tables are present as specified, including the results of all reported significance tests;
   2. that abbreviations for headings or open-ended responses accurately reflect the full content;
   3. that the base for each table is correct against other tables or frequency counts;
   4. that the standard breaks/banner points are checked against source questions;
   5. that all derived data items are checked against their source;
   6. that the figures for subgroups and nets are correct;
   7. that there are no blank tables (i.e., with no data);
   8. weighting (e.g., by test tables);
   9. frequency counts prior to running tables, in order both to ensure the accuracy of data and to determine base sizes for subgroups;
   10. spelling and legibility;
   11. that any statistical analysis used is appropriate and correct, both in its descriptive and inferential aspects.

3. For any subsequent outputs, appropriate checks shall be applied.

13. DELIVERY OF DATA TABLES

13.1. Delivery of Stand-alone Hard or Soft Copy of Data Tables

1. When data are reported to the client, such as in a stand-alone hard or soft copy of data tables, the following shall be taken into account, as appropriate:

   1. reference to the actual source question to which the data pertains;
   2. inclusion of a description of any weighting method applied to the data;
   3. clear identification of any subgroups used;
   4. availability of the bases for each question, so that the number of respondents who have actually answered the question is identifiable;
   5. the number or proportion of respondents who replied "don't know" or gave "no answer";
   6. availability of both weighted and unweighted bases;
   7. clear and complete definition and explanation of all variables used in the analysis of the data, including any significance testing, indexing, scoring, scaling and calculations of means, median, modes and standard deviations;
   8. the types of statistical tests being used and their level of precision;
   9. information on cell suppression and other measures to assure confidentiality;
   10. warnings on results which are unreliable due to very small sample sizes.

13.2. Electronic Data Delivery

1. The researcher shall provide the client with a data file.

2. The following shall be checked prior to data release:

   1. compatibility of the file format with the software specification agreed with the client;

   2. completeness (i.e., the correct number of files and records are in each file);

   3. inclusion of all appropriate documentation to allow for replication of the data analysis and additional analyses, including where applicable:

      1. labelling of the contents of the file, i.e., fully labelled variables and value labels;
      2. identification and description of any computed or recoded variables, and instructions on limitations of use;
      3. labelled weighting variables and a description of how these were applied;
      4. all personal identifiers per the Personal Information Protection and Electronic Documents Act have been removed from the files; (i.e. any information that could identify specific individuals);

   4. encryption of files upon request;

   5. presence of viruses in the file.

13.3. Respondent Confidentiality

1. To ensure respondent confidentiality, tabulated data must in no case show demographic or respondent characteristic categories with a column or row total of less than 10 respondents.

2. Particular care must be taken when respondent verbatim statements are included in data tables. Protection of confidentiality may require that larger minimum row and column totals are applied.

14. DATA SECURITY

14.1. Retention of Technical Data

1. The researcher must maintain the technical data on all studies for a period of three years, so that if requested, the study can be replicated.

2. Technical data not already included in the Survey Report/Appendix that must be maintained includes, but is not limited to:

   1. The information pertaining to data processing and analysis, such as:

      1. raw data files;
      2. other electronic files;
      3. code frames;
      4. project files including project management information and survey programming files;
      5. emails and other correspondence.

   2. Where data has been edited, cleaned, recoded or changed in any other way from the format, content and layout of its original format, the original data, final data and programme files, including all documentation related to changes to the data (as a minimum) shall be kept so that the final data set can be easily reconstructed.

14.2. Protection of Data/Servers

1. Protection against illegal or unsanctioned access: Researchers must use up-to-date technologies to protect survey data collected or stored on servers against illegal or unsanctioned access by third parties (i.e. "hacking"). The researcher must also control access to all databases on which any data relating to the survey is stored so that only individuals with the appropriate security clearance are able to access the database, either by using a password or other form of access control (such as biometric controls).

2. Protection against legally-sanctioned access: Because some jurisdictions allow their authorities, under certain circumstances, to access all data stored on servers located in that jurisdiction (e.g., in the United States under provisions of the Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism Act, known as USA PATRIOT Act: see the Treasury Board Secretariat's overview), the researcher must ensure that all call centres and interviewers employed in survey data collection are located in Canada. The researcher must also ensure that all databases containing any information related to the survey are stored on servers and back-up servers located solely in Canada.

   1. If the client has first consented in writing, these call centres and/or servers may be located in another country where:

      1. equivalent protections are given to personal information as in Canada under legislation such as the Privacy Act, R.S. 1985, c. P-21, and the Personal Information Protection and Electronic Documents Act, S.C. 2000, c. 5, and under any applicable policies of the Government of Canada; and
      2. the laws do not allow the government of that country or any other entity or person to seek or obtain the right to view or copy any information relating to the survey without first obtaining the client's written consent.

   2. In connection with giving consent to locating a call centre and/or server in another country, the client may, at their option, require the researcher to provide a legal opinion (from a lawyer qualified in the foreign country) that the laws in that country meet the requirements of paragraph 2) a) above, or may require the researcher to pay for the Government of Canada to obtain such a legal opinion. The Government of Canada has the right to reject any request to store survey data in a country other than Canada if there is any reason to be concerned about the security, privacy, or integrity of the data. The Government of Canada may also require that any data sent or processed outside of Canada be encrypted with Government of Canada-approved cryptography and that the private key required to decrypt the data be kept in Canada in accordance with key management and storage processes approved by the Government of Canada.

   3. The researcher must ensure that all servers, including back-up servers, on which any data relating to the survey is stored are physically and logically independent (meaning there is no direct or indirect connection of any kind) from all other databases, unless those databases are located in Canada (or in another country approved by the client under paragraph 2) a)) and otherwise meet the requirements of this section.

   4. The researcher must ensure that all data relating to the survey is processed only in Canada or in another country approved by the client under paragraph 2) a).

   5. The researcher must ensure that all domestic network traffic (meaning traffic or transmissions initiated in one part of Canada to a destination or individual located in another part of Canada) is routed exclusively through Canada, unless the client has first consented in writing to an alternate route. The client will only consider requests to route domestic traffic through another country that meets the requirements of paragraph 2) a).

3. The researcher must not subcontract (including to an affiliate) any function that involves providing a subcontractor with access to any data relating to the survey unless the client first consents in writing.

4. Protection of data from international surveys: When the target population for a survey comprises individuals resident in a jurisdiction outside Canada, researchers must ensure that data is stored in a manner consistent with the relevant laws of that jurisdiction.

5. Protection against physical damage to servers: Researchers must also put in place measures to ensure the "physical" security of data and servers.

14.3 Temporary Storage of Data on Servers

If the temporary storage of data collected takes place on a server that is operated by another provider, the researcher must place the provider under the obligation to take the necessary steps to ensure that the requirements described in subsection 14.2. are met. Temporary storage of the collected data on the server must be terminated at the earliest possible time.

14.4 Transmission of Data Internationally

Before data is sent over the Internet to another country, researchers must check with competent authorities that the data transfer is permissible. The recipient may need to provide safeguards necessary for the protection of the data and ensure compliance with the requirements described in subsection 14.2.

14.5 In the Event of Any Data Breach

In the event of any data breach, the client must be informed immediately and provided with details about both the nature and the extent of the data breach.

15. MANDATORY SURVEY REPORT REQUIREMENTS

This section describes the survey report requirements that are mandatory under the Public Opinion Research Contract Regulations for all public opinion research conducted by an external supplier for the Government of Canada.

15.1. Library and Archives Canada

It is a legal requirement for institutions to send written public opinion research reports to Library and Archives Canada (LAC) within six months of the completion of the data collection for all reports.

1. Pursuant to the Public Opinion Research Contract Regulations and the Procedures for Planning and Contracting Public Opinion Research, and LAC deposit instructions, final reports must include the following at a minimum:

   1. on the covering page,

      1. the title of the project;
      2. the name of the research firm who entered into the contract;
      3. the contract number and the award date;
      4. the POR Registration Number;
      5. the delivery date (the date the final version of the report was received) by the project authority;
      6. the name of the client department or agency sponsoring the research;
      7. the departmental contact information in the form of a generic email address;
      8. if a report is provided in English only, the statement "Ce rapport est aussi disponible en français sur demande."; and
      9. if a report is provided in French only, the statement "This report is also available in English on request.";

   2. a narrative executive summary, in English and French, submitted separately and consisting of, at a minimum,

      1. a statement of the research purpose and objectives;
      2. a summary of key findings, except where the person who entered into the contract is not responsible for the design, development of the methodology and analysis of the research;
      3. a brief description of the methodology used;
      4. a statement as to the extent to which the findings can be extrapolated to a broader audience;
      5. an indication of total expenditures of the POR study;
      6. an outline of how the results were used, if possible; and if not
      7. how the information is expected to be used;

   3. appendices containing,

      1. a full set of tabulated data, which, to ensure respondent confidentiality, must in no case show demographic or respondent characteristic categories with a column or row total of less than 10 respondents;
      2. sample size, sampling procedures and dates of research fieldwork;
      3. if applicable, weighting procedures, the confidence interval and the margin of error;
      4. if applicable, the response rate/participation rate and method of calculation;
      5. the research instruments in all languages and, if applicable, the test material in all languages in which they were used and tested; and
      6. all other information about the execution of the fieldwork that would be needed to replicate the research initiative.

15.2. General

The following minimum details shall be documented in the project report. These allow the reader to understand the way the research project was conducted and the implications of its results.

15.3. Background

1. Detailed description of background including, at minimum:

   1. purpose;
   2. how the research will be used.

2. Objectives, research questions.

15.4. Sample

1. Detailed description of the sample including:

   1. the target group for the research project;
   2. the achieved sample size against projected sample size and reasons, if relevant, for not obtaining the projected sample;
   3. the sample source (e.g., lists) and sampling method, including the procedure for selecting respondents;
   4. the weighting procedures, if applicable.

2. For non-probability samples, provide a rationale for choosing a non-probability sample.

15.5. Data Collection

1. Detailed description of methodology including:

   1. the dates of fieldwork (each phase/wave if applicable);

   2. the average survey duration and the range;

   3. the data collection method(s), and if applicable:

      1. the type and amount of incentives;
      2. any accessibility provisions for respondents using adaptive technologies.

2. For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method. When at least some respondents were recruited and interviewed via their cell phones, provide a rationale for using this approach in addition to, or in place of, landline phones.

15.6. Quality Controls

1. The estimating and imputation procedures, if applicable.

2. A brief summary of other quality controls and procedures used.

3. For multi-mode surveys, detailed description of any data quality issues arising from combining data collected via different modes/instruments.

15.7. Results

1. An executive summary of key results and conclusions, linked to the survey objectives and research questions.

2. For probability samples, state the level of precision, including the margin of error and confidence interval for the total sample and any key sub-groups.

3. For non-probability samples and census surveys, the report must contain a statement on why no margin of sampling error is reported. Templates for such statements are provided in subsection 4.3.3., paragraph 2), for non-probability sample surveys and in subsection 4.6., paragraph 2) for census surveys.

4. For multi-mode surveys (see subsection 4.5.), the report must discuss whether there are any data quality issues arising from combining data collected via different modes. This could include, for example, discussion of possible impacts of mode on key survey variables, the impact of any differences in response rate/participation rate by mode, and non-response bias analyses by mode.

5. For all types of surveys (probability sample, non-probability sample, census) the report must state the response/participation rate achieved (see subsection 8.4.).

6. All survey reports must contain a discussion of the potential for non-response bias for the survey as a whole and for key survey variables.

   1. When external sources of comparison exist, as is the case when conducting a survey of the general population, the analysis of non-response bias will consist of a comparison of at least three variables in the survey sample with the equivalent parameters of the population, normally available from Statistics Canada. No extra surveys or interviews are required.
   2. If no external sources of comparison exist, key variables in the sample could alternatively be compared to variables in the sampling frame. If no such variables exist, only then should the non-response bias discussion be based on comparison of early versus late responders or on observations made during data collection about characteristics of non-responders.

7. The final dispositions of cases in sufficient detail to allow secondary analysis of outcome rates (see subsection 8.4.).

8. The results that are based on subgroups and the number of cases used in subgroup analysis.

15.8. Appendices

The following materials must be provided in appendices attached to the final report:

1. study materials, including, when relevant, the recruiting instruments, questionnaires, descriptions or representations of any visual or auditory aids, and other data collection documents, in all languages in which the research was conducted;

2. a version of the questionnaires displaying any instructions (e.g., skip, terminate, etc.) needed to understand the logic and flow of the questionnaire;

3. full set of tabulated data (see LAC requirements, subsection 15.1., paragraph 1) c) i.);

4. a statement of political neutrality signed by the researcher, as an appendix.

ENDNOTES

Important Notes

S0N1:

Survey projects may involve more than one methodology (e.g., on-line and telephone surveys). As such, it may be necessary to consult more than one set of standards.

Section 4.

S4N1:

The design effect due to weighting equals one plus the variance of the weights. The 'effective' sample size for which to calculate the margin of sampling error is the actual sample size divided by the design effect. (See: Spencer, B. 2000. An approximate design effect for unequal weighting when measurements may correlate with selection probabilities. Survey Methodology Vol. 26, No. 2, pp. 137-138.)