The Standards for the Conduct of Government of Canada Public Opinion Research - Online Surveys

2009-11-26

Table of Contents

• Important Notes
• 1. Proposal Documentation
• 2. Questionnaire Design
• 3. Pretesting
• 4. Sampling Procedures
• 5. Survey Respondents
• 6. Interviewing Children and Young People
• 7. Data Collection
• 8. Participation Rate/Response Rate
• 9. Coding
• 10. Data Editing/Imputation
• 11. Data Analysis/Reporting
• 12. Delivery of Data Tables
• 13. Retention of Technical Data
• 14. Data Security
• 15. Survey Reports

1. Applicability of these Standards

These standards apply to custom online public opinion research surveys conducted for the Government of Canada including questions added to omnibus/syndicated surveys. Custom studies are those in which ownership of the data rests with Canada. The standards do not apply to syndicated studies (those in which ownership of the data rests with the Researcher) unless the Government of Canada contracts to place questions in addition to those already included in those studies.

These standards are to be understood as in addition to relevant laws, regulations, and policies, e.g. the Personal Information Protection and Electronic Documents Act, the Privacy Act, the Federal Accountability Act, and the Communication Policy of the Government of Canada

Note: Survey projects may involve more than one methodology (e.g., online and telephone surveys). As such, it may be necessary to consult more than one set of standards.

2. Applicability of other Standards

Where no relevant Government of Canada Standards exist, Researchers must meet or exceed industry standards including those of the Marketing Research and Intelligence Association.

3. Exceptions

These standards were developed with "typical" studies in mind. It is recognized, however, that public opinion research surveys have individual characteristics, which can sometimes clash with a "one size fits all" approach. Therefore, while departments and agencies are generally encouraged to apply the standards, they can also choose to waive a standard when they believe the characteristics of a study warrant it.

This document notes some of the more common instances in which it would be appropriate to waive or modify standards.

As departments and agencies are responsible for the quality of the research they undertake no related procedure is required. In keeping with due diligence however, they should discuss the intention to waive standards with the Public Opinion Research Directorate. They should also consider specifying which standard is being waived and the related rationale in statements of work and other documents such as Researcher proposals and final reports.

4. Two-year Review

The following standards have been developed to help ensure uniform high quality in public opinion research surveys conducted for the Government of Canada. It is recognized that changing technologies and other factors may necessitate their revision over time. As such, the intention is to review the standards after an initial two-year period.

1. Proposal Documentation

Introduction

Purpose

1. Describe the Researcher's understanding of the problem/issues to be investigated and how the Government of Canada will use this information.

Research Objectives

1. Detail the information needs/research questions the research will address.

Technical Specifications of the Research

Overview

1. Provide a brief statement summarizing:
   1. data collection method, including rationale for proposed methodology;
   2. total sample size;
   3. target population.
2. The Researcher must recommend a research design that they believe is cost efficient, will produce quality results, and is appropriate to meet the objectives of the research. The Researcher must also provide a rationale for their recommendation.

   If the Researcher's design is the same as that described in the Statement of Work the Researcher must state that they believe the design in the Statement of Work is cost efficient, will produce quality results, and is appropriate to meet the objectives of the research.

Sample/Sampling Details

1. Provide details related to target population:
   1. the definition of the target population in terms of its specific characteristics and geographic scope, including the assumed incidence of the population and any key sub-groups, and how the incidence was determined/obtained (e.g. supplied by the Client);
   2. the total sample size and the sample sizes of any key sub-groups.
2. Describe the sample frame, including:
   1. the sample source;
   2. sampling procedures;
   3. any known sampling limitations and how these might affect the findings.
3. Explain respondent selection procedures.
4. Indicate the number of recontact attempts and explain recontact attempt procedures.
5. Define respondent eligibility/screening criteria, including any quota controls.
6. For non-probability samples:
   1. provide rationale for choosing a non-probability sample

Response Rate/Participation Rate/Error Rate

1. State the target participation rate/response rate (10 percent range) for the sample(s). State the expected participation/response rate for each mode when more than one mode is proposed.
2. For probability samples, state the level of precision, including the margin of error and confidence interval for the sample(s).
3. Indicate any other potential source of error based on the study design that might affect the accuracy of the data.

Description of Data Collection

1. State the method of data collection.
2. Provide details on incentives/honoraria where applicable, including rationale.
3. For Access Panels, a description of the following must be provided, at minimum:
   1. Active panel size (provide the definition of "active");
   2. Panel recruitment;
   3. Panel monitoring;
   4. Panel maintenance;
   5. Panel refreshment.

   Note: When multiple panels are to be used in the execution of the survey, this must be disclosed.

4. Describe how language requirements will be addressed, if other than English and French.
5. Describe the planned fieldwork validation methods and procedures.
6. Describe how:
   1. the rights of children, youth and vulnerable respondents will be respected if applicable;
7. For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method.

Questionnaire Design

1. Provide the approximate duration of the questionnaire in minutes.
2. Specify the maximum number of open-ends.
3. Describe how the questionnaire will be pre-tested, including:
   1. the objectives of the pre-test;
   2. the method for the pre-test;
   3. the total number of pre-test questionnaires to be completed in total and by key sub-groups (e.g., language, age, gender);
   4. how the results of the pre-test will be documented and communicated to the Government of Canada.

Note: If no pre-test is to be conducted, a rationale must be provided.

Description of Data Processing/Data Management

1. Where known in advance, describe any weighting required.

Data Analysis/Reporting

1. Briefly describe how the data will be analyzed related to the objectives/research questions, including any special analyses (e.g., segmentation).

Deliverables

1. List all deliverables including their scope, format, means of delivery and number of copies, including at minimum:
   1. questionnaire(s), including pre-test, if relevant;
   2. data tabulation/processing;
   3. the report format(s), including the number of copies, language of report;
   4. the nature, location and number of presentations, including the language of presentations.

Project Schedule

1. Provide a detailed work plan with dates and identify responsibilities.

Project Cost

1. Cost information must be presented in the format designated by PWGSC.

2. Questionnaire Design

Standards

1. Survey questionnaires must be designed:
   1. to collect only the information essential to the objectives of the study;
   2. to minimize the burden placed on respondents while maximizing data quality; and
   3. to be completed in a maximum of 20 minutes uninterupted. This does not including pauses or interruptions. Exceptions could include projects with specialized audiences and those with pre-arranged interviews when the respondent is aware the survey will take longer than 20 minutes. Questionnaire durations of 15 minutes or less are strongly encouraged in order to minimize respondent burden.
2. The following are required elements of all Government of Canada telephone survey questionnaires:
   1. inform respondents of (i) the general subject and purpose of the study and (ii) the expected duration of the interview;
   2. identify the Government of Canada or the department/agency sponsoring the survey as well as the researcher and interviewer;
   3. inform respondents that their participation in the study is voluntary and completely confidential;
   4. inform respondents that their responses remain anonymous;
   5. inform respondents that the survey is registered with the National Survey Registration System, if requested. Approval of an online survey questionnaire must include approval of the appearance and functioning of the questionnaire in its online form - i.e., as it would be experienced by respondents. .

   Examples of exceptions: The sponsor would not be mentioned until the end of the survey in particular cases; other languages would be mentioned when the survey is available in non-official languages; and information about the availability of reports could be provided to specialized respondents when the incentive to participate outweighs the addition of time to the introduction.

   In addition to any other material used to communicate with respondents (e.g. invitations), the following introduction could be used for online questionnaires:

   The Government of Canada is conducting this research survey on SUBJECT. NAME OF FIRM has been hired to administer the survey. The survey takes about ## minutes and is voluntary and completely confidential. Your answers will remain anonymous. This survey is also available in OTHER OFFICIAL LANGUAGE.

3. The following standard demographic questions must be included in online surveys, unless particular research objectives justify alternative wording.

   The data from the age, education, and language questions (along with the recording of geographic location and sex) allows comparison with Statistics Canada census data for the purpose of non-response analysis. The data, along with that from the employment status and income questions, also facilitate the comparison of results between Government of Canada public opinion research studies. (See section 8 for further detail on non-response bias.)

   This does not preclude the use of other demographic questions

   • Gender
     What is your gender?
     • Female
     • Male
   • In what year were you born?
   • Education
     Which of the following diplomas or degrees have you completed?
     • None
     • High School diploma or equivalent
     • Registered Apprenticeship or other trades certificate or diploma
     • College, CEGEP or other non-university certificate or diploma
     • University degree, certificate or diploma
   • Mother Tongue
     What is the language you first learned at home as a child and still understand? [ACCEPT UP TO TWO RESPONSES]
     • English
     • French
     • Other [SPECIFY OR DO NOT SPECIFY DEPENDING ON THE NEEDS OF THE STUDY]

     and/or

   • Language Spoken at Home
     What language do you speak most often at home? [ACCEPT UP TO TWO RESPONSES]
     • English
     • French
     • Other [SPECIFY OR DO NOT SPECIFY DEPENDING ON THE NEEDS OF THE STUDY]
   • Employment Status
     Which of the following categories best describes your current employment status? Are you [ACCEPT ONE ANSWER ONLY]
     • Working full-time, that is, 35 or more hours per week
     • Working part-time, that is, less than 35 hours per week
     • Self-employed
     • Unemployed, but looking for work
     • A student attending school full-time
     • Retired
     • Not in the workforce [FULL-TIME HOMEMAKER, UNEMPLOYED, NOT WORKING FOR WORK]
     • [IF VOLUNTEERED: Other -- DO NOT SPECIFY]
   • Household Income
     Which of the following categories best describes your total household income? That is, the total income of all persons in your household combined, before taxes?
     • Under $20,000
     • $20,000 to just under $ 40,000
     • $40,000 to just under $ 60,000
     • $60,000 to just under $ 80,000
     • $80,000 to just under $100,000
     • $100,000 to just under $150,000
     • $150,000 and above

3. Pretesting

Standards

1. Pre-testing of all components of a new or revised survey questionnaire that may influence data quality and respondent behaviour is required. This includes the online appearance and functionality of the questionnaire.
2. Pre-testing must include probing that invites participants recruited for this purpose to provide input about their comprehension of and reaction to the questions. For example, a short series of questions could be included at the end of the pre-test survey. Researchers and Clients must agree in advance as to whether probing will take place during or after administering the survey. If requested by the client a cognitive pre-test must be conducted.
3. The sociodemographic characteristics of the targeted participants must be approved by the Client before recruiting begins.
4. Provision is to be made for the Client to monitor pretests if they are conducted by telephone or in-person should they want to.
5. A minimum of 10 pre-test sessions are to be completed in each language in which the final survey will be fielded. An exception could be projects with small target groups.
6. Pre-test completions shall not be included in the final dataset. An exception could be projects with:
   1. hard-to-reach target groups; or
   2. when no changes are made to the questionnaire; and
   3. the questionnaire was the exact same as in the final survey.
7. The pre-test(s) must be documented, provided to the Client before the questionnaire is finalized. The final report must include (at minimum):
   1. a description of the pre-test approach and number of interviews completed.

4. Sampling Procedures

General

All Researchers must:

1. clearly state the target group (universe) definition for the research study; in the case of Internet surveys this includes explicit identification of whether or not non-Internet users are part of the target group definition;
2. clearly state the method(s) used to obtain a sample of this target group, including whether the method was a probability survey, a non-probability survey, or an attempted census.

Probability Surveys

1. The list or sample source must be clearly stated, including any of its limitations/exclusions in representing the universe for the target sample and the potential for bias.
2. A full description of the sample design and selection procedures will be stated including:
   1. sample stratification variables (if any);
   2. any multi-stage sampling steps taken;
   3. at each sampling stage, the method of attaining a systematic random selection shall be explained, and any subsets of the universe that have been excluded or underrepresented shall be stated (e.g., Internet non-users);

      Note: Whenever possible, an estimate of the percentage of the universe that has been excluded or underrepresented must be provided;

   4. the number of attempted recontacts and procedure for attempted recontact must be stated.
   5. respondent eligibility/screening criteria will be defined, including any quota controls used (e.g., gender).
3. Assuming that proper probability sampling procedures have been followed, the sampling error must then be stated based upon a given sample size at a given confidence level, but Researchers must take care to:
   1. not mislead Clients into believing that a sampling error quoted on the total sample will be the same as that based upon a subset of the total sample.
   2. where possible, express sampling error in terms relevant to the specific nature of the most important or typical variables in a survey.
4. State that there are many potential non-sampling sources of error and include reference to other possible sources of error in any study in order to not give a misleading impression of overall accuracy and precision.

Justification of Use of Non-probability Surveys

1. When a choice is made to use a non-probability sample, that choice must be justified, in both the research proposal and the research report. The justification must take into account the statistical limitations in reporting on data from a non-probability sample, and limitations in generalizing the results to the target group population.

Sampling for Non-probability Samples

1. As for probability sampling, the list or sample source must be stated, including its limitations in covering the universe for the target sample.
2. The precise quota control targets and screening criteria must also be stated including the source of such targets (e.g., census data or other data source).
3. Deviations from target achievement must be shown in the report (i.e., actual versus target).

Statistical Treatment of Non-probability Samples

1. There can be no statements made about margins of sampling error on population estimates when non-probability samples are used.
2. The survey report must contain a statement on why no margin of sampling error is reported, based on the following template:

   Respondents for this survey were selected from among those who have [volunteered to participate/registered to participate] in online surveys. The results of such surveys cannot be described as statistically representative of the target population. [If weighting was done, state the following sentence on weighting:] The data have been weighted to reflect the demographic composition of (target population). Because the sample is based on those who initially self-selected for participation [in the panel], no estimates of sampling error can be calculated.;

   This statement must be prominently placed in descriptions of the methodology in the survey report, including the executive summary.

3. In general, for non-probability surveys it is not appropriate to use statistical significance tests since tests of significance should be used to analyze probability samples. However, an exception can be made regarding the use of tests of significance with non-probability samples when the objective is to establish the extent of the relationship among variables. If tests of significance are used with non-probability samples, it must be clearly noted that conclusions from these tests cannot be generalized to any population.

   Any use of descriptive statistics must clearly indicate that they are not formally generalizable to any group other than the sample studied, and there cannot be any formal statistical inferences about how the descriptive statistics for the sample represent any larger population.

   The exception to the rule against reporting statistical significance tests of differences is non-probability surveys that employ an experimental design in which respondents are randomly assigned to different cells in the experimental design. In this case, it is appropriate to use statistical significance tests to compare results from different cells in the design.

Quota Sampling

Quota sampling techniques are typically used for panel surveys and personal intercept studies to achieve sample representivity. Quotas may also be used to control representivity on other data collection methodologies.

1. A full description of the regional, demographic or other classification variable controls used for balancing the sample to achieve representivity must be described.
2. The precise quota control targets and screening criteria must also be stated including the source of such targets (e.g., census data or other data source).
3. Deviations from target achievement must be shown in the report (i.e., actual versus target).

Multi-Mode Surveys

Note: Multi-mode surveys are ones where different methods of questionnaire administration are used. They will often involve a combination of online and telephone methods, although there are other possibilities (e.g., in-person, mail, fax).

When a survey is conducted using multiple modes of questionnaire administration:

1. The reasons for using a multi-mode rather than a single-mode method must be stated, both in the research proposal and the survey report.
2. When the plan is to combine data collected via different modes in the data analyses, then steps must be taken to ensure as much comparability as possible across the different survey modes in terms of question wording and presentation of response options.
3. Steps must be taken to ensure avoidance of duplicate respondents in different modes. The steps taken, and the results, must be documented.
4. The survey report must discuss whether there are any data quality issues arising from combining data collected via different modes. This could include, for example, discussion of possible impacts of mode on key survey variables, the impact of any differences in participation rate/response rate by mode, and non-response bias analyses by mode.

Census Surveys

Note: In a census survey, an attempt is made to collect data from every member of a population. For example, an organization might want to do a survey of all of its employees. In this case, the population is "all of the organization's employees", and this would qualify as an attempted census survey if all employees are invited to participate in the survey.

The list or sample source must be clearly stated, including any of its limitations/exclusions in representing the universe for the target sample and the potential for bias.

Note: Whenever possible, an estimate of the percentage of the universe that has been excluded or underrepresented must be provided.

1. The number of attempted recontacts and procedure for attempted recontact must be stated.
2. Do not state a margin of sampling error, as this does not apply to census surveys because no sample is drawn.

5. Survey Respondents

Misleading and Deceptive Statements

1. In obtaining the necessary agreement from Respondents, the Researcher must not mislead them about the nature of the research or the uses which will be made of the findings. In particular, the Researcher must avoid deceptive statements that would be harmful or create a nuisance to the Respondent.

Use of Survey Information

1. Survey introductions or a survey description to which a link has been provided must assure Respondents that data will be collected only for research purposes. Any other purpose, such as rectifying a specific customer complaint, must have the proven express consent of the respondent. Researchers must not under any circumstances use personal information for direct marketing or other sales approaches to the respondent.

Duration of the Online Survey

1. Respondents must be informed, at the beginning of the survey, about the expected length of time the questionnaire is likely to take to complete.

Email Invitations to Respond

1. Researchers must reduce any inconvenience or irritation their email invitations might cause the recipient by clearly stating its purpose in the first sentence and keeping the total message as brief as possible.

Links to Privacy and Cookie Policies

1. Any links to data protection, privacy policy or cookie policy statements must be given at the start of the questionnaire.

Disclosure of the Identity of the Researcher

1. Respondents must be told the identity of the Researcher carrying out the project and given contact information so that they can, without difficulty, recontact the Researcher should they wish to do so.

Disclosure of List Sources

1. Where lists are used for sample selection, the source of the list must be disclosed. Researchers must ensure that lists are permission-based for research purposes and that the data are as up-to-date as possible.

Protection of Respondent Anonymity and Use of Information

1. The anonymity of Respondents must always be preserved unless they have given their informed and express consent to the contrary. If these Respondents have given permission for data to be passed on in a form which allows them to be personally identified, the Researcher must ensure that the information will be used for research purposes only, OR, if requested, to rectify a customer complaint. Such personally identified information must not be used for subsequent non-research purposes such as direct marketing, list-building, credit rating, fund-raising or other marketing activities relating to those individual Respondents.

Respondent's Email Address is Personal Information

1. A Respondent's email address is personal information and must be protected in the same way as other identifiers.

Disclosure of the Use of Cookies, Log Files or Software

1. Researchers must have a readily accessible policy statement concerning the use of cookies, log files and, if applicable, software. This statement may be either included in their privacy policy or it may appear in a separate document. Software must not be installed on Respondents' computers without their knowledge or consent. In addition, Respondents must be able to remove the Researcher's software easily from their machines (e.g., for Windows users, the software must appear in the Add/Remove Programs folder in their Control Panel).

6. Interviewing Children and Young People

General

1. Children may be familiar with using the Internet, but research has found them to be naïve and trusting, happily disclosing information about themselves or their households without realizing the implications of doing so. Parent groups, consumer groups and legislators are particularly concerned about potential exploitation of children on the Internet and it is for this reason that guidelines place greater burdens on Researchers than would be the case in adult research. While validating respondent identity and age can be a challenge in online research, it is important that Researchers make every effort to do so.
2. A "child" is defined as "under the age of 13" and a "young person" as "aged 13-15."

Observation of Laws and National Codes

1. Researchers must observe all relevant laws and national codes specifically relating to children and young people although it is recognized that the identification of children and young people is not possible with certainty on the Internet at this time.

Adult Consent

1. Permission of a responsible adult must be obtained before interviewing children and young people under the age of 16 years.

Consent

1. Researchers must ensure that the principle of consent is met, so if Internet research is conducted, special measures must be taken to ensure and record the consent.

Process for Obtaining Consent: Online Panels or Other Approved Lists

1. In cases where interviews with children of adult online panelists or children of other online list members are desired, the following measures must be implemented.

   The email invitation to the adult panelist or list member must contain the following:

   1. A notice stipulating that the online survey is intended for the child within the household.
   2. Name and contact details of the Researcher.
   3. The nature of the data to be collected from the child.
   4. An explanation of how the data will be used.

Process for Obtaining Consent: Recruiting Children from Web sites

1. In cases where children are being recruited from Web sites, the following measures must be implemented:
   1. For Web sites aimed at children, a notice to children, informing them of the requirement for adult consent must be shown at the beginning of the survey. This notice must be clear and prominent and must include an explanation of the subject matter and nature of the research and details of the research Researcher undertaking it, with contact information. To obtain consent, the notice must request the adult's contact information (e.g. email address). It must also refer to the fact that consent will be verified.
   2. Questionnaires on Web sites aimed at children must require a child to give their age before any other information is requested. If the age given is less than 13 years, the child must be excluded from giving further information until the appropriate consent has been obtained.
   3. For Web sites aimed at adults, a notice to parent or guardian, seeking their consent for their child to be asked to participate in the research, must be posted on the Web site. This notice must include:
      1. A heading explaining that this is a notice for parents.
      2. Name and contact details of the agency/agencies and the name of the Client (if the Client agrees).
      3. The nature of the data to be collected from the child.
      4. An explanation of how the data will be used.
      5. A description of the procedure for giving and verifying consent.
      6. A request for a parent's contact email address, address or phone number for verification of consent.

Parent Contact Details

1. It is permissible to ask children to provide contact details for their parents in order for consent to be sought as long as this purpose is made clear in the request for information.

Acceptable Forms of Consent

1. A return email from parent or guardian giving their consent is acceptable, as long as additional steps are taken to ensure that the consent actually came from a parent — for example, following up with an email, letter or phone call.

Situations When Parental Consent is NOT Required

1. Prior parental consent will not be required to:
   1. Collect a child's or parent's email address solely to provide notice of data collection and request consent.
   2. Collect a child's age for screening and exclusion purposes. If this screening leads to the decision that a child does qualify for interview, parental consent must then be sought to continue with the interview.

EMails to Children

1. Email communications must not be addressed to children without verifiable and explicit prior consent.

Types of Information Collected

1. Personal information relating to other people (for example, parents) must not be collected from children.

Sensitive Questions

1. Asking questions on topics generally regarded as sensitive must be avoided wherever possible and in any case handled with extreme care. The Researcher must specify the nature of any potentially sensitive questions when seeking parental consent for their children's participation in the research.

Policies Must be Understandable

1. All data protection, privacy policy, consent and other notices must be capable of being understood by children.

7. Data Collection

Collection of Email Addresses

1. Researchers are prohibited from using any subterfuge in obtaining email addresses of potential respondents, such as collecting email addresses from public domains, using technologies or techniques to collect email addresses without individuals' awareness, and collecting email addresses under the guise of some other activity. An exception to the above is that it is acceptable to collect email addresses from public domains for business-to-business research relevant to their professional interests.

Data Collection & Recruitment Techniques

1. Researchers must not make use of surreptitious, misleading or unsolicited data collection or recruitment techniques – including using spambots, spiders, sniffers or other ‘agents' that collect personal information without the Respondent's explicit awareness.

Misleading Email Return Addresses

1. Researchers are prohibited from using false or misleading return email addresses, including spoofing the "from" label of email messages, when recruiting Respondents over the Internet.

Opt-out

1. A Respondent must be able to refuse participation in the survey via a suitable option, and to refuse further contact by email in connection with the survey.

Acess Panel Participation Frequency

1. To be a respondent to a Government of Canada survey, panel members may not have participated in any Government of Canada survey as a member of that panel or a survey on similar subject matter in the previous 30 days. An exception would be when a study's research design requires a follow-up survey with respondents and the respondent has agreed at the time of the initial survey to be recontacted for that purpose.

Incentives/Honoraria

1. The details of any incentives/honoraria to be used for an online survey must be provided in both the proposal and report documentation, including:
   1. the type of incentive/honoraria (e.g., monetary , non-monetary);
   2. the nature of the incentive – e.g., for monetary, prize, points, donations, direct payments, etc.
   3. the estimated dollar value of the incentives to be disbursed

Monitoring Of Online Survey Fieldwork

1. Each online survey must be closely monitored throughout the fieldwork to ensure that responses are valid, that the survey is administered consistently throughout the data collection period, and that the responses are being recorded accurately.

Detecting And Dealing With Satisficing

1. The access panel provider, working with the client, shall implement procedures to identify and remove fraudulent and inattentive panel members or respondents, documenting these procedures and actions taken.

Attempted Recontacts

1. Where recontact is feasible, there must be attempts made to recontact nonrespondents using an appropriate contact procedure.
2. In general, the intensity of the recontact effort must balance both potential improvements to data quality and degree of intrusiveness in terms of respecting the person's right to choose not to participate.
3. In the case of surveys that do not directly target public servants and where recontact by email is judged the appropriate method, a maximum of three attempted recontacts may be made.

Ensure Respondents Answer A Survey Only Once

1. For any online survey research project, there must be documented procedures to limit the possibility that respondents can answer a survey for which they have been selected, more than once.

8. Participation Rate/Response Rate

Participation Rate/Response Rate Targets

1. The online survey must be designed to achieve the highest practical rates of success.

Monitoring Response Rate/Participation Rate During Data Collection

1. Monitoring of contact dispositions/reasons for non-response shall be carried out on an ongoing basis, throughout the entire field period.
2. This information will be provided to the Client upon request.

Non-response Bias Analyses

1. All survey reports must contain a discussion of the potential for non-response bias for the survey as a whole and for key survey variables. Non-response bias is the systematic difference between true population values and the average result from all possible samples owing to non-response.
2. The analysis of non-response bias will consist of a comparison of at least three variables in the survey sample with the equivalent parameters of the population, normally available from Statistics Canada. No extra surveys or interviews are required.
3. If no external sources of comparison exist, key variables in the sample could alternatively be compared to variables in the sampling frame. If no such variables exist, only then should the non-response bias discussion be based on comparison of early versus late responders or on observations made during data collection about characteristics of non-responders.

Calculation Of Participation Rate and Response Rate

1. Participation rate or level must be calculated for non-probability surveys.
2. Response rate or level must be calculated for probability and attempted census surveys.

The calculations for participation rate/response rate for online surveys must be done as recommended by the MRIA. The results must be included in the survey report. The survey report must also show a record of contact dispositions that includes the categories required to comply with the MRIA calculation formulas.

• Total number of sample units invited to participate (a)
  (e.g. email addresses, online users etc.)
  • Invitations method can include email, pop-up, etc
• Invalid (b)
  Undelivered
  • Includes bounce backs, etc., included only when data is available
• (U) Unresolved
  Did not respond (g)
• (IS) In-scope – non-responding
  Qualified respondent break-off (e)
• (R) In-scope – Responding units
  Disqualified (f)
  Quota filled (h)
  Completed interviews (d)
• Calculation of Contact Rate
  Contact Rate = (R+IS) ÷ (U+IS+R)
  • Contact rate can be used for both probability and non-probability surveys
• Final Rate calculation for probability surveys
  Response Rate = R ÷ (U+IS+R)
  • The term "Response Rate" can ONLY be used to describe respondent cooperation on probability surveys (or in the case of a census of the universe). These might include employee or customer surveys as well as general population surveys where the sample is drawn from a well defined sampling frame using consistent random sampling method and by which each sampling unit in the frame has a known probability of being selected.
• Final Rate calculation for non- probability surveys
  Participation Rate = R ÷ (U+IS+R)
  • Non-probability surveys can include access panels and convenience samples where the sampling units are not drawn from a well defined sample frame. The sampling method used could be random but the selection probability cannot be determined.

1. Total invitations (broadcast or pop-ups)
2. Undeliverables (nil in pop-ups)
3. Net usable invitations (c = a−b)
4. Total completes
5. Qualified breakoffs
6. Disqualified
7. Not responded
8. Quota filled

c = d+e+f+g+h

Contact rate = (d+e+f+h) ÷ c
Participation rate = (d+f+h) ÷ c

9. Coding

Developing Code Frames

1. The initial code list/frame shall be developed based on a systematic review of a minimum of 10% of open-ended responses and 50% of partial open-ended responses, where a frame does not already exist. The initial code list/frame must be provided to the Client for approval upon request.
2. The Researcher shall ensure that coders working on the project are provided with instructions and training that shall include, as a minimum:
   1. an overview of the project;
   2. identification of questions or variables to be coded;
   3. the minimum proportion or number of a sample (and its make-up) used to produce code frames;
   4. where necessary or appropriate, specific sub-groups required to develop code frames (e.g., by region, user or non-user);
   5. guidelines for the inclusion of codes in the code frame (e.g., decisions or rules regarding what must be included or excluded from a given code);
   6. any use to be made of code frames from a previous project or stage;
   7. any other requirements or special instructions specific to the project.

Code Frame Approval/Coding Procedures

1. Where "don't know" and "no response" have been used, these shall be distinguishable from each other.
2. The Researcher shall have clear rules or guidelines for the treatment of responses in "other" or catch-all categories; if the "other" or catch-all category exceeds 10% of responses to be coded, the responses must be reviewed with a view to reducing the size of the group.
3. Researchers shall ensure that there is a systematic method of verifying a minimum of 10% of questionnaires coded per project and the verification shall be undertaken by a second person.

10. Data Editing/Imputation

Standards

1. An accurate record of any changes made to the original data set shall be kept. No data shall be assumed/imputed without the knowledge and approval of the Researcher's project manager. Comparison to the original data source shall be the first step in the process. Any imputation processes, including the logic of the imputation method(s) used shall be documented and provided to the Client with the initial data tables. All edit specifications shall be documented.
2. Where forced editing is used, the logic of the forcing shall be documented and test runs carried out, with the results documented to show that the forcing has the desired effect.
3. Data editing/imputation must be used cautiously. The degree and impact of imputation must be considered when analyzing the data, as the imputation methods used may have a significant impact on distributions of data and the variance of estimates.
4. The Researcher shall include documentation of any imputation/forced editing, both in a technical appendix and in the final report.

11. Data Analysis/Reporting

Inferences and Comparisons

1. Researchers must base statements of comparisons and other statistical conclusions derived from survey data on acceptable statistical practice.

Analysis records

1. The Researcher shall keep accurate and descriptive records of the analysis process, to ensure that any analysis undertaken can be replicated at a later date.

Data analysis verification

1. The Researcher shall have in place procedures to ensure the tabulations and other outputs have been checked.
2. As a minimum, these checks shall verify:
   1. completeness, i.e., that all tables are present as specified, including the results of all reported significance tests;
   2. that abbreviations for headings or open-ended responses accurately reflect the full content;
   3. that the base for each table is correct against other tables or frequency counts;
   4. that the standard breaks/banner points are checked against source questions;
   5. that all derived data items are checked against their source;
   6. that the figures for sub-groups and nets are correct;
   7. that there are no blank tables (i.e., with no data);
   8. weighting (e.g., by test tables);
   9. frequency counts prior to running tables, in order both to ensure the accuracy of data and to determine base sizes for sub-groups;
   10. spelling and legibility;
   11. that any statistical analysis used is appropriate and correct, both in its descriptive and inferential aspects.
3. For any subsequent outputs, appropriate checks shall be applied.

12. Delivery of Data Tables

Delivery of Stand-alone Hard or Soft Copy of Data Tables

1. When data are reported to the Client, such as in a stand-alone hard or soft copy of data tables, the following shall be taken into account, as appropriate:
   1. reference to the actual source question to which the data pertains;
   2. inclusion of a description of any weighting method applied to the data;
   3. clear identification of any sub-groups used;
   4. availability of the bases for each question, so that the number of respondents who have actually answered the question is identifiable;
   5. the number or proportion of respondents who replied "don't know" or gave "no answer";
   6. availability of both weighted and unweighted bases;
   7. clear and complete definition and explanation of all variables used in the analysis of the data, including any significance testing, indexing, scoring, scaling and calculations of means, median, modes and standard deviations;
   8. the types of statistical tests being used and their level of precision;
   9. information on cell suppression and other measures to assure confidentiality;
   10. warnings on results which are unreliable due to very small sample sizes.

Electronic Data Delivery

1. The Researcher shall provide the Client with a data file.
2. The following shall be checked prior to data release:
   1. compatibility of the file format with the software specification agreed with the Client;
   2. completeness (i.e., the correct number of files and records are in each file);
   3. inclusion of all appropriate documentation to allow for replication of the data analysis and additional analyses, including where applicable:
      1. labelling of the contents of the file, i.e., fully labelled variables and value labels;
      2. identification and description of any computed or recoded variables, and instructions on limitations of use;
      3. labelled weighting variables and a description of how these were applied;
      4. all personal identifiers per PIPEDA have been removed from the files;
   4. encryption of files upon request;
   5. presence of viruses in the file.

13. Retention of Technical Data

Standards

1. The Researcher must maintain the technical data on all studies for a period of three years, so that if requested, the study can be replicated. For online surveys, this also will include how the questionnaire was presented and a representation of any visual/audio materials used in the survey.
2. Technical data not already included in the Survey Report/Appendix that must be maintained includes, but is not limited to:
   1. Data pertaining to data processing and analysis may include, but is not limited to:
      1. raw data files;
      2. other electronic files;
      3. code frames;
      4. project files including project management information and survey programming files;
      5. emails and other correspondence.
   2. Where data has been edited, cleaned, recoded or changed in any other way from the format, content and layout of its original format, the original data, final data and programme files, including all documentation related to changes to the data (as a minimum) shall be kept so that the final data set can be easily reconstructed.

14. Data Security

Protection of Data/Servers

1. Researchers must use up-to-date technologies to protect the personal data collected or stored on Web sites or servers. In particular, panel registration pages, and online surveys that collect sensitive personal information, must use Secure Socket Layer (SSL) or an equivalent level of protection.
2. Researchers must also put in place measures to ensure the "physical" security of data and servers.

Temporary Storage of Data on Servers

1. If the temporary storage of data collected takes place on a server that is operated by another provider, the Researcher must place the provider under the obligation to take the necessary technical precautions to ensure that third parties cannot access the data on the server or during data transfer. Temporary storage of the collected data on the server must be terminated at the earliest possible time.

Data Storage on Servers Outside of Canada

1. When data is stored on servers outside of Canada, Researchers must ensure that commitments with respect to privacy can be maintained.

Transmission of Data Internationally

1. Before data is sent over the Internet to another country, Researchers must check with competent authorities that the data transfer is permissible. The recipient may need to provide safeguards necessary for the protection of the data.

Disclosure of Respondents' Emails in Batch Transfers

1. Researchers must have adequate safeguards in place to ensure that when emails are sent in batches, the addresses of the respondents are not revealed.

In the Event of Any Data Breach

1. In the event of any data breach, the Client must be informed immediately and provided with details about both the nature and the extent of the data breach.

15. Survey Reports

Library and Archives Canada

1. It is a legal requirement for institutions to send written public opinion research reports to Library and Archives Canada within six months of the completion of the data collection for all reports.
2. Pursuant to the Public Opinion Research Contract Regulations, final reports must include the following at a minimum:
   1. on the covering page, the title of the project, the name of the person who entered into the contract, the contract number and the award date;
   2. a narrative executive summary consisting of, at a minimum:
      1. a statement of the researches purpose and objectives,
      2. a summary of key findings, except where the person who entered into the contract is not responsible for the design, development of the methodology and analysis of the research,
      3. a brief description of the methodology used, and
      4. a statement as to the extent to which the findings can be extrapolated to a broader audience;
   3. for quantitative research, appendices containing:
      1. a full set of tabulated data,
      2. sample size, sampling procedures and dates of research fieldwork,
      3. if applicable, weighting procedures, the confidence interval and the margin of error,
      4. if applicable, the participation rate/response rate and method of calculation,
      5. the research instruments used, and
      6. all other information about the execution of the fieldwork that would be needed to replicate the research initiative.

General

The following minimum details shall be documented in the project report. These allow the reader to understand the way the research project was conducted and the implications of its results.

Background

1. Detailed description of background including, at minimum:
   1. purpose, how the research will be used.
2. Objectives, research questions.

Sample

1. Detailed description of the sample including:
   1. the target group for the research project;
   2. the achieved sample size against projected sample size and reasons, if relevant, for not obtaining the projected sample;
   3. the sample source and sampling method, including the procedure for selecting respondents;
   4. the weighting procedures, if applicable.
2. For non-probability samples, provide:
   1. rationale for choosing a non-probability sample.

Data Collection

1. Detailed description of methodology including:
   1. the dates of fieldwork;
   2. the data collection method(s), and if applicable:
      1. the type and amount of incentives;
   3. the target group for the research project;
   4. the achieved sample size against projected sample size and reasons, if relevant, for not obtaining the projected sample;
   5. the sample source and sampling method, including the procedure for selecting respondents;
   6. the weighting procedures, if applicable.
2. For non-probability samples, provide:
   1. rationale for choosing a non-probability sample.
3. For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method.

Quality Controls

1. The estimating and imputation procedures, if applicable.
2. A brief summary of other quality controls and procedures used.
3. For multi-mode surveys, detailed description of any data quality issues arising from combining data collected via different modes/instruments.

Results

1. An executive summary of key results and conclusions, linked to the survey objectives, research questions.
2. For probability samples, state the level of precision, including the margin of error and confidence interval for the total sample and any key sub-groups.
3. For non-probability samples and attempted census surveys, the report must contain a statement on why no margin of sampling error is reported; non-probability-based surveys must use the prescribed template.
4. The contact dispositions and participation rate (in the case of non-probability samples) and response rate (in the case of probability samples and attempted censes).
5. The results that are based on sub-groups and the number of cases used in sub-group analysis.

Appendix 1: Study Materials

Study Materials, including the invitations, recruitment instruments, questionnaires, descriptions or representations of any visual or auditory aids, and other relevant data collection documents, in all languages in which the research was conducted. There must be a version of the questionnaires displaying any instructions (e.g., skip, terminate, etc.) needed to understand the logic and flow of the questionnaire. Screen shot examples of the questionnaire's various types of graphic displays must be included to show how the questionnaire appeared to respondents.