The Standards for the Conduct of Government of Canada Public Opinion Research - Telephone Surveys

2009-11-26

Table of Contents

• Important Notes
• 1. Proposal Documentation
• 2. Questionnaire Design
• 3. Pretesting
• 4. Development Of Sampling Frames And Sampling
• 5. Survey Respondents
• 6. Data Collection From Children or Vulnerable Respondents
• 7. Data Collection
• 8. Response Rates / Participation Rates
• 9. Data Entry
• 10. Coding
• 11. Data Editing/Imputation
• 12. Data Analysis
• 13. Delivery of Data Tables
• 14. Back-up, Retention/Security of Data
• 15. Survey Reports

1. Applicability of these standards

These standards apply to custom telephone public opinion research surveys conducted for the Government of Canada including questions added to omnibus/syndicated surveys. Custom studies are those in which ownership of the data rests with Canada. The standards do not apply to syndicated studies (those in which ownership of the data rests with the Researcher) unless the Government of Canada contracts to place questions in addition to those already included in those studies.

These standards are to be understood as in addition to relevant laws, regulations, and policies, e.g. the Personal Information Protection and Electronic Documents Act, the Privacy Act, the Federal Accountability Act, and the Communication Policy of the Government of Canada.

Note: Survey projects may involve more than one methodology (e.g., telephone and on-line surveys). As such, it may be necessary to consult more than one set of standards.

2. Applicability of other standards

Where no relevant Government of Canada Standards exist, Researchers must meet or exceed industry standards including those of the Marketing Research and Intelligence Association.

3. Exceptions

These standards were developed with "typical" studies in mind. It is recognized, however, that public opinion research surveys have individual characteristics, which can sometimes clash with a "one size fits all" approach. Therefore, while departments and agencies are generally encouraged to apply the standards, they can also choose to waive a standard when they believe the characteristics of a study warrant it.

This document notes some of the more common instances in which it would be appropriate to waive or modify standards.

As departments and agencies are responsible for the quality of the research they undertake no related procedure is required. In keeping with due diligence however, they should discuss the intention to waive standards with the Public Opinion Research Directorate. They should also consider specifying which standard is being waived and the related rationale in statements of work and other documents such as Researcher proposals and final reports.

4. Two-year review

The following standards have been developed to help ensure uniform high quality in public opinion research surveys conducted for the Government of Canada. It is recognized that changing technologies and other factors may necessitate their revision over time. As such, the intention is to review the standards after an initial two-year period.

1. Proposal Documentation

Introduction

Purpose

1. Describe the Researcher's understanding of the problem/issues to be investigated and how the Government of Canada will use this information.

Research Objectives

1. Detail the information needs/research questions the research will address.

Technical Specifications of the Research

Overview

1. Provide a brief statement summarizing:
   1. data collection method, including the method(s) used to obtain a sample (probability survey, a non-probability survey, or an attempted census) and rationale for proposed methodology;
   2. expected number of survey completions;
   3. target population.
2. The Researcher must recommend a research design that they believe is cost efficient, will produce quality results, and is appropriate to meet the objectives of the research. The Researcher must also provide a rationale for their recommendation.

   If the Researcher's design is the same as that described in the Statement of Work the Researcher must state that they believe the design in the Statement of Work is cost efficient, will produce quality results, and is appropriate to meet the objectives of the research.

Sample/Sampling Details

1. Provide details related to target population:
   1. the definition of the target population in terms of its specific characteristics and geographic scope, including the assumed incidence of the population and any key sub-groups, and how the incidence was determined/obtained (e.g. supplied by the Client);
   2. the total sample size and the sample sizes of any key sub-groups.
2. Describe the sample frame, including:
   1. the sample source;
   2. sampling procedures;
   3. any known sampling limitations and how these might affect the findings.
3. Explain respondent selection procedures.
4. Indicate the number of call-backs and explain call-back procedures.
5. Define respondent eligibility/screening criteria, including any quota controls.

Response Rate / Participation Rate /Error Rate

1. State the expected response rate / participation rate (range of 10%) for the sample(s). State expected response rates / participation rates for each mode when more than one mode is proposed.
2. State the level of precision, including, if relevant, the margin of sampling error and confidence interval for the sample(s).
3. Indicate any other potential source of error based on the study design that might affect the accuracy of the data.

Description of Data Collection

1. State the method of data collection (in this case telephone interviewing).
2. Provide details on incentives/honoraria where applicable, including rationale.
3. Describe how language requirements will be addressed, if other than English or French.
4. Describe the planned fieldwork validation methods and procedures.
5. Describe how:
   1. the rights of children, youth and vulnerable respondents will be respected if applicable.
6. For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method.

Questionnaire Design

1. Consistent with the proposal costing section, provide the duration of the questionnaire in minutes. Specify the maximum number of open-ends.
2. Describe how the questionnaire will be pre-tested, including:
   1. the objectives of the pre-test;
   2. the method for the pre-test, including the description of the types of interviewers/researchers that will conduct the pre-test interviews;
   3. the total number of interviews to be completed in total and by key sub-groups (e.g., language, age, gender);
   4. how the results of the pre-test will be documented and communicated to the Government of Canada.

   Note: If no pre-test is to be conducted, a rationale must be provided.

Description of Data Processing/Data Management

1. Where known in advance, describe any weighting required.

Data Analysis/Reporting

1. Briefly describe how the data will be analyzed related to the objectives/research questions, including any special analyses (e.g., segmentation).

Deliverables

1. List all deliverables including their scope, format, means of delivery and number of copies, including at minimum:
   1. questionnaire(s), including pre-test, if relevant;
   2. data tabulation/processing;
   3. the report format(s), including the number of copies, language of report;
   4. the nature, location and number of presentations, including the language of presentations.

Project Schedule

1. Provide a detailed work plan with dates and identify responsibilities.

Project Cost

1. Cost information must be presented in the format designated by PWGSC.

2. Questionnaire Design

Standards

1. Survey questionnaires must be designed:
   1. To collect only the information essential to the objectives of the study;
   2. To minimize the burden placed on respondents while maximizing data quality;
   3. To be completed in a maximum of 20 minutes. Exceptions could include projects with specialized audiences and those with pre-arranged interviews when the respondent is aware the survey will take longer than 20 minutes. Questionnaire durations of 15 minutes or less are strongly encouraged in order to minimize respondent burden.
2. The following are required elements of all Government of Canada telephone survey questionnaires:
   1. inform respondents of (i) the general subject and purpose of the study and (ii) the expected length of the interview;
   2. identify the Government of Canada or the department/agency sponsoring the survey as well as the Researcher and interviewer;
   3. inform respondents that their participation in the study is voluntary and completely confidential
   4. inform respondents that their responses remain anonymous;
   5. inform respondents that the survey is registered with the National Survey Registration System, if requested.

   Examples of exceptions: The sponsor would not be mentioned until the end of the survey in particular cases (e.g. advertising post-tests); other languages would be mentioned when the survey is available in non-official languages; and information about the availability of reports could be provided to specialized respondents when the incentive to participate outweighs the addition of time to the introduction.

   Note: Respondents would need to have the introduction read again after choosing another interview language. Interviewers must have information readily available to inform respondents about how respondents were chosen, how privacy is protected, and where the survey results can be obtained.

   For example the following introduction could be used for telephone questionnaires:

   Good (_________), the Government of Canada is conducting a research survey on SUBJECT. My name is FIRST NAME of NAME OF FIRM, the company hired to do the survey. The survey takes about ## minutes and is voluntary and completely confidential. Your answers will remain anonymous. May I continue? Would you prefer to do the survey in English or French?

3. The following standard demographic questions must be included in telephone surveys, unless particular research objectives justify alternative wording.

   The data from the age, education, and language questions (along with the recording of geographic location and sex) allows comparison with Statistics Canada census data for the purpose of non-response analysis. The data, along with that from the employment status and income questions, also facilitate the comparison of results between Government of Canada public opinion research studies. (See section 8 for further detail on non-response bias.)

   • Gender [Do not ask: record based on interviewer observation]
   • In what year were you born? (Record year - XXXX)
   • Education
     Which of the following diplomas or degrees have you completed? (READ LIST)
     • None
     • High School diploma or equivalent
     • Registered Apprenticeship or other trades certificate or diploma
     • College, CEGEP or other non-university certificate or diploma
     • University degree, certificate or diploma
     • (DO NOT READ) Refused
   • Mother Tongue
     What is the language you first learned at home as a child and still understand? [READ LIST - ACCEPT UP TO TWO RESPONSES]
     • English
     • French
     • Other [SPECIFY OR DO NOT SPECIFY DEPENDING ON THE NEEDS OF THE STUDY]
     • [DO NOT READ] Don't know/ Refused

     and/or

   • Language Spoken at Home
     What language do you speak most often at home? [READ LIST - ACCEPT UP TO TWO RESPONSES]
     • English
     • French
     • Other [SPECIFY OR DO NOT SPECIFY DEPENDING ON THE NEEDS OF THE STUDY])
     • [DO NOT READ] Don't know/ Refused
   • Employment Status
     Which of the following categories best describes your current employment status? Are you [READ LIST – ACCEPT ONE ANSWER ONLY]
     • Working full-time, that is, 35 or more hours per week
     • Working part-time, that is, less than 35 hours per week
     • Self-employed
     • Unemployed, but looking for work
     • A student attending school full-time
     • Retired
     • Not in the workforce [FULL-TIME HOMEMAKER, UNEMPLOYED, NOT WORKING FOR WORK])
     • [DO NOT READ] [IF VOLUNTEERED: Other -- DO NOT SPECIFY]
     • [DO NOT READ] Refused
   • Household Income
     Which of the following categories best describes your total household income? That is, the total income of all persons in your household combined, before taxes [READ LIST]?
     • Under $20,000
     • $20,000 to just under $ 40,000
     • $40,000 to just under $ 60,000
     • $60,000 to just under $ 80,000
     • $80,000 to just under $100,000
     • $100,000 to just under $150,000
     • $150,000 and above
     • (DO NOT READ) Refused

3. Pretesting

Standards

1. Pre-testing of all components of a new or revised survey questionnaire that may influence data quality, respondent behaviour and interviewer performance is required.
2. Pre-testing must include probing that invites participants recruited for this purpose to provide input about their comprehension of and reaction to the questions. For example, a short series of questions could be included at the end of the pre-test survey. Researchers and Clients must agree in advance as to whether probing will take place during or after administering the survey. If requested by the client a full cognitive pre-test must be conducted.
3. The socio-demographic characteristics of the targeted participants must be approved by the Client before recruiting begins.
4. Provision is to be made for the Client to monitor pre-tests conducted by telephone or in-person should they want to.
5. A minimum of 10 pre-test sessions are to be completed in each language in which the final survey will be fielded. An exception could be projects with small target groups.
6. Pre-test completions shall not be included in the final dataset. An exception could be projects with:
   1. hard-to-reach target groups; or
   2. when no changes are made to the questionnaire; and
   3. the interview was conducted in the exact same manner as in the final survey.
7. The pre-test(s) must be documented, provided to the Client before the questionnaire is finalized. The final report must include (at minimum):
   1. a description of the pre-test approach and number of interviews completed.

4. Development Of Sampling Frames And Sampling

Sampling Procedures

1. It must be clearly identified whether probability-based sampling methods or non-probability sampling methods will be used. Researchers must clearly state the target group (universe) definition for the research study (including whether or not individuals without a landline telephone are part of the definition).
2. In the case of probability-based samples, the method used to obtain a representative sample of this target group must be clearly described.
3. The use of non-probability/convenience sampling may not be described as a representative sample. If the cost of sampling low incidence target groups prohibits the use of the usual quota controls to ensure sample representivity, then such sampling limitations must be clearly stated (as they should for any sampling methodology).

Probability Sampling

1. The list or sample source must be clearly stated, including any of its limitations/exclusions in representing the universe for the target sample and the potential for bias.
2. National random samples must not exclude any area codes, including the territories.
3. A full description of the sample design and selection procedures will be stated including:
   1. sample stratification variables (if any);
   2. any multi-stage sampling steps taken (e.g., enumeration areas, followed by households, followed by respondents);
   3. at each sampling stage, the method of attaining a systematic random selection shall be explained, and any subsets of the universe that have been excluded or underrepresented shall be stated (e.g., cell only phone households), although whenever possible, an estimate of the percentage of the universe that has been excluded or underrepresented must be provided;
   4. the number of call-backs and the call-back procedure must be stated;
   5. respondent eligibility/screening criteria will be defined, including any quota controls used (e.g., gender).
4. Records of the disposition of the contact sample must be retained to show:
   1. the size of the original contact sample;
   2. the proportion of the contact sample that was unusable/invalid (e.g., listing errors, number not in service);
   3. the proportion of the valid sample that was unreachable after the specified number of call-backs;
   4. other reasons for non-contact (e.g., language [other than one of the two official languages], deafness);
   5. the refusal rate among reached sample;
   6. the proportion of unqualified sample;
   7. mid-interview terminations;
   8. the total number of completed interviews.
5. Assuming that proper probability sampling procedures have been followed, the sampling error must then be stated based upon a given sample size at a given confidence level, but Researchers must take care to:
   1. not mislead Clients into believing that a sampling error quoted on the total sample will be the same as that based upon a subset of the total sample;
   2. where possible, express sampling error for key subsets in a survey;
   3. state that there are other possible sources of error in any study (e.g., question wording errors, interviewer errors, respondent effects) in order to not give a misleading impression of overall accuracy and precision.

Justification for Use of Non-probability Surveys

1. When a choice is made to use a non-probability sample, that choice must be justified, in both the research proposal and the research report. The justification must take into account the statistical limitations in reporting on data from a non-probability sample, and limitations in generalizing the results to the target group population.

Sampling for Non-probability Samples

1. As for probability sampling, the list or sample source must be stated, including its limitations in covering the universe for the target sample.
2. The precise quota control targets and screening criteria must also be stated including the source of such targets (e.g., census data or other data source).
3. Deviations from target achievement must be shown in the report (i.e., actual versus target).

Statistical Treatment of Non-probability Samples

1. There can be no statements made about margins of sampling error on population estimates when non-probability samples are used.
2. The survey report must contain a statement on why no margin of sampling error is reported, based on the following template:

   "Respondents for this survey were selected from among those who have [volunteered to participate/registered to participate] in telephone surveys. The results of such surveys cannot be described as statistically representative of the target population. [If weighting was done, state the following sentence on weighting:] The data have been weighted to reflect the demographic composition of (target population). Because the sample is based on those who initially self-selected for participation [in the panel], no estimates of sampling error can be calculated."

   This statement must be prominently placed in descriptions of the methodology in the survey report, including the executive summary.

3. In general, for non-probability surveys it is not appropriate to use statistical significance tests since tests of significance should be used to analyze probability samples. However, an exception can be made regarding the use of tests of significance with non-probability samples when the objective is to establish the extent of the relationship among variables. If tests of significance are used with non-probability samples, it must be clearly noted that conclusions from these tests cannot be generalized to any population.

   Any use of descriptive statistics must clearly indicate that they are not formally generalizable to any group other than the sample studied, and there cannot be any formal statistical inferences about how the descriptive statistics for the sample represent any larger population.

   The exception to the rule against reporting statistical significance tests of differences is non-probability surveys that employ an experimental design in which respondents are randomly assigned to different cells in the experimental design. In this case, it is appropriate to use statistical significance tests to compare results from different cells in the design.

Quota Sampling

Quota sampling techniques are typically used for panel surveys and personal intercept studies to achieve sample representivity. Quotas may also be used to control representivity on other data collection methodologies.

1. A full description of the regional, demographic or other classification variable controls used for balancing the sample to achieve representivity must be described.
2. The precise quota control targets and screening criteria must also be stated including the source of such targets (e.g., census data or other data source).
3. Deviations from target achievement must be shown in the report (i.e., actual versus target).

Multi-Mode Surveys

Note: Multi-mode surveys are ones where different methods of questionnaire administration are used. They will often involve a combination of online and telephone methods, although there are other possibilities (e.g., in-person, mail, fax).

When a survey is conducted using multiple modes of questionnaire administration:

1. The reasons for using a multi-mode rather than a single-mode method must be stated, both in the research proposal and the survey report.
2. When the plan is to combine data collected via different modes in the data analyses, then steps must be taken to ensure as much comparability as possible across the different survey modes in terms of question wording and presentation of response options.
3. Steps must be taken to ensure avoidance of duplicate respondents in different modes. The steps taken, and the results, must be documented.

The survey report must discuss whether there are any data quality issues arising from combining data collected via different modes. This could include, for example, discussion of possible impacts of mode on key survey variables, the impact of any differences in response rate / participation rate by mode, and non-response bias analyses by mode.

Attempted Census Surveys

Note: In a census survey, an attempt is made to collect data from every member of a population. For example, an organization might want to do a survey of all of its employees. In this case, the population is "all of the organization's employees", and this would qualify as an attempted census survey if all employees are invited to participate in the survey.

The list or sample source must be clearly stated, including any of its limitations/exclusions in representing the universe for the target sample and the potential for bias.

Note: Whenever possible, an estimate of the percentage of the universe that has been excluded or underrepresented must be provided.

1. The number of attempted recontacts and procedure for attempted recontact must be stated.
2. Do not state a margin of sampling error, as this does not apply to census surveys because no sample is drawn.

5. Survey Respondents

Consent

1. Researchers must ensure that respondents are aware that their participation in the survey at all stages is totally voluntary. In order to do that, Researchers must make sure that the respondents fully understand the purpose of the call. Interviewers must therefore always clearly state their name (first name only or unique identifier) and the name of the research organization they are working for and answer any questions the respondents may have in an honest and non-deceptive way.
2. All respondents must be advised that the interview may be monitored and/or taped for quality control purposes.
3. When interviewing children (under 13 years of age) and young persons (13-15 years of age), Researchers must first obtain parental/guardian consent. Special care shall be exercised for respondents who are considered to be otherwise vulnerable. (Details on data collection from children or vulnerable respondents are provided in section 6.)

Right to Refuse

1. Researchers must respect the right of a respondent to refuse to participate in a survey, to refuse to respond to a question or to terminate the interview at any point. Researchers may, on the other hand, use reasonable methods to obtain the respondent's participation. This includes explaining the purpose of the research and offering to call back the respondents at a time that is more suitable to them if they are unable or unwilling to participate at the time of initial contact.

Avoidance of Harassment

1. The Researcher will take all reasonable steps to see that respondents are not in any way hindered or embarrassed by any interview, and that they are not in any way adversely affected as a result of it. Researchers must address sensitive subject matter in a way that will minimize the discomfort and apprehension of both respondents and interviewers.

Respondent Anonymity and Confidentiality

1. Respondent confidentiality must be preserved. Individually, respondents are not to be identified by name or any other identifying information, nor identified with their specific responses in any study without their verbal or written consent.
   1. Researchers must assure respondents of the fact that their identity will not be divulged to the Client. Respondents are entitled to expect such assurance and that this assurance will be respected by the Researcher.

      Under no circumstances are questionnaires or other material (e.g., magnetic media) containing respondent identity, or information that might allow respondents to be identified, to be released by Researchers to Clients or other third parties.

      An exception to this rulecould be considered if a respondent to a customer research study (using a Client-supplied list) asks the interviewer to pass along some information, concerns or requests to the Client. In such a case, once the respondent permission is granted, some personal information such as the respondent's name and phone number could be given to the Client.

   2. When recording or observation techniques are used, Researchers must advise the respondent at the beginning of the interview (or during the recruiting stage if at all possible) that such techniques are used. Respondents' confidentiality must not be jeopardized by the use of such methods.
   3. Client-supplied lists provided for specific projects must not be used for any other projects or for adding names to the Researcher's databases. If they have been updated during the project those lists must be returned to the Client upon completion of the project. Otherwise, the lists must be destroyed (electronic and hardcopy).
   4. A respondent's personal data must not be used, disclosed, nor collected in any manner incompatible with the intended purpose of the research. This applies to all Researchers' personnel.

Care must be taken to keep information secure whether on hard copy, on computer or stored electronically.
The individual respondent's identity must not be revealed to the Client either alone or associated with a particular response without the express permission of the respondent sometime during the interview. If such permission is granted, the interviewer must record it.

Retaining Public Confidence

1. Researchers must do everything in their power to gain, retain and increase public confidence in research organizations and the public opinion research conducted for the Government of Canada. Interviews are to be conducted free of embarrassment, in an environment of trust and goodwill, and with a recognition of the respondents'; right to privacy should they choose not to respond to a question or participate in the survey.
2. Using the respondent's local time, telephone interviewing can only be conducted from 9:00 to 21:00 Monday to Friday, 10:00 to 21:00 Saturday, and 12:00 to 21:00 Sunday and statutory holidays except when a pre-arranged interview is requested by the respondent outside these hours.
3. There must be no or minimal (one second) pause before the interviewer acknowledges that a potential respondent has answered the telephone.
4. Because of response burden due to the high number of calls needed for Interactive Voice Response (IVR) surveys, Researchers can use IVR only for calls that are pre-arranged with respondents or when the particular information to be collected is essential for important decision making and cannot be obtained through other means. When IVR is used, the same information required for interviews conducted by live interviewers (sponsor, researcher, participation is voluntary, assurance of confidentiality, etc.) must be included in the IVR survey introduction.
5. The name, address and telephone number of the research company must be provided if requested by the respondent.
6. Interviewers must answer in an honest and non-deceptive manner questions that respondents may have.
7. Interviews shall be limited to the legitimate gathering of information relevant to the research. Unless explicitly stated at the outset and agreed upon by the respondent, information will not be used to sell or develop leads for marketing purposes or to compile mailing lists.

6. Data Collection From Children or Vulnerable Respondents

General

1. Researchers must take special care when interviewing children and young people and respondents who are considered to be otherwise vulnerable. A "child" is defined as "under the age of 13" and a "young person" as "aged 13-15."

Standardization of Guidelines

1. When interviewing a child or young person, interviewers must obtain the consent of the parent or responsible adult (guardian, etc.) before the child/young person is approached for an interview. Sufficient information must be provided to the person responsible for the child/young person in order for them to make an adequate decision about giving such consent. The name of the person giving the consent must be recorded but normally it is not necessary to obtain the permission in writing.
2. The adult consent allows the interviewer to approach the child/young person but does not give him the permission to interview them as the child/young person must have their own opportunity to refuse to take part in the research.
3. When conducting telephone surveys, it might be difficult to determine the age of the respondent and to obtain the appropriate consent to do the interview. Despite those difficulties, Researchers must still try to comply with the rules set out in this guideline.
4. An exception is when it is not possible to obtain parental/guardian consent (e.g., street kids) and the benefits of the information are judged to outweigh the possible harm to respondents (e.g., youth engaged in risky behaviour).

Children's Safety

1. The welfare of the children and young people themselves is the most important thing to consider. They must not be disturbed or harmed by the interview experience. Therefore, sensitive topics or issues that could upset the children and young people (e.g., relationships with other children or with parents, sexual activities, use of drugs or alcohol) must be dealt with using special care.
2. It is recommended that some responsible adult (beside the interviewer) remains close – not necessarily in the same room – while the interview is carried out. The interviewer and researchers must be protected against any misunderstandings or possible allegations of misconduct arising from their dealings with the children or young people taking part in that project.

Types of Information Collected

1. Personal information relating to other people (for example, parents) must not be collected from children.

7. Data Collection

Project Briefing

1. Fieldworkers shall be given a briefing and/or instructions for each project (or the specifics for each wave of a project), even if they start to take part after it has already started.
   1. The briefing and/or instructions shall include more detail (than is already part of the Introduction for survey respondents) on the background to the project, e.g., why it is being done, by whom and how the information will be used.
   2. The briefing and/or instructions shall be the responsibility of a member of staff with a full understanding of requirements for the project.
   3. While it is preferable to have briefing and/or instructions delivered face-to-face, these may be delivered by phone, by tape (audio or video), in writing or otherwise but records shall be kept to show the coverage of the briefing and instructions and that all fieldworkers allocated to the project have been briefed/instructed. Note that every effort must be made to avoid briefing and/or instruction in writing only.
2. The briefing is to be of sufficient detail as to ensure that each interviewer is familiar with the questionnaire, sample and respondent selection, and potential problem areas in the administration of the survey. Interviewers must be familiar with the question wording before commencing with the interviews. Normal standards of practice include rehearsal interviews and pre-testing.
3. The briefing and/or instructions shall include, as appropriate, the following information:
   1. the fieldwork dates;
   2. the sampling procedures;
   3. quotas to be covered;
   4. methodology;
   5. other special requirements of the project;
   6. instructions / conditions for administration of the questionnaire.

Call-Backs

There will be a minimum of eight call-backs made before retiring a telephone number and substituting it with another number. The call backs must be made at varying days and times over a minimum seven-day period. An exception could be made when the field period is shorter as a result of the need to assess recall of particular events or activities.

Arranged Call-Backs

Every effort should be made to ensure that the respondent is called back if an appointment has been arranged and that the date and time of that appointment are respected.

Refusals

No attempt will be made to call-back refusals.

Fieldwork Validation General

1. The purpose of validation is to establish that data collection by fieldworkers has been carried out to project instructions including following questionnaires and general requirements.
2. Validation shall be carried out as soon as practical after the fieldwork and before the resulting data are processed and/or reported to Clients.
3. Clients must be given an opportunity to arrange "checks" on the quality/validity of fieldwork subject to any legal constraints that may apply.

Validation Methods

1. Depending on the type of data collection concerned, validation shall be undertaken as specified below, by means of checking the data records produced (e.g., questionnaires, CATI data files) and/or respondent re-contact (also termed back checking) and/or monitoring.
2. Checking of data records may include, as appropriate, completeness of the data records, keeping to samples/quotas, consistency of responses and comparison of responses against normal data or between fieldworkers. Checks may be made manually or by computer. This may be linked to the checking of associated records.
3. Respondent re-contact shall include confirmation that the interview or equivalent took place and that instructions were followed, the length of the interview and responses to key questions including demographics and other qualifying questions related to quotas, etc. Re-contact may be by any medium involving direct communication with the respondent (e.g., face-to-face, by phone, by post or by email).
4. Monitoring shall be carried out on an ongoing basis, throughout the field period. Monitoring shall involve listening to interviews at the time they are being carried out, using appropriate equipment or listening to recordings of the interviews. Both interviewer and respondent shall be audible. Interviewers shall be aware that any interview may be monitored but not know whether a specific interview is being monitored. Where interviewing is multilingual, staff fluent in the relevant languages shall carry out monitoring. Monitoring is particularly suitable for phone interviews carried out from suitably equipped central locations (phone units).

Validation Levels

1. Validation shall be carried out to at least the levels below (depending on the type of validation). Validation levels shall be calculated on the basis of achieved interviews and achieved validations.
2. For all projects where data collection is by interview, the Researcher shall validate the data collection by either monitoring or re-contact to the levels shown below, whether or not data records are also checked:
   1. the minimum monitoring validation level shall be 10%of the interviews/cases with at least 75% of the whole interview monitored/listened to;
   2. the minimum re-contact validation level shall be 10% of the interviews/cases.
3. Every fieldworker working on a project must be validated or monitored.
4. In exceptional cases it can be organizationally impossible to carry out re-contact or monitoring to the required level, or at all, or it may be considered contrary to respondents' interest. In such cases project records shall explain why this is the case and what other steps (e.g., checking data records) have been taken to validate data collection.

8. Response Rates / Participation Rates

Response Rate / Participation Rate Targets

1. The telephone survey must be designed to achieve the highest practical rates of success.

Monitoring Response Rate / Participation Rate During Data Collection

1. Monitoring of call dispositions/reasons for non-response shall be carried out on an ongoing basis, throughout the entire field period.
2. This information will be provided to the Client upon request.

Non-Response Bias Analyses

1. All survey reports must contain a discussion of the potential for non-response bias for the survey as a whole and for key survey variables. Non-response bias is the systematic difference between true population values and the average result from all possible samples owing to non-response.
2. The analysis of non-response bias will consist of a comparison of at least three variables in the survey sample with the equivalent parameters of the population, normally available from Statistics Canada. No extra surveys or interviews are required.
3. If no external sources of comparison exist, key variables in the sample could alternatively be compared to variables in the sampling frame. If no such variables exist, only then should the non-response bias discussion be based on comparison of early versus late responders or on observations made during data collection about characteristics of non-responders.

Calculation of Response Rate / Participation Rate

1. Response rate or level must be calculated for probability and attempted census surveys.
2. Participation rate or level must be calculated for non-probability surveys.

The calculation for response rate for telephone surveys must be done as recommended by the MRIA. The results must be included in the survey report. The survey report must also show a record of contact dispositions that includes the categories required to comply with the MRIA calculation formulas. The same formula used to calculate response rate for telephone surveys should be used to calculate the participation rate for telephone surveys (see below).

The response rate / participation rate together with the following record of call disposition must be included in the survey final report.

Empirical Calculation for Data Collection

• Total Numbers Attempted
• Invalid
  
• NIS, fax/modem, business/non-res.

Unresolved (U)

• Busy
• No answer, answering machine

In-scope - non-responding (IS)

• Household refusal
  
• Respondent refusal
• Language problem
• Illness, incapable
  
• Selected respondent not available
  
• Qualified respondent break-off

In-scope - Responding units (R)

• Language disqualify
  
• No one 18+
  
• Other disqualify
  
• Completed interviews

Response Rate / Participation Rate = R/(U+IS+R)

9. Data Entry

Electronic Data Entry

1. It shall be the responsibility of the Researcher to ensure that data entry or capture specifications for CATI are correct as specified and accurate, based on the Client-approved questionnaire.
2. Upon request, the Researcher will provide the Client with the CATI version of the questionnaire.

Data Entry Verification for Paper Documents

1. The Researcher shall document the level of verification to be carried out. A systematic method of verifying data entry shall be carried out on a project or stage/wave. The minimum total percentage verification per project shall be 10% of entries. Procedures shall ensure that there is a systematic method of verification of each operator's work and the verification shall be undertaken by a second person.
2. If an individual operator's work contains frequent errors, that individual's work (on the project) shall be 100% verified/re-worked. If necessary, appropriate retraining shall be given to that operator until error rates are acceptable.
3. The Researcher shall define the meaning of frequent errors and document that definition.

10. Coding

Developing Code Frames

1. The initial code list/frame shall be developed based on a systematic review of a minimum of 10% of open-ended responses and 50% of partial open-ended responses, where a frame does not already exist. The initial code list/frame must be provided to the Client for approval upon request.
2. The Researcher shall ensure that coders working on the project are provided with instructions and training that shall include, as a minimum:
   1. an overview of the project;
   2. identification of questions or variables to be coded;
   3. the minimum proportion or number of a sample (and its make-up) used to produce code frames;
   4. where necessary or appropriate, specific sub-groups required to develop code frames (e.g., by region, user or non-user);
   5. guidelines for the inclusion of codes in the code frame (e.g., decisions or rules regarding what must be included or excluded from a given code);
   6. any use to be made of code frames from a previous project or stage;
   7. any other requirements or special instructions specific to the project.

Code Frame Approval/Coding Procedures

1. Where "don't know" and "no response" have been used, these shall be distinguishable from each other.
2. The Researcher shall have clear rules or guidelines for the treatment of responses in "other" or catch-all categories; if the "other" or catch-all category exceeds 10% of responses to be coded, the responses must be reviewed with a view to reducing the size of the group.
3. Researchers shall ensure that there is a systematic method of verifying a minimum of 10% of questionnaires coded per project and the verification shall be undertaken by a second person.

11. Data Editing/Imputation

Editing Data/Imputation

1. An accurate record of any changes made to the original data set shall be kept. Comparison to the original data source shall be the first step in the process. Any imputation processes, including the logic of the imputation method(s) used shall be documented and provided to the Client. All edit specifications shall be documented.
2. Where forced editing is used, the logic of the forcing shall be documented and test runs carried out, with the results documented to show that the forcing has the desired effect.
3. Data editing/imputation must be used cautiously. The degree and impact of imputation must be considered when analyzing the data, as the imputation methods used may have a significant impact on distributions of data and the variance of estimates.
4. The Researcher shall include documentation of any imputation/forced editing, both in a technical appendix and in the final report.

Data Editing of Paper Documents Prior to Data Entry

1. Where paper documents are hand edited prior to data entry, it shall be possible to distinguish the original answers of the respondent or interviewer from the codes or answers allocated by the person(s) carrying out the editing.
2. When this type of editing is used, the logic and rules being applied shall be documented and any staff working on this element of the project shall be briefed as to the types of checks and corrections they may carry out.

12. Data Analysis

Inferences and Comparisons

Researchers must base statements of comparisons and other statistical conclusions derived from survey data on acceptable statistical practice.

Analysis Records

1. The Researcher shall keep accurate and descriptive records of the analysis process for a minimum of three years, to ensure that any analysis undertaken can be replicated at a later date.

Data Analysis Verification

1. The Researcher shall have in place procedures to ensure the tabulations and other outputs have been checked.
2. As a minimum, these checks shall verify:
   1. completeness, i.e., that all tables are present as specified, including the results of all reported significance tests;
   2. that abbreviations for headings or open-ended responses accurately reflect the full content;
   3. that the base for each table is correct against other tables or frequency counts;
   4. that the standard breaks/banner points are checked against source questions;
   5. that all derived data items are checked against their source;
   6. that the figures for sub-groups and nets are correct;
   7. that there are no blank tables (i.e., with no data);
   8. weighting (e.g., by test tables);
   9. frequency counts prior to running tables, in order both to ensure the accuracy of data and to determine base sizes for sub-groups;
   10. spelling and legibility;
   11. that any statistical analysis used is appropriate and correct, both in its descriptive and inferential aspects.
3. For any subsequent outputs, appropriate checks shall be applied.

13. Delivery of Data Tables

Delivery of Stand-alone Hard or Soft Copy of Data Tables

1. When data are reported to the Client, such as in a stand-alone hard or soft copy of data tables, the following shall be taken into account, as appropriate:
   1. reference to the actual source question to which the data pertains;
   2. inclusion of a description of any weighting method applied to the data;
   3. clear identification of any sub-groups used;
   4. availability of the bases for each question, so that the number of respondents who have actually answered the question is identifiable;
   5. the number or proportion of respondents who replied "don't know" or gave "no answer";
   6. availability of both weighted and unweighted bases;
   7. clear and complete definition and explanation of all variables used in the analysis of the data, including any significance testing, indexing, scoring, scaling and calculations of means, median, modes and standard deviations;
   8. the types of statistical tests being used and their level of precision;
   9. information on cell suppression and other measures to assure confidentiality;
   10. warnings on results which are unreliable due to very small sample sizes.

Electronic Data Delivery

1. The Researcher shall provide the Client with a data file.
2. The following shall be checked prior to data release:
   1. compatibility of the file format with the software specification agreed with the Client;
   2. completeness (i.e., the correct number of files and records are in each file);
   3. inclusion of all appropriate documentation to allow for replication of the data analysis and additional analyses, including where applicable:
      1. labelling of the contents of the file, i.e., fully labelled variables and value labels;
      2. identification and description of any computed or recoded variables, and instructions on limitations of use;
      3. labelled weighting variables and a description of how these were applied;
      4. all personal identifiers per PIPEDA have been removed from the files;
   4. encryption of files upon request;
   5. presence of viruses in the file.

14. Back-up, Retention/Security of Data

Back-up, Security/ Retention of Data

1. All data shall be held securely to prevent unwarranted access, damage or accidental loss.
2. The Researcher shall ensure that all data are stored and retained in accordance with the applicable law and legislative regulations, and any negotiated agreement with the Client.
3. The Researcher must maintain the technical data on all studies for a period of three years, so that if requested, the study can be replicated.
4. Technical data not already included in the Survey Report/Appendix that must be maintained includes, but is not limited to:
   1. raw data files;
   2. other electronic files;
   3. hard copy questionnaires and any materials/visual aids used by interviewers in the execution of interviews (e.g., cards, lists, FAQs);
   4. code frames;
   5. project files including project management information;
   6. emails and other correspondence.
5. The Researcher shall establish and maintain procedures to ensure that data processing computer files are clearly identified.
6. Where data has been edited, cleaned, recoded or changed in any other way from the format, content and layout of its original format, the original data, final data and programme files, including all documentation related to changes to the data (as a minimum) shall be kept so that the final data set can be easily reconstructed.
7. Extra analyses may be requested after the main processing and such analysis shall conform to the data processing requirements of this standard.
8. For any proposed release of tables or data, procedures shall be in place to minimize the risk of disclosure of respondent identifiable information.

15. Survey Reports

Library and Archives Canada

It is a legal requirement for institutions to send written public opinion research reports to Library and Archives Canada within six months of the completion of the data collection for all reports.

Pursuant to the Public Opinion Research Contract Regulations, final reports must include the following at a minimum:

1. on the covering page, the title of the project, the name of the person who entered into the contract, the contract number and the award date;
2. a narrative executive summary consisting of, at a minimum:
   1. a statement of the researches purpose and objectives,
   2. a summary of key findings, except where the person who entered into the contract is not responsible for the design, development of the methodology and analysis of the research,
   3. a brief description of the methodology used, and
   4. a statement as to the extent to which the findings can be extrapolated to a broader audience;
3. for quantitative research, appendices containing:
   1. a full set of tabulated data,
   2. sample size, sampling procedures and dates of research fieldwork,
   3. if applicable, weighting procedures, the confidence interval and the margin of error,
   4. the response rate / participation rate and method of calculation,
   5. the research instruments used, and
   6. all other information about the execution of the fieldwork that would be needed to replicate the research initiative

General

The following minimum details shall be documented in the project report. These allow the reader to understand the way the research project was conducted and the implications of its results.

Background

1. Detailed description of background including, at minimum:
   1. purpose, how the research will be used.
2. Objectives, research questions.

Sample

1. Detailed description of the sample including:
   1. the target group for the research project;
   2. the achieved sample size against projected sample size and reasons, if relevant, for not obtaining the projected sample;
   3. the sample source and sampling method, including the procedure for selecting respondents;
   4. the weighting procedures, if applicable.
2. For non-probability samples, provide:
   1. rationale for choosing a non-probability sample.

Data Collection

1. Detailed description of methodology including:
   1. the dates of fieldwork (each phase/wave if applicable);
   2. the average survey duration and the range;
   3. the data collection method(s), and if applicable:
      1. the type and amount of incentives;
      2. any accessibility provisions for respondents using adaptive technologies.
2. For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method.

Quality Controls

1. The estimating and imputation procedures, if applicable.
2. A brief summary of other quality controls and procedures used.
3. For multi-mode surveys, detailed description of any data quality issues arising from combining data collected via different modes/instruments.

Results

1. For probability samples, state the level of precision, including the margin of error and confidence interval for the total sample and any key sub-groups.
2. For non-probability samples and attempted census surveys, the report must contain a statement on why no margin of sampling error is reported.
3. The call dispositions and response rate (in the case of probability samples and attempted censes) and participation rate (in the case of non-probability samples).
4. The results that are based on sub-groups and the number of cases used in sub-group analysis.

Appendix 1: Study Materials

Study Materials, including the recruiting instruments, questionnaires, descriptions or representations of any visual or auditory aids, and other relevant data collection documents, in all languages in which the research was conducted.

There must be a version of the questionnaires displaying any instructions (e.g., skip, terminate, etc.) needed to understand the logic and flow of the questionnaire.