Note
The sections in blue text presented in 
is the proposed text from the members of the advisory committee.
Prepared for Public Services and Procurement Canada
Supplier: Sage Research Corporation
Contract number: EP363-182149/001/CY
Contract value: $95,106.45
Award date: December 7, 2017
Delivery date: October 26, 2018
Registration number: POR 058-17
This public opinion research report presents the results of a research panel conducted with knowledgeable, leading professionals from the private sector, Statistics Canada and academic institutions, between April 11, 2018 and August 22, 2018.
This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Public Services and Procurement Canada for all other uses. For more information on this report, please contact Public Services and Procurement Canada at questions@tpsgc-pwgsc.gc.ca or at:
Public Services and Procurement Canada
Portage III Tower A
16A1-11 Laurier Street
Gatineau QC K1A 0S5
Cette publication est aussi disponible en français sous le titre : Comité consultative sur la qualité de la recherche quantitative sur l’opinion publique au gouvernement du Canada
Catalogue Number: P103-14/2019E-PDF
International Standard Book Number (ISBN): 978-0-660-28290-9
Related publication (registration number: POR 058-17):
Catalogue Number: P103-14/2019F-PDF (Final Report, French)
International Standard Book Number (ISBN): 978-0-660-28291-6 (Final Report, French)
© Her Majesty the Queen in Right of Canada, as represented by the Minister of Public Works and Government Services, 2019
The sections in blue text presented in is the proposed text from the members of the advisory committee.
The Public Opinion Research Directorate (PORD) is a mandatory common service provider responsible for giving advice on legislation, policies, research methodology and accepted industry practices. Under the Policy on Communications and Federal Identity, PORD also has the responsibility for developing and maintaining Government of Canada standards. The standards for both telephone and online public opinion surveys were first developed and implemented in 2009 and based on 2 separate advisory panels, one related to telephone and the other related to online research. These standards were later revised in 2013.
Given the ongoing changes in the public opinion research industry, PORD is undertaking a review of its quantitative standards for the conduct of Government of Canada public opinion research.
The project involved convening an advisory panel composed of knowledgeable, leading professionals from the private sector, Statistics Canada and academic institutions to provide advice on potential standards and best practices for public opinion survey research (POR) conducted using telephone and/or online quantitative methods.
The advisory panel addressed and provided guidance on standards for the following topics:
For the topics listed above, the intent is to help PORD (a) to revise existing standards and guidelines and, as appropriate, create new standards and guidelines to guide the quality of survey research undertaken on behalf of the Government of Canada, and (b) to equip PORD with expert advice on dealing with evolving research methodologies.
The advisory panel on Government of Canada quantitative public opinion research consisted of 10 members drawn from the private sector, academics experienced with market research, and Statistics Canada. Members of the panel were recruited by PORD, with assistance from Sage Research.
The advisory panel’s work took place between April 11, 2018 and August 22, 2018. The advisory panel process consisted of an initial web conference followed by 3 online discussion boards. Panel members then reviewed 4 working reports summarizing the results and proposing the guidance to put into the final report of the advisory panel.
The advisory panel’s recommended guidance for quantitative research is expressed as standards and guidelines, together with supporting commentary.
While it was not a mandate of the advisory panel to reach consensus, it did so on quite a few aspects of standards and guidelines for quantitative research.
Qualitative research is designed to reveal a rich range of opinions and interpretations rather than to measure what percentage of the target population holds a given opinion. Advisory panel members gave their personal opinions and experiences on the issues discussed, and were not speaking on behalf of their organization or industry.
Contract Value: $95,106.45, including Harmonized Sales Tax (HST)
I hereby certify as Senior Officer of Sage Research Corporation that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications and Federal Identity Policy of the Government of Canada and Directive on the Management of Communications. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, and standings with the electorate or ratings of the performance of a political party or its leaders.
Anita Pollak
President
Sage Research Corporation
The panel provided input on the following topics:
There were recommended changes to the following sections of the standards:
Section 4 sampling procedures includes standards for probability sampling, non-probability sampling and a census but does not give definitions of these types of sampling procedures.
Section 4.1.2 (online)/4.1.1 (telephone) should be expanded (a) to include both definitions and examples of probability and non-probability sampling, and (b) a definition of a census.
The objective is to revise the standards to emphasize the importance of striving for representativeness in non-probability surveys, and to explain in the proposal how this will be done. The current standards (sections 4.3.2 sampling procedures and 1.2.2 proposal documentation) address this objective to some extent, but the intent is to make the requirement more explicit and detailed.
The majority of the panel members agreed with including text to emphasize the importance of taking steps to improve the representativeness of non-probability survey results (in both sections 4.3.2 and 1.2.2). However, no consensus was reached on whether this should be a standard or a guideline.
Section 1.2.4 #3 proposal documentation lists the required information when an online sample provider is used. This section should be expanded to require separate and more specific proposal information for both probability and non-probability online samples.
There was agreement on the types of information that should be required in the proposal for online probability samples. For online non-probability samples, there were different points of view on some of the specific information disclosure requirements.
The panel provided input on the following topics:
There were recommended changes to the following sections of the standards:
The panel was asked to clarify the use of statistical measures for non-probability surveys in light of recent developments in the application of alternative measures of statistical precision.
Section 4.3.3 of sampling procedures should include (a) revised wording to further clarify that margins of sampling error do not apply to non-probability survey data; (b) limitations on use of alternative measures of precision (e.g. Bayesian credible intervals) for non-probability surveys; (c) documentation requirements in both the proposal and the survey report when alternative measures of precision are used.
The objective was to determine whether to expand on the current requirements for reporting level of precision for probability surveys (section 14.7.2/15.7.2).
There were several proposals for revised wording, but no consensus on the most appropriate wording.
In section 4.6 in sampling procedures, there are 2 statements made about the statistical treatment of census survey results. The panel was asked to comment on whether the 2 statements were consistent.
It was agreed that part of section 4.6.3 should be deleted as it is inconsistent with the standard to not use inferential statistical tests in a census survey.
The panel was also asked to consider whether or not a survey ceases being a census if the response rate falls below a certain level.
There was agreement that a census with a response rate of less than 100% is still a census, albeit perhaps better described as an attempted census. Margin of sampling error does not apply but other sources of survey error can still be present, such as non-response bias.
The panel provided input on the following topics and recommended changes to section 2 questionnaire design (2.1.2; 2.1.3):
There is a general requirement (section 2.1.2) to inform respondents at the beginning of a survey of the confidentiality of their questionnaire responses, but the current standards do not state any specific wording for how to preface the block of demographic questions located near the end of the questionnaire.
Add a requirement for an introduction to the block of demographic questions that addresses confidentiality. There was agreement on most of the wording for this introduction, but no consensus was reached on use of the terms “confidential” and/or “anonymous” in this introduction.
“Prefer not to answer” (and its related forms) should be removed as a required listed response category in online surveys, but retained as an optional response category.
Some of the required demographic questions have a relatively long list of response options. This includes the questions on age, education and household income.
Revise section 2.1.3 to allow for modifying the wording of a demographic question to allow the interviewer to instruct a respondent to stop at the category that applies to them.
In the currently mandated question, gender information is collected very differently in telephone versus online surveys. The telephone version does not actually ask a question about gender, but rather relies on interviewer observation. The online version asks a question and so is based on respondent self-classification rather than interviewer classification. Neither the telephone nor online survey versions offer an “other” answer option.
Revise the mandated question for all POR surveys (telephone and online) (a) to include an “other” answer option, and (b) require the gender question to be read to respondents in telephone surveys.
There are currently 2 mandated questions for language (mother tongue and language spoken most often at home), with discretion for the researcher to use one or both questions depending on the survey objectives.
Revise this to require only language spoken most often at home.
Revise 2 of the current age categories (35-49 and 50-54) to 35-44 and 45-54 to get a more even distribution of the age categories.
There was discussion but no consensus on whether the current age category 18-34 should be split into 2 categories (18-34 and 25-34) or left as is.
In order to better align the response options with the school systems in both Quebec and the rest of Canada, combine “grade 8 or less” and “some high school” into a single category, “less than high school diploma or equivalent.”
Revise the question wording to specify a time frame of “last year” for household income.
This information can sometimes be useful in quota controls or weighting. Add a requirement to section 2.1.3 (a) to include a question on household phone status in all telephone surveys.
The current standards do not address the possibility and implications of an online survey being completed on a mobile device. The panel was asked to provide input on revising the standards to address the following areas:
There were recommended changes to the following sections of the online standards:
The default expectation should be that an online POR survey sample will include respondents using either a computer or a mobile device for the survey and that surveys have a mobile-friendly version of the questionnaire.
Additions to sections 1.2.2 and 1.2.5 in proposal documentation are recommended to make these expectations explicit.
There were 3 potential revisions/additions to section 2 questionnaire design considered with respect to questionnaire design in online surveys where mobile devices may be used.
The consensus was that a standard is not appropriate given that research on what design approach is best for question design/layout across devices is inconclusive, and optimal design approach can vary across surveys and for different questions within a survey. However, the addition of a guideline highlighting the options available to researchers could added in section 2 questionnaire design.
The standard for online questionnaire duration is 20 minutes, but an average duration of 15 minutes or less is “strongly encouraged.” The panel considered whether the standard for survey duration should be left as is, or revised to specify a shorter duration for mobile-friendly surveys.
No change was recommended to the existing standard on survey duration.
Most agreed that a list of examples be added to section 2 questionnaire design as a useful reminder to researchers of the elements that make a questionnaire more mobile-friendly.
The current standard specifies the total number of pre-test completions, but does not break this down by device type.
The panel considered potential revisions/additions to section 3 pre-testing with respect to online surveys where both mobile devices and computers may be used.
Section 3 pre-testing should be revised to include a requirement for pre-testing on both computers and mobile devices when a survey can be completed on both types of devices. Several alternative options for how to word the requirement were proposed.
The consensus was that section 3.1.5, which requires a minimum of 10 pre-test interviews in each language, should be left as is with the understanding that the pre-test would include a sample of different devices.
In a survey that allows completion on both mobile devices and computers, there is the potential for a “mode” effect. That is, the different designs/layouts for a given question could cause different response distributions.
The panel considered whether or not there should be any requirement to collect information on device type, and any requirement to conduct an analysis for mode effects by device type.
A standard should be added to section 7 data collection requiring collection of data on the type of device used by respondents to complete a survey.
The panel did not support adding a standard requiring an analysis of mode effects for each survey (section 14.6 quality controls). The view was that not enough is known about device type mode effects at this time to specify analytic requirements for individual surveys and “research on research” needs to be done using the aggregated device data collected across surveys to determine what if any standard would be appropriate for analysis of potential mode effects.
Users of mobile devices may incur costs to participate in a research survey. The current standards do not have any requirements as to how such costs should be handled.
The consensus was that there should not be a standard about covering respondent costs associated with using a mobile device: (a) respondents always have a choice whether or not to participate in a Government of Canada (GC) Public Opinion Research (POR) online or telephone survey; (b) the current standards require certain information be given about the survey (e.g. length), so respondents are able to make an informed choice about whether or not to participate; (c) unless compensation is set at an arbitrary fixed amount for all mobile users, the logistics of determining the amount to compensate each respondent and documenting this for billing purposes would be very complex and difficult, if not impossible.
An important issue in sampling for telephone surveys is the inclusion of cell phone users and landline users. This can affect coverage of the survey population, the sampling frame(s) used for the survey, and possibly weighting. A telephone probability sample of the general Canadian adult population must include a cell phone sample. The panel was asked to consider revisions to the telephone standards related to proposal documentation and sampling procedures:
There were recommended changes to the following sections of the standards:
There were 3 potential revisions/additions to section 1 proposal documentation in the telephone standards considered by the panel.
Consider revising the text in section 1.2.3 #1 to require stating an estimated response/participation rate for both cell phones and landline in surveys where both device types can be used.
The panel was asked to consider whether there should be any revisions to section 1.2.4 #7 in proposal documentation, which states that a rationale must be given when the sample includes interviews on cell phones. The current language overly downplays the importance of including cell phone users in the sample.
The wording of section 1.2.4 #7 should be revised to acknowledge the importance of cell phone samples in telephone surveys. There were several alternative proposals on the approach to take.
The current standard in sampling procedures section 4.2.3c addresses disclosure of coverage issues in probability samples, and gives as an example a sample of cell phone only households.
Add landline-only samples as another example in section 4.2.3c given the growing number of cell phone only households, a landline-only sample could have substantial coverage error.
The telephone standards for call-backs in section 7 data collection (7.2) require a minimum of 8 call-backs to be made before a telephone number is retired. Some concern has been expressed that 8 call-backs is too many, and might be perceived as harassment. The current standard also (a) does not provide a definition of what constitutes a call-back, and (b) does not differentiate between call-backs to landlines and cell phones. The panel was asked to consider what should be the standard for number of call-backs, including whether there should be a different standard for respondents reached on a cell phone.
There were 2 main recommendations for revisions to section 7.2:
The panel opted to apply the same call-back requirement to both cell phones and home phones.
The panel was asked to consider revisions to the telephone standards in section 5.3 use of interactive voice response in the following areas:
The panel was also asked to provide input on standards related to:
There were recommended changes to the following sections of the standards:
Section 5.3.1 discourages, but does not forbid, use of IVR surveys for POR. It also suggests circumstances when IVR may be an appropriate methodology. The panel considered whether there should be any changes to this sub-section on the use of interactive voice response.
The majority suggested adding more examples of situations when IVR as a data collection method may be acceptable while maintaining the principle that IVR is not a preferred method for GC POR surveys.
Section 5.3.2 states that the information disclosure requirements for IVR surveys are the same as for interviewer-administered surveys, and similarly requires that the information be provided in the survey introduction. Because IVR surveys are typically shorter than interviewer administered surveys, the panel was asked to comment on (a) whether the required elements for telephone survey introductions should be revised or shortened for IVR surveys, and (b) the possibility of moving some of the information disclosures to the end of the survey.
Most panelists said the required information in the survey introduction should be the same for IVR surveys as for other surveys. There was no consensus on where in the questionnaire the various required disclosures should be made. However if it is decided that certain types of information can be disclosed later in a survey, that option should be available to all surveys, and not just to IVR surveys.
The standard for survey duration states surveys must be completed in 20 minutes, and strongly encourages a duration of 15 minutes or less.
A guideline should be added to questionnaire design section 2.1.1 to encourage an IVR survey duration of 5 to 7 minutes or less.
The call-back requirements in section 7.2. call-backs do not make any distinction between interviewer-administered surveys and IVR surveys. The panel considered whether there should be any changes to this section specific to IVR surveys.
There were several alternative proposals, ranging from a recommendation to exempt IVR surveys from the call-back requirements for interviewer-administered surveys, to requiring IVR surveys to have the same call-back requirements as interviewer-administered surveys.
The current standards already address multi-mode surveys to some extent. The panel was asked to provide input on possible revisions to the standards in the following areas related to multi-mode surveys:
There were recommended changes to the following sections of the standards:
The primary concern associated with multi-mode surveys is the potential for mode bias. The panel considered whether and how the proposal documentation requirements need to be elaborated to make it more clear in the proposal that the issue of potential mode bias is recognized and that steps will be taken to address this.
Additions were recommended to:
Revise section 4.5 in sampling procedures to emphasize the value of using similar modes of data collection to minimize the risk of mode biases.
There is no current standard for questionnaire design specific to multi-mode surveys.
Revise section 2.1 in questionnaire design (a) to encourage comparability across modes in question wording and presentation of response options, and (b) to highlight the value that including benchmark questions can have for enabling detection of mode biases.
The current section 3 pre-testing does not make any specific references to separate pre-tests by mode in a multi-mode survey. The panel was asked to consider whether there should be a requirement for a minimum number of pre-test interviews in English and French for each mode in a multi-mode survey.
There were several different points of view on this matter, and no consensus was reached.
Currently in section 8 outcome rates there is no standard for how to calculate outcome rates for a multi-mode survey.
A standard should be added outlining the general principles for calculating and reporting on outcome rates for multi-mode surveys. Research designs which do not allow calculation of either of the mandatory outcome rates (response rate or participation rate) should not be permitted for GC POR surveys.
The standard for reporting on data collection in sections 14.5.2/15.5.2 of mandatory survey report requirements should be updated using the updated language in sub-section 1.2.4 #7 in proposal documentation.
Section 14.6.3/15.6.3 quality controls should be revised (a) to ensure decisions made about combining or not combining data across modes are clear, and (b) to require descriptions of any adjustments made to the data to mitigate mode effects.
Section 6 data collection from children, young people or vulnerable respondents does not make any reference to whether or how incentives are used for this survey population. Section 7 data collection (7.5 [telephone]/7.6 [online]) that deals with incentives/honoraria also does not refer to this population.
Guidance should be added to section 7.5/7.6 to address such matters as who will receive the incentive and getting parental consent.
The panel was asked to provide input on possible revisions/additions to the standards in the following areas:
There were recommended changes to the following sections of the standards:
Online and mobile methodologies create possibilities for collecting various types of personal data “passively” that is, without direct interaction with respondents. The issue considered was what passive data collection is allowed and under what circumstances is it allowed in the context of surveys? The panel was asked to consider if the current standards are sufficient to address these questions associated with passive data collection in surveys.
The panel endorsed a revision to the standards in data collection section 7.2 (a) to explicitly define “passive data collection” and provide examples of “personal information”, and (b) to note exceptions where the passive data collection is legally permissible.
The online and telephone survey standards do not currently have any standards pertaining specifically to respondent photographs, videos or audio recordings.
The panel endorsed the addition of standards in section 5 retaining public confidence to clarify (a) that photographs and recordings are considered to be personal data and need to be treated as such, and (b) the responsibility of researchers when the survey involves asking respondents to generate photographs and/or recordings.
The current telephone standards section 5.2.1 avoidance of harassment, has a standard focused on sensitivity of the survey subject matter, but it does not directly address issues potentially caused by the setting of the interview. Because respondents are increasingly likely to answer calls using a mobile phone, there can be issues with them using the phone in problematic settings (e.g. driving, walking in a public space). On both mobile phones and fixed-location phones, they may be in a setting where they can be overheard.
Most panelists supported adding a guideline to determine if a telephone survey respondent is in a location where they can take the call, for both cell and landline users (section 2, questionnaire design).
The current standards in section 13/14 data security require taking steps to protect against data breaches (the loss of or unauthorized access to/disclosure of personal or organization information). The relevant sections are: 13.2 (online)/14.2 (telephone) protection of data servers; 13.3/14.3 temporary storage of data on servers; (c) 13.6/14.5 in the event of any data breach.
The panel was asked to identify any revisions or additions to the standards, and/or any guidelines that should be included.
The existing standards pertaining to privacy and security of data, including data breaches, are appropriate.
There were 2 main areas identified for additional standards or guidelines:
The current standards in section 13/14, data security require that survey data be stored in Canada.
This is a complex area: it requires expertise in the legal and regulatory framework affecting data access and use not only in Canada but in other countries as well where servers might be located, and it requires an understanding of GC policies in this area. The panel did not consider itself to be experts in these areas. For the most part there were no suggested changes to the current standards. However, one suggestion was for the GC to have a pre-approved list of countries that satisfy the conditions set out in in the current standards and that are acceptable for cloud storage of GC POR data.
The panel considered whether there are any additional standards required for surveys that use a social media venue as either a sample source or to administer a survey.
The current standards, together with the various changes recommended elsewhere by the panel, are sufficient to ensure that any such surveys meet the quality requirements for GC POR surveys. Therefore no additional standards are needed for surveys that use a social media venue for either sampling or survey administration.
The online and telephone standards do not contain any standards or guidelines pertaining to accessibility.
The panel considered whether a statement should be added to the standards about the importance of accessibility, and what if any specific guidelines might be provided for online and telephone surveys. Note that according to PORD, the Treasury Board Secretariat (TBS) is working on a proposed policy for accessibility standards specific to all devices used to access online surveys. The results of this development work will probably be available in a year or so. When the TBS policy is finalized, it will take precedence.
The majority of panelists supported adding a general guideline encouraging accessibility including providing guidelines on some examples of steps that could be taken to improve accessibility in online or telephone surveys.
The Public Opinion Research Directorate (PORD) is a mandatory common service provider responsible for giving advice on legislation, policies, research methodologies and accepted industry practices. Under the Policy on Communications and Federal Identity, PORD also has the responsibility for developing and maintaining Government of Canada standards. The standards for both telephone and online public opinion surveys were first developed and implemented in 2009 based on 2 separate advisory panels, 1 related to telephone and 1 related to online research. These standards were later revised in 2013.
Given the ongoing changes in the public opinion research industry, PORD is undertaking a review of its quantitative standards for the conduct of Government of Canada public opinion research.
The project involved convening an advisory panel composed of knowledgeable, leading professionals from the private sector, Statistics Canada and academic institutions to provide advice on potential standards and best practices for public opinion survey research (POR) conducted using telephone and/or online quantitative methods.
The advisory panel addressed and provided guidance on standards for the following topics: sampling
For the topics listed above, the intent is to help PORD (a) to revise existing standards and guidelines and, as appropriate, create new standards and guidelines to guide the quality of survey research undertaken on behalf of the Government of Canada, and (b) to equip PORD with expert advice on dealing with evolving research methodologies.
PORD provides coordination and advisory services for public opinion research (POR). These services are mandatory for POR contracted by institutions listed in schedules 1, 1.1 and 2 of the Financial Administration Act.
The study will assess a broad range of issues that affect quantitative public opinion research, for both telephone and online.
The findings will help PORD to revise existing standards and guidelines and create new standards and guidelines to guide the quality of research undertaken on behalf of the Government of Canada
The advisory panel on Government of Canada quantitative public opinion research consisted of 10 members drawn from the private sector, academics experienced with market research, and Statistics Canada. Members of the panel were recruited by PORD, with assistance from Sage Research.
Advisory panel members gave their personal opinions and experiences on the issues discussed, and were not speaking on behalf of their organization or industry.
Anita Pollak and Rick Robson of Sage Research Corporation facilitated the proceedings of the advisory panel. Their roles included:
The advisory panel was provided with copies of the current Government of Canada standards for the conduct of telephone and online POR, and with additional background materials pertinent to the panel’s deliberations. Panelists were also provided with a literature review previously prepared for PORD (Literature Review in Support of the 2017 Quantitative Standards Review) which contained discussion of some of the topics the panel was asked to address.
The advisory panel’s work took place between April 11, 2018 and August 22, 2018. Panel members then reviewed 4 working reports summarizing the results and proposing the guidance to put into the final report of the advisory panel.
The advisory panel process consisted of an initial web conference followed by 3 online discussion boards. Panel members then reviewed 4 working reports summarizing the results and proposing the guidance to put into the final report of the advisory panel.
An online discussion board method was selected to allow panelists to log on and participate in the discussions at their convenience. For each discussion board, the moderators provided background information to the panelists and posted questions online. Each panelist logged on multiple times over the course of the discussion board:
The dates of each component are shown below. Note that Sage Research informally extended the end dates for most discussion boards to allow certain panel members needing extra time to provide their input:
The advisory panel’s recommended guidance for quantitative research is expressed as standards and guidelines, together with supporting commentary.
Practices that are requirements for all online studies conducted by the Government of Canada; these are typically stated using the word “must.”
Practices that are recommended, but would not be requirements (that is, known good practices or criteria that serve as a checklist to ensure quality research but are not necessarily applied to every study; these are typically stated using the word “should.”)
While it was not a mandate of the advisory panel to reach consensus, it did so on quite a few aspects of standards and guidelines for quantitative research.
Panel members were provided for review and comment 4 working reports summarizing the results and proposing the guidance to put into the final report of the advisory panel.
Section 4 of the standards is sampling procedures, and it includes standards for both probability sampling and non-probability sampling. Section 4 does not give definitions of these 2 types of sampling procedures. This section may be revised to include definitions and some examples to help users determine whether a sample is a probability sample or a non-probability sample.
Probability sampling is a method of sampling that allows inferences to be made about the population based on observations from a sample. In order to be able to make inferences, the sample should not be subject to selection bias. Probability sampling avoids this bias by randomly selecting units from the population (using a computer or table of random numbers). It is important to note that random does not mean arbitrary. In particular, the interviewers do not arbitrarily choose respondents since then sampling would be subject to their personal biases. Random means that selection is unbiased (it is based on chance). With probability sampling, it is never left up to the discretion of the interviewer to subjectively decide who should be sampled.
There are 2 main criteria for probability sampling: one is that the units be randomly selected, the second is that all units in the survey population have a non-zero inclusion probability in the sample and that these probabilities can be calculated. It is not necessary for all units to have the same inclusion probability, indeed, in most complex surveys, the inclusion probability varies from unit to unit. (p. 91)
Non-probability sampling is a method of selecting units from a population using a subjective (i.e., nonrandom) method. Since non-probability sampling does not require a complete survey frame, it is a fast, easy and inexpensive way of obtaining data. The problem with non-probability sampling is that it is unclear whether or not it is possible to generalize the results from the sample to the population. The reason for this is that the selection of units from the population for a non-probability sample can result in large biases.
Due to selection bias and (usually) the absence of a frame, an individual’s inclusion probability cannot be calculated for non-probability samples, so there is no way of producing reliable estimates or estimates of their sampling error. In order to make inferences about the population, it is necessary to assume that the sample is representative of the population. This usually requires assuming that the characteristics of the population follow some model or are evenly or randomly distributed over the population. This is often dangerous due to the difficulty of assessing whether or not these assumptions hold.
The panel was asked to comment on the following proposed expansion of section 4.1.2, giving definitions and examples of 3 types of samples: probability, non-probability and census.
4. Sampling procedures 1
4.1. General [from online standards]
All researchers must:
- clearly state the target group (universe) definition for the research study; in the case of Internet surveys this includes explicit identification of whether or not non-Internet users are part of the target group definition
- clearly state the method(s) used to obtain a sample of this target group, including whether the method is a probability survey, a non-probability survey, or a census.
- probability sample: respondents are randomly selected from the survey's target population, and each respondent's probability of inclusion can be calculated. Probability sampling is a method for obtaining a sample projectable to the target population. Some examples:
- random-digit-dial (RDD) telephone survey of Canadians
- random sampling from a list of all members of the target population
- random sampling from a panel that is itself a probability sample of the target population
- website intercept survey in which target population is visitors to the website, and visitors are randomly sampled to take part in a survey
- non-probability sample: a sample that does not meet the requirements of a probability sample (that is, respondents are not randomly selected from the survey's target population, and/or each respondent's probability of inclusion cannot be calculated). Additional steps must be taken to try to make results from a non-probability sample representative of the target population. Some examples:
- a sample drawn from a research panel consisting of people who volunteer to join the panel and do surveys. Note that a sample collected using probabilistic methods from sampling frames that were compiled using non-probability methods is considered a non-probability sample.
- quota sampling, in which the selection of respondents is based on judgment, convenience or some other nonrandom process
- Census: an attempt is made to collect data from every member of the target population. Note that a census can be subject to other types of survey error, notably coverage error and nonresponse, so not every member of the target population may be in the final data set.
Section 4.1.2 should be expanded to include definitions and examples, and the panel largely agreed with the proposed text.
The most contentious sentence is in 4.1.2b pertaining to non-probability surveys: “additional steps must be taken to try to make results from a non-probability sample representative of the target population”. There were differing points of view on to what extent it might be possible to make results from a non-probability survey representative of the target population, and on whether such attempts should be required. The final decision was to drop this sentence from 4.1.2b simply because it is outside the scope of providing definitions and examples. The topic, however, continued to be considered by the panel in the context of another topic, “maximizing representativeness of non-probability surveys”.
Other revisions to the proposed text are:
The following is the suggested revision to section 4.1 (as applied to the online standards):
4. Sampling procedures 2
4.1. General [applied to online standards]
All researchers must:
- clearly state the target
- clearly state the method(s) used to obtain a sample of this target
- probability sample: The sample meets both of the following conditions: (1) respondents are randomly selected from the survey's target population, and (2) all units in the target population have a non-zero probability of being included in the sample and these probabilities can be calculated. Some examples:
- random-digit-dial (RDD) telephone survey of Canadians
- random sampling from a list of all members of the target population
- website intercept survey in which target population is visitors to the website, and visitors are randomly sampled to take part in a survey
- non-probability sample: a sample that does not meet the requirements of a probability sample (that is, respondents are not randomly selected from the survey's target population, and/or each respondent's probability of inclusion cannot be calculated). Some examples:
- a sample drawn from a research panel consisting of people who volunteer to join or opt in to the panel and do surveys. Note that a sample collected using probabilistic methods from sampling frames that were compiled using non-probability methods is considered a non-probability sample.
- quota sampling, in which respondents are selected through some nonrandom process
- census: an attempt is made to collect data from every member of the target population. Note that a census can be subject to other types of survey error, notably coverage error and non-response, so not every member of the target population may be in the final data set.
Surveys based on non-probability sampling have become more common in marketing research, particularly because of the growth of online opt-in panels that provide significant cost savings over telephone probability samples. Public opinion research surveys for the Government of Canada have historically usually used probability sampling, but there may be more usage of non-probability surveys if there is confidence that these can deliver results that are representative of the target population being surveyed.
The objective is to revise the standards to emphasize the importance of striving for representativeness in non-probability surveys, and to explain in the proposal how this will be done.
The current standards address this objective to some extent, but the intent is to make the requirement more explicit and detailed.
Standard 1 proposal documentation, does not contain any explicit language on the importance of maximizing representativeness. Standard 1.2.2 sample/sampling details only says:
There is some relevant language in standards 4.3 non-probability sampling and 4.4 quota sampling.
Standard 4.3.2 sampling for non-probability samples
Standard 4.4 quota sampling
Quota sampling techniques are typically used for panel surveys and personal intercept studies to achieve sample representativeness. Quotas may also be used to control representativeness on other data collection methodologies.
Pew Research recently published 2 studiesFootnote 3 that examined factors affecting the extent to which results from a non-probability survey can be representativeFootnote 4. Some observations on their results:
The following is an excerpt from the summary section of the 2018 Pew Research Report, “For Weighting Online Opt-In Samples, What Matters Most?”
For Weighting Online Opt-In Samples, What Matters most? The right variables make a big difference for accuracy. Complex statistical methods, not so much.
A growing share of polling is conducted with online opt-in samples. This trend has raised some concern within the industry because, while low participation rates pose a challenge for all surveys, the online opt-in variety face additional hurdles. By definition they do not cover the more than 10% of Americans who don’t use the internet. The fact that potential respondents are self-selected means that there is still substantial risk that these samples will not resemble the larger population. To compensate for these challenges, researchers have employed a variety of statistical techniques, such as raking, propensity weighting and matching, to adjust samples so that they more closely match the population on a chosen set of dimensions. Researchers working with online opt-in samples must make a great many decisions when it comes to weighting. What factors should guide these decisions, and which ones are most consequential for data quality?
A new Pew Research Center study adds to the survey field’s broader efforts to shed light on these questions. The study was based on over 30,000 online opt-in panel interviews conducted in June and July of 2016, with 3 vendors, and focuses on national (as opposed to state or local level) estimates. We evaluated 3 different weighting techniques, raking, propensity weighting and matching, both on their own and in combination. Every method was applied using 2 sets of adjustment variables: basic demographics (age, sex, race and ethnicity, education, and geographic region), and a more extensive set that included both demographics and a set of variables associated with political attitudes and engagement (voter registration, political party affiliation, ideology and identification as an evangelical Christian). Each procedure was performed on simulated samples ranging in size from n=2,000 to n=8,000.
The procedures were primarily appraised according to how well they reduced bias on estimates from 24 benchmark questions drawn from high-quality federal surveys. They were also compared in terms of the variability of weighted estimates, accuracy among demographic subgroups, and their effect on a number of attitudinal measures of public opinion.
Among the key findings:
The weighting procedures tested in this report represent only a small fraction of the many possible approaches to weighting opt-in survey data. There are a host of different ways to implement matching and propensity weighting, as well as a variety of similar alternatives to raking (collectively known as calibration methods). We also did not evaluate methods such as multilevel regression and poststratification, which require a separate statistical model for every outcome variable. Add to this the innumerable combinations of variables that could be used in place of those examined here, and it is clear that there is no shortage of alternative protocols that might have produced different results.
But whatever method one might use, successfully correcting bias in opt-in samples requires having the right adjustment variables. What’s more, for at least many of the topics examined here, the “right” adjustment variables include more than the standard set of core demographics. While there can be real, if incremental, benefits from using more sophisticated methods in producing survey estimates, the fact that there was virtually no differentiation between the methods when only demographics were used implies that the use of such methods should not be taken as an indicator of survey accuracy in and of itself. A careful consideration of the factors that differentiate the sample from the population and their association with the survey topic is far more important.
The panel was asked to comment on the following proposed revisions to section 1, proposal documentation, and section 4, sampling.
Proposed revision to section 1.2.2: proposal documentation, sample/sampling details:
1.2.2. Sample/sampling details
- provide details related to target population:
- the definition of the target population in terms of its specific characteristics and geographic scope, including the assumed incidence of the population and any key subgroups and how the incidence was determined/obtained (e.g., supplied by the client)
- the total sample size and the sample sizes of any key subgroups
- describe the sampling procedures, including:
- the sample source
- the sample frame
- whether a sample survey or a census will be conducted and, if a sample, whether probability or non-probability sampling will be applied (see section 4
- explain respondent selection procedures
- indicate the number of re-contact attempts and explain re-contact attempt procedures
- define respondent eligibility/screening criteria, including any quota controls
- for non-probability samples, provide the rationale for choosing a non-probability sample
Proposed revision to section 4.3, non-probability sampling:
Note: this proposal includes eliminating section 4.4, quota sampling, and moving relevant content to section 4.3. non-probability sampling:
For reference, section 4.4, quota sampling, is:
Quota sampling techniques are typically used for panel surveys and personal intercept studies to achieve sample representativeness. Quotas may also be used to control representativeness on other data collection methodologies.
4.3. Non-probability sampling
4.3.1. Justification of use of non-probability samples
- 1. when a choice is made to use a non-probability sample, that choice must be justified, in both the research proposal and the research report. The justification must take into account the statistical limitations in reporting on data from a non-probability sample, and limitations in generalizing the results to the target population.
4.3.2. Sampling for non-probability samples
- as for probability sampling, the list or sample source must be stated, including its limitations in covering the universe for the target sample
- controls on sample composition to maximize representativeness, such as quota sampling
- weighting
- a full description must be provided of the regional, demographic or other classification variables used to maximize the representativeness of the sample and survey results. In selecting variables, also consider their likely correlation with key survey measures (adjustment variables that are uncorrelated with survey measures will do little to improve representativeness). Behavioural or attitudinal variables can also improve representativeness, providing relevant, high quality benchmarks exist for the target population.
- the precise quota control targets and screening criteria must also be stated including the source of such targets (e.g., census data or other
- deviations from target achievement must be shown in the report (i.e., actual versus target)
The following issues were discussed with respect to the proposed text: “if the survey results will be used to make statements about a population, steps must be taken to maximize the representativeness of the survey results with respect to the target population.”
“If the survey results will be used to make statements about a population:” the majority of panelists accepted the premise that survey results from a non-probability survey can possibly be used to make statements about a population, providing appropriate adjustments are made to the sample and/or data. However, a few panelists did not agree with this premise, because they did not think that adequate adjustment methods exist to allow making statements about a population based on a non-probability sample. Under this minority view, this premise should not appear in the standards. This would mean:
Change “maximize the representativeness” to “improve the extent to which the survey results are representative”: among the panelists (a majority) who accepted the premise that survey results from a non-probability survey can possibly be used to make statements about a population, this change was suggested because the steps that would be required to “maximize” representativeness may not be known with any certainty, and even if the steps are credibly known they may be of a complexity that puts the particular POR project over budget. “Improving representativeness” is more practical and realistic than “maximizing representativeness.”
“Must be taken” versus “may be taken”: the panel was split on whether taking steps to improve representativeness should be worded as a requirement (“must”) or a guideline (“may” or “should”).
The reasons for making this a guideline rather than a requirement were (a) for some POR surveys, it may be more practical to identify the ways in which the results are not representative and to interpret the results accordingly than to try to correct for these shortcomings, and (b) there may be uncertainty at the proposal stage about what steps could be taken that would improve representativeness.
The reason for making this a requirement was that given the premise, “if the survey results will be used to make statements about a population”, it is reasonable to require that steps be taken to improve representativeness.
“Option to explain not taking steps to improve representativeness”: regardless of whether taking steps to improve representativeness is worded as a requirement or a guideline, there must be an option to explain why for a particular project it may be decided not to commit to steps to improve representativeness. This decision might be made because at the proposal stage there is uncertainty about what, if any, steps can realistically be taken, or there may be other reasons.
Among the panelists who accepted the premise that survey results from a non-probability survey can possibly be used to make statements about a population, the following are 2 alternative wordings that reflect the split on whether taking steps to improve representativeness should be a requirement or a guideline:
If the survey results will be used to make statements about a population, steps must be taken to improve the extent to which the survey results are representative of the target population, subject to special justifications to the contrary. These steps must be documented in the research proposal and in the survey report. These steps could include:
If at the proposal stage no specific measures to improve representativeness are planned to be taken, the reasons for this must be documented in the research proposal. If subsequently no measures are taken to improve representativeness, the reasons for this must be documented in the survey report.
If the survey results will be used to make statements about a population, steps should be taken to improve the extent to which the survey results are representative of the target population, and any steps taken must be documented in the research proposal and in the survey report.
These steps could include:
If at the proposal stage no specific measures to improve representativeness are planned to be taken, the reasons for this must be documented in the research proposal. If subsequently no measures are taken to improve representativeness, the reasons for this must be documented in the survey report.
There were some comments on the proposed text in 4.3.2, “in selecting variables, also consider their likely correlation with key survey measures (adjustment variables that are uncorrelated with survey measures will do little to improve representativeness). Behavioural or attitudinal variables can also improve representativeness, providing relevant, high quality benchmarks exist for the target population.” One suggestion was to change “high quality benchmarks” to “population statistics” on the grounds that the latter is simpler and more direct about what is required. Some other comments did not involve changes to the proposed text, but rather are observations on difficulties that can arise:
One panelist suggested adding a requirement or guideline when a non-probability sample is being used to show that the proposed approach is based on a methodology that has produced externally validated estimates in the past. The panelist commented that there has been some progress on developing approaches to produce validated results in specific areas such as voting intention. Another panelist, however, argued that this would not be feasible as a requirement, because:
The current standards contain the following requirements for proposal documentation when an online sample provider is used:
1.2.4.
- 3. For access panels, a description of the following must be provided, at minimum (when multiple panels are used, the information must be provided for each):
- active panel size (provide the definition of “active”)
- panel recruitment
- panel monitoring
- panel maintenance
- panel refreshment
The objective is to update and revise this standard, including:
Note that there are other relevant sections in the standards, notably:
The intent is to not duplicate standards in these areas. Therefore, the focus here was specifically on additional proposal documentation requirements when an online sample provider is used who will be providing sample using an access panel or river sampling.
For both probability and non-probability surveys in the revised section 1.2.4 #3/#4, it says the following with respect to providing information in the proposal about panel size:
active panel size (provide the definition of “active”),
PORD asked that the panel comment on how to assess “large enough.”
There is no numeric rule of thumb that can be given in the standards to assess what constitutes “large enough.” There are a variety of project-specific factors that can affect how large the sample of survey invitations needs to be in order to deliver the desired number of completed questionnaires. Factors include incidence rate, response rate, fieldwork duration, exclusion rules involving past survey participation, and data loss resulting from data cleaning. Response rate is itself influenced by a variety of factors such as survey topic, sponsor, questionnaire length, and incentive. Ultimately, there is quite a bit of judgment on the part of the research supplier to estimate the required number of survey invitations for any particular project.
The panel was split on whether or not there should be a requirement to report the active panel size:
For purposes of discussion by the panel, the current section 1.2.4 #3 was substantially revised, including suggesting separate proposal information requirements for probability versus non-probability samples.
The proposed requirements for probability online panel samples were:
1.2.4. Description of data collection
- 3.
- active panel size (provide the definition of “active”),
- how the panel is constructed and maintained such that it is a probability sample
- how a probability sample of the survey's target population will be drawn from the panel
- whether or not respondents will be sourced from more than 1 panel. If more than 1 panel is being used, provide the following:
- the identity of the other panels
- whether or not each of the other panels is a probability-based panel. if the sample includes respondents from non-probability samples, then the sample as a whole is a non-probability sample
- the reasons for sourcing sample from more than 1 panel (e.g. to achieve the desired survey sample size; to get better coverage of the survey population)
- steps to be taken to ensure that people enrolled in more than 1 of the panels are not sent the survey more than once
- the incentive or reward respondents are offered for taking part in the survey. This should be reviewed to ensure it does not risk causing certain types of people to either take or avoid the survey, and thereby potentially bias the sample.
Given that there are now online panels that purport to be probability samples, the panel agreed that there needs to be proposal documentation requirements specific to such samples.
The panel agreed with the proposed text for the information that needs to be in a POR research proposal for a survey using a probability online panel sample.
In this regard, the single most important required item is “how the panel is constructed and maintained such that it is a probability sample.” The majority of panelists were either skeptical that it is actually possible to have probability online panel, or skeptical that it can be as good as a RDD probability sample. The skepticism is based on the characteristics of an opt-in research panel. The people who are willing to join and actively participate in an online opt-in panel over time may be different in some ways from those who are not willing to participate, even if panelists are initially recruited to the panel using probability sampling. Actively participating in a research panel typically means responding to a large number of surveys over time. This high degree of involvement in and exposure to surveys can potentially affect in some ways how panelists respond to survey questionnaires. The requirement to document “how the panel is constructed and maintained such that it is a probability sample” means any concerns are addressed in the POR proposal.
The objective is to update and revise Proposal Documentation Standard 1.2.4 #3 for non-probability samples obtained from an online sample provider. Considerations include:
For discussion purposes, the panel was provided with 3 different listings of what the required proposal documentation might be:
The following pages show the guidance from the 3 sources.
1.2.4. Description of data collection (alternative approaches)
Current 1.2.4
European Society for Opinion and Marketing Research Online Research (pp. 18-19)
- details of how the sample was recruited and a description of the sampling frame and how well the sample covers the target population it is meant to represent
- steps taken to validate sample sources
- the procedures used to “on-board” prospective participants to panels
- cleaning and updating procedures
- any monitoring of survey-taking performance or quality controls to minimize satisficing or fraud and the steps taken if such behaviour is identified (section 7.8 of the standards addresses satisficing. The question is whether this should also be referenced in section 1, proposal documentation).
- whether and how new sources are integrated
- any procedures in place to maximize sample consistency for tracking projects
A proposed revision to 1.2.4 #4 (the non-probability section)
- if a panel is being used, whether or not respondents will be sourced from more than 1 panel. If more than 1 panel is being used, provide the following:
- the identity of the other panels
- the reasons for sourcing sample from more than 1 panel (e.g. to achieve the desired survey sample size; to get better coverage of the survey population; to reduce the risk of bias)
- steps to be taken to ensure that people enrolled in more than 1 of the panels are not sent the survey more than once
- if a panel is being used, describe the recruitment sources used to populate the panel on an ongoing basis, and if possible information on the proportion of panelists accounted for by each source. The goal is to ensure the diversity of the survey population is represented, not only in terms of demographics but also on relevant behaviours and attitudes. In this regard, it can be better to have a large number of diverse sources, to reduce the risk of bias.
- if a router is used to select survey respondents, what measures are taken to guard against, or mitigate, any bias that might arise from using a router. A router is an online system that screens potential respondents and assigns each person to 1 of multiple surveys being fielded at the same time. Routers are important for efficiently fielding multiple surveys. However, there can be a risk that systematically directing people with certain characteristics to other surveys may bias the profile of people directed to your survey.
- if profile data from a panel is used to structure the sample or for key analyses, and these measures are not validated by remeasuring the characteristics in the questionnaire, provide information on the panel's validation processes to ensure the information is accurate and up to date. Panel profile data on panelists can be very helpful for structuring a sample. However, it is a good practice to verify this data by remeasuring the characteristics in the survey questionnaire.
- if the survey is a tracking study, the steps that will be taken to ensure comparability of the sample with previous waves of the tracking study
- the incentive or reward respondents are offered for taking part in the survey. This should be reviewed to ensure it does not risk causing certain types of people to either take or avoid the survey, and thereby potentially bias the sample.
7.8. Detecting and dealing with satisficing
- 1) The access panel provider, working with the client, shall implement procedures to identify and remove fraudulent and inattentive panel members or respondents, documenting these procedures and actions taken.
The current proposal documentation requirements for access panels in section 1.2.4 #3 of the online standards are not sufficient. The current standard is:
1.2.4. Description of data
The basic problem with this standard is that it is too vague, and it is not clear what specific information is supposed to be provided. It also does not given any indication of why the information might matter or how to use it. Another issue is that not all of the information may be available, either because the panel provider considers it to be proprietary, or because of limited time for proposal preparation.
There were differing views within the panel on what information should be required about a non-probability online sample obtained from an online sample provider.
One point of view was that only very basic information should be required in the proposal, such as whether the sample will be drawn from a panel or panels, and the identity of the panel(s). The rationale is that details about a panel are not actionable information for the client, and may not have any clear relationship to the quality of the survey sample. It is also time consuming to compile detailed information, and some of it may not be available. What is arguably more important for the client and for survey quality is what steps will be taken to improve the representativeness of the sample (and this is addressed elsewhere in the standards).
The majority of panelists supported having some information disclosure requirements for a non-probability online sample obtained from an online sample provider, although there were some different points of view on the specifics.
The following are possible elements for a revised section 1.2.4 #3 of the online standards, with commentary.
Introductory paragraph
1.2.4. Description of data collection
When the sample is a non-probability sample drawn from a panel or other online sources and obtained from an online sample provider, a description of the following must be provided, if available. Note that “panel” refers both to panels operated by an online sample provider and to lists available from online sample providers. The phrase “if available” is used because some information may not be available, either because the sample provider considers it to be proprietary, or it is not available at the time the proposal is prepared.
First bullet point
whether all or part of the sample will come from web-. These methods may involve less quality control in terms of validation of respondent identity. The nature of the websites from which respondents are sourced can potentially introduce bias. If these sources are likely to lead to potential bias in the sample composition, this should be noted and any steps taken to mitigate bias should be described. Second bullet point
and if possible the active panel size corresponding to the survey’s target population that is available to be surveyed
The panel size should be large enough to ensure a high likelihood the target sample size of usable completed questionnaires can be obtained, taking into account such factors as likely response rate, exclusion rules involving past survey participation, and sample reduction due to data cleaning.
The panel was split on whether or not this should be a requirement. The rationale for including versus not including it is the same as was discussed earlier in the section on information disclosure requirements for probability panels.
Third bullet point
if a panel is being used, the identity of the panel and whether or not respondents will be sourced from more than 1 panel. If it is not known at the time of proposal preparation if more than 1 panel will be used but the supplier deems this may be needed, this possibility should be noted in the proposal. If at the time of proposal preparation it has already been decided to use more than 1 panel, provide the following:
“If it is not known at the time of proposal preparation […]:” this qualifier is needed because it may not be known with certainty at the time of proposal preparation whether or not multiple panels will be needed. This can depend on discussions with the client after a contract has been awarded on such matters as qualified respondents, incidence, and sample sizes (both in total and for subgroups).
“The identity and sizes of the other panels:” as noted previously, the panel was split on whether information on panel size should be required.
“Steps to be taken to ensure that people enrolled in more than 1 of the panels are not sent the survey more than once:” 1 panelist suggested this requirement be removed because they did not think it is feasible: panels may not want to share information on their members with other panels.
Fourth bullet point
if a panel is being used, describe the recruitment sources used to populate the panel on an ongoing basis, and if possible information on the proportion of panelists accounted for by each source. The goal is to ensure the diversity of the survey population is represented. The panel was split on whether or not information on recruitment sources should be required. Some thought this is useful information, while others did not. The rationale for not having this requirement is that there is already guidance in the standards to outline any steps that will be taken to improve the representativeness of a non-probability sample. The latter is the key information for the research client, and given this, information on recruitment sources is arguably not actionable for the research client. Also, the panel provider may consider recruitment sources to be proprietary information.
Fifth bullet point
if profile data from a panel is used to structure the sample or for key analyses, and these measures are not validated by remeasuring the characteristics in the questionnaire, provide information on the panel’s validation processes to ensure the information is accurate and up to date. Panel profile data on panelists can be very helpful for structuring a sample. However, it is a good practice to verify this data by remeasuring the characteristics in the survey questionnaire. The panel was split on this requirement. The issue was whether or not usage of panel profile data should be allowed without remeasuring the characteristics in the survey questionnaire.
Some panelists said it should be a requirement that the survey questionnaire include questions to verify any panel profile data being used. This would essentially result in eliminating this bullet point, and adding a requirement to this effect to section 2, questionnaire design. This ensures that the data for these characteristics is accurate and up to date.
Some other panelists said that using good quality panel profile data can reduce respondent burden by reducing questionnaire length. The proposed text ensures confidence in the quality of the panel profile data being used in the survey.
Sixth bullet point
the incentive or reward respondents are offered for taking part in the survey. This should be reviewed to ensure it does not risk causing certain types of people to either take or avoid the survey, and thereby potentially bias the sample. The issue here is what can and should be done with respect to the statistical treatment of non-probability surveys.
The current standards are in section 4.3.3, statistical treatment of non-probability samples. Note that this section comes after standards pertaining to justification of the use of a non-probability sample (section 4.3.1), and requirements for information disclosure and improving representativeness (section 4.3.2).
4.3.3. Statistical treatment of non-probability samples
- There can be no statements made about margins of sampling error on population estimates when non-probability samples are used.
- The survey report must contain a statement on why no margin of sampling error is reported, based on the following template: “Respondents for this survey were selected from among those who have [volunteered to participate/registered to participate] in online surveys. The results of such surveys cannot be described as statistically projectable to the target population. [If weighting was done, state the following sentence on weighting:] The data have been weighted to reflect the demographic composition of (target population). Because the sample is based on those who initially self-selected for participation [in the panel], no estimates of sampling error can be calculated.” This statement must be prominently placed in descriptions of the methodology in the survey report, including the executive summary.
- In general, for non-probability surveys it is not appropriate to use statistical significance tests. However, tests of significance with non-probability samples are appropriate when the objective is to establish the extent of the relationship among variables. If tests of significance are used with non-probability samples, it must be clearly noted that conclusions from these tests cannot be generalized to any population. Any use of descriptive statistics must clearly indicate that they are not formally generalizable to any group other than the sample studied, and there cannot be any formal statistical inferences about how the descriptive statistics for the sample represent any larger population. In the case of non-probability surveys that employ an experimental design in which respondents are randomly assigned to different cells, it is appropriate to use statistical significance tests to compare results from different cells.
The 4.3.3 standards are consistent with, albeit more detailed than, the Marketing Research and Intelligence Association (MRIA) standards (note that these apply to research results generally, not just non-probability samples)Footnote 6:
Researchers must not present research results with greater confidence than the data warrants. Instead, as responsible professionals, members must point out the relevant limitations of the research. This includes but is not limited to the following guidelines:
For example, panels of repeat volunteers will not ordinarily qualify as sources of probability samples of the general population."
Until 2015, the American Association for Public Opinion Research (AAPOR) took a similar stance. However, in 2015 AAPOR revised its Code of Professional Ethics and Practices to allow for reporting of measures of precision from non-probability samples.
The change was apparently motivated in part by the 2013 Report of the AAPOR Task Force on Non-probability Sampling, which stated:
We believe that users of non-probability samples should be encouraged to report measures of the precision of their estimates, but suggest that, to avoid confusion, the set of terms be distinct from those currently used in probability sample surveys. The precision of estimates from non-probability samples is not the average deviation over all possible samples, but rather is a model-based measure of deviation from the population value. Ipsos, for example has proposed the credibility interval … for their estimates from an opt-in panel survey. As noted in section 6, the credibility interval is measure of uncertainty that is used with Bayesian methods, and Ipsos described their procedure as Bayesian. Other model-based approaches also produce estimates of precision such as standard errors that could be used and do not refer to the average over all possible samples (the accepted terminology for design-based inferences used in probability samples).
Although the research base does not exist to endorse this particular measure or to urge its adoption across the industry, we believe the industry needs constructive attempts to develop measures that fill the gap created by the unsuitability of the standard margin of error calculation with non-probability samples. Treating estimates as though they had no error at all is not a reasonable option. At this point, it falls to individual researchers to judge the usefulness of this particular measure. Such judgments are only possible when organizations using them fully disclose the full range of information specified in the AAPOR Code of Professional Ethics and Practice along with a detailed description of how the underlying model was specified, its assumptions validated, and the measure calculated.
The relevant section of AAPOR’s Code of Professional Ethics and Practice now reads as follows:
AAPOR has issued detailed guidance in AAPOR Guidance on Reporting Precision for Nonprobability Samples:
Note that the current section 4.3.3 can be interpreted as not permitting alternative methods of statistical inference such as Bayesian credible intervals. This is based on the sentence in 4.3.3 #3, “Any use of descriptive statistics must clearly indicate that they are not formally generalizable to any group other than the sample studies, and there cannot be any formal statistical inferences about how the descriptive statistics for the sample represent any larger population.”
Note that the MRIA guideline does not forbid using Bayesian credible intervals, as the language used refers only to margin of sampling error. Bayesian credible intervals are not a margin of sampling error. The MRIA guidelines are silent on the use of Bayesian credible intervals.
The objective is to update and clarify section 4.3.3 on use of statistical measures with non-probability samples.
The panel was asked to consider the following 2 alternative options with respect to possible revisions of section 4.3.3, statistical treatment of non-probability samples:
For purposes of discussion, the panel considered the following revision to section 4.3.3: statistical treatment of non-probability samples. The proposed text was written for option 2. Note, though, that some of the revisions would apply equally to option 1. The main point of difference is in 4.3.3 #3.
4.3.3. Statistical treatment of non-probability samples
- There can be no statements made about margins of sampling error on population estimates when non-probability samples are used.
- The survey report must contain a statement on why no margin of sampling error is reported, based on the following template: "Respondents for this survey were selected
- for some surveys (e.g., exploratory, internal research) estimating precision may not be important to the research goals. For other surveys precision measures may be relevant, but the researcher may not have the statistical resources to compute them. It is acceptable for researchers working with non-probability samples to decline to report an estimate of precision. In such cases, the report must note that survey estimators have variance, but there has been no attempt to quantify the size.
- if an alternative statistical measure of precision such as Bayesian credible intervals is used:
- optional (the following alternatives are accepted):
- Bayesian credible intervals
- resampling approaches
- Taylor series linearization
- the statistical measure of precision that will be used must be stated in the research proposal, together with a rationale and brief description
- the survey report must provide:
- a detailed description of how the underlying model was specified, its assumptions validated and the measure(s) calculated. Refer to the AAPOR document AAPOR Guidance on Reporting Precision for Nonprobability Samples for the information to provide, as well as an example of the type of statement to make in the report.
- one key assumption is that the survey results are unbiased. This assumption must be prominently noted, together with any limitations on this assumption (see 4.3.1).
- an explanation of how to understand the measure of precision
- if applicable, how tests of statistical significance of differences or relationships based on the alternative method are to be understood
Most panelists supported option 1 (i.e. expressly forbidding the use of alternative measures of precision such as Bayesian credible intervals). There were 2 basic concerns expressed with respect to allowing use of alternative measures:
One panelist supported option 2 (i.e. allowing optional use of alternative measures of precision, but with a significant condition) namely, that the non-probability survey methodology used for a study has been previously validated against observable results. This would mean there is confidence that the non-probability survey is generating reasonable population estimates, and in this context use of an alternative measure of precision is a reasonable option.
One panelist was strongly in favour of option 2, allowing optional use of alternative measures of precision:
The majority of panelists supported a compromise wording for a standard of the following sort: “the use of alternative measures of precision such as Bayesian credible intervals are not allowed for GC POR surveys unless approved by PORD and agreed to by the GC client.” This opens the door to optional use of alternative measures of precision, but explicitly emphasizes the need to ensure buy-in by PORD and the GC client on a case by case basis. It also allows a GC client to include a requirement in a survey’s statement of work for an alternative measure of precision such as Bayesian credible intervals if that is something they want to have. One panelist suggested adding a further condition that any usage of alternative measures of precision must be based on a request in the client’s Statement of Work. One panelist did not support the compromise wording because they did not support any usage of alternative measures of precision for GC POR surveys.
This compromise wording would be accompanied by requirements similar to those in the proposed text given to the panel for section 4.3.3 #3. In this regard, the panel suggested the following modifications to the proposed text:
With these modifications, the majority of panelists supported the following revised section 4.3.3 #3:
4.3.3. Statistical treatment of non-probability samples 1
- 3.
- if a measure of statistical precision is not used, the report must note that survey estimators have variance, but there has been no attempt to quantify the size
- if an alternative statistical measure of precision such as Bayesian credible intervals is used:
- the statistical measure of precision that will be used must be stated in the research proposal, together with a rationale and brief description
- the survey report must include the following:
- a detailed description of how the underlying model was specified, its assumptions validated and the measure(s) calculated. This must include any model assumption about the survey results being unbiased and any limitations on this assumption.
- an explanation of how to understand the measure of precision, and any limitations or cautions in interpreting the measure
- if applicable, how tests of statistical significance of differences or relationships based on the alternative method are to be understood
The text proposed to the panel for section 4.3.3 #1 and #2 was accepted as follows:
4.3.3. Statistical treatment of non-probability samples 2
- There can be no statements made about margins of sampling error on population estimates when non-probability samples are used.
- The survey report must contain a statement on why no margin of sampling error is reported, based on the following template: “Respondents for this survey were selected
PORD asked the panel to consider the points made in a short article by Robert Peterson with the provocative title, “It’s time for pollsters to report margins of error more honestly”Footnote 7. The article lists the following problems with current practices:
First, the reported margin of sampling error may imply that it pertains to an entire poll. In reality, it is only for a single question, even though research shows that many of us think a reported margin pertains to the entire poll.
Second, the single question used to calculate the margin is a fictitious one. It does not reflect the actual response to any question posed in the poll.
And finally, the formula for calculating the margin arbitrarily assumes that the answer to the fictitious question is 50% “yes” and 50% “no.” Only if, by chance, were the polled individuals to respond exactly that way would the reported margin be correct for that question.
The rationale offered for reporting margins of error this way is that a 50/50 split has the largest possible margin. If a poll reports the maximum sampling error possible, it is covering the worst-case scenario (for that [fictitious] question).
Except that the poll hasn’t necessarily done so. If a question has more than 2 possible answers (like, “don’t know” in addition to “yes” and “no”) it requires a different formula. For subgroups such as “males” or “65 and over,” the margins will be larger than that for the overall poll.
Most important, margins of error can be cumulative. Many answers to questions in a poll are related to one another. Collectively, their margins add up to more than that for a single question. In a simple poll with only 4 questions, and a 95% confidence rate for each one, the total margin of sampling error of the poll might be as large as 19%.
The article lists the following types of solutions to these problems:
There are more accurate ways for polls to represent their margins of error. One option is to present none at all. A recent Harvard CAPS-Harris Poll did just that.
Another option is to compute the individual sampling error margins for every question in a poll. Pollsters could then work out an average margin across all poll questions. Such estimates would still possess weaknesses, but they would still be superior to the single fictitious margin currently being reported.
Ultimately, pollsters should devise a multiquestion approach. Such an approach would simultaneously take into account the number, types, weights and relationships of questions, along with individual sampling error margins for actual answers and one for the overall poll.
Section 14.7 (online)/15.7 (telephone) addresses one of the points in the article, namely that margin of sampling error varies by sample size (i.e. the margin of sampling error will be different for subgroups versus the total sample):
14.7/15.7. Mandatory survey report requirements; results
- 2) for probability samples, state the level of precision, including the margin of error and confidence interval for the total sample and any key subgroups
For discussion purposes, the panel was provided with a proposed revision to 14.7.2/15.7.2 that adds a requirement to indicate how the margin of sampling error varies across different observed percentages.
Revised 14.7/15.7. Mandatory survey report requirements; results
- 2) for probability samples, state the level of precision, including the margin of sampling error and confidence interval for the total sample and any key subgroups, and for a selection of different percentage values spanning the range of percentages that appear in the report.
The panel was asked to comment on this proposed revision, and on any other aspects of the Peterson article that might impact the standards.
The panel was split on what the requirements should be for reporting level of precision for probability surveys.
Each of the following 2 options for 14.7/15.7 was supported by several panelists:
option 1: For probability samples, state the level of precision, including the margin of sampling error and confidence interval for an observed percentage of 50% for the total sample and any key subgroups. Results for subgroups will have a larger margin of sampling error because of their smaller sample sizes. Also, the margin of sampling error is highest for questions where 50% of the respondents gave 1 answer and the other 50% gave another answer. The margin of sampling error decreases as the observed percentage for a particular response approaches 0% or 100%.
Applying this standard means that numeric margins of sampling error would be reported for an observed percentage of 50% on the base of the total sample, and on the bases of key subgroups.
One panelist suggested the survey report should also provide a link to a website with a calculator for readers who want to calculate margins of sampling error for other observed percentages or sample sizes.
option 2: For probability samples, state the level of precision, including the margin of sampling error and confidence interval for the total sample and any key subgroups, and for a selection of different percentage values representative of the range of percentages that appear in the report. Results for subgroups will have a larger margin of sampling error because of their smaller sample sizes. Also, the margin of sampling error is highest for questions where 50% of the respondents gave 1 answer and the other 50% gave another answer. The margin of sampling error decreases as the observed percentage for a particular response approaches 0% or 100%.
Option 2 adds to option 1 a requirement for the report to include a table, or narrative equivalent, indicating the margin of sampling error for other observed percentages (i.e. not just for an observed percentage of 50%) and for a representative selection of sample sizes. For example, this could be done as a single table, with sample sizes or key subgroups on 1 dimension, and a selection of observed percentages on the other dimension.
The reason some panelists preferred option 1 over option 2 is that option 2 requires a bit more work. However, some panelists did not feel strongly about this, and could live with either option.
The following options were each supported by 1 panelist:
option 3: For probability samples, state the level of precision, including the margin of sampling error and confidence interval for the total sample for every question summarized in the survey report, and for any subgroup results reported for each question.
The rationale was that with modern data processing software this information is readily available. Some panelists, however, felt this amount of detail is excessive and not necessary.
option 4: For probability samples, state the level of precision, including the margin of sampling error and confidence interval for the total sample for an observed percentage of 50%. Results for subgroups will have a larger margin of sampling error because of their smaller sample sizes. Also, the margin of sampling error is highest for questions where 50% of the respondents gave 1 answer and the other 50% gave another answer. The margin of sampling error decreases as the observed percentage for a particular response approaches 0% or 100%.
The rationale for this option is that it is simple, and is often how margin of sampling error is currently reported for surveys.
PORD has received some industry feedback that part of what the current standards say about the statistical treatment of census survey results may not be correct.
The relevant language is in section 4.6 census surveys. The following is an abridged version of 4.6:
In a census survey, an attempt is made to collect data from every member of a population. For example, an organization might want to do a survey of all of its employees. In this case, the population is “all of the organization’s employees”, and this would qualify as a census survey if all employees are invited to participate in the survey.
The list whereby all members of the target population are to be contacted and invited to respond must be clearly described, including any of its limitations/exclusions in representing that target population. Whenever possible, an estimate of the percentage of the population that is excluded from the list must be provided and the potential impact of their exclusion on the research results considered.…
- The number of attempted re-contacts and procedure for attempted re-contact must be stated.
- Do not state a margin of sampling error, as this does not apply to census surveys because no sample is drawn. The survey report must contain a statement on why no margin of sampling error is reported, based on the following template: “Since the entire population of [target population] was invited to participate in this study there is no margin of sampling error to be estimated or reported. The potential impact of non-sampling error due to non-response is discussed in the results section of the report. [If weighting was done, state the following sentence on weighting:] The data have been weighted to reflect the composition of [the target population (if known) or the sampling frame (e.g., client-supplied list)] on the main known characteristics.”
- There is no need to use inferential statistical tests since the results (frequencies, percentages) reported in a census survey describe the entire target population.[statement #1]. However, it is acceptable to use statistical significance tests to measure differences between subgroups within the target population. [statement #2]. As with any surveys, be they sample or census, the impact on the results of non- sampling error due to non-response must be assessed to the extent possible, and appropriate caveats on the interpretation of the results must be clearly stated.
The feedback was that statement #2 is not consistent with statement #1: “using the same reasoning, a subgroup of the census would still be a census of the subgroup. Statistical significance tests would only be appropriate on random samples of the census survey.”
The panel was asked whether the 2 statements flagged above are inconsistent.
Most panelists said that statement 1 is fine, but statement 2 is not appropriate. The reason for the latter is that if a survey is a census or attempted census of a population, then it is also a census or attempted census of the subgroups within that population. Therefore inferential statistics tests are not needed for either the total sample or subgroups of the sample.
A few panelists noted that there is a school of thought that statistical significance tests using census survey data may be legitimate when trying to determine if differences among groups are meaningful, but this is not a universally accepted practice. It was suggested that when using census survey data, a more useful way to determine if a difference is meaningful is to focus on the effect size and consider whether the effect size is meaningful given the policy or issue involved.
In a census, an attempt is made to contact every individual in the population (subject to coverage error). However, rarely will an interview be completed with every attempted contact (that is, the response rate will usually be less than 100%).
The panel was asked to consider whether or not a survey ceases being a census/attempted census if response rate falls below a certain level.
A census with a response rate of less than 100% is still a census, albeit one better described as an attempted census. Margin of sampling error does not apply. However, other sources of survey error can still be present. In particular, there can be non-response bias affecting survey measures if those not responding to the survey are systematically different from responders in some meaningful way.
Section 2.1.3 of the online and telephone standards gives required questions that must be asked in all surveys of individuals, “unless a convincing argument is made that the research objectives are better served by excluding one or more of them” (there is also an exclusion for business-to-business research where the unit of analysis is the organization).
Section 2.1.3 states:
The wording used for each question must be that provided below, unless a convincing argument is made that particular research objectives require alternative wording. Even in these exceptional cases, the terms used and/or categories applied (e.g., for household income) to capture responses must be those provided below.
Section 2.1.3 states the following rationale for these required questions:
The data from the age, education, and language questions (along with the recording of geographic location and sex) allows comparison with Statistics Canada census dataFootnote * for the purpose of non-response analysis. The data, along with that from the employment status and income questions, also facilitate the comparison of results between Government of Canada public opinion research studies. (See section 8. for further detail on non-response bias.) * A panelist suggested revising this to “ […] allows comparison with official Statistics Canada population counts […]” The reason is that between censuses, Statistics Canada updates population demographic counts using multiple sources of data. So, the latest population demographic counts may not based solely on the previous census.
Comparability to Statistics Canada’s questions and response options is a critical requirement. This is needed for non-response analyses, and it is needed when the variables are used to weight survey data to match the population.
Because of differences between the online (self-completion) and telephone (interviewer-administered) modes, the questions/response options may be somewhat different for the 2 modes.
There is a general requirement (section 2.1.2) to inform respondents at the beginning of a survey of the confidentiality of their questionnaire responses, but the current standards do not state any specific wording for how to preface the block of demographic questions located near the end of the questionnaire.
For reference, the relevant parts of section 2.1.2 are:
2.1.2 The following are required elements of all Government of Canada online survey questionnaire introductions:
- f. inform respondents that their participation in the study is voluntary and completely confidential
- g. inform respondents that their responses remain anonymous. In the exceptional cases where the research objectives require that respondent identity be revealed, the informed consent of the respondent must be obtained.
The Privacy Commissioner has requested the addition of a statement on privacy before the demographics section at the end of a questionnaire, such as this:
“These last few questions are for statistical purposes and will be kept confidential. Your identity will always remain anonymous.”
The panel considered possible wordings for an introduction to the block of demographic questions at the end of a questionnaire. For discussion purposes, the panel started with the following proposed text:
2.1.3 The following statement is required for all Government of Canada telephone/online surveys prior to administering the demographic section of the questionnaire:
With regard to the suggested introductory phrase, “these last few questions are for statistical purposes only”, the panel preferred the following wording instead: “these last few questions will allow us to compare groups of respondents.” It was felt that “compare groups of respondents” would be more easily understandable by respondents than “statistical purposes.”
With regard to the follow-up introductory sentence, the panel was split across the following alternatives:
Note that in all options, the proposed phrase, “and will be combined with the answers from other respondents to this survey”, is not present. It was felt that this adds to survey length without providing any much added meaningful information to respondents.
Choice of “anonymous” and/or “confidential” depends on 2 factors:
How respondents would interpret these words in a survey context is not really known, and it is possible that there are individual differences in interpretation. It should also be kept in mind that in the context of a survey questionnaire, a respondent may not spend much time analyzing the meaning.
While respondents’ interpretations may not really be known, the GC standards should be based on clear intended definitions when choosing the appropriate language. The following is an example of how one might think about these terms:
The key point is that the decision about use of “confidential” or “anonymous” in the introduction to the block of demographic questions near the end of the survey will need to be based on clear intended definitions.
In the current required demographic questions for telephone and online surveys, there is a notable difference in the response option, “prefer not to answer”, and related forms of this response option (i.e. “don’t know”, “refused”):
There was agreement by the panel with the approach used on telephone surveys (that is, the interviewer does not read the “prefer not to answer” response option [or its related forms], but does record this if volunteered by the respondent).
There was disagreement about whether “prefer not to answer” should always be presented as an option in online surveys:
Reasons for this position included:
Reasons for this position included:
The panel concluded that “prefer not to answer” (and its related forms) should be removed as a required listed response option in online surveys, but rather retained as an optional response category. In online surveys where there could be a conditioning effect such that respondent’s may not realize they can choose not to answer, it would be a good practice to inform them of when they have this option.
Some of the required demographic questions have a relatively long list of response options. This includes the questions on age, education and household income.
The panel’s view is that in a telephone interview, the interviewer should not be required to read all the response options before accepting a response, but rather should be allowed to tell the respondent to stop at the category that applies to them. This can make the telephone version of the question wording a bit different from the wording in the online version, but the benefit is a reduction in survey duration for telephone surveys.
Using the household income question as an example, the telephone version of the question could be revised to something like the following: “Please stop me at the category that best describes your total household income, that is, the total income of all persons in your household combined, before taxes last year. (READ LIST)”
The suggestion is to revise section 2.1.3 to allow researchers the option of modifying the wording of a demographic question to allow the interviewer to instruct a respondent to stop at the category that applies to them. For example, the required wording of a question could be modified by prefacing it with an introduction of the form, “Please stop me at the category that best describes your [insert demographic dimension].”
The following is the current mandated question in section 2.1.3 for telephone and online surveys:
Telephone surveys
Gender: [Do not ask: record based on interviewer observation]
Online surveys
Gender: What is your gender?
In the currently mandated questions, gender information is collected very differently in telephone versus online surveys. The telephone version does not actually ask a question about gender, but rather relies on interviewer observation. The online version asks a question and so is based on respondent self-classification rather than interviewer classification. Also, unlike the online version, the telephone version logically does not have a “prefer not to answer” response option (the research firm might have an “indeterminate” response option, though it is not known to what extent firms actually give interviewers this type of option).
Considerations around phrasing of this question include:
Statistics Canada’s 2016 census used the following question (although as noted below this will change in the next census):
What is this person’s sex?
The census questionnaire in other major countries (e.g. U.S., United Kingdom, Australia) also asked a similar question and offered only the 2 answer options shown above on their most recent census data collection cycle:
MRS also notes the following key points:
MRS provided the following examples of questions and a checklist of what questions researchers should ask themselves before deciding what question to ask:
What is your sex?
What is your sex? OR What is your gender?
The MRS goes on to say the following about gender identity questions specifically:
Establishing best practice in developing and asking gender identity questions will need to build on the position and practice of the ONS [Office of National Statistics] and research carried out by the EHRC [Equality and Human Rights Commission]. This will allow the research community to design and implement a consistent and standard gender identity question that can be understood and answered by all people living in the United Kingdom (UK).
Uptal Dholakia in his article “How Should Market Researchers Ask About Gender in Surveys?” in an online blog for Psychology Today (September 2016) points out that:
Any well-designed market research survey is based on 2 core principles: the “principle of accuracy” and the “principle of inclusiveness”. A questionnaire should be designed to gather information accurately, using best practices of survey design that psychometricians have formulated over several decades. But this is not enough. A survey should also be inclusive. When a respondent has finished taking a survey, they should feel like the opinion they have provided will be valued just as much as every other survey taker.
The panel considered whether or not the current required gender questions should be revised, and what revisions might be appropriate.
The mandated question should ask for the respondent’s gender, not sex, and provide the response options of “male”, “female”, and “other”.
Asking for gender rather than sex respects the principle of inclusivity, and is consistent with Statistics Canada’s plan to ask for gender, with an “other” response option, in the next census.
Until the next census data are available, there will be a mismatch between this revised gender question on GC POR surveys and the existing census data based on a question about sex. However, according to a panelist, Statistics Canada has found relatively little difference in responses to the 2 types of question, and relatively low usage of the “other” response option. Therefore, the impacts on weighting are minor and can be managed.
Because of the addition of the “other” response option, the gender question must be asked explicitly in telephone surveys, rather than left to interviewer observation. Asking the question explicitly also respects the participant’s self-classification of their gender. The result is that the same question would be asked in both telephone and online surveys:
Telephone surveys and online surveys
Gender: What is your gender?
A few panelists who supported asking this question in both telephone and online surveys commented that in telephone surveys this question may lead to some complaints, jokes or refusals, and interviewers will need to be prepared for this. For example, a suggestion was to allow an interviewer to “soften” the question by preceding it with something like “just to confirm…” This type of “softening” preserves the required question wording and response options, and therefore would be consistent with the suggested required wording.
Another possible approach to “softening” this question in a telephone survey would be to not require reading the response options. That would mean the “other” category is selected by the interviewer whenever a respondent gives an answer that is neither female only nor male only. While this results in a different presentation of the question in telephone versus online surveys, it seems reasonable to expect that the results would be comparable (i.e. that this would not cause a mode effect).
It was noted that depending on the objectives of some surveys, the “other” response option might be expanded into an open-ended “other (please specify)” option, or expanded into a longer list of response options. This optional wording would still be consistent with the basic requirement to have an “other response” option.
One panelist observed that respondent reaction to the use of the English word gender may be different from reaction to the French word genre in “quel est votre genre?” The thought was that this may be perceived as more awkward or odd in French. Statistics Canada has indicated that they have moved to using the word “genre” in some recent surveys, albeit with (a) an explanation and (b) preceded by a question on the individual’s sex:
Assuming “genre” is used in the next census, then for comparability the same wording should be used in the Standard for the French language version of the gender question.
One panelist had a different opinion on what should be the mandated question, and suggested using a multi-step question that first establishes sex at birth, and then establishes gender. This would look something like the following:
The following are the 2 mandated questions for language for telephone and online surveys in section 2.1.3. The researcher can choose to use both questions or only 1 of the 2 questions, depending on the survey objectives:
Telephone surveys
Mother tongue
What is the language you first learned at home as a child and still understand? [READ LIST — ACCEPT ALL THAT APPLY]
Language spoken at home
What language do you speak most often at home? [READ LIST — ACCEPT ALL THAT APPLY]
Online surveys
Mother tongue
What is the language you first learned at home as a child and still understand? [ACCEPT ALL THAT APPLY]
Language spoken at home
What language do you speak most often at home? [ACCEPT ALL THAT APPLY]
The panel was asked to provide input on what question or questions to require in GC POR surveys for official language of respondents.
Considerations include:
The panel was asked to consider whether both of the existing required language questions should be retained, and to comment on wording and response options. Note that the discussion below does not deal with the issue of whether or not to include a “prefer not to answer” option, as this issue was discussed earlier.
The panel felt that Language used most often at home is the most useful form of the language question and should be the only required question. Adding “mother tongue” would be at the discretion of the researcher. The current wording of the “language used most often at home” question is appropriate.
The current required age question is:
Telephone surveys
In what year were you born? [Record year: XXXX]
[IF PREFERS NOT TO PROVIDE A PRECISE BIRTH YEAR, ASK:]
Would you be willing to tell me in which of the following age categories you belong?
Online surveys
In what year were you born? [YYYY]
[IF PREFERS NOT TO PROVIDE A PRECISE BIRTH YEAR, ASK:]
Would you be willing to indicate in which of the following age categories you belong?
A survey can use more detailed breaks as long as these can be collapsed into the categories above.
In the census, Statistics Canada determines age from birth date.
For reference, the 2016 Census Profile for Canada shows the following age distribution:
| Age range | % of population 18 years or more |
|---|---|
| 18 to 24 years old | 10.9% |
| 25 to 29 years old | 8.1% |
| 30 to 34 years old | 8.3% |
| 35 to 39 years old | 8.1% |
| 40 to 44 years old | 8.0% |
| 45 to 49 years old | 8.4% |
| 50 to 54 years old | 9.5% |
| 55 to 59 years old | 9.3% |
| 60 to 64 years old | 8.1% |
| 65 to 69 years old | 7.0% |
| 70 to 74 years old | 5.1% |
| 75 to 79 years old | 3.6% |
| 80 years or older | 5.4% |
| Age range | % of population 18 years or more |
|---|---|
| 18 to 24 years old | 10.9% |
| 25 to 34 years old | 16.4% |
| 35 to 44 years old | 16.2% |
| 45 to 54 years old | 17.9% |
| 55 to 64 years old | 17.5% |
| 65 years or older | 21.1% |
The panel was asked to consider whether there should be any changes to the required age question, including possible revisions to the break points or whether a smaller number of age categories should be used.
The mandated age question is structured appropriately in that it first asks for date of birth, and only presents age categories if the respondent declines to give their date of birth. Most respondents will answer the date of birth question.
With regard to the follow-up question addressed to respondents who decline to give date of birth, there were suggested changes to the categories:
The current required education question is:
Telephone Surveys
What is the highest level of formal education that you have completed? [READ LIST]
Online Surveys
What is the highest level of formal education that you have completed?
For reference, the 2016 Census Profile for Canada shows the following education categories:
| Education level | % of population |
|---|---|
| No certificate; diploma or degree | 18.3% |
| Secondary (high) school diploma or equivalency certificate | 26.5% |
| Postsecondary certificate; diploma or degree | 55.3% |
| Apprenticeship or trades certificate or diploma | 9.8% |
| College; CEGEP or other non-university certificate or diploma | 19.4% |
| University certificate or diploma below bachelor level | 2.8% |
| University certificate; diploma or degree at bachelor level or above | 23.3% |
| Bachelor's degree | 15.5% |
| University certificate or diploma above bachelor level | 1.6% |
| Degree in medicine; dentistry; veterinary medicine or optometry | 0.7% |
| Master's degree | 4.6% |
| Earned doctorate | 0.8% |
It has been pointed out that the first 2 categories in the current required education question do not line up well with the Quebec education system. PORD provided the following from Wikipedia:
Mandatory elementary education (école primaire) starts with grade 1, through to grade 6. Secondary school (école secondaire) has 5 grades, called secondary I-V (Sec I-V for short) or simply grades 7-11. Students are 12 to 16 years old (age of September 30), unless they repeat a grade. Upon completion of grade 11, students receive their high school diploma from the provincial government.
In Quebec, grade 8 is the start of high school (école secondaire), whereas outside Quebec grade 9 is the start of high school. The view is that the first 2 existing required categories (grade 8 or less, and some high school) are confusing in Quebec, and indeed the 2 categories overlap, since grade 8 is also “some high school” in Quebec.
A possible revised approach is to drop any attempt to distinguish subcategories of less than a high school diploma (such distinctions can always be added on an ad hoc basis for a particular survey). In this scheme, the first 2 response options could be:
The panel considered whether and how to modify the education question, including the proposed revision to combine the 2 “less than high school” response options into 1 category.
The panel supported combining the 2 “less than high school” response options into a single category, “less than a high school diploma or equivalent”. It was noted that these are usually combined when examining the relationship of education to other survey variables. A researcher would still have the option of breaking the “less than high school” category into subcategories should that be useful for a particular survey.
One panelist commented that it is not uncommon for young adults to start but not finish postsecondary education, or to start postsecondary and then take a break for a period of time, or to attend more than 1 postsecondary institution over a period of time before getting a degree. Depending on the point in time they are asked for education, they have more than a high school education but do not have a degree or certificate. For some purposes, it can be useful to include this as a response option. Note that this is still consistent with the mandated education question, in that this subcategory response option can be recoded into the mandated response options.
The current required household income question is:
Telephone Surveys
Which of the following categories best describes your total household income? That is, the total income of all persons in your household combined, before taxes [READ LIST]?
Online Surveys
Which of the following categories best describes your total household income? That is, the total income of all persons in your household combined, before taxes?
It is assumed a survey could optionally use more detailed breaks as long as these can be collapsed into the categories above.
Statistics Canada now sources income data from administrative records, although some Statistics Canada surveys do ask some form of income question.
Statistics Canada reports various income measures, one of which is “total income for private households before tax”, which appears to be equivalent to “total household income” in the questions above.
For reference, based on 2015 data, the Census Profile for Canada shows the following for total income for private households:
| Income of private households | % of population |
|---|---|
| Under $5,000 | 1.6% |
| $5,000 to $9,999 | 1.4% |
| $10,000 to $14,999 | 2.7% |
| $15,000 to $19,999 | 4.0% |
| $20,000 to $24,999 | 4.3% |
| $25,000 to $29,999 | 3.8% |
| $30,000 to $34,999 | 4.3% |
| $35,000 to $39,999 | 4.3% |
| $40,000 to $44,999 | 4.2% |
| $45,000 to $49,999 | 4.1% |
| $50,000 to $59,999 | 7.8% |
| $60,000 to $69,999 | 7.2% |
| $70,000 to $79,999 | 6.6% |
| $80,000 to $89,999 | 5.9% |
| $90,000 to $99,999 | 5.3% |
| $100,000 to $124,999 | 10.4% |
| $125,000 to $149,999 | 7.2% |
| $150,000 to $199,999 | 7.9% |
| $200,000 and over | 6.8% |
| Income of private households: current required breaks | % of population |
|---|---|
| Under $20,000 | 9.7% |
| $20,000 to just under $40,000 | 16.7% |
| $40,000 to just under $60,000 | 16.1% |
| $60,000 to just under $80,000 | 13.7% |
| $80,000 to just under $100,000 | 11.2% |
| $100,000 to just under $150,000 | 17.7% |
| $150,000 and above | 14.7% |
Some other statistics using 2015 data:
| Household size | After tax income | Before tax income |
|---|---|---|
| 1 person | $22,133 | $25,516 |
| 2 persons | $31,301 | $36,084 |
| 3 persons | $38,335 | $44,194 |
| 4 persons | $44,266 | $51,031 |
| 5 persons | $49,491 | $57,054 |
| 6 persons | $54,215 | $62,500 |
| 7 persons | $58,558 | $67,508 |
The panel considered whether there should be any revisions to the household income question or response options. Note that presentation of a “prefer not to answer” response option is discussed elsewhere in the report.
The panel recommended that the question wording be revised to specify a time frame. This reduces ambiguity and uncertainty in how to answer the question among respondents whose household income has changed recently. The specific suggestion was to use a “last year” time frame.
A few Panel members also suggested that a reminder to respondents that household income can come from a variety of sources could provide more accurate data. One panelist suggested a detailed list of different income sources be included and another Panel member suggested that the reference be more generic, i.e. total income from all sources for all people in the household.
The following is a possible question that incorporates both the time frame and the “all income sources” cue: “which of the following best describes your total household income before taxes last year from all sources for all household members?”
The panel agreed with retaining the existing response options.
PORD consulted with MRIA on the value of adding a question in telephone surveys about the type(s) of phone(s) a respondent has access to at home:
The information about phone usage may also be incorporated into the weighting scheme for telephone surveys. The literature review commissioned by PORD notes this possibility (see pp. 13-14 of the review). The review says that, “at the time of writing, there is no consensus on the best approach to weighting dual-frame survey samples.” It goes on to cite an example of an approach that weights by telephone status (cell phone only, landline only, dual phone users), and an approach that does not.
Classification of household telephone status can be quite complicated, depending on the approach taken to weighting, or to how respondents are selected for the interview within a householdFootnote 10. Some examples of complicating factors:
The panel focused on whether and how to ask questions that establish up to a three-way classification of a respondent’s household: landline only household, cell phone only household, and dual landline and cell phone household.
The panel agreed with adding a required question on household phone status, but with the understanding that whether and how it might be used for weighting or quota controls will be decided for each POR project.
The following factors played a role in the panel discussion of whether and how to determine household telephone status.
The literature review commissioned by PORD noted that in 2018 there is no consensus on whether or how to use household phone status in weighting survey data, and referenced an approach that uses this type of weighting and an approach that does not. One panelist noted that depending on the survey topic, household phone status may have no direct relationship to the survey variables of interest. For example, if respondent age (which is correlated with household phone status) has a more direct bearing on survey responses, then it is better to use age in weighting than an “indirect” correlate such as household phone status.
Another complicating factor is that the use of household phone status in weighting or quota control can be difficult because of a scarcity of high quality, up to date population information on household phone status in Canada. Up to date data are important because the relative usage of cell phones and landlines has been changing and continues to change.
The panel agreed with adding a required question on household phone status, but with the understanding that whether and how it might be used for weighting or quota controls will be decided for each POR project.
There are sample frames for cell phones and landlines, so there is information in the sample about whether a number being dialed is a cell phone or a landline.
This information is not, however, sufficient to classify the respondent’s household telephone status, as it only addresses the status of 1 particular phone number. There can be more than 1 phone number associated with a household (e.g. multiple cell phone numbers, and/or a combination of cell phone numbers and a landline number). Therefore, it is necessary to ask questions in the survey to determine the respondent’s household telephone status.
If it is assumed that the sample information on phone type is correct (but see below), then it is possible to ask just 1 question to determine household phone status: “one-question” approach based on using sample information.
If from the cell phone sample, then ask if there is a landline phone in the household or if from the landline sample, then ask about cell phones in the household.
The advantage of this use of sample information is that household telephone status can be determined with only 1 survey question, which is important for reducing survey duration. One panelist favoured this approach because of the importance of keeping telephone surveys as short as possible.
While sample information about type of phone is good, it is not perfect:
The majority of panelists opted not to use sample information about phone type in determining household phone status. In practical terms, this means that at least 2 questions need to be asked to determine household phone status.
In the panel discussion, there were references to usage of 2 different classification approaches:
Both require asking at least 2 questionsFootnote 11 (assuming sample information is not used). Questions that produce the three-way classification will also produce the two-way classification. However, questions designed to produce the two-way classification may not support the three-way classification. The panel opted for questions that produce the three-way classification, with the understanding that if a researcher chooses to use the telephone status question for weighting or quota control, it is their choice as to which classification scheme to use.
“At least 1 cell phone in the household” versus “number of cell phones in the household”
To produce the three-way classification with respect to cell phones, all that is necessary is to determine if there is at least 1 cell phone in the household. However, some panelists said that the “number of cell phones in the household” can have value. This approach answers the question of whether there is at least 1 cell phone in the household (zero cell phones versus 1 or more cell phones), but also determines the number of cell phones. Knowing the number of cell phones can allow a more accurate estimate of the probability that a household could be randomly selected from the cell phone sample frame. The panel agreed on using the “number of cell phones in the household” measure.
How to refer to a landline phone
Historically, “landline” has been used to refer to the phone(s) connected to a telephone line. However, with the increasing incidence of cell phone only households, particularly among younger people, there’s a question of whether one can assume everyone understands what “landline” refers to. Further, there is increasing use of internet-based phone service (Voice over internet protocol (VOIP) service), and it is unclear whether people would consider these to be “landline” phones. The panel recommends that the English questions on household phone status use “home phone” rather than “landline”:
“Landline” versus “home phone” in the standards: The preceding discussion pertained to the terminology used in the required question on household phone status. The panel was also asked whether standards that refer to “landline” should continue to do so, or switch to “home phone” (e.g. in the current telephone standards, see 1.2.4 #7, 4.1.1, and 15.2). The panel was split on this issue, with some preferring to retain “landline” as being technically more precise, and others preferring “home phone” as being consistent with the recommend use of “home phone” in the required question on household phone status. If “home phone” is adopted, the standards should include a definition of what the term means.
Question location
There needs to be flexibility in where the household phone status question, and all of the other required questions, are placed in the questionnaire. Depending on the needs of the survey, they may be placed in the final section of the questionnaire, or they may be placed earlier in the questionnaire if needed for quota control or filtering purposes. The panel’s understanding of section 2.1.3 of the standards is that this flexibility currently exists.
To summarize, the panel suggested the following additions to telephone surveys for type of phone(s) in the household:
In an online survey, it is likely that a sizable percentage of respondents will use, or attempt to use, a mobile device (smartphone or tablet) to complete the questionnaire.
A 2014 AAPOR report on mobile technologies concluded: recognize if you are conducting online surveys, you are conducting mobile surveys.
A non-ignorable and growing percentage of respondents are now accessing online surveys via their mobile browsers (with estimates ranging from 8 to 23% depending on the study), resulting in higher abandonment rates and potentially greater measurement error among these mobile respondents.Footnote 12
With the growth of smartphone ownership and use, the percentages of people completing online surveys on a mobile device has very likely increased since 2014.
In a summary of results from the 2016 General Social Survey (GSS), it was reported by Statistics Canada in The DailyFootnote 13 that three-quarters of Canadians 15+ own a smartphone, although there is substantial variation by age. Most 15-34 year-olds (94%) reported owning a smartphone, compared with 69% of those aged 55 to 64 and 18% of Canadians 75 years and older.
In the 2015 Sage Research report, Best Practices for Improving Cooperation for Online SurveysFootnote 14, the following are some of the conclusions drawn from a review of the research literature:
Some survey companies are implementing technologies to accommodate use of mobile devices in online surveys, for example:
The key point is that adapting questionnaire design for those using a mobile device can potentially improve the survey data in terms of coverage, response rate, reduction of non-response bias, and answer quality.
An online survey can take different approaches to the possibility mobile devices will be usedFootnote 15:
The current standards do not address the possibility and implications of an online survey being completed on a mobile device. The objective for the panel was to provide input to revising the standards to address these matters.
The matters considered by the panel pertaining to usage of mobile devices to complete online surveys were:
The panel considered possible revisions to the following 2 sections of proposal documentation in the online standards, neither of which currently refer to potential use of mobile devices:
The panel said the default expectation should be that an online POR survey sample will include respondents using either a computer or a mobile device.
Because of the belief that this is now an industry norm, a few questioned whether this needs to be stated at all in the standards. However, to be clear that this is an expectation, it is better to state it explicitly. One panelist also commented that there can be some surveys which need to be restricted to computers, i.e. larger screens. The example given was a business survey which has accounting figures in columns. So, there can be cases where a survey sample needs to be restricted to certain device types.
To make this expectation explicit, as well as to give a brief rationale for the expectation, the recommendation was to add the following to section 1.2.2 sample/sampling details of the online standards:
Given the default expectation that the sample for an online survey will include users of computers and mobile devices, the panel said the default expectation should be that surveys have a mobile-friendly version of the questionnaire.
The suggestion was to include the following sort of text in section 1.2.5 questionnaire design of the online standards:
There were 3 revisions/additions to section 2 questionnaire design of the standards considered with respect to questionnaire design in online surveys where mobile devices may be used:
The 2015 Sage Research report, Best Practices for Improving Cooperation for Online Surveys, reviewed the research literature and concluded:
There are 2 approaches to integrating mobile-friendly question designs into an online survey:
These results have practical implications for researchers conducting Web surveys. Our finding of near-comparability in the quality of responses between smartphone and PC users, even for sensitive and burdensome questions, suggest not only that smartphone users should be accommodated in Web surveys but also that the survey instrument should be optimized for small screens. There is a line of thinking, based on universal mode design or unimode design principles (Dillman 2000), that one should aim to display the exact same question formats across devices in order to minimize mode effects. Our results seem to suggest otherwise since we found no evidence that the redesigned (optimized) question formats in the smartphone version introduced device effects or had an adverse effect on data quality.
The notion of comparability that we would instead argue for is a best practices approach … This places the emphasis not on presenting the identical surface-level features, but on using the best practices of each mode or device in order to present the same stimulus to the respondent and minimize error within each mode or device. Carrying out these principles for mobile Web research requires playing to the strengths of smartphones by identifying and using input tools that are user friendly and avoiding any formats that are not (e.g., sliders, drop boxes that turn into pickers). This may also entail using Responsive Web Design (RWD) to adapt a questionnaire to the continuum of different screen sizes rather than binary optimization.
Notably, the same lead author, Antoun, was more cautious in making a recommendation on approach in another article that reviewed research literature from 2007 to 2016 on smartphone optimization:
Another issue for SO [smartphone optimized] surveys relates to the design and release of different layouts in response to the size of the respondent’s screen or web browser. The discussion so far has focused on adapting a questionnaire designed for PCs into a single optimized version for smartphones; but it is important to note that several studies used a responsive design where several versions of the questionnaires were displayed, with the implicit goal of improving response quality within each version (see, e.g., Amin, 2016). This practice has been adopted from web design where a large number of different visual designs and layouts for a single website are deployed (e.g., for small smartphones, large smartphones, small tablets, large tablets, small desktops, and so forth). What is unclear is whether this approach is also effective for surveys where standardization across layouts is a higher priority. A concern is that variations in a particular layout can affect responses (see, e.g., Smyth, Dillman, Christian, and Stern, 2006). Thus, responsive design calls attention to the need to promote comparability across versions, on the one hand, and to minimize error within each version, on the other.
Whether optimization is binary or responsive to a continuum of screen sizes, another issue is whether to design for the biggest or smallest devices first. Almost all of the reviewed studies started from the point of an existing survey that is designed for PCs and then adapted for smartphones. While this approach may prevent major usability problems in the smartphone survey, it does not necessarily produce an optimal design for smartphones (as the word “optimized” implies). Because the smallest screens appear to pose a greater design challenge, the “mobile first” approach may be desirable to the extent that it does not have negative effects on the version of the questionnaire displayed on larger browsers (see, e.g., de Bruijne and Wijnant, 2013a; Tharp, 2015). “Future research on the effect of responsive design, with and without a mobile-first design philosophy, is necessary before any firm conclusions on these different approaches are drawn” Footnote 19.
The bottom line is the last sentence: there doesn’t seem to be a definitive conclusion yet as to which approach is best.
The panel considered the following options with respect to the possibility of a standard encouraging a particular approach to question design/layout across devices:
Modern survey platforms have responsive design built in, so responsive design is essentially already the default option.
The panel’s view was that a standard is not appropriate. Reasons include:
The panel said a guideline could be helpful to highlight the options available to a researcher. A guideline to add to section 2 questionnaire design could be:
Design of an online survey should take into consideration the approach taken to the design and presentation of questions on different devices. Options include:
- responsive design: optimize question design/layout for mobile (small) and computer (large) screens, which can lead to different designs/layouts across devices, but gives the best data quality for each device
- mobile-first: which uses a mobile-friendly design for all devices, large and small, and minimizes the possibility of "mode" effects
- combination: within a survey questionnaire, use responsive design for questions unlikely to be at risk of a mode effect, and use mobile-first for questions where a mode effect is more likely
Because completion times tend to be longer on a mobile device than on a computer, a frequent recommendation is that a mobile-friendly questionnaire should be “short.” However, there is no consensus on what this means in terms of number of minutes.
The current standard for online questionnaire duration is 20 minutes, but an average duration of 15 minutes or less is “strongly encouraged”:
2.1. Standards
- Survey questionnaires must be designed:
- 3) to be completed in a maximum duration of 20 minutes, not including pauses or interruptions. Exceptions could include projects with specialized audiences and those with pre-arranged interviews when the respondent is aware the survey will take longer than 20 minutes. Average questionnaire durations of 15 minutes or less are strongly encouraged in order to minimize respondent burden.
The panel considered whether the standard for survey duration should be left as is, or revised to specify a shorter duration for mobile-friendly surveys.
The panel said the existing standard on survey duration does not need to be changed. The industry is well aware that survey duration needs to be kept as short as possible, and the panel agreed that strongly encouraging questionnaire durations of 15 minutes or less is sufficient.
It was also suggested that advances in mobile technology are reducing albeit not eliminating the differences in time to complete a survey. Mobile communication speeds have improved and screen sizes and resolutions have improved to the point where the relative difficulty of completing surveys on a mobile device is compensated for by the convenience of doing a survey when and where a respondent would like.
It is difficult to state “standards” (i.e. requirements) for what constitutes a mobile-friendly question design:
One option in the standards is not to give any standards or guidelines on mobile-friendly design. The idea is that in proposal documentation there will be a statement of intent to have a mobile-friendly survey. Beyond that, it will be up to the researcher to decide how to implement that for their survey. And, under this view, guidelines do not belong in a standards document.
The other option is to give guidelines (i.e. recommended, but not required, practices or principles). This is certainly possible to do based on the research literature. The idea is that it would be helpful to GC POR researchers to have this guidance stated in the standards document.
For discussion purposes, 2 approaches for guidance were put to the panel.
1. The Sage Research report, Best Practices for Improving Cooperation for Online Surveys, summarized the following guidelines based on a review of the literature (detailed citations are in the report):
To this list could be added:
2. Another option for guidelines is the 5 “design heuristics” proposed by Antoun et al (2017)Footnote 20 based on their review of the literature:
The panel considered whether or not there should be guidelines in the standards on what makes a questionnaire mobile-friendly.
The panel said there should not be any required features (i.e. standards) for mobile-friendly questionnaires. Reasons included:
One view was that the standards should not address this topic at all, but the majority felt that a list of examples, in the form of a guideline, could be a useful reminder to researchers.
The following would be a possibility to include in section 2 questionnaire design:
Section 2
The following are examples of practices that can make a questionnaire more mobile-friendly:
There are 2 types of pre-testing that can be done for an online survey questionnaire where some respondents may use a mobile device while others may use a computer:
The current standards do not specifically refer to requirements for online surveys where the survey is being completed on both computers and mobile devices.
In section 3 of the online standards, the 2 most relevant current standards are:
3. Pre-testing
3.1. Standards
- 1. Pre-testing of all components of a new or revised survey questionnaire that may influence data quality and respondent behaviour is required. This includes the online appearance and functionality of the questionnaire.
- 5. A minimum of 10 pre-test surveys are to be completed in each language in which the final survey will be fielded. An exception could be projects with small survey populations, such as a client-satisfaction survey of a small client base. In such cases the researcher must, in consultation with the client, take steps to ensure that the smaller number of pre-tests are sufficient to guarantee questionnaire quality. For example, a cognitive pre-test may be warranted.
The panel considered the following questions:
An argument for not adding anything is that standard 3.1.1 is sufficiently broad in scope that a requirement for pre-testing of the mobile-friendly aspects of the survey is implied even if not explicitly stated.
An argument in favour of adding standards is to make sure the pre-testing is done appropriately (with the added standards stating the minimum requirements).
For example, a revision could be to add a requirement for some internal pre-testing on different devices:
for mobile-friendly surveys, researchers must also do internal pre-testing on a sample of the types and sizes of devices that respondents might use
On smartphones, the internal pre-testing must look at how questions display in both portrait and landscape mode.
For example, a revision could be to require pre-testing on a sample of different devices:
for online surveys the pre-test must include respondents using at least some of the different device types and sizes that might be used to complete the survey
For example, in a survey that can be completed on either a mobile device or a computer, the sample must include a specified minimum number of both mobile device users and computer users.
Section 3.1.1 of the pre-testing standards should be revised for online surveys completed on both computers and mobile devices to include a requirement to pre-test on different devices.
Section 3.1.5, which requires a minimum of 10 pre-test interviews in each language, should be left as is.
The panel was split in terms of how 3.1.1 should be revised to include a reference to pre-testing on different device types, with more favouring option 1 below than option 2. Option 1 adds a requirement for internal pre-testing on different device types, but leaves the existing external pre-testing requirement as is (i.e. with no requirement to externally pre-test on different device types). Option 2 adds both internal and external pre-testing requirements for different device types.
Require internal pre-testing on different devices but not pre-testing with respondents using different devices (external pre-testing): some degree of internal pre-testing by the research company is a reasonable requirement, but routinely requiring external pre-testing can be impractical and is not usually necessary if the internal pre-testing results are satisfactory.
An issue with mandating an external pre-testing requirement is that the researcher does not know in advance the type of device a respondent will use. Trying to complete a specified number of pre-tests for different device types can substantially increase the total number of people who have to be contacted to complete the minimum quotas by device type.
There were different views on how to express the scope of an internal pre-testing requirement, including:
There was also a difference of opinion as to whether or not the internal pre-testing should require pre-testing in both portrait and landscape modes, although most did not feel that pre-testing in both modes should be required. People who use smartphones are familiar with changing the phone orientation should they feel a need.
Require both internal and external pre-testing on different devices.
The wording for the external pre-test could be:
For online surveys, the pre-test must include respondents using at least some of the different device types that might be used to complete the survey. For example, in a survey that can be completed on either a mobile device or a computer, the sample must include a specified minimum number of both mobile device users and computer users.
Note that it was not suggested to increase the current required minimum number of pre-tests (10 in each language), so the quotas by device types would be within the current required minimum.
In a survey that allows completion on both mobile devices and computers, and particularly one using a “responsive design” approach that can result in different question designs/layouts for different size screens, there is the potential for a “mode” effect. That is, the different designs/layouts for a given question could cause different response distributions. If this possibility is to be explored, then data on device type needs to be collected during the survey.
Also, there is the possibility of a device-type effect as a result of differences in the characteristics of people who use a mobile device to complete a survey versus people who use a computer. For example, a research vendor noted the following in an article advocating for “device agnostic” sampling for online surveys:
Specific pockets of the population gravitate toward mobile and we expect to see the level of systematic non-coverage bias to grow in non-mobile research designs. Some of our testing demonstrates between 20% and 25% of millennials (those born from 1981 to 2000) prefer to access surveys via mobile, which means our non-mobile surveys are missing the views of a substantial portion of this audience. Considering that this group is likely more tech savvy and connected with peers, we expect over time to see biased and inaccurate results when they are excluded from our sampling frameFootnote 21.
This illustrates the importance of a mobile-friendly survey for ensuring good coverage of the population. However, it also means that an exploration of possible effects of question design/layout would need to be done carefully to avoid confounding question design/layout with covariates such as age.
The conclusions from the research literature are:
The panel considered whether or not there should be any requirement to collect information on device type, and any requirement to conduct an analysis for mode effects by device type. With regard to the latter, the following was suggested to the panel for purposes of discussion:
14.6. Quality controls
- 4)
A majority of panelists supported having a standard requiring collection of device information, but did not support having any standard requiring an analysis of mode effects for each survey. This view was paired with a recommendation that “research on research” be done using the aggregated device data collected across surveys to determine what if any standard would be appropriate for analysis of potential mode effects. The view was that not enough is known about device type mode effects at this time to specify analytic requirements for individual surveys, and it might turn out that really there is not a problem that needs to be addressed in the standards. It was noted that if device information is collected, the researcher has the option to explore mode effects if that is judged to be useful for a particular survey.
Under this view, the following standard could be added to section 7.7 monitoring of online survey fieldwork:
7.7. Monitoring of online survey fieldwork
- 3)
Several panelists had a different take on possible requirements for collection of device type information and analysis of potential mode effects by device type:
Users of mobile devices may incur costs to participate in a research survey.
The current standards do not have any requirements as to how such costs should be handled.
The following is a standard/guideline in the ESOMAR/GRBN Guideline on Mobile Research:
3.1.3 Costs
Unlike most other research methods, data subjects may incur costs as a consequence of participating in mobile research that may include charges for data downloads, online access, text messaging, data plan overages, roaming charges, voicemail message retrieval and standard telephone charges. Researchers should design their research so that data subjects incur no costs without express approval. If this is not possible, researchers must be prepared to offer compensation. Such compensation may be cash, mobile money, airtime or other forms of value.
Note the ESOMAR/GRBN explain their use of “must” and “should” as follows:
Throughout this document the word “must” is used to identify mandatory requirements. We use the word “must” when describing a principle or practice that researchers are obliged to follow. The word “should” is used when describing implementation. This usage is meant to recognise that researchers may choose to implement a principle or practice in different ways depending on the design of their research.
The panel considered whether something like the ESOMAR/GRBN 3.1.3 above should be incorporated into the online and telephone standards.
The panel’s view was that there should not be a standard about covering respondent costs associated with using a mobile device, and several reasons were given.
Respondents always have a choice whether or not to participate in a GC POR online or telephone survey. There was some discussion of what to do in the case of a survey where respondents are required to use a mobile device. However, even in this case respondents can choose not to do the survey.
The current standards require certain information be given about the survey (e.g. length), so respondents are able to make an informed choice about whether or not to participate.
It is reasonable to assume that mobile device users know that there are costs to using a mobile device, albeit what the cost is for participating in a particular survey can vary considerably, and may not even be known, or could only be calculated in some arbitrary fashion (e.g. what would the cost be if a person has an unlimited data plan). Since everyone knows at least roughly what their mobile plan and costs are, it is reasonable to assume this is factored into their decision whether or not to participate in a survey.
Unless compensation is set at an arbitrary fixed amount for all mobile users, the logistics of determining the amount to compensate each respondent and documenting this for billing purposes would be very complex and difficult, if not impossible. There is a wide variety of data and voice plans on the market, at different prices. When a plan provides a certain number of minutes or gigabytes, calculating a cost for a survey completed within those parameters would be largely arbitrary, and very difficult to document. Overall, the cost of administration of compensation would be considerable.
In the absence of a standard, research firms would still have the option to address compensation should that prove useful for a particular survey. For example, if response rate is a concern for a particular survey, respondents could be offered an option to complete the survey at a different time or on a computer, or offered a fixed incentive. However, the need for these sorts of actions is best determined on a case by case basis, and in consultation with the GC client concerning cost recovery of such incentives.
An important issue in sampling for telephone surveys is the inclusion of cell phone users and landline users. This can affect coverage of the survey population, the sampling frame(s) used for the survey, and possibly weighting.
The following are some statistics taken from the Canadian Radio-television and Telecommunications Commission (CRTC)’s Communications Monitoring Report 2017:
The CRTC report does not report phone type by age, but back in 2010, the AAPOR Cell Phone Task Force concluded: “young adults in the U.S. aged 18 to 34 years, can no longer be reached successfully via the landline frame.”Footnote 22
These data indicate that a telephone probability sample of the general Canadian adult population must include a cell phone sample.
The PORD literature review for this project noted the following:
A useful guideline would be one that focusses on the outcome of the data collection. That is, the sample ratios chosen should ensure that the proportion of cellphone-only households in the final survey sample reflects that of the population of cellphone-only households at the time of the study.
The 2010 AAPOR Cell Phone Task Force noted a variety of other issues involving surveying cell phone users, and typically noted that there is no definitive “best” resolution. For example:
The following are the current standards in the telephone standards that refer to cell phones:
1. Proposal documentation
1.2. Technical specifications of the research
1.2.3. Response rate/participation rate and error rate
- 1) State the expected response rate/participation rate for the sample(s). State expected response rates/participation rates for each mode when more than 1 mode is proposed (i.e. separately for online and telephone components). Similarly, if a component of the research will recruit respondents via their cell phones, the expected response rate/participation rate for that component must be stated separately as well. The expected response rate/participation rate is an estimation based on various factors such as previous response rates/participation rates, trends etc. The proposal must include a brief discussion of the factors that might cause the actual response/participation rate to fall short of the stated target.
1.2.4. Description of data collection
- 7) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method. When it is proposed to recruit at least some respondents via their cell phones, provide a rationale for using this approach in addition to or in place of landline phones.
4. Sampling procedures 3
4.2. Probability sampling
- 3) A full description of the sample design and selection procedures must be stated including:
- c. at each sampling stage, the method of achieving a probability sample (e.g., random selection) shall be explained, and any subsets of the universe that have been excluded or underrepresented shall be stated (e.g., cell phone only households), although whenever possible, an estimate of the percentage of the universe that has been excluded or underrepresented must be provided;
Note that section 1.2.4 is repeated in section 15.5 data collection (telephone) as part of section 15, mandatory survey report requirements.
The panel was asked to consider whether there should be any revisions to standard 1.2.3 #1 in proposal documentation in the telephone standards, which states requirements to provide information about estimated response/participation rates. Currently it requires an overall estimated response/participation rate, and if relevant an estimated response rate for cell phones:
1. Proposal documentation
1.2. Technical specifications of the research
1.2.3. Response rate/participation rate and error rate
- 5) State the expected response rate/participation rate for the sample(s). State expected response rates/participation rates for each mode when more than 1 mode is proposed (i.e. separately for online and telephone components). Similarly, if a component of the research will recruit respondents via their cell phones, the expected response rate/participation rate for that component must be stated separately as well. The expected response rate/participation rate is an estimation based on various factors such as previous response rates/participation rates, trends etc. The proposal must include a brief discussion of the factors that might cause the actual response/participation rate to fall short of the stated target.
Some panelists said section 1.2.3 is fine as is, while others said for telephone surveys it should require the estimated response rate, if relevant, for both cell phones and landlines. An argument in favour of the latter is that telephone surveys typically include both a cell phone sample and a landline sample.
Response/participation rate estimates are useful at the proposal stage because they are an important driver of the planned level of fieldwork effort needed to achieve the desired sample size.
The standards appropriately do not require reporting the actual response/participation rates achieved for cell phones versus landlines. Calculation of the response/participation rates actually achieved for cell phones versus landlines is subject to error. This is because there is some uncertainty over how to classify phone numbers when an interview is not completed. The sample used for dialing will usually be divided into a cell phone sample and a landline sample, but these classifications are imperfect due to number porting and the existence of exchanges which contain a mix of cell phone and landline numbers.
This standard currently states:
1. Proposal documentation
1.2. Technical specifications of the research
1.2.4. Description of data collection
- 7) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method. When it is proposed to recruit at least some respondents via their cell phones, provide a rationale for using this approach in addition to or in place of landline phones.
This standard states that a rationale must be given when the sample includes interviews on cell phones. The language arguably overly downplays the importance of including cell phone users in the sample. For purposes of discussion, the following revision was suggested to the panel (leaving out the multi-mode component, which was not relevant to the discussion):
1.2.4 #7: The telephone survey should (must?) include interviewing by both cell phone and landline. The sample ratios of cell phone to landline must be stated, and should (must?) ensure that the proportion of cell phone only households in the final survey sample reflects that of the population of cell phone only households at the time of the study. If interviewing is to be done only by landline or only by cell phone, provide the rationale.
The suggested text for section 1.2.4 #7 does not contain a specific recommended numeric ratio for cell phone sample to landline sample, nor should it. There is not good, up-to-date information that would support stating a specific numeric ratio.
For the same reason, the panel deleted the phrase, “ensure that the proportion of cell phone only household in the final survey sample reflects that of the population of cell phone only household at the time of the study”. Again, there is not always good, up-to-date information to make this possible. Further, even if the ratio is known, it would be onerous and expensive to ensure the ratio for population segments or geographic zones more narrow or granular than at the national and provincial/territorial levels.
The panel was split on how to revise section 1.2.4 #7. There were 3 different positions expressed with respect to how to revise the section:
1.2.4 #7: Telephone surveys with the general population or most its subgroups must include interviewing by both cell phone and landline. The ratio of cell phone sample to landline sample must be stated, and the rationale for the ratio must also be provided. If interviewing is to be done only by landline or only by cell phone, provide the rationale. . Because of the large and growing importance of cell phone usage, the view is that including both cell phone and landlines in the sample should be the default requirement. The phrase “with the general population or most of its subgroups” is there both to make the point that interviewing by both cell phone and landline is widely applicable, but also to acknowledge there may be subgroups where cell phone only or landline only would be appropriate. The research firm should give a reason for its recommended sampling ratio, but recognizing that this will likely be based on imperfect population data.
1.2.4 #7: Telephone surveys with the general population or most of its subgroups must include interviewing by both cell phone and landline. The ratio of cell phone sample to landline sample must be stated. If interviewing is to be done only by landline or only be cell phone, provide the rationale. . The reasons for dropping the requirement for a rationale are (a) lack of good, up to date population data, and (b) even with population estimates, a different ratio could be used and be weighted back to population ratio. Note that the versions of section 1.2.4 #7 above use the phrase “sampling ratio” and refer to the “cell phone sample” and the “landline sample.” As a proposal documentation requirement, this language focusses on the ratio of cell phone sample that will be dialed versus landline sample that will be dialed. The ratio does not directly apply to the actual ratio of completed interviews for cell phones versus landlines, which could end up being somewhat different. A panelist suggested an alternative approach is to frame the ratio stated in section 1.2.4 #7 as being what will be achieved in the completed interviews. From this perspective, the language used in section 1.2.4. #7 should be something like the following: The ratio of interviews to be completed by cell phone versus by landline must be stated. The argument for this alternative is that the ratio of the completed interviews states what the client can expect in the completed unweighted data set, rather than the ratio used at the intermediate step of the sample of numbers to be dialed. The choice between these 2 alternatives is significant, as it affects the composition of the sample of numbers to be dialed and quota controls on completion by device type during fieldwork.
The current standard 4.2.3c in sampling procedures addresses disclosure of coverage issues in probability samples, and gives as an example a sample of cell phone only households. The question was whether or not also to include an example of landline-only households.
The panel recommends adding landline-only samples as another example. Given the growing number of cell phone only households, a landline-only sample could have substantial coverage error.
4. Sampling procedures 4
4.2. Probability sampling
- 3. A full description of the sample design and selection procedures must be stated including:
- c. at each sampling stage, the method of achieving a probability sample (e.g., random selection) shall be explained, and any subsets of the universe that have been excluded or underrepresented shall be stated (e.g., cell phone only
The telephone standards for call-backs are in section 7 data collection:
7.2. Call-backs
- There will be a minimum of 8 call-backs made before retiring a telephone number and substituting it with another number. The call backs must be made at varying days and times over a minimum seven-day period. An exception could be made when the field period is shorter as a result of the need to assess recall of particular events or activities.
- Every effort must be made to ensure that the respondent is called back if an appointment has been arranged and that the date and time of that appointment are respected.
- No attempt will be made to call back refusals.
PORD posed a question to the MRIA about the appropriate number of call-backs, as concern had been expressed about whether 8 call-backs is too many, and might be perceived as harassment.
The MRIA Polling Standards for the Canadian MarketplaceFootnote 24 state 8 call attempts as a maximum, as compared to the telephone standards section 7.2 Call-backs which states 8 call-backs as a minimum. Note that the MRIA and section 7.2 differ in 2 ways: (1) MRIA specifies a maximum whereas section 7.2 specifies a minimum, and (2) MRIA specifies call attempts whereas section 7.2 specifies call-backs. The term “call-back” implies there is an initial contact attempt followed by call-backs. Rephrasing the rule in section 7.2 in terms of call attempts, it is stating that there should be a minimum of 9 call attempts.
The MRIA Polling Standards also state a definition of call-backs (no definition is given in section 7.2 of the telephone standards):
MRIA posed the following question to some members: “The MRIA had adopted the policy of a maximum of 8 calls to each potential respondent. See Appendix "L" 8.4.2. If the number of calls allowed are reduced would this affect your research studies? What is your firm's frequency of call-backs? Over what period do you make the (eight) 8 calls?” The response relayed to PORD was:
Members are very sensitive to respondent fatigue and aim to regulate the frequency of calls to the same respondents.
The additional calls to respondents are usually made] after exhausting the list of potential respondents and calls are made to those whom the researcher was unable to contact in the initial call.
It is extremely important to recognize the difference between dispositions for call attempts. Not every call attempt should be considered a call-back. For example, 8 call backs, all yielding a busy signal, is very different than 8 call backs all resulting in an answer and call back request. At the same time, the number of call backs must be large enough to provide a reasonable expectation of equal probability of selection for all primary sample units. We have a maximum of 7 call backs, but do consider some call dispositions to be partial call-backs (e.g., a busy signal counts as 1/3 of a call-back). So theoretically we could call a phone number up to 21 times (21 busy signals). Usually phone numbers are resolved after 6 - 10 attempts.
Refusal conversion dialing must also be considered and whether these conversion attempts are considered within the call back limit.
The panel was also asked to comment on whether the call-back requirements should be the same for respondents using a cell phone. The MRIA Polling Standards do not specify different call-back requirements for respondents using a cell phone versus a landline. However, the MRIA Framework for Live Telephone StandardsFootnote 25 states that consideration should be given to making a smaller number of call attempts to respondents using a cell phone:
The questions considered were:
With regard to terminology, there is a difference between “call-backs” and “call attempts”: the implication is that “call attempts” equals 1 plus the number of “call-backs.” There is also some ambiguity in the meaning of “call-back”, in that it can be interpreted in a general way, or more narrowly when the person answering the phone has requested to be called back. The panel’s recommendation is that the updated standards refer to “call attempts” rather than “call backs” on the grounds that the meaning is more straightforward and it aligns with the terminology used in the MRIA Code of Conduct.
The panel’s view is that a minimum of 8 call-backs (9 call attempts) is excessive, and a smaller number should be specified.
The following were considerations in the discussion:
One panelist suggested the standards should specify both a minimum and a maximum (numeric values were not specified), while most suggested specifying only a minimum number of call-backs. A reason for the latter is that research firms do not have a default urge to do a large number of call-backs. Call-backs cost money and time, there are rapidly diminishing returns for response rate, and firms do not want to harass people. Research firms are motivated, for good reasons, to limit the number of call-backs. Further, there can be circumstances when a larger number of call-backs is warranted, such as a limited number of sample units, or a hard-to-reach target population. Specifying only a minimum number of call-backs gives research firms more flexibility to adjust the number of call-backs upwards if or as needed.
The majority of panelists concluded that the appropriate minimum number of call attempts is 6 (meaning 5 call-backs), while several opted for a minimum of 5 call attempts (meaning 4 call-backs).
There was some discussion about call-backs after an appointment has been made. This is in reference to section 7.2.3, “every effort must be made to ensure that the respondent is called back if an appointment has been arranged and that the date and time of the appointment are respected.” A few panelists suggested specifying that at least 2 call-backs be made to reach the respondent. Some other panelists said this level of specificity is not needed for this situation because the research firm would already likely be motivated to make “every effort” to interview someone who has expressed interest in participating.
The panel’s view was that it is not necessary to put a definition of call attempts in the standards. In this context, there was discussion of whether or not a busy signal should be treated as a call attempt, and there was disagreement on this point. There were 2 alternative points of view:
Because the standard specifies only a minimum number of call-backs, variability across research firms in treatment of busy signals is not an issue: some will treat it as a call attempt and perhaps end up closer to the minimum number of call attempts, while others will not treat it as a call attempt and perhaps end up doing more call attempts. Since busy signals are unobtrusive for respondents, the extra dialing would not risk harassment.
In an Interactive Voice Response (IVR) telephone survey, a computer is programmed with a questionnaire, calls are made automatically, and the recorded questions are read by the computer. There are no live interviewers.
The literature review commissioned by PORD described the following advantages and disadvantages of IVR surveys:
Section 5.3 use of Interactive Voice Response (IVR) in the telephone standards discourages, but does not forbid, use of IVR surveys for POR. It also suggests circumstances when IVR may be an appropriate methodology. The standard states that IVR surveys have the same requirements as interviewer surveys for the survey introduction, respondent opt-out, times when calls can be made, and delay in acknowledging an answered call.
5. Retaining public confidence
5.3. Use of Interactive Voice Response
- Characteristics of Interactive Voice Response (IVR) surveys, including the impersonal style conveyed by automation, put that method in conflict with the manner in which the Government of Canada wishes to engage Canadians. IVR can therefore be used only when a convincing case is made that the specific information to be collected is essential for making important decisions and cannot be obtained through other means. For example, IVR may be judged acceptable when the opinions of a hard-to-reach (low incidence) group are critical to the issue at hand and the very high call volume made economical by IVR is likely to markedly increase participation from members of that group in the survey. (IVR may also be used whenever respondents have agreed beforehand to this method.)
- When IVR is used, the same information required for interviews conducted by live interviewers (sponsor, researcher, participation is voluntary, assurance of confidentiality, etc.) must be included in the IVR survey introduction. Respondents must also be provided early in the introduction with an easy method to opt out of the survey (e.g., by pressing a specific key) so that the call is terminated gracefully and no more calls are made to that number. The same requirements for the time-of-day of calls and delay in acknowledging an answered call (see section 5.2.) apply to the use of IVR.
The panel considered whether there should be changes to section 5.3.1 use of Interactive Voice Response (IVR).
Within the panel, there was a range of views on the use of IVR for GC POR surveys, including:
The majority of panelists were somewhere in between these 2 views.
Specific considerations pertinent to revising section 5.3.1 included:
Based on the considerations above, the majority of panelists supported the following modification to section 5.3.1 (subject to further modification based on CRTC regulations):
5.3. Use of Interactive Voice Response
- Characteristics of Interactive Voice Response (IVR) surveys, including the impersonal style conveyed by automation, put that method in conflict with the manner in which the Government of Canada wishes to engage Canadians. IVR can therefore be used only when a convincing case is made that the specific information to be collected is essential for making important decisions and cannot be obtained through other means. For example:
- IVR may be judged acceptable when the opinions of a hard-to-reach (low incidence) group are critical to the issue at hand and the very high call volume made economical by IVR is likely to markedly increase participation from members of that group in the survey
- IVR may be judged acceptable when it is part of a mixed-mode design, for example when respondents identified through another survey mode are directed to an IVR, or a design in which there is switching between a live interviewer and IVR
- IVR may also be used whenever respondents have agreed beforehand to this method
- [See next section]
- There must be no or minimal (one second) pause before the
Section 5.3.2 states that the information disclosure requirements for IVR surveys are the same as for interviewer-administered surveys, and similarly requires that the information be provided in the survey introduction (it also addresses time of day and time-to-respond, which are addressed in the revised 5.3.1 and so are not shown here):
5. Retaining public confidence
5.3. Use of Interactive Voice Response
- 2) When IVR is used, the same information required for interviews conducted by live interviewers (sponsor, researcher, participation is voluntary, assurance of confidentiality, etc.) must be included in the IVR survey introduction. Respondents must also be provided early in the introduction with an easy method to opt out of the survey (e.g., by pressing a specific key) so that the call is terminated gracefully and no more calls are made to that number.
For reference, information disclosure requirements are described in section 2 questionnaire design:
2. Questionnaire design
2.1. Standards
- 2) the following are required elements of all Government of Canada telephone survey questionnaire introductions:
- follow the Official Languages Act and policies; in particular, introductions must include an active offer of both official languages (English and French); the language that is used first will depend on the province in which the respondent resides (e.g., in Quebec “Bonjour/Hello”; in the Rest of Canada “Hello/Bonjour”)
- identify the Government of Canada or the department/agency sponsoring the survey
- inform respondents of the general subject and purpose of the study. The subject and purpose of the study may be expressed in very general terms so long as these terms cannot be construed as an attempt to misinform respondents. When the researcher makes a convincing case that the subject and purpose of the survey is such that stating it at the outset will affect respondents’ willingness to participate, stating them may be deferred to the conclusion of the study.
- inform respondents of the expected length of the interview
- identify the researcher and interviewer (a pseudonym may be used for the interviewer as long as the individual remains identifiable by management internally for quality control purposes)
- inform respondents that their participation in the study is voluntary and completely confidential
- inform respondents that their responses remain anonymous. In the exceptional cases where the research objectives require that respondent identity be revealed, the informed consent of the respondent must be obtained
- inform respondents that the survey is registered with the Research Registration System maintained by the MRIA and provide information sufficient for respondents to access the System’s Research Verification Service (project registration number and service’s website URL, e-mail or phone number), if requested. Examples of exceptions: the sponsor would not be mentioned until the end of the survey in particular cases (e.g., advertising post-tests); other languages would be mentioned when the survey is available in non-official languages; and information about the availability of reports could be provided to specialized respondents when the incentive to participate outweighs the additional length of the introduction. Note: if requested by the respondent, interviewers need to be prepared to repeat the instructions. Interviewers must have information readily available to inform respondents about how respondents were chosen, how privacy is protected, and where the survey results can be obtained.
PORD has found that IVR surveys are less than 20 minutes, and typically shorter than interviewer-administered surveys. In this context, PORD asked that the panel comment on whether the required elements for telephone survey introductions should be revised or shortened for IVR surveys, and to comment on the possibility of moving some of the information disclosures to the end of the survey, as follows:
Hello/bonjour, this is (survey company) calling on behalf of the Government of Canada/government department. Pour continuez en francais appuyez sur le (number). We are conducting a (number) minute research survey on SUBJECT. Your participation is voluntary and completely confidential and anonymous. The name, telephone and web address of whom to contact for additional information about this research project are available at the end of the survey. To continue press (number). To end this call now hang up or press (number).
(At the end of the call) If you would like further information on this study please contact (name and phone number) or go to (website). To repeat this information press (number).
Further information could include:
Most panelists said the required information in the survey introduction should be the same for IVR surveys as for other surveys.
Many supported the principle that if information disclosure requirements are allowed to be different for IVR surveys (either in terms of what is disclosed or where the disclosure occurs) the same allowances should be available for other surveys. Shortening the survey introduction using either method would be done to help improve response rate, and presumably any type of survey could benefit, not just IVR surveys. A few suggested additional conditions:
In the context of support for retaining the existing information disclosure requirements, there was no consensus on where in the questionnaire the information should be made available. Views included:
The standard for survey duration for telephone states surveys must be completed in 20 minutes, and strongly encourages a duration of 15 minutes or less.
2. Questionnaire design
2.1. Standards
- 1) Survey questionnaires must be designed:
- c. to be completed in a maximum duration of 20 minutes. Exceptions could include projects with specialized audiences and those with pre-arranged interviews when the respondent is aware the survey will take longer than 20 minutes. Average questionnaire durations of 15 minutes or less are strongly encouraged in order to minimize respondent burden.
The literature review commissioned by PORD suggests that IVR surveys are better suited to shorter surveys:
One of the main drawbacks of using the IVR methodology is the need for a short, simply constructed questionnaire. This is one of the reasons this methodology is well-suited to election polling and measuring voter intention. 1 or 2 clear and concise questions can be asked when using IVR and these questions will have simple (and few) response options from which to select. For example, “press 1 if you know who you intend to vote for on Election Day and 2 if you do not”.
For a research project that intends to explore several topics with respondents and/or topics in a more in-depth manner (e.g., to uncover reasons for voter intentions or to gain insights on salient election issues with voters), an IVR survey would not be the appropriate methodology. (p. 18)
The panel considered whether the standard for survey duration should be modified for IVR surveys.
The panel agreed that IVR surveys should be kept short, and recommended adding a guideline to standard 2.1.1. in the telephone standards. It was noted that IVR surveys are typically 7 minutes or less in duration, and usually less than 5 minutes. A revised version of 2.1.1 in the telephone standards could be:
2. Questionnaire design
2.1. Standards
- 1) Survey questionnaires must be designed:
- c. to be completed in a maximum duration of 20 minutes. Exceptions could include projects with specialized audiences and those with pre-arranged interviews when the respondent is aware the survey will take longer than 20 minutes. Average questionnaire durations of 15 minutes or less are strongly encouraged in order to minimize respondent burden.
For the value of “X minutes”, the majority of panelists suggested 5 minutes as a guideline, while several suggested 7 minutes. Among the latter, the comment was that 5 minutes is a good target for the main questionnaire, but the required information in the survey introduction may necessitate a guideline of 7 minutes.
The call-back requirements in section 7.2 call-backs do not make any distinction between interviewer-administered surveys and IVR surveys.
7.2. Call-backs
- 1) There will be a minimum of 8 call-backs made before retiring a telephone number and substituting it with another number. The call backs must be made at varying days and times over a minimum seven-day period. An exception could be made when the field period is shorter as a result of the need to assess recall of particular events or activities.
In the literature review commissioned by PORD, under drawbacks of using IVR, it states:
In addition to limits on the length and complexity of the survey questionnaire, the quality of the sample can be questionable. This, however, is not unique to IVR. All survey research requires complete sampling frames (little to no coverage error), sound sampling strategies (simple random sampling or stratified random sampling), and appropriate sample control (an adequate number of call-backs that vary by time of day/day of the week to maximize the response rate). Since speed is one of the key advantages of using IVR, sample control measures are not as stringent (there is no time to call back a number in the sample multiple times). The survey sample will include whoever could be reached on the night of the data collection, which introduces the possibility of non-response bias. While survey weights will be applied to the survey sample post-data collection to ensure it reflects the demographic profile of target population, this will not address attitudinal differences that might exist between survey respondents and non-respondents. (p. 18)
The literature review implies that it may often be the case that few, if any, call-backs are made in an IVR survey. It connects this to using IVR for its speed advantage, and note that Standard 7.2.1 includes an exemption from the 8 call-back requirement when speed of fieldwork is important: “An exception could be made when the field period is shorter as a result of the need to assess recall of particular events or activities.”
The panel considered whether there should be any changes to section 7.2 call-backs specific to IVR surveys.
There were different views on whether and how to give call-back requirements specific to IVR surveys:
For reference, the current standards referring to multi-mode surveys are:
1. Proposal documentation
1.2.4. Description of data collection
- 7) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method.
4. Sampling procedures 5
4.5. Multi-Mode Surveys
Multi-mode surveys are ones where different methods of questionnaire administration are used. They will often involve a combination of online and telephone methods, although there are other possibilities (e.g., in-person, mail, fax).
When a survey is conducted using multiple modes of questionnaire administration:
- The reasons for using a multi-mode rather than a single-mode method must be stated, both in the research proposal and the survey report.
- When the plan is to combine data collected via different modes in the data analyses, then steps must be taken to ensure as much comparability as possible across the different survey modes in terms of question wording and presentation of response options.
- Steps must be taken to ensure avoidance of duplicate respondents in different modes. The steps taken, and the results, must be documented.
14 (online)/15 (telephone). Mandatory survey report requirements 1
14.5/15.5 Data collection
- 2) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method.
14.6/15.6 Quality controls
- 3) For multi-mode surveys, detailed description of any data quality issues arising from combining data collected via different modes/instruments.
The panel was asked to comment on possible revisions to the standards in the following areas:
In section 1, proposal documentation, the requirements specific to multi-mode surveys are:
1. Proposal documentation 1
1.2.4. Description of data collection
- 7) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method.
Other components of section 1 will ensure that the methodology employed for each mode will be described even though they don’t explicitly refer to multi-mode survey designs (e.g. section 1.2.2 sample/sampling details, and other elements in section 1.2.4).
The primary concern associated with multi-mode surveys is the potential for mode bias (that is, getting different response distributions for the same question area due specifically to characteristics of the mode). For example, it has been found that social desirability effects tend to be stronger in interviewer-administered surveys (e.g. telephone) than in self-completion surveys (e.g. online). Differences across mode in how questions and response options are presented could potentially cause different response distributions. Mode bias can vary by question (e.g. some questions may be more prone to social desirability effects than others, and some questions may be more similar in design across modes than other questions). These types of mode effects pose challenges for combining or comparing data across modes.
There can also be differences in response due to different types of people using the different modes, and indeed improving population coverage is a reason to consider doing multi-mode surveys in some circumstances. That said, this can also make it more challenging to detect mode bias.
The issue is whether and how the proposal documentation requirements need to be elaborated to make it more clear in the proposal that the issue of potential mode bias is recognized and that steps will be taken to address this. The existing requirement only indirectly refers to dealing with the potential for mode bias. Note that section 4, Sampling, is more explicit, but perhaps more can be done in section 1 of the standards.
The panel was asked to comment on the following potential revisions to proposal documentation standards:
1. Proposal documentation 2
1.2.4. Description of data collection
- 7) for multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method,
1.2.7. Data analysis
- 1) briefly describe how the data will be analyzed in order to address the objectives/research questions, including any special analyses (e.g., segmentation)
- a. for multi-mode surveys, state whether the plan is to report results based on combining the data across modes, or to report results separately by mode
There are 3 basic scenarios where respondents use different modes:
The standards use the term “multi-mode” to refer to scenario #1. However, some researchers describe this as “mixed-mode” and use “multi-mode” for scenario #2. The issue of mode bias arises with scenario #1. Given the ambiguity of terminology, the standards should include a definition of “multi-mode” to ensure everyone interprets it the same way.
The panel agreed with the proposed revision to 1.2.4, which adds a requirement to give a rationale for the modes that will be used, and to describe steps that will be taken to reduce the likelihood of mode biases and to facilitate detection of any mode biases:
1. Proposal documentation 3
1.2.4. Description of data collection
- 7) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method,
Most panelists agreed with the suggested addition of a requirement to 1.2.7 to state whether the plan is to combine data across modes or to report the results separately by mode:
1. Proposal documentation
1.2.7. Data analysis
- 1) Briefly describe how the data will be analyzed in order to address the objectives/research questions, including any special analyses (e.g., segmentation)
- a)
A few noted that plans can change once the data have been collected and examined for mode effects. The suggested wording above is acceptable as long as it is understood that plans can change. However, 1 panelist suggested the following revision to make this more explicit:
For multi-mode surveys, state that analysis will be conducted to determine the feasibility of combining the data across modes and that the conclusions of this analysis will be summarized in the report. The existing standard for multi-mode surveys in sampling procedures is:
4. Sampling procedures 6
4.5. Multi-mode surveys
Multi-mode surveys are ones where different methods of questionnaire administration are used. They will often involve a combination of online and telephone methods, although there are other possibilities (e.g., in-person, mail, fax).
When a survey is conducted using multiple modes of questionnaire administration:
- The reasons for using a multi-mode rather than a single-mode method must be stated, both in the research proposal and the survey report.
- When the plan is to combine data collected via different modes in the data analyses, then steps must be taken to ensure as much comparability as possible across the different survey modes in terms of question wording and presentation of response options.
- Steps must be taken to ensure avoidance of duplicate respondents in different modes. The steps taken, and the results, must be documented.
There is no current standard for questionnaire design specific to multi-mode surveys.
For purposes of panel discussion, changes were suggested in order (a) to distinguish between sampling-related topics and questionnaire-related topics, (b) to increase the prominence of the value of using similar modes of survey administration, (c) to clarify that one needs to be concerned about mode biases when comparing results by mode as well as when combining data across modes, and (d) to highlight the value that benchmark questions can have for enabling detection of mode biases.
The proposed revisions to section 4 sampling procedures were:
4. Sampling procedures 7
4.5. Multi-mode surveys
Multi-mode surveys are ones where different methods of questionnaire administration are used (e.g. some combination of telephone, online, in-person, or mail).
When a survey is conducted using multiple modes of questionnaire administration:
- The reasons for using a multi-mode rather than a single-mode method must be stated, both in the research proposal and the survey report.
- The risk of mode biases can be lower if the modes of administration are similar (i.e. both interviewer-administered [e.g. telephone and in-person] or both self-administered [e.g. online and mail]).
- Steps must be taken to ensure avoidance of duplicate respondents in different modes. The steps taken, and the results, must be documented.
The proposed revisions to section 2 questionnaire design were as follows -- this would be a separately numbered item in section 2, shown here as “x”:
2. Questionnaire design
2.1. Standards
- x)
When a survey is conducted using multiple modes of questionnaire administration:
- a) when the plan is to combine or compare data collected via different modes in the data analyses, then steps must be taken to ensure as much comparability as possible across the different survey modes in terms of question wording and presentation of response options
- b) include questions for which there are high quality data on the population and which therefore can be used as benchmarks for exploring the possibility of mode biases. The required demographic questions can serve this purpose (see section 1.2.4). Also try to identify any other high quality benchmarks available (whether demographic, behavioural or attitudinal) that might correlate with key measures in the survey. The latter can help better identify any mode biases in the key survey measures.
The panel agreed with the proposed revision to section 4.5, which emphasizes the value of using similar modes of administration:
4. Sampling procedures 8
4.5. Multi-mode surveys
Multi-mode surveys are ones where different methods of questionnaire administration are used (e.g. some combination of telephone, online, in-person, or mail).
When a survey is conducted using multiple modes of questionnaire administration:
- the reasons for using a multi-mode rather than a single-mode method must be stated, both in the research proposal and the survey report
- the risk of mode biases can be lower if the modes of administration are similar (i.e. both interviewer-administered [e.g. telephone and in-person] or both self-administered [e.g. online and mail])
- steps must be taken to ensure avoidance of duplicate respondents in different modes. The steps taken, and the results, must be documented
The panel agreed with the suggested addition to section 2.1, with 1 modification. The point was made that while use of behavioural and attitudinal benchmarks can be useful in detecting mode bias, this analysis needs to control for any differences in demographic variables. The standard was revised to include this cautionary note:
2. Questionnaire design
2.1. Standards
- x)
- a) when the plan is to combine or compare data collected via different modes in the data analyses, then steps must be taken to ensure as much comparability as possible across the different survey modes in terms of question wording and presentation of response options
- b) Include questions for which there are high quality data on the population and which therefore can be used as benchmarks for exploring the possibility of mode biases. The required demographic questions can serve this purpose (see Section 1.2.4). Also try to identify any other high quality benchmarks available (whether demographic, behavioural or attitudinal) that might correlate with key measures in the survey. After controlling for any differences by mode on key demographics, the latter can help better identify any mode biases in the key survey measures.
Two other important issues came up in the panel discussion, although these probably do not have direct implications for standards.
The “don’t know” response option (and related forms such as “prefer not to answer”) poses a challenge when 1 mode is interviewer-administered (e.g. telephone) and the other mode is self-completion (e.g. online). Typically, “don’t know” is not explicitly presented in an interviewer-administered survey. In an online self-completion mode, this response option can be presented, although it does not have to be. Explicitly presenting the “don’t know” option in 1 mode but not the other likely will affect the response distribution. Not presenting the “don’t know” option in the online mode is not exactly comparable to the telephone approach, because the respondent is either forced to choose 1 of the substantive response options, or they may think they have to choose 1 of those options (i.e. if they do not realize they can proceed without answering the question). These issues support the panel’s previous addition to 4.5, that “the risk of mode biases can be lower if the modes of administration are similar (i.e. both interviewer-administered [e.g. telephone and in-person] or both self-administered [e.g. online and mail]).”
If a mode effect exists, the standards require disclosure and discussion of this in the report, but there may not be any way to mitigate this effect in the results.
The current section 3 standard for pre-testing does not make any specific references to multi-mode surveys.
For reference, section 3 pre-testing (online version) is:
3. Pre-testing
3.1. Standards
- Pre-testing of all components of a new or revised survey questionnaire that may influence data quality and respondent behaviour is required. This includes the online appearance and functionality of the questionnaire.
- The client must be given the opportunity to test and approve the online survey prior to launch.
- Pre-testing must include probing that invites participants recruited for this purpose to provide input about their comprehension of and reaction to the questions. For example, a short series of questions could be included at the end of the pre-test survey. Researchers and clients must agree in advance as to whether probing will take place during or after administering the survey. If requested by the client a cognitive pre-test must be conducted.
- To help ensure questionnaire effectiveness with subgroups where there is reason for concern (e.g., due to language, age, level of education, etc.), the socio-demographic characteristics of the targeted participants must be approved by the client before recruiting begins.
- A minimum of 10 pre-test surveys are to be completed in each language in which the final survey will be fielded. An exception could be projects with small survey populations, such as a client-satisfaction survey of a small client base. In such cases the researcher must, in consultation with the client, take steps to ensure that the smaller number of pre-tests are sufficient to guarantee questionnaire quality. For example, a cognitive pre-test may be warranted.
- Pre-test completions shall not be included in the final dataset. An exception could be projects with:
- hard-to-reach target groups, or
- when no changes are made to the questionnaire.
- Documentation of the pre-test(s) must be provided to the client before the questionnaire is finalized. The documentation must include (at minimum):
- a description of the pre-test approach and number of interviews completed
- findings and any resulting modifications
- average survey completion time
- a statement of whether or not pre-test cases will be retained in the final data set
The final research report must include this same information.
The panel was asked to consider whether for a multi-mode survey there should be a requirement for a minimum number of pre-test interviews in English and French for each mode.
Based on the current 3.1 standards, a full pre-test in each mode would mean doing 10 English and 10 French surveys in each mode.
One panelist recommended a standard of doing a full pre-test in each mode, and another recommended doing a full pre-test unless the questionnaires are identical across modes (which may not commonly be the case).
Other panelists said it depends on how similar the different modes are to each other:
Currently in section 8 outcome rates there is no standard for how to calculate outcome rates for a multi-mode survey.
The panel was asked to consider adding a standard to section 8 outcome rates that addresses how to approach determining outcome rates for a multi-mode study. For discussion purposes, the following added standard to section 8 was suggested:
8. Outcome rates
- 1) The general principles for calculating the response rate for mixed-mode surveys are (a) calculate a response rate for each mode or stage of sampling, (b) use the same general response rate formula for each mode or stage of sampling, and (c) calculate a single aggregated response rate. The specific details of how this is done will be adapted to the particular multi-mode survey design.
There are 2 problems with the suggested addition to section 8 outcome rates:
The following addresses these issues:
8. Outcome rates
- 1) The general principles for calculating the mandatory outcome rates for mixed-mode surveys are (a) calculate the mandatory outcome rate for each mode or stage of sampling, and (b) if appropriate, calculate a single aggregated outcome rate. The specific details of how this is done will be adapted to the particular multi-mode design. An acceptable approach could involve presenting separate outcome measures for each mode or stage of sampling.
Also discussed were research designs where no outcome rate can be calculated, and therefore section 8 of the standards cannot be applied. Two such scenarios are:
In neither case is it clear how to calculate a meaningful outcome rate.
The use of sample routers was addressed in a 2016 AAPOR report, Evaluating Quality in Today’s Complex Environment, and it noted the following:
More recently, online sample providers have moved aggressively to the use of sample routers (software that screens potential respondents and assigns them to one of many waiting surveys). […] This can improve the efficiency of fieldwork, but there is little published research on the effect of routers on bias. One concern is the degree to which sample composition is affected by other surveys that are active in the router at the same time.
The panel recommended that research designs which do not allow calculation of either of the mandatory outcome rates should not be permitted for GC POR surveys. That is, for GC POR surveys, the sample of respondents to be contacted should be well defined, and each respondent should be invited to complete a single survey questionnaire.
The existing standards in the Mandatory Survey Report Requirements referring to multi-mode surveys are:
14 (online)/15 (telephone). Mandatory survey report requirements 2
14.5/15.5 Data collection
- 2) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method.
14.6/15.6 Quality controls
- 3) For multi-mode surveys, detailed description of any data quality issues arising from combining data collected via different modes/instruments.
Section 14.5/15.5.2 should be updated based on the panel discussion of section 1.2.4 in proposal documentation, where this same language was used. Based on the panel’s recommendation for Section 1.2.4, section 14.5/15.5 should be revised as follows:
14 (online)/15 (telephone). Mandatory survey report requirements 3
14.5/15.5 Data collection
- 2) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method,
The panel considered the following possible expansion of section 14.6/15.6 quality controls, with the intent (a) to ensure there is clarity as to decisions made about combining or not combining data across modes, and (b) to require descriptions of any adjustments made to the data to mitigate mode effects:
14.6/15.6 Quality controls
- 3) For multi-mode surveys, detailed description of any data quality issues arising from combining data collected via different modes/instruments.
- a)
- b) description of any steps taken to mitigate mode effects in the survey results
The panel agreed with the suggested revision to section 14.6/15/6, although there was a minor disagreement over the wording of part “b”. Some agreed with the wording proposed, but another view was to use the following wording: “Description of any adjustments made to the data to mitigate mode effects in the survey results. The argument in favour of “steps” was that there might be sequential operations taken on the data to mitigate mode effects (e.g. data cleaning, weighting).”
Section 6 data collection from children, young people or vulnerable respondents does not make any reference to whether or how incentives are used for this survey population.
Section 7.5 (telephone)/7.6 (online) does not refer to this population either:
7.5/6. Incentives/honoraria 1
- 1) The details of any incentives/honoraria to be used for any telephone/online survey must be provided in both the proposal and report documentation, including:
- the type of incentive/honoraria (e.g., monetary, non-monetary)
- the nature of the incentive (e.g., for monetary, prize, points, donations, direct payments)
- the estimated dollar value of the incentives to be disbursed
GC’s qualitative research standards contain 1 standard pertaining to incentives for this target population:
5. Participant recruiting
5.4. Incentives
- 7) Ethical considerations should be taken into account especially when providing incentives to youth, or vulnerable groups (e.g. payment is not coercive, or exposes young or vulnerable persons to a risk that they would otherwise have avoided).
Note that the standards define children and young people as follows:
People 16 and over do not require the consent of a parent or responsible adult (guardian, etc.) before being invited to participate in a survey.
The panel considered whether or not there should be guidance on incentives/honoraria for children, young people or vulnerable individuals who participate in a telephone or online survey. For discussion purposes, the following addition to the Incentives/Honoraria standard was suggested.
7.5/6. Incentives/honoraria 2
- The details of any incentives/honoraria to be used for an telephone/online survey must be provided in both the proposal and report documentation, including:
- the type of incentive/honoraria (e.g., monetary, non-monetary)
- the nature of the incentive (e.g., for monetary, prize, points, donations, direct payments)
- the estimated dollar value of the incentives to be disbursed
- decide in advance who will receive the incentive (the parent or responsible adult [guardian, etc.], the respondent, or if both will receive an incentive)
- the parent or responsible adult must agree to the incentive, regardless of who is receiving the incentive
- ethical considerations should be taken into account when providing incentives to children, youth, or vulnerable groups (e.g. payment is not coercive, or exposes young or vulnerable persons to a risk that they would otherwise have avoided)
There should be guidance in the telephone and online standards on incentives/honoraria for children, young people and vulnerable respondents.
The majority of panelists agreed with the suggested wording as is, but there were 2 suggested changes by a few panelists:
Online and mobile methodologies create possibilities for collecting various types of personal data “passively”, that is without direct interaction with respondents. The issue to be considered is, in the context of surveys, what passive data collection is allowed and under what circumstances is it allowed?
The ESOMAR/GRBN Guideline on Mobile ResearchFootnote 26 states:
Mobile applications are capable of collecting a broad range of personal data without direct interaction with data subjects. Examples include web use and browsing history, app usage statistics, loyalty card data, geolocation, social media data, data from wearables and internet of things (IoT) and other data generated by or obtained from mobile devices.
At least some of these capabilities can also apply to online surveys where the device used by the respondent is a computer.
The guideline notes an important exception to what constitutes “personal” data involving certain device characteristics:
While it is possible to passively detect the type of device a data subject is using, this is not personal data as long as the purpose is to optimize app performance and survey rendering.
The existing standards address relevant general principles of protection of respondent anonymity and confidentiality, and address passive data collection to some extent.
Section 5.1.4, protection of anonymity and confidentiality, states the general principle: “the anonymity of respondents must always be preserved unless they have given their informed and explicit consent to the contrary.”
5.1.4. Protection of anonymity and confidentiality
- 1) The anonymity of respondents must always be preserved unless they have given their informed and explicit consent to the contrary. If these respondents have given informed consent for data to be passed on in a form which allows them to be personally identified, the researcher must ensure that the information will be used for research purposes only, OR, if requested by the respondent, to resolve a customer complaint. The same holds true when respondents’ answers are collated, with their informed consent, with pre-existing data that allows such identification (e.g., with an administrative data base). Moreover, such personally identifiable information must not be used for any purpose unrelated to the current study, such as direct marketing, list-building, credit rating, fund-raising, or any marketing activities directed at those individual respondents.
Section 5.1.4 establishes that any personal data collected passively must be handled in a way that protects the anonymity and confidentiality of the survey respondents.
Section 5.3, privacy issues specific to online survey research, states a requirement to have an accessible policy statement “concerning the use of cookies, log files and, if applicable, software.” This ensures respondents are informed of certain types (but not all types) of passive data collection.
- Researchers must have a readily accessible policy statement concerning the use of cookies, log files and, if applicable, software. This statement may be either included in their privacy policy or it may appear in a separate document. Software must not be installed on respondents’ computers without their knowledge or consent. In addition, respondents must be able to remove the researcher’s software easily from their machines (e.g., for Windows users, the software must appear in the add/remove programs folder in their control panel).
- Any links to data protection, privacy policy or cookie policy statements must be given at the start of the questionnaire.
Section 7.2, data collection and recruitment techniques (online standards), addresses passive data collection with its reference to forbidding “surreptitious” or “unsolicited” data collection (terms which appear to refer to passive data collection generally). The principle is that passive data collection can only be done with the respondent’s awareness and presumably consent (although the standard doesn’t say “consent”):
7.2 Data collection and recruitment techniques
- Researchers must not make use of surreptitious, misleading or unsolicited data collection or recruitment techniques (including using spambots, spiders, sniffers or other ‘agents’ that collect personal information without the respondent’s explicit awareness).
It may be that the above standards are sufficient to address issues associated with passive data collection in surveys. Alternatively, it may be that the existing standards are not sufficiently explicit. To aid in Panel consideration of this topic, standard 7.2 was revised as follow:
Passive data collection refers to collecting personal information about respondents without direct interaction with them.
Researchers must not make use of data collection or recruitment techniques using passive data collection methods unless survey respondents or potential respondents have first given informed consent, or unless collecting the information is legally permissible or is permissible under the Terms of Use of the website, service or application from which the data are sourced. Examples of passive data collection of personal information include, but are not limited to, web use and browsing history, app usage statistics, geolocation, personally identifiable biometric data, social media data, data from wearables and IoT (internet of things), and other data generated by and obtained from respondents’ mobile devices or computers. Passive detection of the type of device a respondent is using is not personal data as long as the purpose is to optimize app performance and survey rendering.
Some things considered when formulating this possible revision to 7.2:
The panel considered whether the current standards adequately address passive data collection of personal information done in conjunction with a survey. For discussion purposes, the panel considered the proposed revision to 7.2 Data Collection and Recruitment Techniques.
The panel endorsed the proposed revision to 7.2 as an improvement over the current version, with 1 modification. The modification is to the last paragraph stating that device type is not considered to be personal information. In order to optimize survey rendering of an online survey, it is also useful to have information on browser characteristics. The last paragraph should be revised as follows:
One panelist suggested checking the proposed revision to 7.2 with the Office of the Privacy Commissioner of Canada to see if there are any other best practices that should be incorporated into the standard.
Although this topic pertained specifically to online surveys, there was some discussion of whether a standard on passive data collection needs to be added to the telephone standards. Overall, it is not clear that there are any legally permissible forms of passive data collection in telephone surveys, and so it appears that it is not necessary to add anything about passive data collection to the telephone standards.
Note that an audio recording of a telephone survey call can be considered a form of data collection. However, section 5.1.4 #4 of the telephone standards states that the respondent must be informed of the recording, so “passive” audio recording is not permitted:
5.1.4 Protection of anonymity and confidentiality
- 4) When recording or observation techniques are used, researchers must advise the respondent at the beginning of the interview of this fact. Respondents’ confidentiality must not be jeopardized by the use of such methods.
The online and telephone survey standards do not currently have any standards pertaining specifically to respondent photographs, videos or audio recordings.
The ESOMAR/GRBN Guideline on Mobile Research section 3.4.2 states the following about photographs and recordings:
Photographs, video and audio recordings are considered to be personal data and therefore must be gathered, processed and stored as such. They can only be shared with a client if the data subject gives his or her prior consent with knowledge of the specific purpose for which it will be used. When potentially identifying information has been removed (such as through pixelisation or voice modification technology) so that it is no longer considered personal data it can be shared with a client provided the client agrees to make no attempt to identify the individual.
Researchers must not instruct data subjects (or those that may be acting as data collectors) to engage in surveillance of individuals or public places. Data subjects should be given specific limited tasks (e.g. capturing interactions with friends with their consent, or images of objects or displays) that do not involve monitoring a particular area where personal data would be captured without the consent of the individuals present. When recorded observation of a location is undertaken, clear and legible signs indicating that the area is under observation along with the contact details for the researcher or research organization performing the research should be posted and images of individuals must be pixelated or deleted as soon as possible. Cameras should be situated so that they monitor only the areas intended for observation. (pp. 10-11)
The following is a slightly revised version of the ESOMAR/GRBN guideline that emphasizes the survey context (as the panel is only considering revisions to the telephone and online survey standards):
Revised to emphasize survey context: photographs, video and audio recordings from survey respondents are considered to be personal data and therefore must be gathered, processed and stored as such. They can only be shared with a client if the respondent gives his or her prior consent with knowledge of the specific purpose for which it will be used. When potentially identifying information has been removed (such as through pixelisation or voice modification technology) so that it is no longer considered personal data it can be shared with a client provided the client agrees to make no attempt to identify the individual.
Researchers must not instruct survey respondents to engage in surveillance of individuals or public places. Respondents should be given specific limited tasks (e.g. capturing interactions with friends with their consent, or images of objects or displays) that do not involve monitoring a particular area where personal data would be captured without the consent of the individuals present. When recorded observation of a location is undertaken, clear and legible signs indicating that the area is under observation along with the contact details for the researcher or research organization performing the research should be posted and images of individuals must be pixelated or deleted as soon as possible. Cameras should be situated so that they monitor only the areas intended for observation. (pp. 10-11)
The panel considered whether or not the modified ESOMAR/GRBN Guideline on Mobile Research section 3.4.2 should be added to the online and telephone survey standards.
The first paragraph of the modified ESOMAR/GRBN guidance should be incorporated into the updated standards. One panelist also noted that while at present this mainly applies to online surveys, it can apply to telephone surveys where the respondent is using a smart phone. For example, the respondent could be asked to take a picture of something and send it to the survey company (e.g. attached to a SMS message sent to the calling phone number). Therefore, it should be included in both the online and telephone standards.
With regard to the second paragraph of the ESOMAR/GRBN guidance, there was discussion of whether this is a sufficiently common scenario to justify inclusion in the standards. The first sentence forbids instructing respondents to engage in “surveillance of individuals or public places.” The issue some had with this is that asking for “surveillance” would be very unlikely in GC POR, and so it may not be worth addressing in the standards.
The second sentence in the second paragraph addresses a more realistic scenario of instructing a respondent to take an image or recording of something relevant to the survey topic. Particularly because of the prevalence of smartphones, this may increasingly be one of the tools used by survey researchers for certain types of survey topics (e.g. a visitor survey). A simplified version of the ESOMAR/GRBN text is worth including in the updated standards. The following is a possible wording:
Revised second paragraph: if respondents are asked to make recordings as part of their survey input , they should be given specific limited tasks (e.g. capturing interactions with friends with their consent, or images of objects or displays) that do not involve monitoring a particular area where personal data would be captured without the consent of the individuals present.
PORD requested the panel consider whether there needs to be an addition to the telephone standards related to sensitivity to the respondent’s setting. In this regard, there are 2 relevant guidelines in the ESOMAR/GRBN Guideline on Mobile Research (p. 7):
3.1.1 Safety
When calling mobile phones researchers may sometimes contact potential data subjects who are engaged in an activity or in a setting not normally encountered in fixed-line calling. This might include driving a vehicle, operating machinery or walking in a public space. The researcher should confirm whether the individual is in a situation where it is legal, safe and convenient to take the call. If the researcher does not receive confirmation, then the call should be terminated while allowing the possibility of making further attempts at another time.
3.1.2 Confidentiality and sensitive data
A researcher might contact a potential data subject who is engaged in an activity or situation where others may overhear the call. In this case, the researcher must consider the nature of the research content in light of the possibility that the data subject might be overheard and personal information or behaviour inadvertently disclosed or responses modified as a result of their situation. If appropriate, the call should be rescheduled to another time or location when confidentiality will not to be compromised.
Note that ESOMAR/GRBN 3.1.1 above is specific to mobile phones, but 3.1.2 could apply to either mobile or fixed-location phones.
The current telephone standards, in section 5.2.1 avoidance of harassment, has a standard focused on sensitivity of the survey subject matter, but it does not directly address issues caused by the setting of the interview:
5.2. Avoidance of harassment
- The researcher must take all reasonable steps to ensure that respondents are not in any way hindered or embarrassed by any interview, and that they are not in any way adversely affected as a result of it. Researchers must address sensitive subject matter in a way that will minimize the discomfort and apprehension of both respondents and interviewers.
Because respondents are increasingly likely to answer calls using a mobile phone, there can be issues with them using the phone in problematic settings (e.g. driving, walking in a public space). On both mobile phones and fixed-location phones, they may be in a setting where they can be overheard.
The panel considered whether the existing standards are sufficient, or whether there should be an additional standard or guideline for the interviewer to confirm the respondent is in a location where they are comfortable taking the call.
Most panelists supported adding a guideline to determine if a telephone survey respondent is in a location where they can take the call. One panelist suggested the following wording:
The reasons most recommended a guideline rather than a standard is that while this is a good practice, it can be problematic when a survey is at risk of being overly long. Further, it was suggested that often the benefit of asking the question may be small because:
One panelist said this should be a standard (i.e. required) for all telephone surveys. The reason is that especially for respondents using a cell phone, some may be tempted to do the survey even though they are in a setting such as driving where it is not entirely safe to do so. In order to avoid potentially contributing to unsafe behaviour, it was suggested a standard is needed. Further, it was suggested that cell phone samples are not guaranteed to be 100% cell phones, so this should be required for all telephone surveys, not just the portion of the survey directed to a cell phone sample.
A data breach is the loss of or unauthorized access to/disclosure of personal or organizational information.
The current standards require taking steps to protect against data breaches. The objective for the panel is to identify any revisions or additions to the standards, and/or any guidelines that should be included.
The current standards are:
13.2 (Online)/14.2 (Telephone). Protection of data/servers
- 1) Protection against illegal or unsanctioned access: Researchers must use up-to-date technologies to protect survey data collected or stored on Web sites or servers against illegal or unsanctioned access by third parties (i.e. “hacking”). The researcher must also control access to all databases on which any data relating to the survey is stored so that only individuals with the appropriate security clearance are able to access the database, either by using a password or other form of access restriction (such as biometric controls).
- 3) The researcher must not subcontract (including to an affiliate) any function that involves providing a subcontractor with access to any data relating to the survey unless the client first consents in writing.
- 5) Protection against physical damage to servers: Researchers must also put in place measures to ensure the “physical” security of data and servers.
13.3/14.3 Temporary storage of data on servers
- 1) If the temporary storage of data collected takes place on a server that is operated by another provider, the researcher must place the provider under the obligation to take the necessary steps to ensure that the requirements described in subsection 13.2 are met. Temporary storage of the collected data on the server must be terminated at the earliest possible time.
13.6/14.5 In the event of any data breach
- 1) In the event of any data breach, the client must be informed immediately and provided with details about both the nature and the extent of the data breach.
The literature review commissioned by PORD (see pp. 28-30) cites the following as a framework for considering data protection:
The panel considered whether any revisions are needed to the current standards pertaining to the data privacy and security standards cited above.
The existing standards pertaining to privacy and security of data, including data breaches, are appropriate.
There were 2 areas where there was some discussion of the need for additional standards or guidelines: encryption of data on portable devices, and data retention limits.
There should be guidance on encryption of survey data stored on portable devices (laptops, USB drives and other portable media). The panel distinguished between survey data files that contain personally identifiable information (PII) and those that do not. There should be a requirement to encrypt survey data on portable devices with PII, and a guideline to encrypt when there is not PII.
The current standards address the minimum time for which survey data must be retained (retention of technical data: 13.1 of online standards, and 14.1 of telephone standards). However, there is no standard requiring destruction of PII after a period of time.
Limits on data retention are addressed in some other contexts:
It was suggested that the standards should add rules about retention limits on PII, although the panel was divided on whether this should be a standard or a guideline, and on whether it should leave the time frame for retention general in nature (as is the case with MRIA and PIPEDA), or state a specific time frame. With regard to the latter, 1 panelist suggested a guideline of the following sort:
PII data for a project (either received from a client for purposes of data collection, or collected as part of the study) should be retained for no more than 1 year following the end of the contract unless there are legitimate reasons for retaining said data beyond 1 year.
The rationale for stating a specific time frame is that it is easier to explain should a respondent enquire about this. The specific time frame could be something other than a year, but 1 year arguably gives sufficient latitude for the client and supplier to deal with any survey-related queries or issues that might arise.
Some suggestions for additional guidelines included:
The current standards require that survey data be stored in Canada:
13.2/14.2 Protection of data/servers
- 2) Protection against legally-sanctioned access: because some jurisdictions allow their authorities, under certain circumstances, to access all data stored on servers located in that jurisdiction (e.g., in the United States under provisions of the Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism Act, known as USA PATRIOT Act: see the Treasury Board Secretariat ’s overview), the researcher must ensure that all the databases containing any information related to the survey are stored on servers and back- up servers located solely in Canada.
- If the client has first consented in writing, these servers may be located in another country where:
- equivalent protections are given to personal information as in Canada under legislation such as the Privacy Act, R.S. 1985, c. P-21, and the Personal Information Protection and Electronic Documents Act, S.C. 2000, c. 5, and under any applicable policies of the Government of Canada
- the laws do not allow the government of that country or any other entity or person to seek or obtain the right to view or copy any information relating to the survey without first obtaining the client’s written consent
- In connection with giving consent to locating a database in another country, the client may, at their option, require the researcher to provide a legal opinion (from a lawyer qualified in the foreign country) that the laws in that country meet the requirements of paragraph 2) a), above, or may require the researcher to pay for the Government of Canada to obtain such a legal opinion. The Government of Canada has the right to reject any request to store survey data in a country other than Canada if there is any reason to be concerned about the security, privacy, or integrity of the data. The Government of Canada may also require that any data sent or processed outside of Canada be encrypted with Government of Canada-approved cryptography and that the private key required to decrypt the data be kept in Canada in accordance with key management and storage processes approved by the Government of Canada.
- The researcher must ensure that all servers, including back-up servers, on which any data relating to the survey is stored are physically and logically independent (meaning there is no direct or indirect connection of any kind) from all other databases, unless those databases are located in Canada (or in another country approved by the client under paragraph 2) a)) and otherwise meet the requirements of this section.
- The researcher must ensure that all data relating to the survey is processed only in Canada or in another country approved by the client under paragraph 2) a).
- The researcher must ensure that all domestic network traffic (meaning traffic or transmissions initiated in one part of Canada to a destination or individual located in another part of Canada) is routed exclusively through Canada, unless the client has first consented in writing to an alternate route. The client will only consider requests to route domestic traffic through another country that meets the requirements of paragraph 2) a).
The panel considered whether any other standards are required with respect to cloud-based storage, either in terms of location of servers/back-up servers or any other aspects of data security specific to cloud-based storage.
For the most part there were no suggested changes to the current standards. This is a complex area: it requires expertise in the legal and regulatory framework affecting data access and use not only in Canada but in other countries as well where servers might be located, and it requires an understanding of GC policies in this area. The panel did not consider itself to be experts in these areas.
There was some discussion about allowing cloud storage in countries with equivalent standards. The current standards permit this providing certain conditions are met as detailed in 2a and 2b. However, it was suggested that applying 2b on a case by case basis is cumbersome and time consuming. A better approach would be for the GC to have a pre-approved list of countries that satisfy the conditions set out in 2a and are acceptable for cloud storage of GC POR data. It was noted that this is an approach taken by the European Commission with respect to protection of personal information in non-EU countriesFootnote 28.
Some other points to note:
A document relevant to this topic is MRIA’s Appendix C: Guideline on Social Media Research. Note however that the panel focus was limited to usage of social media only in connection with conducting online or telephone surveys, and that also meet PORD’s definition of public opinion researchFootnote 29. Under this definition (1) there must be attitudinal/opinion questions in the research, and (2) the research must be based on asking questions. The MRIA document deals with social media research more generally, not just survey research.
The Introduction to the MRIA’s Appendix C: Guideline on Social Media Research states:
The concept of consumers generating their own content on the internet has become ubiquitous. This has created new opportunities for researchers to observe, interact and gather information. Many techniques have been developed to leverage social media such as community panels, crowd-sourcing, co-creation, netnography, blog mining and web scraping. It is likely that many more will evolve over the coming years as the Internet continues to change.
Many of the research possibilities referred to above fall outside the scope of PORD’s online and telephone standards, because the activities do not qualify as public opinion research surveys. However, particularly in the case of market research online communities (MROCs), it is possible to do an online survey and perhaps even a telephone survey. And, it is possible that other types of social media venues can be platforms for sampling and for administering surveys.
If an online or telephone survey is done using a social media venue as the sample source, and perhaps additionally as the medium for administering an online survey, the research project would have to conform to all of the relevant standards, that is, the online survey standards or the telephone survey standards.
The question is whether anything needs to be added to the standards to cover online or telephone surveys that make use of social media (meaning, the sample is sourced from a social media, and the additional possibility that the survey is administered via the social media venue).
One perspective is that nothing really needs to be added to the standards for social media-based surveys. The standards lay out requirements for proposal documentation, questionnaire design, sampling, retaining public confidence, data collection, data security, etc. Adhering to these standards could be considered sufficient for an acceptable social media-based survey conducted for the Government of Canada.
A different perspective is that there may be some issues specific to social media-based surveys that are not clearly covered by the standards.
The MRIA’s Appendix C: Guideline on Social Media Research covers social media research broadly, not just surveys. The document describes “key principles” for researchers. These principles are consistent with both the online and telephone standards, but are explained specifically in terms of application to social media:
Also of particular interest for the survey standards is section 3 of the MRIA document, Recommendations for Specific Social Media, and within this, section 3.2 private social media area issues and section 3.3 market research social media area issues. These address aspects of permission, informed consent and privacy specific to these types of social media venues used for research.
The panel considered whether there are any additional standards required for surveys that use a social media venue as a sample source or to administer a survey.
No additional standards are needed for surveys that use a social media venue as a sample source or to administer a survey. The current standards, together with the various changes recommended elsewhere by the panel, are sufficient to ensure that any such surveys meet the quality requirements for GC POR surveys.
MRIA’s Appendix C: Guideline on Social Media Research on social media states the following principles (all of which are already addressed in the current standards).
If anything were to be added to the survey standards pertaining to social media, it would be more in the way of reminders to take care to follow the standards on matters where there is concern that use of social media may be problematic. For example, depending on the social media venue being used, there may be particular concern that one is dealing with “real people” (i.e. not bots) or that the people are who they say they are. In this case, one could add to section 4, sampling procedures, something like the following: if the research involves sourcing sample from a social media platform, the researcher should outline the additional measures they have taken to confirm respondent eligibility and authenticity.
The online and telephone standards do not contain any standards or guidelines pertaining to accessibility.
The Web Accessibility Initiative (WAI) home page provides definitions of accessibility, usability and inclusion:
According to PORD, the Treasury Board Secretariat (TBS) is working on a proposed policy for accessibility standards specific to all devices used to access online surveys. The results of this development work will probably be available in a year or so. When the TBS policy is finalized, it will take precedence.
Note that the panel made recommendations for standards and guidelines for mobile-friendly questionnaires, and these are to some extent relevant to accessibility of online surveys.
On pages 25-28 of the literature review commissioned by PORD there is a section on accessibility. In this section, there are 3 lists of guidelines from various sources. In the literature review, they are associated with making mobile surveys accessible, but many of the items pertain to online surveys generally. The 3 lists are as follows:
There are currently no accessibility standards in the telephone standards. There are ways to make telephone surveys more accessible, such as providing options for TTY or alternative modes such as online or mail, and allowing use of proxy respondents.
The panel considered whether a statement should be added to the standards about the importance of the principles of accessibility (including literacy considerations, usability and inclusion, and what if any specific guidelines might be provided for online and telephone surveys).
The majority of panelists supported stating a general guideline encouraging accessibility, followed by some examples of steps that could be taken to improve accessibility in online or telephone surveys. These examples would be guidelines, not requirements. The general guideline could be:
All Government of Canada online or telephone surveys should strive to be as accessible as possible for all respondents. Research suppliers are expected to work closely with the client team to accommodate the needs of a respondent for whom standard data collection practices do not allow for an equivalent or adequate participation in a research study for which they would otherwise qualify.
Examples could be helpful to clients and suppliers, and could include:
[After the general statement:] There are many aspects to survey accessibility. Some examples to consider include:
Online surveys
Telephone surveys
Several panelists had somewhat different recommendations:
The following contains background information and the questions for the group discussion. More questions may be developed for this discussion board as it progresses and based on the views and suggestions of panel members.
You will need to refer to this background for the detailed commentary on the questions shown on the discussion board. The discussion board displays only the questions. This background also provides, for example:
This type of detail has not been included in the discussion board questions.
Please answer all the relevant questions; it is important to know the point of view of each member of the advisory panel.
Wherever possible, we need to have as detailed answers as possible, particularly in areas where there is possible disagreement between Panel members on either a principle stated, or the language used to express the principle.
We need all members of the panel to begin posting answers to the questions as soon as possible and shortly after the discussion board is up and running. Otherwise, there will be little if any time left to debate issues, to try to get clarification or more detail from the panel, or to come to some agreement on issues where Panel members have different views.
It is the mandate of the advisory panel to reach consensus where possible even though it is not an essential outcome of deliberations. Where there is initial disagreement, please talk to one another to see if a consensus can be found. Think of lack of consensus as a last resort.
Just to remind you about the terminology being used:
Wherever possible for the sake of simplicity, the standards or guidelines should be the same for Government of Canada POR telephone and online surveys.
The Public Opinion Research Directorate (PORD) asks that the following criteria be applied when formulating standards and guidelines.
In general, we wish to adhere to the following criteria when revising/developing standards:
You will see various citations in the footnotes of this background. We have tried to make the background “self-sufficient” so that it is not necessary to read the cited articles. However, if you would like to get any of these documents, let us know which ones you would like and we’ll email them to you.
The following are other documents panel members will need to refer to, and so are being provided to members.
Section 4 of the standards is sampling procedures, and it includes standards for both probability sampling and non-probability sampling. Section 4 does not give definitions of these 2 types of sampling procedures. This section may be revised to include definitions and some examples to help users determine whether a sample is a probability sample or a non-probability sample. The objectives is to formulate appropriate definitions and examples to include in the standards.
Probability sampling is a method of sampling that allows inferences to be made about the population based on observations from a sample. In order to be able to make inferences, the sample should not be subject to selection bias. Probability sampling avoids this bias by randomly selecting units from the population (using a computer or table of random numbers). It is important to note that random does not mean arbitrary. In particular, the interviewers do not arbitrarily choose respondents since then sampling would be subject to their personal biases. Random means that selection is unbiased (it is based on chance). With probability sampling, it is never left up to the discretion of the interviewer to subjectively decide who should be sampled.
There are 2 main criteria for probability sampling: one is that the units be randomly selected, the second is that all units in the survey population have a non-zero inclusion probability in the sample and that these probabilities can be calculated. It is not necessary for all units to have the same inclusion probability, indeed, in most complex surveys, the inclusion probability varies from unit to unit. (p. 91)
Non-probability sampling is a method of selecting units from a population using a subjective (i.e., nonrandom) method. Since non-probability sampling does not require a complete survey frame, it is a fast, easy and inexpensive way of obtaining data. The problem with non-probability sampling is that it is unclear whether or not it is possible to generalize the results from the sample to the population. The reason for this is that the selection of units from the population for a non-probability sample can result in large biases.
Due to selection bias and (usually) the absence of a frame, an individual’s inclusion probability cannot be calculated for non-probability samples, so there is no way of producing reliable estimates or estimates of their sampling error. In order to make inferences about the population, it is necessary to assume that the sample is representative of the population. This usually requires assuming that the characteristics of the population follow some model or are evenly or randomly distributed over the population. This is often dangerous due to the difficulty of assessing whether or not these assumptions hold.
All researchers must:
- clearly state the target group (universe) definition for the research study; in the case of Internet surveys this includes explicit identification of whether or not non-Internet users are part of the target group definition
- clearly state the method(s) used to obtain a sample of this target group, including whether the method is a probability survey, a non-probability survey, or a census.
- probability sample: respondents are randomly selected from the survey’s target population, and each respondent’s probability of inclusion can be calculated. Probability sampling is a method for obtaining a sample projectable to the target population.
Some examples:
- random-digit-dial (RDD) telephone survey of Canadians
- random sampling from a list of all members of the target population
- random sampling from a panel that is itself a probability sample of the target population
- website intercept survey in which target population is visitors to the website, and visitors are randomly sampled to take part in a survey
- non-probability sample: a sample that does not meet the requirements of a probability sample (that is, respondents are not randomly selected from the survey’s target population, and/or each respondent’s probability of inclusion cannot be calculated). Additional steps must be taken to try to make results from a non-probability sample representative of the target population.
Some examples:
- a sample drawn from a research panel consisting of people who volunteer to join the panel and do surveys. Note that a sample collected using probabilistic methods from sampling frames that were compiled using non-probability methods is considered a non-probability sample.
- quota sampling, in which the selection of respondents is based on judgment, convenience or some other nonrandom process
- Census: an attempt is made to collect data from every member of the target population. Note that a census can be subject to other types of survey error, notably coverage error and nonresponse, so not every member of the target population may be in the final data set.
A1. Do you have any suggestions for changes to the proposed revision to section 4.1 #2?
Surveys based on non-probability sampling have become more common in marketing research, particularly because of the growth of online opt-in panels that provide significant cost savings over telephone probability samples. Public opinion research surveys for the Government of Canada have historically usually used probability sampling, but there may be more usage of non-probability surveys if there is confidence that these can deliver results that are representative of the target population being surveyed.
The objective is to revise the standards to emphasize the importance of striving for representativeness in non-probability surveys, and to explain in the proposal how this will be done.
The current standards address this objective to some extent, but the intent is to make the requirement more explicit and detailed.
Standard #1 proposal documentation, does not contain any explicit language on the importance of maximizing representativeness.
Standard 1.2.2 Sample/sampling details only says:
Standard 4.3.2 sampling for non-probability samples
Standard 4.4 quota sampling
Quota sampling techniques are typically used for panel surveys and personal intercept studies to achieve sample representativeness. Quotas may also be used to control representativeness on other data collection methodologies.
Pew Research has recently published 2 studies Footnote 31 that examined factors affecting the extent to which results from a non-probability survey can be representative Footnote 32. Some observations on their results:
The following is an excerpt from summary section of the 2018 Pew Research report, For Weighting Online Opt-In Samples, What Matters Most?
For Weighting Online Opt-In Samples, What Matters most? The right variables make a big difference for accuracy. Complex statistical methods, not so much.
A growing share of polling is conducted with online opt-in samples. This trend has raised some concern within the industry because, while low participation rates pose a challenge for all surveys, the online opt-in variety face additional hurdles. By definition they do not cover the more than 10% of Americans who don’t use the internet. The fact that potential respondents are self-selected means that there is still substantial risk that these samples will not resemble the larger population. To compensate for these challenges, researchers have employed a variety of statistical techniques, such as raking, propensity weighting and matching, to adjust samples so that they more closely match the population on a chosen set of dimensions. Researchers working with online opt-in samples must make a great many decisions when it comes to weighting. What factors should guide these decisions, and which ones are most consequential for data quality?
A new Pew Research Center study adds to the survey field’s broader efforts to shed light on these questions. The study was based on over 30,000 online opt-in panel interviews conducted in June and July of 2016, with 3 vendors, and focuses on national (as opposed to state or local level) estimates. We evaluated 3 different weighting techniques, raking, propensity weighting and matching, both on their own and in combination. Every method was applied using 2 sets of adjustment variables: basic demographics (age, sex, race and ethnicity, education, and geographic region), and a more extensive set that included both demographics and a set of variables associated with political attitudes and engagement (voter registration, political party affiliation, ideology and identification as an evangelical Christian). Each procedure was performed on simulated samples ranging in size from n=2,000 to n=8,000.
The procedures were primarily appraised according to how well they reduced bias on estimates from 24 benchmark questions drawn from high-quality federal surveys. They were also compared in terms of the variability of weighted estimates, accuracy among demographic subgroups, and their effect on a number of attitudinal measures of public opinion.
Among the key findings:
- even the most effective adjustment procedures were unable to remove most of the bias
- The study tested a variety of elaborate weighting adjustments to online opt-in surveys with sample sizes as large as 8,000 interviews. Across all of these scenarios, none of the evaluated procedures reduced the average estimated bias across 24 benchmarks below 6 percentage points (down from 8.4 points unweighted). This means that even the most effective adjustment strategy was only able to remove about 30% of the original bias.
- when it comes to accuracy, choosing the right variables for weighting is more important than choosing the right statistical method
- Adding a set of politically focused variables to the weighting adjustment reduced the average estimated bias by an additional 1.4 percentage points relative to adjusting only on basic demographics (e.g., age, education, race). While that might seem small, a difference of 1.4 points in the average implies that about 36 percentage points of bias were removed overall, but spread out across all 24 variables. Benchmarks most strongly associated with the political adjustment variables saw the largest improvements. In contrast, the use of more complex statistical methods never reduced the average estimated bias by than 0.3 points beyond what was achieved with raking, the most basic statistical method evaluated.
- the benefits of adding political variables to adjustment differ by survey topic Footnote 33
- Perhaps not surprisingly, benchmarks related to political engagement saw the largest improvement with the addition of political adjustment variables. Unweighted, these benchmarks had an average estimated bias of 22.3 percentage points, more than any other topic. While demographic weighting reduced the average bias by an average of 2.9 points, the effect of adding political adjustment variables was 4 times as large, reducing bias by 11.7 points and cutting the average estimated bias nearly in half (to 10.6 percentage points). Benchmarks pertaining to civic engagement and technology use also benefited disproportionately from political adjustment variables, though to a lesser degree. For benchmarks related to family composition and other personal characteristics, variable selection made little difference and proved mildly detrimental for questions of personal finance.
- the most basic weighting method (raking) performs nearly as well as more elaborate techniques based on matching
- When weighting on both demographic and political variables, methods based on matching resulted in the lowest average bias across the full set of 24 benchmarks (either in combination with raking at smaller sample sizes [n=less than 4,000] or on its own when the sample size was larger). Even so, procedures that only used raking (the least complex method evaluated) performed nearly as well, coming in 0.1 to 0.3 points behind the most effective method, depending on sample size. For benchmarks related to political engagement, the benefits from the more complex approach are somewhat larger than for other topics, doing between 0.5 and 1.2 points better than raking depending on sample size, but nowhere near the magnitude of improvement derived from weighting on political variables in addition to demographics. If the data necessary to perform matching are readily available and the process can be made routine, then a combination of matching and other methods like raking is likely worthwhile, providing incremental but real improvements. In other situations, such marginal improvements may not be worth the additional statistical labor.
- very large sample sizes do not fix the shortcomings of online opt-in samples
- While an online opt-in survey with 8,000 interviews may sound more impressive than one with 2,000, this study finds virtually no difference in accuracy. When adjusting on both demographic and political variables, the most effective procedure at n=8,000 was only 0.2 points better than the most effective procedure at n=2,000. While a large sample size may reduce the variability of estimates (i.e., the modeled margin of error), this is of little help from a “total survey error” perspective. For example, raking on demographic and political variables, the average modeled margin of error across all 24 benchmark variables is ±1.8 percentage points when n=2,000 and ±0.5 points when n=8,000, but the average bias holds steady at 6.3 points. As the sample size increases, estimates become less dispersed and more tightly clustered, but they are often more tightly clustered around the wrong (biased) value.
The weighting procedures tested in this report represent only a small fraction of the many possible approaches to weighting opt-in survey data. There are a host of different ways to implement matching and propensity weighting, as well as a variety of similar alternatives to raking (collectively known as calibration methods). We also did not evaluate methods such as multilevel regression and poststratification, which require a separate statistical model for every outcome variable. Add to this the innumerable combinations of variables that could be used in place of those examined here, and it is clear that there is no shortage of alternative protocols that might have produced different results.
But whatever method one might use, successfully correcting bias in opt-in samples requires having the right adjustment variables. What’s more, for at least many of the topics examined here, the “right” adjustment variables include more than the standard set of core demographics. While there can be real, if incremental, benefits from using more sophisticated methods in producing survey estimates, the fact that there was virtually no differentiation between the methods when only demographics were used implies that the use of such methods should not be taken as an indicator of survey accuracy in and of itself. A careful consideration of the factors that differentiate the sample from the population and their association with the survey topic is far more important.
Revisions are proposed for section 1, proposal documentation, and section 4, sampling. The bold text is new/revised material.
- Provide details related to target population:
- the definition of the target population in terms of its specific characteristics and geographic scope, including the assumed incidence of the population and any key sub-groups and how the incidence was determined/obtained (e.g., supplied by the client)
- the total sample size and the sample sizes of any key sub-groups
- Describe the sampling procedures, including:
- the sample source
- the sample frame
- whether a sample survey or a census will be conducted and, if a sample, whether probability or non-probability sampling will be applied (see section 4
- Explain respondent selection procedures
- Indicate the number of re-contact attempts and explain re-contact attempt procedures
- Define respondent eligibility/screening criteria, including any quota controls
- For non-probability samples, provide the rationale for choosing a non-probability sample
Note this proposal includes eliminating section 4.4, quota sampling, and moving relevant content to section 4.3. non-probability sampling:
For reference, section 4.4, quota sampling, is:
Quota sampling techniques are typically used for panel surveys and personal intercept studies to achieve sample representativeness. Quotas may also be used to control representativeness on other data collection methodologies.
4.3. Non-probability sampling
4.3.1. Justification of use of non-probability samples
1) When a choice is made to use a non-probability sample, that choice must be justified, in both the research proposal and the research report. The justification must take into account the statistical limitations in reporting on data from a non-probability sample, and limitations in generalizing the results to the target population.
4.3.2. Sampling for non-probability samples
- As for probability sampling, the list or sample source must be stated, including its limitations in covering the universe for the target sample.
- controls on sample composition to maximize representativeness, such as quota sampling
- weighting
- A full description must be provided of the regional, demographic or other classification variables used to maximize the representativeness of the sample and survey results. In selecting variables, also consider their likely correlation with key survey measures (adjustment variables that are uncorrelated with survey measures will do little to improve representativeness). Behavioural or attitudinal variables can also improve representativeness, providing relevant, high quality benchmarks exist for the target population.
- The precise quota control targets and screening criteria must also be stated including the source of such targets (e.g., census data or other
- Deviations from target achievement must be shown in the report (i.e., actual versus target).
The issue here is what can and should be done with respect to the statistical treatment of non-probability surveys.
The current standards are in section 4.3.3, statistical treatment of non-probability samples. Note that this section comes after standards pertaining to justification of the use of a non-probability sample (4.3.1), and requirements for information disclosure and maximizing representativeness (section 4.3.2, which was the subject of the previous question to the panel).
- There can be no statements made about margins of sampling error on population estimates when non-probability samples are used.
- The survey report must contain a statement on why no margin of sampling error is reported, based on the following template: “Respondents for this survey were selected from among those who have [volunteered to participate/registered to participate] in online surveys. The results of such surveys cannot be described as statistically projectable to the target population. [If weighting was done, state the following sentence on weighting:] The data have been weighted to reflect the demographic composition of (target population). Because the sample is based on those who initially self-selected for participation [in the panel], no estimates of sampling error can be calculated.”. This statement must be prominently placed in descriptions of the methodology in the survey report, including the executive summary.
- In general, for non-probability surveys it is not appropriate to use statistical significance tests. However, tests of significance with non-probability samples are appropriate when the objective is to establish the extent of the relationship among variables. If tests of significance are used with non-probability samples, it must be clearly noted that conclusions from these tests cannot be generalized to any population. Any use of descriptive statistics must clearly indicate that they are not formally generalizable to any group other than the sample studied, and there cannot be any formal statistical inferences about how the descriptive statistics for the sample represent any larger population. In the case of non-probability surveys that employ an experimental design in which respondents are randomly assigned to different cells, it is appropriate to use statistical significance tests to compare results from different cells.
The 4.3.3 standards are consistent with, albeit more detailed than, the MRIA standards (note that these apply to research results generally, not just non-probability samples): Footnote 34
MRIA Code of Conduct
"Researchers must not present research results with greater confidence than the data warrants. Instead, as responsible professionals, members must point out the relevant limitations of the research. This includes but is not limited to the following guidelines:
- disclosing relevant potential sources of error, both sampling and non-sampling (e.g. response, non-response, measurement, coverage, etc.)
- being explicit about the assumptions made about data accuracy when employing quota or stratification methods with probability samples
- refraining from making unqualified statements about confidence intervals or margins of sampling error on population estimates when probability samples are not used
For example, panels of repeat volunteers will not ordinarily qualify as sources of probability samples of the general population."
Until 2015, the American Association for Public Opinion Research (AAPOR) took a similar stance. However, in 2015 AAPOR revised its Code of Professional Ethics and Practices to allow for reporting of measures of precision from non-probability samples.
The change was apparently motivated in part by the 2013 Report of the AAPOR Task Force on Non-probability Sampling, which stated:
We believe that users of non-probability samples should be encouraged to report measures of the precision of their estimates, but suggest that, to avoid confusion, the set of terms be distinct from those currently used in probability sample surveys. The precision of estimates from non-probability samples is not the average deviation over all possible samples, but rather is a model-based measure of deviation from the population value. Ipsos, for example has proposed the credibility interval (Ipsos, 2012) for their estimates from an opt-in panel survey. As noted in Section 6, the credibility interval is measure of uncertainty that is used with Bayesian methods, and Ipsos described their procedure as Bayesian. Other model-based approaches also produce estimates of precision such as standard errors that could be used and do not refer to the average over all possible samples (the accepted terminology for design-based inferences used in probability samples).
Although the research base does not exist to endorse this particular measure or to urge its adoption across the industry, we believe the industry needs constructive attempts to develop measures that fills the gap created by the unsuitability of the standard margin of error calculation with non-probability samples. Treating estimates as though they had no error at all is not a reasonable option. At this point, it falls to individual researchers to judge the usefulness of this particular measure. Such judgments are only possible when organizations using them fully disclose the full range of information specified in the AAPOR Code of Professional Ethics and Practice along with a detailed description of how the underlying model was specified, its assumptions validated, and the measure calculated.
The relevant section of AAPOR’s Code of Professional Ethics and Practice now reads as follows:
III. Standards for disclosure
A. Disclosure items for surveys
10. Sample sizes (by sampling frame if more than one was used) and a discussion of the precision of the findings.
For probability samples, the estimates of sampling error will be reported, and the discussion will state whether or not the reported margins of sampling error or statistical analyses have been adjusted for the design effect due to weighting, clustering, or other factors.
Disclosure requirements for non-probability samples are different because the precision of estimates from such samples is a model-based measure (rather than the average deviation from the population value over all possible samples). Reports of non-probability samples will only provide measures of precision if they are accompanied by a detailed description of how the underlying model was specified, its assumptions validated and the measure(s) calculated. To avoid confusion, it is best to avoid using the term “margin of error” or “margin of sampling error” in conjunction with non-probability samples.
AAPOR has issued detailed guidance in AAPOR Guidance on Reporting Precision for Nonprobability Samples (a copy has been provided to the advisory panel):
Note the current 4.3.3 standard can be interpreted as not permitting alternative methods of statistical inference such as Bayesian credible intervals. This is based on the sentence in 4.3.3 #3, Any use of descriptive statistics must clearly indicate that they are not formally generalizable to any group other than the sample studies, and there cannot be any formal statistical inferences about how the descriptive statistics for the sample represent any larger population.
Note that the MRIA guidelines does not forbid using Bayesian credible intervals, as the language used refers only to margin of sampling error. Bayesian credible intervals are not a margin of sampling error. The MRIA guidelines are silent on the use of Bayesian credible intervals.
The objective is to update and clarify section 4.3.3 on use of statistical measures with non-probability samples.
Note that 4.3.3 should be reviewed in the context of the discussion of 4.1 and 4.3.2, where revisions associated with non-probability sampling were discussed.
There are 2 alternatives for revising 4.3.3:
The detailed proposal below is written for option 2. Note, though, that some of the revisions (in bold) would apply equally to option 1. The main point of difference is in 4.3.3 #3.
- There can be no statements made about margins of sampling error on population estimates when non-probability samples are used.
- The survey report must contain a statement on why no margin of sampling error is reported, based on the following template: “Respondents for this survey were selected
- For some surveys (e.g., exploratory, internal research) estimating precision may not be important to the research goals. For other surveys precision measures may be relevant, but the researcher may not have the statistical resources to compute them. It is acceptable for researchers working with non-probability samples to decline to report an estimate of precision. In such cases, the report must note that survey estimators have variance, but there has been no attempt to quantify the size.
- If an alternative statistical measure of precision such as Bayesian credible intervals is used:
- optional: the following alternatives are accepted:
- Bayesian credible intervals
- resampling approaches
- Taylor series linearization
- the statistical measure of precision that will be used must be stated in the research proposal, together with a rationale and brief description
- the survey report must provide:
- a detailed description of how the underlying model was specified, its assumptions validated and the measure(s) calculated. Refer to the AAPOR document AAPOR Guidance on Reporting Precision for Nonprobability Samples for the information to provide, as well as an example of the type of statement to make in the report.
- one key assumption is that the survey results are unbiased. This assumption must be prominently noted, together with any limitations on this assumption (see 4.3.1).
- an explanation of how to understand the measure of precision
- if applicable, how tests of statistical significance of differences or relationships based on the alternative method are to be understood
PORD has received some industry feedback that part of what the standards say about the statistical treatment of census survey results may not be correct.
The relevant language is in section 4.6 census surveys. The following is an abridged version of 4.6:
In a census survey, an attempt is made to collect data from every member of a population. For example, an organization might want to do a survey of all of its employees. In this case, the population is “all of the organization’s employees”, and this would qualify as a census survey if all employees are invited to participate in the survey.
The list whereby all members of the target population are to be contacted and invited to respond must be clearly described, including any of its limitations/exclusions in representing that target population. Whenever possible, an estimate of the percentage of the population that is excluded from the list must be provided and the potential impact of their exclusion on the research results considered. …
- The number of attempted re-contacts and procedure for attempted re-contact must be stated.
- Do not state a margin of sampling error, as this does not apply to census surveys because no sample is drawn. The survey report must contain a statement on why no margin of sampling error is reported, based on the following template: “Since the entire population of [target population] was invited to participate in this study there is no margin of sampling error to be estimated or reported. The potential impact of non-sampling error due to non-response is discussed in the results section of the report. [If weighting was done, state the following sentence on weighting:] The data have been weighted to reflect the composition of [the target population (if known) or the sampling frame (e.g., client-supplied list)] on the main known characteristics.”
The feedback was that statement #2 is not consistent with statement #1: “using the same reasoning, a sub-group of the census would still be a census of the sub-group. Statistical significance tests would only be appropriate on random samples of the census survey.”
D1.1 Are the 2 statements in section 4.6, census surveys, about statistical treatment of results inconsistent? That is:
If inconsistent, or not quite right, what changes do you suggest to the language in 4.6, census surveys?
D2.1 In a census, an attempt is made to contact every individual in the population (subject to coverage error). However, rarely will an interview be completed with every attempted contact (that is, the response rate will usually be less than 100%).
Does the survey cease being a census/attempted census if response rate falls below a certain level?
If so, what is the numeric response rate threshold to put in the standards? And if the response rate is below the threshold, does “margin of sampling error” come into play in terms of calculating confidence intervals and significance tests? Or, is some other measure of statistical precision used?
PORD would like the panel to consider the points made in a short article by Robert Peterson with the provocative title, “It’s time for pollsters to report margins of error more honestly”. A copy has been sent to panel members.
The article lists the following problems with current practices:
The article lists the following types of solutions to these problems:
Standard 14.7 (online)/15.7 (telephone) addresses one of the points in the article, namely that margin of sampling error varies depending upon the observed percentage:
2) For probability samples, state the level of precision, including the margin of error and confidence interval for the total sample and any key sub-groups.
The proposed revision to 14.7/15.7 #2 adds a requirement to indicate how the margin of sampling error varies across different observed percentages. It does not address all of the issues raised in Peterson article, nor does it embrace the recommended solutions (as we understand those).
1) For probability samples, state the level of precision, including the margin of sampling error and confidence interval for the total sample and any key sub-groups,
E1. Do you think the standards should be revised to address issues raised in the Peterson article, It’s time for pollsters to report margins of error more honestly? If so, which issues should be better addressed in the standards? What do you think of the proposed revision to section 14.7 (online)/15.7 (telephone), which would now require reporting level of precision both for a range of sample sizes and observed percentages?
The current standards contain the following requirements for proposal documentation when an online sample provider is used:
3) For access panels, a description of the following must be provided, at minimum (when multiple panels are used, the information must be provided for each):
- active panel size (provide the definition of “active”)
- panel recruitment
- panel monitoring
- panel maintenance
- panel refreshment
The objective is to update and revise this standard, including:
Note that there are other relevant sections in the standards, notably:
The intent is to not duplicate standards in these areas. Therefore, the focus here is specifically on additional proposal documentation requirements when an online sample provider is used who will be providing sample using an access panel or river sampling.
In 2008, the advisory panel on online public opinion survey quality recommended a number of standards and guidelines pertaining to access panels. For the most part, these were not incorporated into the online standards. However, PORD has asked that the current Panel consider whether any aspects of the earlier recommendations should be incorporated into the updated online standards. The relevant section of the earlier panel’s report is provided.
The following are some references, which we can send upon request:
Note that the proposed revision below splits the current #3 into 2 sections (now numbered #3 and #4, in order to address both probability panels and non-probability sources).
- active panel size (provide the definition of “active”),
- how the panel is constructed and maintained such that it is a probability sample
- how a probability sample of the survey’s target population will be drawn from the panel
- whether or not respondents will be sourced from more than 1 panel. If more than 1 panel is being used, provide the following:
- the identity of the other panels
- whether or not each of the other panels is a probability-based panel. If the sample includes respondents from non-probability samples, then the sample as a whole is a non-probability sample.
- the reasons for sourcing sample from more than 1 panel (e.g. to achieve the desired survey sample size; to get better coverage of the survey population)
- steps to be taken to ensure that people enrolled in more than 1 of the panels are not sent the survey more than once
- the incentive or reward respondents are offered for taking part in the survey. This should be reviewed to ensure it does not risk causing certain types of people to either take or avoid the survey, and thereby potentially bias the sample.
4)
- if a panel is being used, the active panel size (provide the definition of “active”), and if possible the active panel size corresponding to the survey’s target population. The panel size should be large enough to ensure a high likelihood the target sample size of usable completed questionnaires can be obtained, taking into account such factors as likely response rate, exclusion rules involving past survey participation, and sample reduction due to data cleaning.
- if a panel is being used, whether or not respondents will be sourced from more than 1 panel. If more than 1 panel is being used, provide the following:
- the identity of the other panels
- the reasons for sourcing sample from more than 1 panel (e.g. to achieve the desired survey sample size; to get better coverage of the survey population; to reduce the risk of bias)
- steps to be taken to ensure that people enrolled in more than 1 of the panels are not sent the survey more than once
- if a panel is being used, describe the recruitment sources used to populate the panel on an ongoing basis, and if possible information on the proportion of panelists accounted for by each source. The goal is to ensure the diversity of the survey population is represented, not only in terms of demographics but also on relevant behaviours and attitudes. In this regard, it can be better to have a large number of diverse sources, to reduce the risk of bias.
- if a router is used to select survey respondents, what measures are taken to guard against, or mitigate, any bias that might arise from using a router. A router is an online system that screens potential respondents and assigns each person to 1 of multiple surveys being fielded at the same time. Routers are important for efficiently fielding multiple surveys. However, there can be a risk that systematically directing people with certain characteristics to other surveys may bias the profile of people directed to your survey.
- if profile data from a panel is used to structure the sample or for key analyses, and these measures are not validated by remeasuring the characteristics in the questionnaire, provide information on the panel’s validation processes to ensure the information is accurate and up to date. Panel profile data on panelists can be very helpful for structuring a sample. However, it is a good practice to verify this data by remeasuring the characteristics in the survey questionnaire.
- the incentive or reward respondents are offered for taking part in the survey. This should be reviewed to ensure it does not risk causing certain types of people to either take or avoid the survey, and thereby potentially bias the sample.
As noted at the beginning of this section, the objective is to update and revise this proposal documentation standard 1.2.4 #3, with the focus here being on non-probability samples obtained from an online sample provider. The criteria to apply include:
To this list of criteria we also add “simplicity”: simpler/shorter language is preferred as long as it meets the objectives of being clear and useful.
PORD has asked that the panel draw upon 3 different listings of what the required proposal documentation should be, when formulating suggested requirements for non-probability samples obtained from an online sample provider:
The following pages show the guidance from the 3 sources. As you can see, they differ in length/complexity.
As noted earlier, PORD has also asked that the panel consider the suggestions of the earlier Advisory Panel on Online Public Opinion Survey Quality. The relevant excerpt has been sent separately to the panel.
F3. Focusing on non-probability samples obtained from an online sample provider: this is complicated but we ask that you take a crack at it. Please recommend revisions to section 1.2.4 #3, for non-probability samples (that is, what information to give about the sample in proposal documentation). The background provides some alternatives to pick and choose from (current standard, ESOMAR, proposed revision 1.2.4 #4, suggestions from the 2008 Advisory Panel on Online Public Opinion Survey Quality). The criteria to apply are:
Current 1.2.4
ESOMAR Online Research (pp. 18-19)
- details of how the sample was recruited and a description of the sampling frame and how well the sample covers the target population it is meant to represent
- steps taken to validate sample sources
- the procedures used to “on-board” prospective participants to panels
- cleaning and updating procedures
- any monitoring of survey-taking performance or quality controls to minimize satisficing or fraud and the steps taken if such behaviour is identified (section 7.8 of the standards addresses satisficing. The question is whether this should also be referenced in section 1, proposal documentation)
- whether and how new sources are integrated
- any procedures in place to maximize sample consistency for tracking projects
A proposed revision to 1.2.4 #4 (the non-probability section)
- if a panel is being used, the active panel size (provide the definition of “active”), and if possible the active panel size corresponding to the survey’s target population. The panel size should be large enough to ensure a high likelihood the target sample size of usable completed questionnaires can be obtained, taking into account such factors as likely response rate, exclusion rules involving past survey participation, and sample reduction due to data cleaning.
- if a panel is being used, whether or not respondents will be sourced from more than 1 panel. If more than 1 panel is being used, provide the following:
- the identity of the other panels
- the reasons for sourcing sample from more than 1 panel (e.g. to achieve the desired survey sample size; to get better coverage of the survey population; to reduce the risk of bias)
- steps to be taken to ensure that people enrolled in more than 1 of the panels are not sent the survey more than once
- if a panel is being used, describe the recruitment sources used to populate the panel on an ongoing basis, and if possible information on the proportion of panelists accounted for by each source. The goal is to ensure the diversity of the survey population is represented, not only in terms of demographics but also on relevant behaviours and attitudes. In this regard, it can be better to have a large number of diverse sources, to reduce the risk of bias.
- if a router is used to select survey respondents, what measures are taken to guard against, or mitigate, any bias that might arise from using a router. A router is an online system that screens potential respondents and assigns each person to one of multiple surveys being fielded at the same time. Routers are important for efficiently fielding multiple surveys. However, there can be a risk that systematically directing people with certain characteristics to other surveys may bias the profile of people directed to your survey.
- if profile data from a panel is used to structure the sample or for key analyses, and these measures are not validated by remeasuring the characteristics in the questionnaire, provide information on the panel’s validation processes to ensure the information is accurate and up to date. Panel profile data on panelists can be very helpful for structuring a sample. However, it is a good practice to verify this data by remeasuring the characteristics in the survey questionnaire.
- if the survey is a tracking study, the steps that will be taken to ensure comparability of the sample with previous waves of the tracking study
- the incentive or reward respondents are offered for taking part in the survey. This should be reviewed to ensure it does not risk causing certain types of people to either take or avoid the survey, and thereby potentially bias the sample.
- 1) The access panel provider, working with the client, shall implement procedures to identify and remove fraudulent and inattentive panel members or respondents, documenting these procedures and actions taken.
Section 2.1 #3 of the online and telephone standards gives required questions that must be asked in all surveys of individuals, “unless a convincing argument is made that the research objectives are better served by excluding one or more of them” (there is also an exclusion for B-to-B research where the unit of analysis is the organization).
Section 2.1 #3 states:
The wording used for each question must be that provided below, unless a convincing argument is made that particular research objectives require alternative wording. Even in these exceptional cases, the terms used and/or categories applied (e.g., for household income) to capture responses must be those provided below.
Section 2.1 #3 states the following rationale for these required questions:
The data from the age, education, and language questions (along with the recording of geographic location and sex) allows comparison with Statistics Canada census data for the purpose of non-response analysis. The data, along with that from the employment status and income questions, also facilitate the comparison of results between Government of Canada public opinion research studies. (See section 8. for further detail on non-response bias.)
Note that comparability to Statistics Canada’s questions and response options is a critical requirement. This is needed for non-response analyses, and it is needed when the variables are used to weight survey data to match the population.
Because of differences between the online (self-completion) and telephone (interviewer-administered) modes, the questions/response options may be somewhat different for the 2 modes.
In this regard, an issue for the panel to keep in mind throughout section F when there are differences in wording between the online and telephone modes is what constitutes comparability with Statistics Canada’s questions and response options. A key example: we can access the self-completion version of the 2016 census questionnaire, but it is not always clear what Statistics Canada would consider to be a comparable question in a telephone survey.
There is a general requirement (section 2.1 #2) to inform respondents at the beginning of a survey of the confidentiality of their questionnaire responses, but the current standards do not state any specific wording for how to preface asking the required demographic questions (which are often asked near the end of the survey).
For reference, the relevant parts of section 2.1 #2 are:
2.1. #2 The following are required elements of all Government of Canada online survey questionnaire introductions:
The Privacy Commissioner has requested the addition of a statement on privacy before the demographics section of questionnaires, such as the one below.
These last few questions are for statistical purposes and will be kept confidential. Your identity will always remain anonymous.
The proposal is to insert a new standard just before the current 2.1 #3 as follows (the current 2.1 #3 would become 2.1 #4:
2.1. #3 The following statement is required for all Government of Canada telephone/online surveys prior to administering the demographic section of the questionnaire:
These last few questions are for statistical purposes only. Your answers will be kept anonymous and confidential and will be combined with the answers from other respondents to this survey.
G1.1 The first sentence in the proposed addition of a new section 2.1 #3 is, “These last few questions are for statistical purposes only.” Is “statistical purposes” the best phrase to use, or is there a different phrase that would be more clear/effective? For example, “analytical purposes”, or “purposes of analysis”?
G1.2 Do you have any suggestions for revising the proposed section 2.1 #3?
The following is the current mandated question in section 2.1 #3 for telephone and online surveys:
Telephone surveys
Gender: [Do not ask: record based on interviewer observation]
Online Surveys
Gender: What is your gender?
Note: The fact that the answer options are different by methodology may be problematic for mixed-mode surveys as well as for weighting (i.e. telephone is based on the interviewer’s opinion and gives 2 answer options, while the online is based on respondent self-classification and gives 3 answer options). It is also an issue in its own right if there is an interest or a need to compare different surveys using the 2 different methods.
There are several issues related to the current ‘gender’ question for the panel to consider:
MRS also notes the following key points:
MRS provided the following examples of questions and a checklist of what questions researchers should ask themselves before deciding what question to ask:
What is your sex?
What is your sex? OR What is your gender?
Checklist - Questions to ask
The MRS goes on to say the following about gender identity questions specifically:
Establishing best practice in developing and asking gender identity questions will need to build on the position and practice of the ONS [Office of National Statistics which is the United Kingdom (UK) equivalent of Statistics Canada] and research carried out by the EHRC [Equality and Human Rights Commission]. This will allow the research community to design and implement a consistent and standard gender identity question that can be understood and answered by all people living in the U.K.
Uptal Dholakia in his article How Should Market Researchers Ask About Gender in Surveys? in an online blog for Psychology Today (September 2016) points out that: Any well-designed market research survey is based on 2 core principles: the principle of accuracy and the principle of inclusiveness. A questionnaire should be designed to gather information accurately, using best practices of survey design that psychometricians have formulated over several decades. But this is not enough. A survey should also be inclusive. When a respondent has finished taking a survey, they should feel like the opinion they have provided will be valued just as much as every other survey taker.
We propose the advisory panel consider 2 alternative options as a starting point, which the panel can of course revise as they see fit:
Option 1: status quo (leave the standards unchanged). The status quo is to ask about “gender”, as follows:
Option 2: More closely match Statistics Canada question(s)
The advantages of option 2 (see proposed text below) are:
Neither the telephone nor the online question exactly match the Statistics Canada question because of the possibility of a “prefer not to answer” response.
Telephone surveys (sex)
[Do not ask: record based on interviewer observation]
If uncertain based on voice, ask:
What is your sex? [read list]
Online surveys (sex)
What is your sex?
The following are the 2 mandated questions for language for telephone and online surveys in section 2.1 #3. The researcher can choose to use both questions or 1 of the 2 questions, depending on the survey objectives:
Telephone surveys
Mother tongue:
What is the language you first learned at home as a child and still understand? [READ LIST — ACCEPT ALL THAT APPLY]
Language spoken at home:
What language do you speak most often at home? [READ LIST — ACCEPT ALL THAT APPLY]
Online surveys
Mother tongue:
What is the language you first learned at home as a child and still understand? [ACCEPT ALL THAT APPLY]
Language spoken at home:
What language do you speak most often at home? [ACCEPT ALL THAT APPLY]
PORD has requested input on what question or questions to ask to best get at official language of respondents.
Considerations include:
The following are 3 options for the panel to consider:
Telephone surveys
Mother tongue:
What is the language you first learned at home in childhood and still understand? [READ LIST — ACCEPT ALL THAT APPLY]
Or do you prefer not to answer Language spoken at home:
What language do you speak most often at home? [READ LIST — ACCEPT ALL THAT APPLY]
Or do you prefer not to answer Online surveys
Mother tongue:
What is the language you first learned at home in childhood and still understand? [ACCEPT ALL THAT APPLY]
Language spoken at home:
What language do you speak most often at home? [ACCEPT ALL THAT APPLY]
option 3: make language spoken at home the only mandatory question and remove mother tongue question from the standards
Telephone surveys
Language spoken at home:
What language do you speak most often at home? [READ LIST — ACCEPT ALL THAT APPLY]
Or do you prefer not to answer Online surveys
Language spoken at home:
What language do you speak most often at home? [ACCEPT ALL THAT APPLY]
The current required age question is:
Telephone surveys
In what year were you born? [Record year: XXXX]
[IF PREFERS NOT TO PROVIDE A PRECISE BIRTH YEAR, ASK:]
Would you be willing to tell me in which of the following age categories you belong?
Online Surveys
In what year were you born?
[IF PREFERS NOT TO PROVIDE A PRECISE BIRTH YEAR, ASK:]
Would you be willing to indicate in which of the following age categories you belong?
Note that we assume a survey could use more detailed breaks as long as these can be collapsed into the categories above (although the standards do not explicitly say this).
In the census, Statistics Canada determines age from birth date.
For reference, the 2016 Census Profile for Canada shows the following age distribution:
| Age range | % of population |
|---|---|
| 0 to 17 years old | 20.0% |
| 18 to 24 years old | 8.8% |
| 25 to 29 years old | 6.5% |
| 30 to 34 years old | 6.6% |
| 40 to 44 years old | 6.4% |
| 45 to 49 years old | 6.7% |
| 50 to 54 years old | 7.6% |
| 55 to 59 years old | 7.5% |
| 60 to 64 years old | 6.5% |
| 65 to 69 years old | 5.6% |
| 70 to 74 years old | 4.0% |
| 75 to 79 years old | 2.9% |
| 80 years or older | 4.3% |
| Age range | % of population 18 years or more |
|---|---|
| 18 to 24 years old | 10.9% |
| 25 to 29 years old | 8.1% |
| 30 to 34 years old | 8.3% |
| 35 to 39 years old | 8.1% |
| 40 to 44 years old | 8.0% |
| 45 to 49 years old | 8.4% |
| 50 to 54 years old | 9.5% |
| 55 to 59 years old | 9.3% |
| 60 to 64 years old | 8.1% |
| 65 to 69 years old | 7.0% |
| 70 to 74 years old | 5.1% |
| 75 to 79 years old | 3.6% |
| 80 years and older | 5.4% |
| Age range | % of population 18 years or more |
|---|---|
| 18 to 24 years old | 10.9% |
| 25 to 34 years old | 16.4% |
| 35 to 44 years old | 16.2% |
| 45 to 54 years old | 17.9% |
| 55 to 64 years old | 17.5% |
| 65 years or older | 21.1% |
G4.1 Should the age categories be left the way they are in section 2.1 #3, or do you suggest revising them? Revisions could include revisions to the break points, and revisions to the total number of age categories.
PORD is particularly interested in whether a smaller number of categories could be used, and what those categories would be (what would you suggest a smaller number of categories consist of)?
The current required education question is:
Telephone Surveys
What is the highest level of formal education that you have completed? [READ LIST]
Online surveys
What is the highest level of formal education that you have completed?
For reference, the 2016 Census Profile for Canada shows the following education categories:
| Education level | % of population |
|---|---|
| No certificate; diploma or degree | 18.3% |
| Secondary (high) school diploma or equivalency certificate | 26.5% |
| Postsecondary certificate; diploma or degree | 55.3% |
| Apprenticeship or trades certificate or diploma | 9.8% |
| College; CEGEP or other non-university certificate or diploma | 19.4% |
| University certificate or diploma below bachelor level | 2.8% |
| University certificate; diploma or degree at bachelor level or above | 23.3% |
| Bachelor's degree | 15.5% |
| University certificate or diploma above bachelor level | 1.6% |
| Degree in medicine; dentistry; veterinary medicine or optometry | 0.7% |
| Master's degree | 4.6% |
| Earned doctorate | 0.8% |
It has been pointed out that the first 2 categories in the current required education question do not line up well with the Quebec education system. PORD provided the following from Wikipedia:
Mandatory elementary education (école primaire) starts with grade 1, through to grade 6. Secondary school (école secondaire) has 5 grades, called secondary I-V (Sec I-V for short) or simply grades 7-11. Students are 12 to 16 years old (age of September 30), unless they repeat a grade. Upon completion of grade 11, students receive their high school diploma from the provincial government.
So, in Quebec, grade 8 is the start of high school (école secondaire), whereas outside Quebec grade 9 is the start of high school. The view is that the first 2 existing required categories (grade 8 or less, and some high school) are confusing in Quebec, and indeed the 2 categories overlap, since grade 8 is also “some high school” in Quebec.
One approach is simply to drop any attempt in the required question to distinguish subcategories of less than a high school diploma (such distinctions can always be added on an ad hoc basis for a particular survey).
In this scheme, the first 2 response options could be:
G5.1 In the context of the differences between the education systems between Quebec and other parts of Canada, are the first 2 required education response options in section 2.1 fine as is, or do they need to be revised?
If revised, what do you suggest the response options should be? What do you think of the proposal to eliminate having subcategories of “less than high school”?
The current required household income question is:
Telephone surveys
Which of the following categories best describes your total household income? That is, the total income of all persons in your household combined, before taxes [READ LIST]?
Online surveys
Which of the following categories best describes your total household income? That is, the total income of all persons in your household combined, before taxes?
Note that we assume a survey could use more detailed breaks as long as these can be collapsed into the categories above (although the standards do not explicitly say this).
Statistics Canada now sources income data from administrative records, so there is no Statistics Canada question to use as a model.
Statistics Canada reports various income measures, one of which is “total income for private households before tax”, which appears to be equivalent to “total household income” in the questions above.
For reference, based on Statistics Canada 2015 data, the Census Profile for Canada shows the following for total income for private households:
| Income of private households | % of population |
|---|---|
| Under $5,000 | 1.6% |
| $5,000 to $9,999 | 1.4% |
| $10,000 to $14,999 | 2.7% |
| $15,000 to $19,999 | 4.0% |
| $20,000 to $24,999 | 4.3% |
| $25,000 to $29,999 | 3.8% |
| $30,000 to $34,999 | 4.3% |
| $35,000 to $39,999 | 4.3% |
| $40,000 to $44,999 | 4.2% |
| $45,000 to $49,999 | 4.1% |
| $50,000 to $59,999 | 7.8% |
| $60,000 to $69,999 | 7.2% |
| $70,000 to $79,999 | 6.6% |
| $80,000 to $89,999 | 5.9% |
| $90,000 to $99,999 | 5.3% |
| $100,000 to $124,999 | 10.4% |
| $125,000 to $149,999 | 7.2% |
| $150,000 to $199,999 | 7.9% |
| $200,000 and over | 6.8% |
| Income of private households: current required breaks | % of population |
|---|---|
| Under $20,000 | 9.7% |
| $20,000 to just under $40,000 | 16.7% |
| $40,000 to just under $60,000 | 16.1% |
| $60,000 to just under $80,000 | 13.7% |
| $80,000 to just under $100,000 | 11.2% |
| $100,000 to just under $150,000 | 17.7% |
| $150,000 and above | 14.7% |
Some other statistics using 2015 data:
The dollar threshold for low income varies with household size. Statistics Canada has the following table showing the low-income thresholds as a function of household size Footnote 36:
| Household size | After tax income | Before tax income |
|---|---|---|
| 1 person | $22,133 | $25,516 |
| 2 persons | $31,301 | $36,084 |
| 3 persons | $38,335 | $44,194 |
| 4 persons | $44,266 | $51,031 |
| 5 persons | $49,491 | $57,054 |
| 6 persons | $54,215 | $62,500 |
| 7 persons | $58,558 | $67,508 |
G6.1 Should the household income categories be left the way they are in section 2.1 #3, or do you suggest revising them? Revisions could include revisions to the break points, and revisions to the total number of income categories.
PORD consulted with MRIA as to the value of adding a question in telephone surveys about type(s) of phone(s) a respondent has access to at home:
The information about phone usage may also be incorporated into the weighting scheme for telephone surveys. The literature review commissioned by PORD notes this possibility (see pp. 13-14 of the review). The review says that, “At the time of writing, there is no consensus on the best approach to weighting dual-frame survey samples.” It goes on to cite an example of an approach that weights by telephone status (cell phone only, landline only, dual phone users), and an approach that does not.
Classification of household telephone status can be quite complicated, depending on the approach taken to weighting, or to how respondents are to be selected for the interview within a household Footnote 37.
Some examples of complicating factors:
The proposal below is to keep things simple and to classify the household telephone status of respondents in a survey of individuals (not surveys of businesses/organizations) into 3 basic categories: landline only, cell phone only, or dual users (household has both cell phone users and has a landline). Note that this is a classification of the household, not of the individual respondent. It does not attempt to differentiate personal versus business use of a cell phone.
A terminology issue to consider is how to refer to a landline phone. Does everyone know what “landline” refers to? If a household has a Voice over Internet Protocol (VOIP) phone and no telco landline, would they say they have a “landline”? Do people know what “VOIP”, or “IP”, phone is? Would people interpret “regular phone” to mean anything other than a cell phone?
The following is a possible question that could be added to section 2.1 #3
Type of home phone(s): The intent of these 2 questions is to classify the respondent’s household into 1 of 3 mutually exclusive categories: cell phone only, landline only, dual cell phone and landline.
G7.1 Should a question on type of home phone service be added as a requirement for all telephone surveys of individuals in section 2.1 #3?
G7.2 Do you have suggestions for any revisions to the suggested approach to determining type of household phone service?
In an online survey, it is likely that a sizable percentage of respondents will use, or attempt to use, a mobile device (smartphone or tablet) to complete the questionnaire.
A 2014 AAPOR report on mobile technologies concluded:
Recognize If You Are Conducting Online Surveys, You Are Conducting Mobile Surveys: A non-ignorable and growing percentage of respondents are now accessing online surveys via their mobile browsers (with estimates ranging from 8 to 23% depending on the study), resulting in higher abandonment rates and potentially greater measurement error among these mobile respondents. Footnote 38
With the growth of smartphone ownership and use, the percentages of people completing online surveys on a mobile device has very likely increased since 2014.
In a summary of results from the 2016 General Social Survey (GSS), it was reported by Statistics Canada in The Daily Footnote 39 that three-quarters of Canadians 15+ own a smartphone, although there is substantial variation by age (we haven’t been able to find detailed tables on the Statistics Canada website, so we’re stuck with the gaps in age below):
| Age range | % of population |
|---|---|
| Canadians aged 15 or more | 76% |
| 15 to 34 years old | 94% |
| 55 to 64 years old | 69% |
| 75 years or older | 18% |
In the 2015 Sage Research report, Best Practices for Improving Cooperation for Online Surveys Footnote 40, the following are some of the conclusions drawn from a review of the research literature:
Some survey companies are implementing technologies to accommodate use of mobile devices in online surveys, for example:
The key point is that adapting questionnaire design for those using a mobile device can potentially improve the survey data in terms of coverage, response rate, reduction of non-response bias, and answer quality.
An online survey can take different approaches to the possibility mobile devices will be used Footnote 41:
The current standards do not address the possibility and implications of an online survey being completed on a mobile device. The objective for the panel is to revise the standards to address these matters.
As we see it, the matters to address in the standards pertaining to usage of mobile devices to complete online surveys are:
There are proposed revisions to 2 sections of proposal documentation relating to use of mobile devices in online surveys:
The proposal is to add another numbered item to 1.2.2. For now, the number is referred to as “x.”
1.2.2. Sample/sampling details (in online standards)
- x) State whether respondents will be able to complete the questionnaire on (a) either a computer or a mobile device (smartphone or tablet), (b) a computer only, or (c) on a mobile device only.
- If the intent is to limit completion of the survey to only a computer or to only a mobile device, describe the reasons for this. It is likely that without intervention, some people will complete the survey on a computer, and some will use a mobile device. This has implications for questionnaire design because mobile devices have smaller screens. Restricting device usage can potentially impact coverage of the survey’s target population.
Note that item #2 in 1.2.2 (describe the sampling procedures) includes a reference to section 4, sampling procedures, where there are requirements to address any issues with representativeness, so it would be redundant to also flag those issues here in item “x” above.
The proposal is to add another numbered item to 1.2.5. For now, the number is referred to as “x.”
1.2.5. Questionnaire design (in online standards)
x) If it is possible the survey will be completed on a mobile device by at least some respondents, then indicate whether or not the intent is to provide a mobile-friendly version of the questionnaire. A mobile-friendly questionnaire can increase response rates and improve data quality. a) If the intent is not to have a mobile-friendly version of the questionnaire, describe the reasons for this.
A comment: we expect that people will usually intend to have a mobile-friendly questionnaire, so perhaps the main reason to have this standard is simply to remind people that this is something they need to consider when they get to the questionnaire design stage of the project.
There are 3 revisions/additions to section 2 of the standards to consider with respect to questionnaire design in online surveys where mobile devices may be used:
The 2015 Sage Research report, Best Practices for Improving Cooperation for Online Surveys, reviewed the research literature and concluded:
There are 2 approaches to integrating mobile-friendly question designs into an online survey:
These results have practical implications for researchers conducting Web surveys. Our finding of near-comparability in the quality of responses between smartphone and PC users, even for sensitive and burdensome questions, suggest not only that smartphone users should be accommodated in Web surveys but also that the survey instrument should be optimized for small screens. There is a line of thinking, based on universal mode design or unimode design principles (Dillman 2000), that one should aim to display the exact same question formats across devices in order to minimize mode effects. Our results seem to suggest otherwise since we found no evidence that the redesigned (optimized) question formats in the smartphone version introduced device effects or had an adverse effect on data quality.
The notion of comparability that we would instead argue for is a best practices approach […] This places the emphasis not on presenting the identical surface-level features, but on using the best practices of each mode or device in order to present the same stimulus to the respondent and minimize error within each mode or device. Carrying out these principles for mobile Web research requires playing to the strengths of smartphones by identifying and using input tools that are user friendly and avoiding any formats that are not (e.g., sliders, drop boxes that turn into pickers). This may also entail using Responsive Web Design (RWD) to adapt a questionnaire to the continuum of different screen sizes rather than binary optimization.
Notably, the same lead author, Antoun, was more cautious in making a recommendation on approach in another article that reviewed research literature from 2007 to 2016 on smartphone optimization:
Another issue for SO [smartphone optimized] surveys relates to the design and release of different layouts in response to the size of the respondent’s screen or web browser. The discussion so far has focused on adapting a questionnaire designed for PCs into a single optimized version for smartphones; but it is important to note that several studies used a responsive design where several versions of the questionnaires were displayed, with the implicit goal of improving response quality within each version (see, e.g., Amin, 2016). This practice has been adopted from web design where a large number of different visual designs and layouts for a single website are deployed (e.g., for small smartphones, large smartphones, small tablets, large tablets, small desktops, and so forth). What is unclear is whether this approach is also effective for surveys where standardization across layouts is a higher priority. A concern is that variations in a particular layout can affect responses (see, e.g., Smyth, Dillman, Christian, and Stern, 2006). Thus, responsive design calls attention to the need to promote comparability across versions, on the one hand, and to minimize error within each version, on the other.
Whether optimization is binary or responsive to a continuum of screen sizes, another issue is whether to design for the biggest or smallest devices first. Almost all of the reviewed studies started from the point of an existing survey that is designed for PCs and then adapted for smartphones. While this approach may prevent major usability problems in the smartphone survey, it does not necessarily produce an optimal design for smartphones (as the word “optimized” implies). Because the smallest screens appear to pose a greater design challenge, the “mobile first” approach may be desirable to the extent that it does not have negative effects on the version of the questionnaire displayed on larger browsers (see, e.g., de Bruijne and Wijnant, 2013a; Tharp, 2015). Future research on the effect of responsive design, with and without a mobile-first design philosophy, is necessary before any firm conclusions on these different approaches are drawn. Footnote 45
The bottom line is the last sentence: there doesn’t seem to be a definitive conclusion yet as to which approach is best.
Questions posted on the discussion board
A2.1 Should there be a standard encouraging a particular approach to question design/layout across devices (computers and mobile devices), and if so, what should it be?
Options could be:
Because completion times tend to be longer on a mobile device than on a computer, a frequent recommendation is that a mobile-friendly questionnaire should be “short.” However, there is no consensus on what this means in terms of number of minutes.
The current Standard for online questionnaire duration is 20 minutes, but an average duration of 15 minutes or less is “strongly encouraged”:
2.1. Standards
- Survey questionnaires must be designed:
- c) to be completed in a maximum duration of 20 minutes, not including pauses or interruptions. Exceptions could include projects with specialized audiences and those with pre-arranged interviews when the respondent is aware the survey will take longer than 20 minutes. Average questionnaire durations of 15 minutes or less are strongly encouraged in order to minimize respondent burden.
Questions posted on the discussion board
One option in the standards is not to give any standards or guidelines on mobile-friendly design. The idea is that in Proposal Documentation there will be a statement of intent to have a mobile-friendly survey. Beyond that, it will be up to the researcher to decide how to implement that for their survey. And, under this view, guidelines do not belong in a standards document.
The other option is to give guidelines (i.e. recommended, but not required, practices or principles). This is certainly possible to do based on the research literature. The idea is that it would be helpful to GC POR researchers to have this guidance stated in the standards document.
For reference, 2 approaches for guidance are shown here.
Questions posted on the discussion board
A2.3 In section 2 of the standards, questionnaire design, should there be guidelines (recommendations, but not requirements) in the standards on what makes a questionnaire mobile-friendly, or is that not necessary/appropriate in the standards?
In the event guidelines are included in the standards, what do you suggest these be? Possibilities were outlined in the background. It would also be helpful if you could take a crack at writing them as you would like them to appear in the standards.
There are 2 types of pre-testing that should be done for an online survey questionnaire where some respondents may use a mobile device:
The current standards do not specifically refer to requirements for online surveys where mobile devices may be used.
In section 3 of the online standards, the 2 most relevant current standards are:
3. Pre-testing
3.1. Standards
- 1) Pre-testing of all components of a new or revised survey questionnaire that may influence data quality and respondent behaviour is required. This includes the online appearance and functionality of the questionnaire.
- 5) A minimum of 10 pre-test surveys are to be completed in each language in which the final survey will be fielded. An exception could be projects with small survey populations, such as a client-satisfaction survey of a small client base. In such cases the researcher must, in consultation with the client, take steps to ensure that the smaller number of pre-tests are sufficient to guarantee questionnaire quality. For example, a cognitive pre-test may be warranted.
Should standards specific to completion of an online survey on mobile devices be added, or not?
What, if any, revision should be made to standard 3.1.1?
For example, a revision could be:
1) Pre-testing of all components of a new or revised survey questionnaire that may influence data quality and respondent behaviour is required. This includes the online appearance and functionality of the questionnaire.
What, if any, revision should be made to standard 3.1.5?
For example, a revision could be:
In a survey that allows completion on both mobile devices and computers, and particularly one using a “responsive design” approach that can result in different question designs/layouts for different size screens, there is the potential for a “mode” effect. That is, the different designs/layouts for a given question could cause different response distributions. If this possibility is to be explored, then data on device type needs to be collected during the survey.
Also, there is the possibility of a device-type effect as a result of differences in the characteristics of people who use a mobile device to complete a survey versus people who use a computer. For example, a research vendor noted the following in an article advocating for “device agnostic” sampling for online surveys:
Specific pockets of the population gravitate toward mobile and we expect to see the level of systematic non-coverage bias to grow in non-mobile research designs. Some of our testing demonstrates between 20% and 25% of millennials (those born from 1981 to 2000) prefer to access surveys via mobile, which means our non-mobile surveys are missing the views of a substantial portion of this audience. Considering that this group is likely more tech savvy and connected with peers, we expect over time to see biased and inaccurate results when they are excluded from our sampling frame. Footnote 47
This illustrates the importance of a mobile-friendly survey for ensuring good coverage of the population. However, it also means that an exploration of possible effects of question design/layout would need to be done carefully to avoid confounding question design/layout with covariates such as age.
As noted earlier, the conclusions from the research literature are:
A4.1 Should the standards have a requirement (or guideline) for researchers to look at whether different question designs/layouts in smartphones versus computers affected response distributions in the survey data?
A4.2 If analysis of device type or screen size effects is required or encouraged in the standards, how should section 7.7, monitoring of online survey fieldwork, be revised? What types of information on devices should be gathered? Should the information be collected for all online surveys, or only those with certain characteristics (e.g. different question design/layout by device type)? For example, a possible addition to 7.7 could be the following guideline (or should it be a standard, i.e. a requirement?):
7.7. Monitoring of online survey fieldwork
A5.1 To summarize, possible revisions to the standards pertaining to use of mobile devices in online surveys have been discussed for:
Are there any other areas where standards or guidelines should be added or revised with respect to usage of mobile devices in online surveys? Note that Section B of the discussion board will deal with covering respondent costs for using a mobile device.
Users of mobile devices may incur costs to participate in a research survey.
The current standards do not have any requirements as to how such costs should be handled.
The following is a standard/guideline in the ESOMAR/GBRN Guideline on Mobile Research:
3.1.3 Costs
Unlike most other research methods, data subjects may incur costs as a consequence of participating in mobile research that may include charges for data downloads, online access, text messaging, data plan overages, roaming charges, voicemail message retrieval and standard telephone charges. Researchers should design their research so that data subjects incur no costs without express approval. If this is not possible, researchers must be prepared to offer compensation. Such compensation may be cash, mobile money, airtime or other forms of value.
Note the ESOMAR/GBRN explain their use of “must” and “should” as follows:
Throughout this document the word “must” is used to identify mandatory requirements. We use the word “must” when describing a principle or practice that researchers are obliged to follow. The word “should” is used when describing implementation. This usage is meant to recognise that researchers may choose to implement a principle or practice in different ways depending on the design of their research.
The objective for the panel is to determine whether something like the ESOMAR/GBRN 3.1.3 above should be incorporated into the online and telephone standards.
The following is slight re-wording of the ESOMAR/GBRN (we suggest you use this as a starting point for making suggestions):
Proposal: Respondents using a mobile device may incur costs as a consequence of participating in mobile research that may include charges for data downloads, online access, text messaging, data plan overages, roaming charges, voicemail message retrieval and standard telephone charges. Researchers should design their research so that data subjects incur no costs without their express approval. If this is not possible, researchers must be prepared to offer compensation. Such compensation may be cash, mobile money, airtime or other forms of value.
B1. Should the online and telephone standards have a section about covering respondent costs of using a mobile device? If so, what do you think of the proposed text in the background based on a ESOMAR/GBRN standard? Please revise as you see fit.
An important issue in sampling for telephone surveys is the inclusion of cell phone users and landline users. This can affect coverage of the survey population, the sampling frame(s) used for the survey, and possibly weighting. The objective for the panel is to identify any revisions to standards pertaining to this issue. Of particular interest is what, if anything, to say about the proportion of cell phone versus landline interviews in a probability survey of the Canadian population.
The following are some statistics taken from the CRTC’s Communications Monitoring Report 2017:
The CRTC report does not report phone type by age, but back in 2010, the AAPOR Cell Phone Task Force concluded: “young adults in the U.S. aged 18 to 34 years, can no longer be reached successfully via the landline frame.” Footnote 48
These data indicate that a telephone probability sample of the general Canadian adult population must include a cell phone sample.
For this section, please review 3. Dual-Frame Probability Telephone Surveys, in the literature review commissioned by PORD (pp. 12-14, and recommendation on pp. 31-32 for Sample frame proportions for dual-frame surveys). Some points made in the literature review:
In Canada, when conducting a telephone survey of the general public, it is common to apply 1 set of demographic weights to survey data thereby treating cellphone and landline interviews the same. A dual-sample frame is used to minimize the potential for coverage error resulting from the incomplete landline sample frame… From this perspective, survey samples resulting from dual-frame sampling only require weighting adjustments to bring the demographic distributions in line with those of the population. (p.14)
Sample frame proportions for dual-frame surveys: The proportion of landline and cellphone numbers included in dual-frame samples varies as researchers try to find the optimal balance and as the incidence of exclusive cellphone use increases. For this reason, it is recommended that PORD not put in place prescriptive standards for dual-frame sample proportions. There will not be one ratio of landline-to-cell sample that will apply to all public opinion research studies. Other factors, such as budget, target audience, and field period, will influence the proportions. A useful guideline would be one that focuses on the outcome of the data collection. That is, the sample ratios chosen should ensure that the proportion of cellphone-only households in the final survey sample reflects that of the population of cellphone-only households at the time of the study.
The 2010 AAPOR Cell Phone Task Force noted a variety of other issues involving surveying cell phone users, and typically noted that there is no definitive “best” resolution. For example:
The following are the current standards in the telephone standards that refer to cell phones:
1. Proposal documentation
1.2. Technical specifications of the research
1.2.3. Response rate/participation rate and error rate
- 1) State the expected response rate/participation rate for the sample(s). State expected response rates/participation rates for each mode when more than 1 mode is proposed (i.e. separately for online and telephone components). Similarly, if a component of the research will recruit respondents via their cell phones, the expected response rate/participation rate for that component must be stated separately as well. The expected response rate/participation rate is an estimation based on various factors such as previous response rates/participation rates, trends etc. The proposal must include a brief discussion of the factors that might cause the actual response/participation rate to fall short of the stated target.
1.2.4. Description of data collection
- 7) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method. When it is proposed to recruit at least some respondents via their cell phones, provide a rationale for using this approach in addition to or in place of landline phones.
4. Sampling procedures
4.2. Probability sampling
- 3) A full description of the sample design and selection procedures must be stated including:
- c) at each sampling stage, the method of achieving a probability sample (e.g., random selection) shall be explained, and any subsets of the universe that have been excluded or underrepresented shall be stated (e.g., cell phone only households), although whenever possible, an estimate of the percentage of the universe that has been excluded or underrepresented must be provided;
Note that standard 1.2.4 is repeated in standard 15.5 data collection (telephone) as part of section 15, mandatory survey report requirements.
There may not be a need for this level of specificity, because the general requirements are sufficient to trigger disclosure of the information (i.e. the researchers will translate the general requirements into the information that needs to be stated about cell phone/landline for the particular study in question). Alternatively, perhaps the standards should address the cell phone/landline topic in a more specific way.
Are there any additional design matters or information disclosure requirements pertinent to the cell phone/landline topic that you think should be added to either section 1 proposal documentation or section 4 sampling procedures? If so, please indicate whether these should be standards (requirements) or guidelines (suggestions).
The telephone standards for call-backs are in section 7, data collection:
7.2. Call-backs
- There will be a minimum of 8 call-backs made before retiring a telephone number and substituting it with another number. The call backs must be made at varying days and times over a minimum seven-day period. An exception could be made when the field period is shorter as a result of the need to assess recall of particular events or activities.
- Every effort must be made to ensure that the respondent is called back if an appointment has been arranged and that the date and time of that appointment are respected.
- No attempt will be made to call back refusals.
The panel is being asked for input on:
PORD posed this question to the MRIA, as concern had been expressed about whether 8 call-backs is too many, and might be perceived as harassment.
The MRIA Polling Standards for the Canadian Marketplace Footnote 50 state 8 call-backs as a maximum, as compared to section 7.2 call-backs which states it as a minimum. The standards also state a definition of call-backs (no definition is given in Section 7.2 of the telephone standards).
Make no more than 8 calls to the same telephone number. This number includes:
MRIA posed the following question to some members: The MRIA had adopted the policy of a maximum of 8 calls to each potential respondent. See Appendix "L" 8.4.2. If the number of calls allowed are reduced would this affect your research studies? What is your firm's frequency of call-backs (over what period do you make the [eight] 8 calls)? The response relayed to PORD was:
Members are very sensitive to respondent fatigue and aim to regulate the frequency of calls to the same respondents.
The additional calls to respondents are usually after exhausting the list of potential respondents and calls are made to those whom the researcher was unable to contact in the initial call.
It is extremely important to recognize the difference between dispositions for call attempts. Not every call attempt should be considered a call-back. For example, 8 call backs, all yielding a busy signal, is very different than 8 call backs all resulting in an answer and call back request. At the same time, the number of call backs must be large enough to provide a reasonable expectation of equal probability of selection for all primary sample units. We have a maximum of 7 call backs, but do consider some call dispositions to be partial call-backs (e.g., a busy signal counts as 1/3 of a call-back). So theoretically we could call a phone number up to 21 times (21 busy signals). Usually phone numbers are resolved after 6 - 10 attempts.
Refusal conversion dialing must also be considered and whether these conversion attempts are considered within the call back limit.
PORD also would like the panel to comment on whether the call-back requirements should be the same for respondents using a cell phone.
The call-back requirements in section 7.2 call-backs do not make any distinction between interviewer-administered surveys and IVR surveys.
7.2. Call-backs
- 1) There will be a minimum of 8 call-backs made before retiring a telephone number and substituting it with another number. The call backs must be made at varying days and times over a minimum seven-day period. An exception could be made when the field period is shorter as a result of the need to assess recall of particular events or activities.
In the literature review commission by PORD, under drawbacks of using IVR, it states:
In addition to limits on the length and complexity of the survey questionnaire, the quality of the sample can be questionable. This, however, is not unique to IVR. All survey research requires complete sampling frames (little to no coverage error), sound sampling strategies (simple random sampling or stratified random sampling), and appropriate sample control (an adequate number of call-backs that vary by time of day/day of the week to maximize the response rate). Since speed is one of the key advantages of using IVR, sample control measures are not as stringent (there is no time to call back a number in the sample multiple times). The survey sample will include whoever could be reached on the night of the data collection, which introduces the possibility of non-response bias. While survey weights will be applied to the survey sample post-data collection to ensure it reflects the demographic profile of target population, this will not address attitudinal differences that might exist between survey respondents and non-respondents. (p. 18)
The literature review implies that it may often be the case that few, if any, call-backs are made in an IVR survey. It connects this to using IVR for its speed advantage, and note that Standard 7.2 #1 includes an exemption from the 8 call-back requirement when speed of fieldwork is important: “An exception could be made when the field period is shorter as a result of the need to assess recall of particular events or activities.”
D2.1 Should any there be any changes to section 7.2 call-backs specific to IVR surveys?
In an Interactive Voice Response (IVR) telephone survey, a computer is programmed with a questionnaire, calls are made automatically, and the recorded questions are read by the computer. There are no live interviewers.
The objectives for the panel are:
For this topic, please review the section Interactive Voice Response (IVR) surveys in the literature review commissioned by PORD (pp. 17-19). To summarize, the literature review identifies the following:
Section 5.3 use of Interactive Voice Response (IVR) in the telephone standards discourages, but does not forbid, use of IVR surveys for POR. It also suggests circumstances when IVR may be an appropriate methodology. The standard states that IVR surveys have the same requirements as interviewer surveys for the survey introduction, respondent opt-out, times when calls can be made, and delay in acknowledging an answered call. The standard does not explicitly state that call-back requirements apply to IVR surveys (see D3 for a question about this).
5. Retaining public confidence
5.3. Use of Interactive Voice Response
- 1) Characteristics of Interactive Voice Response (IVR) surveys, including the impersonal style conveyed by automation, put that method in conflict with the manner in which the Government of Canada wishes to engage Canadians. IVR can therefore be used only when a convincing case is made that the specific information to be collected is essential for making important decisions and cannot be obtained through other means. For example, IVR may be judged acceptable when the opinions of a hard-to-reach (low incidence) group are critical to the issue at hand and the very high call volume made economical by IVR is likely to markedly increase participation from members of that group in the survey. (IVR may also be used whenever respondents have agreed beforehand to this method.)
- 2) When IVR is used, the same information required for interviews conducted by live interviewers (sponsor, researcher, participation is voluntary, assurance of confidentiality, etc.) must be included in the IVR survey introduction. Respondents must also be provided early in the introduction with an easy method to opt out of the survey (e.g., by pressing a specific key) so that the call is terminated gracefully and no more calls are made to that number. The same requirements for the time-of-day of calls and delay in acknowledging an answered call (see section 5.2.) apply to the use of IVR.
Note that the relevant parts of section 5.2 referred to above are as follows. Note also that the next discussion board will take another look at #2 (hours of calling) for telephone surveys generally, so our interest here is only in issues specific to IVR surveys.
5.2. Avoidance of harassment
- 2) Using the respondent’s local time, telephone interviewing can only be conducted from 9:00 to 21:00 Monday to Friday, 10:00 to 21:00 Saturday, and 12:00 to 21:00 Sunday and statutory holidays except when a pre-arranged interview is requested by the respondent outside these hours.
- 3) There must be no or minimal (one second) pause before the interviewer acknowledges that a potential respondent has answered the telephone.
E1.1 Do you have suggestions for revisions to section 5.3 use of Interactive Voice Response (IVR)? Considerations could include:
The standard for survey duration for telephone states surveys must be completed in 20 minutes, and strongly encourages a duration of 15 minutes or less.
2. Questionnaire design
2.1. Standards
- 1) Survey questionnaires must be designed:
- c) to be completed in a maximum duration of 20 minutes. Exceptions could include projects with specialized audiences and those with pre-arranged interviews when the respondent is aware the survey will take longer than 20 minutes. Average questionnaire durations of 15 minutes or less are strongly encouraged in order to minimize respondent burden.
The literature review commissioned by PORD suggests that IVR surveys are better suited to shorter surveys:
One of the main drawbacks of using the IVR methodology is the need for a short, simply constructed questionnaire. This is one of the reasons this methodology is well-suited to election polling and measuring voter intention. 1 or 2 clear and concise questions can be asked when using IVR and these questions will have simple (and few) response options from which to select. For example, “press 1 if you know who you intend to vote for on Election Day and 2 if you do not”.
For a research project that intends to explore several topics with respondents and/or topics in a more in-depth manner (e.g., to uncover reasons for voter intentions or to gain insights on salient election issues with voters), an IVR survey would not be the appropriate methodology. (p.18)
PORD has found that typically IVR surveys are less than 20 minutes. PORD also noted that they have seen IVR being used more in the past year for recruiting. In this context, PORD has raised a question about whether the required elements for telephone survey introductions should be revised or shortened for IVR surveys. Note that exempting IVR surveys from any of these elements would be significant, and would need strong justification. The current standard is:
2. Questionnaire design
2.1. Standards
- 2) The following are required elements of all Government of Canada telephone survey questionnaire introductions:
- follow the Official Languages Act and Policies; in particular, introductions must include an active offer of both official languages (English and French); the language that is used first will depend on the province in which the respondent resides (e.g., in Quebec “Bonjour/Hello”; in the Rest of Canada “Hello/Bonjour”)
- identify the Government of Canada or the department/agency sponsoring the survey
- inform respondents of the general subject and purpose of the study. The subject and purpose of the study may be expressed in very general terms so long as these terms cannot be construed as an attempt to misinform respondents. When the researcher makes a convincing case that the subject and purpose of the survey is such that stating it at the outset will affect respondents’ willingness to participate, stating them may be deferred to the conclusion of the study
- inform respondents of the expected length of the interview
- identify the researcher and interviewer (a pseudonym may be used for the interviewer as long as the individual remains identifiable by management internally for quality control purposes)
- inform respondents that their participation in the study is voluntary and completely confidential
- inform respondents that their responses remain anonymous. In the exceptional cases where the research objectives require that respondent identity be revealed, the informed consent of the respondent must be obtained
- inform respondents that the survey is registered with the Research Registration System maintained by the MRIA and provide information sufficient for respondents to access the System’s Research Verification Service (project registration number and service’s website URL, e-mail or phone number), if requested
Examples of exceptions: the sponsor would not be mentioned until the end of the survey in particular cases (e.g., advertising post-tests); other languages would be mentioned when the survey is available in non-official languages; and information about the availability of reports could be provided to specialized respondents when the incentive to participate outweighs the additional length of the introduction.
Note: If requested by the respondent, interviewers need to be prepared to repeat the instructions. Interviewers must have information readily available to inform respondents about how respondents were chosen, how privacy is protected, and where the survey results can be obtained.
The current standard for IVR surveys specifies that all of these items must be included in an IVR survey:
5.3. Use of Interactive Voice Response (IVR)
- 2) When IVR is used, the same information required for interviews conducted by live interviewers (sponsor, researcher, participation is voluntary, assurance of confidentiality, etc.) must be included in the IVR survey introduction. […]
E2.1 PORD has raised a question about whether the required elements for telephone survey introductions should be revised or shortened for IVR surveys (see Background for listing). Note that exempting IVR surveys from any of these elements would be significant, and would need strong justification. Should any of the required elements be omitted or revised for IVR surveys?
E2.2 PORD has asked that the panel comment on the following type of introduction for an IVR survey. This approach moves some of the required information disclosures to the “end of the survey”, where the information can then be accessed by calling a number or going to a website:
Introduction for an IVR survey: Hello/bonjour, this is (survey company) calling on behalf of the Government of Canada/government department. Pour continuez en francais appuyez sur le (number). We are conducting a (number) minute research survey on SUBJECT. Your participation is voluntary and completely confidential and anonymous. The name, telephone and web address of whom to contact for additional information about this research project are available at the end of the survey. To continue press (number). To end this call now hang up or press (number).
(At the end of the call) If you would like further information on this study please contact (name and phone number) or go to (website). To repeat this information press (number).
Further information could include:
What do you think of this type of introduction for an IVR survey, which for IVR surveys (but not interviewer surveys) moves some information disclosures to the “end of the survey” in a form that requires the person to take an additional action to get that information? Any suggested revisions to the above text?
PORD would like to clarify and strengthen the standards for multi-mode surveys.
For reference, the current standards referring to multi-mode surveys are:
1. Proposal documentation
1.2.4. Description of data collection
7) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method.
4. Sampling procedures
4.5. Multi-mode surveys
Multi-mode surveys are ones where different methods of questionnaire administration are used. They will often involve a combination of online and telephone methods, although there are other possibilities (e.g., in-person, mail, fax).
When a survey is conducted using multiple modes of questionnaire administration:
- The reasons for using a multi-mode rather than a single-mode method must be stated, both in the research proposal and the survey report.
- When the plan is to combine data collected via different modes in the data analyses, then steps must be taken to ensure as much comparability as possible across the different survey modes in terms of question wording and presentation of response options.
- Steps must be taken to ensure avoidance of duplicate respondents in different modes. The steps taken, and the results, must be documented.
14 (online)/15 (telephone). Mandatory survey report requirements
14.5/15.5 Data collection
- 2) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method.
14.6/15.6 Quality controls
- 3) For multi-mode surveys, detailed description of any data quality issues arising from combining data collected via different modes/instruments.
The proposed revisions the panel will be asked to comment on involve the following topics:
- clarify what needs to be stated in the proposal
- incorporate high quality benchmark questions in order facilitate detection of mode bias
- encourage use of similar modes of administration, i.e. both self-administered or both interviewer-administered, in order to reduce mode effects
- pre-testing requirements for multi-mode surveys
- clarify the reporting requirements in terms of (a) any adjustments made to mitigate mode biases, and (b) the decision to combine versus not combine the data across modes
- clarify calculation of outcome rates for multi-mode
The final question to the panel will be whether there are any other suggestions for revisions or additions to standards with respect to multi-mode surveys.
In section 1, proposal documentation, the requirements specific to multi-mode surveys are:
1. Proposal documentation
1.2.4. Description of data collection
- 7) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method.
Other components of section 1 will ensure that the methodology employed for each mode will be described even though they don’t explicitly refer to multi-mode survey designs (e.g. 1.2.2 Sample/Sampling Details, and other elements in 1.2.4).
The primary concern associated with multi-mode surveys is the potential for mode bias (that is, getting different response distributions for the same question due specifically to characteristics of the mode). For example, it has been found that social desirability effects tend to be stronger in interviewer-administered surveys (e.g. telephone) than in self-completion surveys (e.g. online). Differences across mode in how questions and response options are presented could potentially cause different response distributions. Mode bias can vary by question (e.g. some questions may be more prone to social desirability effects than others, and some questions may be more similar in design across modes than other questions). These types of mode effects pose challenges for combining or comparing data across modes.
Note that differences in response due to different types of people using the different modes is not a problem, and indeed improving population coverage is a reason to consider doing multi-mode surveys in some circumstances.
The issue is whether and how the proposal documentation requirements need to be elaborated to make it more clear in the proposal that the issue of potential mode bias is recognized and that steps will be taken to address this. The existing requirement only indirectly refers to dealing with the potential for mode bias. Note that section 4, Sampling, is more explicit, but perhaps more can be done in section 1 of the standards.
1. Proposal documentation
1.2.4. Description of data collection
- 7) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method,
1.2.7. Data analysis
- 1) Briefly describe how the data will be analyzed in order to address the objectives/research questions, including any special analyses (e.g., segmentation)
- a) For multi-mode surveys, state whether the plan is to report results based on combining the data across modes, or to report results separately by mode.
The existing standard for multi-mode surveys in sampling procedures is:
4. Sampling procedures
4.5. Multi-mode surveys
Multi-mode surveys are ones where different methods of questionnaire administration are used. They will often involve a combination of online and telephone methods, although there are other possibilities (e.g., in-person, mail, fax).
When a survey is conducted using multiple modes of questionnaire administration:
- The reasons for using a multi-mode rather than a single-mode method must be stated, both in the research proposal and the survey report.
- When the plan is to combine data collected via different modes in the data analyses, then steps must be taken to ensure as much comparability as possible across the different survey modes in terms of question wording and presentation of response options.
- Steps must be taken to ensure avoidance of duplicate respondents in different modes. The steps taken, and the results, must be documented.
There is no current standard for questionnaire design specific to multi-mode surveys.
The intent in the proposed changes below are to (a) distinguish between sampling-related topics and questionnaire-related topics, (b) increase the prominence of the value of using similar modes of survey administration, (c) clarify that one needs to be concerned about mode biases when comparing results by mode as well as when combining data across modes, and (d) to highlight the value that benchmark questions can have for enabling detection of mode biases.
The proposed revisions to sampling procedures are:
4. Sampling procedures
4.5. Multi-mode surveys
Multi-mode surveys are ones where different methods of questionnaire administration are used (e.g. some combination of telephone, online, in-person, or mail).
When a survey is conducted using multiple modes of questionnaire administration:
- The reasons for using a multi-mode rather than a single-mode method must be stated, both in the research proposal and the survey report.
- The rationale for the specific modes used must be stated, both in the research proposal and the survey report.
- a) The risk of mode biases can be lower if the modes of administration are similar (i.e. both interviewer-administered [e.g. telephone and in-person] or both self-administered [e.g. online and mail]).
- Steps must be taken to ensure avoidance of duplicate respondents in different modes. The steps taken, and the results, must be documented.
The proposed revisions to section 2 questionnaire design are as follows: this would be a separately numbered item in section 2, and for now the number is shown as “x”; for now we’ve used the same introductory explanation of multi-mode surveys as in 4.5):
2. Questionnaire design
2.1. Standards
- x) Multi-mode surveys are ones where different methods of questionnaire administration are used (e.g., some combination of telephone, online, in-person, or mail). When a survey is conducted using multiple modes of questionnaire administration:
- When the plan is to combine or compare data collected via different modes in the data analyses, then steps must be taken to ensure as much comparability as possible across the different survey modes in terms of question wording and presentation of response options.
- Include questions for which there are high quality data on the population and which therefore can be used as benchmarks for exploring the possibility of mode biases. The required demographic questions can serve this purpose (see section 1.2.4). Also try to identify any other high quality benchmarks available (whether demographic, behavioural or attitudinal) that might correlate with key measures in the survey. The latter can help better identify any mode biases in the key survey measures.
The current section 3 standard for pre-testing does not make any specific references to multi-mode surveys.
For reference, section 3 pre-testing (online version) is:
3. Pre-testing
3.1. Standards
- Pre-testing of all components of a new or revised survey questionnaire that may influence data quality and respondent behaviour is required. This includes the online appearance and functionality of the questionnaire.
- The client must be given the opportunity to test and approve the online survey prior to launch.
- Pre-testing must include probing that invites participants recruited for this purpose to provide input about their comprehension of and reaction to the questions. For example, a short series of questions could be included at the end of the pre-test survey. Researchers and clients must agree in advance as to whether probing will take place during or after administering the survey. If requested by the client a cognitive pre-test must be conducted.
- To help ensure questionnaire effectiveness with subgroups where there is reason for concern (e.g., due to language, age, level of education, etc.), the socio- demographic characteristics of the targeted participants must be approved by the client before recruiting begins.
- A minimum of 10 pre-test surveys are to be completed in each language in which the final survey will be fielded. An exception could be projects with small survey populations, such as a client-satisfaction survey of a small client base. In such cases the researcher must, in consultation with the client, take steps to ensure that the smaller number of pre-tests are sufficient to guarantee questionnaire quality. For example, a cognitive pre-test may be warranted.
- Pre-test completions shall not be included in the final dataset. An exception could be projects with:
- hard-to-reach target groups, or
- when no changes are made to the questionnaire.
- Documentation of the pre-test(s) must be provided to the client before the questionnaire is finalized. The documentation must include (at minimum):
- a description of the pre-test approach and number of interviews completed
- findings and any resulting modifications
- average survey completion time
- a statement of whether or not pre-test cases will be retained in the final data set
The final research report must include this same information.
F3.1 Section 3.1.3 of pre-testing stipulates a minimum 10 English and 10 French pre-test interviews, and there is no reference to multi-mode surveys. In the case of a multi-mode survey, should there be a requirement for a minimum number of pre-test interviews in English and French for each mode? If so, what should be the minimum number of interviews per mode in English and French?
Currently in section 8 outcome rates there is no standard for how to report outcome rates for a multi-mode survey.
The following proposal is based on general recommendations in the AAPOR document, Standard Definitions: Final dispositions of case codes and outcome rates for surveys Footnote 51. A more detailed discussion can be found in the Sage Research report, Best Practices for Improving Cooperation for Online Surveys Footnote 52.
The proposed text below would be part of the “definitional” part of section 8 (8.1 to 8.4), and before the “mandatory requirements” part (8.5). For now it is labeled “8.x”.
8. Outcome rates
8.x. Multi-mode surveys
- 1) The general principles for calculating the response rate for mixed-mode surveys are (a) calculate a response rate for each mode or stage of sampling, (b) use the same general response rate formula for each mode or stage of sampling, and (c) calculate a single aggregated response rate. The specific details of how this is done will be adapted to the particular multi-mode survey design.
F4.1 Currently in section 8 outcome rates there is no standard for how to report outcome rates for a multi-mode survey. The background proposes a standard consisting of 3 principles: (a) calculate a response rate for each mode or stage of sampling, (b) use the same general response rate formula for each mode or stage of sampling, and (c) calculate a single aggregated response rate. Do you have any suggestions for changes to the proposed standard?
The existing standards referring to multi-mode surveys are:
14 (online)/15 (telephone). Mandatory survey report requirements
14.5/15.5 Data collection
- 2) For multi-mode surveys, provide a rationale for using a multi-mode rather than a single-mode method.
14.6/15.6 Quality controls
- 3) For multi-mode surveys, detailed description of any data quality issues arising from combining data collected via different modes/instruments.
14.5/15.5 #2 will be updated based on the panel discussion of 1.2.4 in proposal documentation, where this same language was used.
The intent of the proposed revisions to 14.6/15.6 are to (a) ensure there is clarity as to decisions made about combining or not combining data across modes, and (b) to require descriptions of any adjustments made to the data to mitigate mode effects.
Proposed revision to 14.6/15.6 quality controls
14.6/15.6 quality controls
- 3) For multi-mode surveys, detailed description of any data quality issues arising from combining data collected via different modes/instruments. The discussion must also include:
- Rationale for decisions about combining or not combining data from different modes in reporting the results
- Description of any steps taken to mitigate mode effects in the survey results
F5.1 The existing standard in the quality controls section of mandatory reporting requirements addresses “description of any data quality issues arising from combining data collected via different modes/instruments.” The background proposes an additional requirement: description of any steps taken to mitigate mode effects in the survey results.
Is this additional necessary/useful? Any changes to the wording?
F6.1 Do you have suggestions for any other revisions to the standards to address issues associated with multi-mode surveys?
Section 6 data collection from children, young people or vulnerable respondents does not make any reference to whether or how incentives are used for this survey population.
Section 7.5 (telephone)/7.6 (online) does not refer to this population either:
7.5/6. Incentives/honoraria
- 1) The details of any incentives/honoraria to be used for an telephone/online survey must be provided in both the proposal and report documentation, including:
- the type of incentive/honoraria (e.g., monetary, non-monetary)
- the nature of the incentive (e.g., for monetary, prize, points, donations, direct payments)
- the estimated dollar value of the incentives to be disbursed
GC’s Qualitative Research standards contain 1 standard pertaining to incentives for this target population:
5. Participant recruiting
5.4. Incentives
- 7) Ethical considerations should be taken into account especially when providing incentives to youth, or vulnerable groups (e.g. payment is not coercive, or exposes young or vulnerable persons to a risk that they would otherwise have avoided).
PORD received some input from MRIA. MRIA posed the following question to some members (note that it’s not clear if the context of the question was survey research, or if it included qualitative research as well): Information gathered from audit of the MRIA Gold Seal members and speaking with other members, shows that contact with children and young people are through their parents and guardians. Please provide your comments on how your firm offers incentives to children and young people. PORD relayed the following response from MRIA:
Perform household level research, so incentives are offered to all members of the household regardless of age.
The incentive is always in cash; recognizing that young people may not be able to cash a cheque themselves.
Some members offer the incentives to the parent/guardian on behalf of the child.
Gift cards related to gaming are offered by some members.
PORD would like the panel to consider whether the telephone and online standards should have guidance on incentives for surveys of children, young people or vulnerable respondents.
The standards define children and young people as follows:
If panel members think there should be guidance for this survey population, the following is an addition to 7.5/6 Incentives/Honoraria that is a starting point for the panel’s consideration:
7.5/6. Incentives/honoraria
- The details of any incentives/honoraria to be used for an telephone/online survey must be provided in both the proposal and report documentation, including:
- the type of incentive/honoraria (e.g., monetary, non-monetary)
- the nature of the incentive (e.g., for monetary, prize, points, donations, direct payments)
- the estimated dollar value of the incentives to be disbursed
- decide in advance who will receive the incentive (the parent or responsible adult [guardian, etc.], the respondent, or if both will receive an incentive)
- the parent or responsible adult must agree to the incentive, regardless of who is receiving the incentive
- ethical considerations should be taken into account when providing incentives to children, youth, or vulnerable groups (e.g. payment is not coercive, or exposes young or vulnerable persons to a risk that they would otherwise have avoided)
G1. There are currently no standards pertaining specifically to incentives when the respondents are children, young people or vulnerable individuals. Should there be any standards/guidelines on this topic? If so, what should these be? The Background gives a possible standard you might want to edit.
A data breach is the loss of or unauthorized access to/disclosure of personal or organizational information.
The current standards require taking steps to protect against data breaches. The objective for the panel is to identify any revisions or additions to the standards, and/or any guidelines that should be included.
The current standards are:
- 1) Protection against illegal or unsanctioned access: Researchers must use up-to-date technologies to protect survey data collected or stored on Web sites or servers against illegal or unsanctioned access by third parties (i.e. “hacking”). The researcher must also control access to all databases on which any data relating to the survey is stored so that only individuals with the appropriate security clearance are able to access the database, either by using a password or other form of access restriction (such as biometric controls).
- 3) The researcher must not subcontract (including to an affiliate) any function that involves providing a subcontractor with access to any data relating to the survey unless the client first consents in writing.
- 5) Protection against physical damage to servers: Researchers must also put in place measures to ensure the “physical” security of data and servers.
1) If the temporary storage of data collected takes place on a server that is operated by another provider, the researcher must place the provider under the obligation to take the necessary steps to ensure that the requirements described in subsection 13.2. are met. Temporary storage of the collected data on the server must be terminated at the earliest possible time.
1) In the event of any data breach, the client must be informed immediately and provided with details about both the nature and the extent of the data breach.
The literature review commissioned by PORD (see pp. 28-30) cites the following as a framework for considering data protection:
PORD would like to know whether this framework suggests any revisions to the current standards cited above.
A1.1 The background lists the current standards pertinent to protection against data breaches, and cites a data protection framework in the literature review commissioned by PORD.
Does the framework in the literature review suggest any revisions/additions to the current standards, or any guidelines, with respect to preventing data breaches? Or, are the existing standards sufficient? For example, should anything more be stated with respect to: destruction of information
Note: section 13.1.1/14.1.1 deals with retention of technical data but does not deal with either (a) destruction of data in general or (b) destruction of information in the event of a data breach.
encryption
Online and mobile methodologies create possibilities for collecting various types of personal data “passively”,that is without direct interaction with respondents. The issue to be considered is, in the context of surveys, what passive data collection is allowed and under what circumstances is it allowed?
The ESOMAR/GBRN Guideline on Mobile Research Footnote 53 states:
Mobile applications are capable of collecting a broad range of personal data without direct interaction with data subjects. Examples include web use and browsing history, app usage statistics, loyalty card data, geolocation, social media data, data from wearables and IoT and other data generated by or obtained from mobile devices.
At least some of these capabilities can also apply to online surveys where the device used by the respondent is a computer.
The Guideline notes an important exception to what constitutes “personal” data involving certain device characteristics:
While it is possible to passively detect the type of device a data subject is using, this is not personal data as long as the purpose is to optimize app performance and survey rendering.
The existing standards address relevant general principles of protection of respondent anonymity and confidentiality, and address passive data collection to some extent.
section 5.1.4, protection of anonymity and confidentiality, states the general principle: the anonymity of respondents must always be preserved unless they have given their informed and explicit consent to the contrary
1) The anonymity of respondents must always be preserved unless they have given their informed and explicit consent to the contrary. If these respondents have given informed consent for data to be passed on in a form which allows them to be personally identified, the researcher must ensure that the information will be used for research purposes only, OR, if requested by the respondent, to resolve a customer complaint. The same holds true when respondents’ answers are collated, with their informed consent, with pre-existing data that allows such identification (e.g., with an administrative data base). Moreover, such personally identifiable information must not be used for any purpose unrelated to the current study, such as direct marketing, list-building, credit rating, fund-raising, or any marketing activities directed at those individual respondents.
Section 5.1.4 establishes that any personal data collected passively must be handled in a way that protects the anonymity and confidentiality of the survey respondents.
section 5.3, privacy issues specific to online survey research, states a requirement to have an accessible policy statement “concerning the use of cookies, log files and, if applicable, software”
This ensures respondents are informed of certain types (but not all types) of passive data collection.
- 1) Researchers must have a readily accessible policy statement concerning the use of cookies, log files and, if applicable, software. This statement may be either included in their privacy policy or it may appear in a separate document. Software must not be installed on respondents’ computers without their knowledge or consent. In addition, respondents must be able to remove the researcher’s software easily from their machines (e.g., for Windows users, the software must appear in the Add/Remove Programs folder in their Control Panel).
- 2) Any links to data protection, privacy policy or cookie policy statements must be given at the start of the questionnaire.
section 7.2, data collection and recruitment techniques (online standards), addresses passive data collection with its reference to forbidding “surreptitious” or “unsolicited” data collection (terms which on the face of it would appear to refer to passive data collection generally)
The principle is that passive data collection can only be done with the respondent’s awareness and presumably consent (although the standard doesn’t say “consent”):
- 1) Researchers must not make use of surreptitious, misleading or unsolicited data collection or recruitment techniques (including using spambots, spiders, sniffers or other ‘agents’ that collect personal information without the respondent’s explicit awareness).
It may be that the above standards are sufficient to address issues associated with passive data collection in surveys, and the panel will be asked if this is the case.
Alternatively, it may be that the existing standards are not considered to be sufficiently explicit. To aid in panel consideration of this topic, we took a crack at reformulating standard 7.2:
- 1) Passive data collection refers to collecting personal information about respondents without direct interaction with them.
Researchers must not make use of data collection or recruitment techniques using passive data collection methods unless survey respondents or potential respondents have first given informed consent, or unless collecting the information is legally permissible or is permissible under the Terms of Use of the website, service or application from which the data are sourced.
Examples of passive data collection of personal information include, but are not limited to, web use and browsing history, app usage statistics, geolocation, personally identifiable biometric data, social media data, data from wearables and IoT (internet of things), and other data generated by and obtained from respondents’ mobile devices or computers.
Passive detection of the type of device a respondent is using is not personal data as long as the purpose is to optimize app performance and survey rendering.
Some things considered when formulating this possible revision to 7.2:
Also, does the proposed phrase, “website, service or application”, clearly include panels/communities, or does it need revision?
Note: by including panels (and perhaps MROCs) within the scope of “passive data collection of personal information”, should the proposed standard also be incorporated into the telephone standards? Most of the possibilities for passive data collection involve apply to online surveys. Are there passive data collection issues with telephone surveys?
For example, facial coding involves recording a data subject's face as he or she completes a survey or similar task. Eye tracking, virtual reality headsets and other wearable devices may be used in a similar way. (p.10)
To address this, the proposed revision to 7.2 includes “personally identifiable biometric data” (not all biometric data is personally identifiable; for example, an eye-tracking gaze pattern superimposed on the object being looked at does not identify the individual).
A2.1 Online and mobile methodologies create possibilities for collecting various types of personal data “passively”, that is without direct interaction with respondents.
Do the current standards, as outlined in the background, adequately address passive data collection of personal information done in conjunction with a survey?
To facilitate discussion, the background gives an alternative to Standard 7.2 data collection and recruitment techniques that deals with passive data collection in a different way. What do you think of this alternative? If you think something like this is needed, please revise as you see fit.
For ease of reference, the alternative version is:
Possible revision of 7.2. data collection and recruitment techniques
Researchers must not make use of data collection or recruitment techniques using passive data collection methods unless survey respondents or potential respondents have first given informed consent, or unless collecting the information is legally permissible or is permissible under the Terms of Use of the website, service or application from which the data are sourced.
Examples of passive data collection of personal information include, but are not limited to, web use and browsing history, app usage statistics, geolocation, personally identifiable biometric data, social media data, data from wearables and IoT (internet of things), and other data generated by and obtained from respondents’ mobile devices or computers.
Passive detection of the type of device a respondent is using is not personal data as long as the purpose is to optimize app performance and survey rendering.
The online and telephone survey standards do not currently have any standards pertaining specifically to respondent photographs, videos or audio recordings.
The ESOMAR/GBRN Guideline on Mobile Research section 3.4.2 states the following about photographs and recordings:
Photographs, video and audio recordings are considered to be personal data and therefore must be gathered, processed and stored as such. They can only be shared with a client if the data subject gives his or her prior consent with knowledge of the specific purpose for which it will be used. When potentially identifying information has been removed (such as through pixelisation or voice modification technology) so that it is no longer considered personal data it can be shared with a client provided the client agrees to make no attempt to identify the individual.
Researchers must not instruct data subjects (or those that may be acting as data collectors) to engage in surveillance of individuals or public places. Data subjects should be given specific limited tasks (e.g. capturing interactions with friends with their consent, or images of objects or displays) that do not involve monitoring a particular area where personal data would be captured without the consent of the individuals present. When recorded observation of a location is undertaken, clear and legible signs indicating that the area is under observation along with the contact details for the researcher or research organization performing the research should be posted and images of individuals must be pixelated or deleted as soon as possible. Cameras should be situated so that they monitor only the areas intended for observation. (pp. 10-11)
The following is a slightly revised version of the ESOMAR/GBRN guideline that emphasizes the survey context (as the panel is only considering revisions to the telephone and online survey standards):
Researchers must not instruct
A3.1 The background gives a standard pertaining to photographs, videos and recordings closely based on an ESOMAR/GBRN standard.
The online and telephone survey standards do not currently have any standards pertaining specifically to respondent photographs, videos or audio recordings. Do the online and telephone survey standards need a standard on this topic?
If a standard is needed, what do you think of the modified ESOMAR/GBRN standard in the Background? Please revise as you see fit.
PORD has requested the panel consider whether there needs to be an addition to the telephone standards related to sensitivity to the respondent’s setting. In this regard, there are 2 relevant guidelines in the ESOMAR/GBRN Guideline on Mobile Research (p. 7):
3.1.1 Safety: When calling mobile phones researchers may sometimes contact potential data subjects who are engaged in an activity or in a setting not normally encountered in fixed-line calling. This might include driving a vehicle, operating machinery or walking in a public space. The researcher should confirm whether the individual is in a situation where it is legal, safe and convenient to take the call. If the researcher does not receive confirmation, then the call should be terminated while allowing the possibility of making further attempts at another time.
3.1.2 Confidentiality and sensitive data: A researcher might contact a potential data subject who is engaged in an activity or situation where others may overhear the call. In this case, the researcher must consider the nature of the research content in light of the possibility that the data subject might be overheard and personal information or behaviour inadvertently disclosed or responses modified as a result of their situation. If appropriate, the call should be rescheduled to another time or location when confidentiality will not to be compromised.
Note that the ESOMAR/GBRN 3.1.1 above is specific to mobile phones, but 3.1.2 could apply to either mobile or fixed-location phones.
The current telephone standards, in section 5.2 #1 Avoidance of Harassment, has a standard focused on sensitivity of the survey subject matter, but it does not directly address issues caused by the setting of the interview:
1) The researcher must take all reasonable steps to ensure that respondents are not in any way hindered or embarrassed by any interview, and that they are not in any way adversely affected as a result of it. Researchers must address sensitive subject matter in a way that will minimize the discomfort and apprehension of both respondents and interviewers.
A4.1 Because respondents are increasingly likely to answer calls using a mobile phone, there can be issues with them using the phone in problematic settings (e.g. driving, walking in a public space). On both mobile phones and fixed-location phones, they may be in a setting where they can be overheard.
Does there need to be a standard (or guideline) for the interviewer to confirm the respondent is in a location where they are comfortable taking the call? Or, is it reasonable that it is the responsibility of the respondent to raise this if they have an issue, and not have a standard (guideline)?
If a standard or guideline is needed, how would you word it?
A starting point for language: the interviewer must (should?) confirm with a respondent that the respondent is in a location where they are comfortable doing the interview.
The current standards require that survey data must be stored in Canada:
- 2) Protection against legally-sanctioned access: because some jurisdictions allow their authorities, under certain circumstances, to access all data stored on servers located in that jurisdiction (e.g., in the United States under provisions of the Uniting and Strengthening America by Providing Appropriate Tools Required to Intercept and Obstruct Terrorism Act, known as USA PATRIOT Act: see the Treasury Board Secretariat ’s overview ), the researcher must ensure that all the databases containing any information related to the survey are stored on servers and back- up servers located solely in Canada.
- If the client has first consented in writing, these servers may be located in another country where:
- equivalent protections are given to personal information as in Canada under legislation such as the Privacy Act, R.S. 1985, c. P-21, and the Personal Information Protection and Electronic Documents Act, S.C. 2000, c. 5, and under any applicable policies of the Government of Canada
- the laws do not allow the government of that country or any other entity or person to seek or obtain the right to view or copy any information relating to the survey without first obtaining the client’s written consent
- In connection with giving consent to locating a database in another country, the client may, at their option, require the researcher to provide a legal opinion (from a lawyer qualified in the foreign country) that the laws in that country meet the requirements of paragraph 2) a), above, or may require the researcher to pay for the Government of Canada to obtain such a legal opinion. The Government of Canada has the right to reject any request to store survey data in a country other than Canada if there is any reason to be concerned about the security, privacy, or integrity of the data. The Government of Canada may also require that any data sent or processed outside of Canada be encrypted with Government of Canada-approved cryptography and that the private key required to decrypt the data be kept in Canada in accordance with key management and storage processes approved by the Government of Canada.
- The researcher must ensure that all servers, including back-up servers, on which any data relating to the survey is stored are physically and logically independent (meaning there is no direct or indirect connection of any kind) from all other databases, unless those databases are located in Canada (or in another country approved by the client under paragraph 2) a)) and otherwise meet the requirements of this section.
- The researcher must ensure that all data relating to the survey is processed only in Canada or in another country approved by the client under paragraph 2) a).
- The researcher must ensure that all domestic network traffic (meaning traffic or transmissions initiated in one part of Canada to a destination or individual located in another part of Canada) is routed exclusively through Canada, unless the client has first consented in writing to an alternate route. The client will only consider requests to route domestic traffic through another country that meets the requirements of paragraph 2) a).
A5.1 The current standards require that survey data must be stored in Canada. PORD would like to know if the panel thinks any other standards are required with respect to cloud-based storage, either in terms of location of servers/back-up servers or any other aspects of data security specific to cloud-based storage. Any suggestions?
The online and telephone standards do not contain any standards or guidelines pertaining to accessibility or literacy.
The “B” series of questions address several aspects of this topic. We suggest you look over the whole series of questions before answering the question in this section (B1).
PORD refers to the Web Accessibility Initiative (WAI) home page for definitions of accessibility, usability and inclusion:
PORD has asked the panel to consider: [does the panel] see a way of condensing the discussion around accessibility, usability and inclusion with concepts of literacy […]?
B1.1 Should a statement be put into the standards about the importance of the principles of accessibility (including literacy considerations, usability and inclusion)?
If yes:
The current online standards do not contain any standards or guidelines pertaining to the accessibility of online surveys to people with disabilities or to other people who may have difficulty completing an online survey.
PORD has communicated to us that: PORD has been advised that Treasury Board Secretariat is working on a proposed policy for accessibility standards specific to all devices used to access online surveys. The results of this development work will probably be a year in the making.
When the Treasury Board Secretariat policy is finalized, it will take precedence.
Note that the panel has previously considered the question of whether there should be guidelines for mobile-friendly questionnaires. Depending on Panel decisions there, this could address some accessibility issues.
On pages 25-28 of the literature review commissioned by PORD is a section on accessibility. In this section, there are 3 lists of guidelines from various sources. In the literature review, they are associated with making mobile surveys accessible, but you will see that many of the items pertain to online surveys generally.
The 3 lists are as follows (see the literature review for full context):
Page 26: Web Content Accessibility Guidelines
Pages 26-27: W3C’s Web Accessibility Initiative “that could apply to survey research”
Page 27 “commercial tools” list
The question to the panel is whether any of the above items suggest standards or guidelines to include in the update of the online standards. Some thoughts about how to approach this question:
B2.1 There are currently no accessibility standards in the online standards. The background lists various accessibility-related practices listed in the literature review commissioned by PORD, and some factors to consider when evaluating these items. Are there any that you think should be incorporated into the online standards update? Please explain your selections and any suggested wording modifications.
The current online standards do not contain any standards or guidelines pertaining to literacy.
The literature review commission by PORD states the following about literacy and survey accessibility:
pp. 27-28: usability research provides some guidance on how to design online surveys that are accessible to people with low literacy skills. For example, make questions short in length and use plain language (i.e., avoid the use of jargon, 3 or more nouns in a row, the passive voice, and verb-noun phrases). This is a best practice for survey questionnaire design in general, but one that is particularly salient when the target population has low literacy skills.
In addition to language, online surveys allow survey researchers to use design elements to help the survey respondent understand and complete the survey. For example, the following style and design conventions can be helpful: using a larger font size, having a clean, simple design (one that is not encumbered by too many colours or graphics), and using bolded font to draw attention to “working” words that convey the meaning of the question or instructions that are important for a respondent (and facilitate his/her reading of the information).
p. 28: finally, when it comes to literacy, questionnaires should use plain language to the extent possible and warranted (for a survey of the general public this would be imperative, but for a survey of a specialized population, this may not be possible or desirable depending on the subject matter).
In a footnote, the literature review refers to The Canadian Style Guide, section 13, plain language, which gives a variety of guidelines.
B3.1 The current online standards do not contain any standards or guidelines pertaining to literacy. The literature review commissioned by PORD recommends the following:
When it comes to literacy, questionnaires should use plain language to the extent possible and warranted.
Should there be a literacy standard or guideline in the online standards? If so, what do you suggest it be, and is it a standard, or a guideline (literature review recommendation is a guideline). And, should there be examples of “plain language”, or not? Or, a reference to The Canadian Style Guide on Plain Language?
The current telephone standards do not contain any standards or guidelines pertaining to the accessibility of telephone surveys to people with disabilities or to other people who may have difficulty completing a telephone survey.
For this section, please scan MRIA’s Appendix C: Guideline on Social Media Research. Note that the panel focus is limited to usage of social media only in connection with conducting online or telephone surveys, and that also meet PORD’s definition of public opinion research Footnote 54. Under this definition (1) there must be attitudinal/opinion questions in the research, and (2) the research must be based on asking questions. So, research consisting of web scraping is not considered to be public opinion research.
The focus of the advisory panel is on revising and updating the standards for telephone surveys and online surveys.
The Introduction to the MRIA’s Appendix C: Guideline on Social Media Research states:
The concept of consumers generating their own content on the internet has become ubiquitous. This has created new opportunities for researchers to observe, interact and gather information. Many techniques have been developed to leverage social media such as community panels, crowd-sourcing, co-creation, netnography, blog mining and web scraping. It is likely that many more will evolve over the coming years as the Internet continues to change.
Many of the research possibilities referred to above fall outside the scope of PORD’s online and telephone standards, because the activities do not qualify as public opinion research surveys. However, particularly in the case of market research online communities (MROCs), it is possible to do an online survey and perhaps even a telephone survey. And, it is possible that other types of social media venues can be platforms for sampling and administering surveys.
If an online or telephone survey is done using a social media venue as the sample source, and perhaps additionally as the medium for administering an online survey, the research project would have to conform to all of the relevant standards, that is, the online survey standards or the telephone survey standards.
The question is whether anything needs to be added to the standards to cover online or telephone surveys that make use of social media (meaning, the sample is sourced from a social media, and the additional possibility that the survey is administered via the social media venue).
One perspective is that nothing really needs to be added to the standards for social media-based surveys. The standards lay out requirements for proposal documentation, questionnaire design, sampling, retaining public confidence, data collection, data security, etc. Adhering to these standards could be considered sufficient for an acceptable social media-based survey conducted for the Government of Canada.
A different perspective is that there are arguably some issues specific to social media-based surveys that are not clearly covered by the standards. The MRIA’s Code of Conduct includes Appendix C: Guideline on Social Media Research, which is based on ESOMAR’s guideline. This code of conduct is intended to cover social media research broadly, not just surveys, but surveys are within its scope (MROCs are of particular interest in this context).
The MRIA guideline describes “key principles” for researchers. These principles are consistent with the standards, but are explained specifically in terms of application to social media:
Also of particular interest for the survey standards is section 3, recommendations for specific social media, and within this, section 3.2 private social media area issues and section 3.3 market research social media area issues. These address aspects of permission, informed consent and privacy specific to these types of social media venues used for research.
C1. It would be possible to add to the standards a requirement to adhere to the MRIA’s Appendix C: Guideline on Social Media Research in cases where a survey sample is sourced from a social media venue, and perhaps additionally fielded on a social media venue.
For example, in section 5 retaining public confidence, an item could be added to section 5.1 respondent rights, such as:
5.1.6. Surveys using social media as the sample source or medium of survey administration
Do you think it is necessary or useful to add something like the above to the standards, or not? If yes, do you have any suggestions for revisions to the above proposed wording?
C2. The Government of Canada holds itself to high standards for the conduct of public opinion research. In this context, adherence to the MRIA Guideline on Social Media Research would be a minimum requirement. Is the MRIA social media guideline sufficient as it pertains to online or telephone public opinion research surveys, or is there some part of it that you think should be made stronger for Government of Canada surveys that make use of a social media venue?
Antoun, C., Couper, M., Conrad, F. (2017). Effects of mobile versus PC web on survey response quality. Public Opinion Quarterly, Vol 81, Special Issue, 2017, pp. 280-306