POR Registration Number: 007-14
PWGSC Contract Number: G9292-150494/001/CY
Contract Award Date: May 22, 2014
Delivery Date: June 5, 2014

Employment and Social Development Canada
Canadians’ Views of the Temporary Foreign Worker Program

Methodology Report

Prepared by:
Harris/Decima

Prepared for:
Employment and Social Development Canada

Ce rapport est aussi disponible en français sur demande

For more information on this report, please contact: nc-por-rop-gd@hrsdc-rhdcc.gc.ca

Executive Summary

Harris/Decima is pleased to present this methodology report to Employment and Social Development Canada (ESDC) highlighting the public opinion research conducted with Canadians about the Temporary Foreign Worker Program (TFWP).

The department had identified the need to conduct public opinion research to understand awareness and opinions about the TFWP.  The results will be used to help inform policy making.

To meet the research objectives, a telephone survey using both landline and cell sample was conducted with 1,984 Canadian adults, 18+ years. The research was conducted between May 23 and May 25, 2014 (in English and French) and the survey took an average of 12 minutes to complete. A sample of this size yields a margin of error of +/-2.2%, 19 times out of 20. A detailed description of the survey methodology used to complete this research, including sample design, survey administration, and response rates (along with margin of error), is outlined in the methodological report. This report contains all the details necessary to replicate this study in the future.

The total cost of the research was $95,102.57 (including HST).

I hereby certify as Senior Officer of Harris/Decima that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research. Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate or ratings of the performance of a political party or its leaders.

Doug Anderson, Senior Vice President, Harris/Decima
(613) 751-5052
danderson@harrisdecima.com

Survey Methodology

Overview of Methodology

This research consisted of a telephone survey with the Canadian adult general population. Specifically, 1,984 Canadians were interviewed by telephone using a Random Digit Dialing (RDD) approach and therefore utilized probability sampling. A sample of this size drawn from the Canadian population would be expected to provide results accurate to within plus or minus 2.2 percent in 19 out of 20 samples. A dual 80/20 landline and cell phone sampling approach was used.

The sampling plan was designed to obtain a distribution reflective of the general population with overall quotas placed on gender, age and region. 

Details regarding the approach used for completing this research are outlined below.

Questionnaire Design

Harris/Decima reviewed the questionnaire provided by ESDC. The overall length of the survey was 12 minutes and included one open-ended question.

Survey Pre-tests

Prior to being finalized, the telephone survey was pre-tested on May 22, 2014 in both official languages to ensure it elicited the required information. In total, 10 interviews were conducted in English and 18 interviews were conducted in French. On average, the survey took 12 minutes to complete. All calling was completed from Harris/Decima’s Montreal call-centre. 

Following the pretest, the data was reviewed by checking frequencies and skip logic to ensure the survey instrument was programmed properly. Minimal revisions were made to the questionnaire after the pretest. Frequencies were monitored closely after full field began to ensure that no issues were encountered with the new programming.

Sample Design and Selection

The sample for this survey was designed to complete 2,000 interviews with Canadians. The sample was stratified by region, with overall quotas set for gender and age, to allow for meaningful sub-group analysis and to ensure that weighting factors were within the acceptable research standards.

The landline sample was drawn using ASDE’s SurveySampler technology, which uses random digit dialing (RDD) to ensure that all residential listings in Canadian provinces have an opportunity to be selected for inclusion in the survey. For mobile phone sample, we currently purchase lists targeted for cell phone exchanges (the first 3-digits after the area code).  This type of sample is the equivalent of RDD, as it is randomly pulled from dedicated cell phone exchanges.

Survey Administration

The telephone survey was conducted with 1,984 respondents in English or French using computer-assisted-telephone-interviewing (CATI) technology, from Harris/Decima's facilities in Montreal. The survey was completed between May 23 and May 25, 2014. The average length of time required to complete the survey was 12 minutes. All interviewing was conducted by fully trained and supervised interviewers, and a minimum of 5 percent of all completed interviews were independently monitored and validated in real time (with 75% of the survey monitored to count towards the 5%). 

Harris/Decima informed all survey participants of the general purpose of the research, identified both the sponsor (Government of Canada) and the research supplier, and informed participants that their participation in the study was voluntary and completely confidential. Furthermore, the survey was registered with the National Survey Registration System.

Harris/Decima used Confirmit’s Horizons CATI program for data collection. The software provided complete control over entry flow, including skips, valid ranges, and logical error-trapping. The Horizons system imported sample directly from databases – meaning there was no need for re-entry and therefore no entry errors. Moreover, the system automated all scheduling and call-back tasks, ensuring that every appointment was set within project time limitations and that an interviewer was available for every call-back.

Sample Distribution

A sample of 1,984 drawn from the Canadian population would be expected to provide results accurate to within plus or minus 2.2 percent in 95 out of 100 samples, as presented below:

Sample Disposition and Response Rate

Two sample sources, landline and cell phone, were used to complete the general population survey. A total of 134,782 Canadian households were dialed for the landline sample, of which 1,634 qualified as eligible and completed the survey (adults 18 years and older). The overall response rate achieved for this sample was 1.42%. For the cell phone sample (exchange dedicated to cell phones), a total of 46,955 numbers were dialed. Of these, a total of 383 respondents qualified and completed the survey. The response rate reflects the compressed data collection timeframe.

The following report on sample disposition and response rate follows MRIA guidelines, which are set up to establish consistency in reporting across the market research industry.

Non-response bias

The calculated response rate of this survey was 1.42% for landline and 0.92% for cell phone, which is expected for a telephone survey conducted in a condensed field window of three days. In order to maximize the response rate while undertaking the study within the constraints of field time, sample size and budget, the following steps were taken:

1. Callback scheduling was varied to maximize the possibility of finding someone at home; and
2. Flexible callbacks and appointments were offered to respondents so they could respond to the survey at their most convenient time. Daytime interviewing was scheduled to pick up any appointments that were made for daytime hours.

Response rates for telephone surveys in Canada and elsewhere have been steadily declining for many years and the trend appears to be continuing.  Research has thus far indicated that response rates are a poor indicator of survey quality, yet there remains a valid concern that the universe of individuals ultimately providing responses has an increasing chance of being different from those who are not included in the final dataset.  Fundamentally, once a household’s phone number is drawn into the sample frame, there are only three ways that the number ends up as a non-response:

• The phone number is not attempted at a time when the potential respondent is available;
• The survey sample is completed before the phone number needs to be attempted or re-attempted; or
• The respondent chooses not to answer or participate.

By implementing the callback measures described above, the risk of failing to provide a viable opportunity for an interview is mitigated. 

However, the concern remains that the high proportion of households that are ultimately non-participants in a study may be different from the survey sample in a way that influences the results of the survey.

In order to investigate whether non-response bias may be having an impact on the results, two forms of tests have been applied:

Comparing Sample Profile to Universe Profile.  Using Statistics Canada data from the 2006 and 2011 Census as the factual description of the universe being sampled, the demographic characteristics of the weighted final sample were examined in order to identify any differences and, where any may exist, to provide PCO with the ability to examine whether these had a statistically significant impact on the findings.

Comparison of Early and Late Responders.  Using the information on the specific call attempt which resulted in the completed interview, an analysis was undertaken to investigate whether those who responded on the first attempt differed from those who responded only after at least one callback attempt. The callback strategy is specifically implemented to mitigate the risk that non-response is caused by an insufficient sampling attempt.  This is built upon the logical hypothesis that those who require multiple attempts in order to be a respondent may be different from those who respond immediately and therefore may be at least somewhat similar to non-responders.  At the very least, it is clear that if multiple attempts had not been made to contact these households, the respondent would have been considered a non-responder.  Therefore, an analysis was undertaken to identify any differences and, where any may exist, examine whether these had a statistically significant impact on the findings.

Comparing Sample Profile to Universe Profile

The profile of the final sample (both weighted and unweighted) of Canadians was compared to the available population data. As is typically found with telephone surveys in Canada, the final sample over-represents those with higher levels of education.

Using this information, ESDC will be able to compare education groups and identify significant differences, if any exist.

Comparison of Early and Late Responders

A comparison of “early” and “late” responders to the survey was undertaken. Early responders are those who completed the survey upon first contact; late responders required two or more callbacks in order to secure their participation. A higher proportion of early responders were from Quebec and Ontario than among late responders. Conversely, a higher proportion of late responders were from the western provinces. This is due to the combination of the very short field period for this project, the higher number of completes targeted in Ontario and Quebec, and the time differences between the provinces. In order to approach the stated quotas the field team focussed their initial efforts on completing enough interviews in Quebec and Ontario to make the targets achievable. Additionally, a higher proportion of late responders had an income of $80000 or more. ESDC can compare the statistically significant differences between the groups identified and determine whether the differences would have made an impact on the overall analysis.

Non-Response Bias Data

The following table presents a profile of the final weighted and unweighted sample and how it compares to the Canadian population (18 years and over) on measured regional and demographic characteristics, based on the most recent (2011) census figures.

Conclusion

Harris/Decima has provided ESDC with a discussion of the non-response bias. ESDC will further investigate whether the non-response to this survey has affected the final weighted sample to the extent that different conclusions would have been drawn from this study.

Data Analysis

Upon completion of data collection, Harris/Decima cleaned, coded, and weighted the data.  As requested by ESDC, a weighted data file and a set of cross-tabulation banners were provided.  Our data analysis procedures are outlined below:

Data Validity and Integrity Checks: Our custom system immediately identifies cases where the interview length is unrealistically short, contradicts established facts or presents patterns of response deserving attention.  As a result, we can determine whether a case should be excluded from the final sample if necessary.  All of these checks are performed manually and cleaned out of the data in the back end of the project. Harris/Decima uses a checklist to ensure all data that is delivered to the client has gone through a rigorous quality control process.

Data Cleaning: Harris/Decima analysts have considerable experience in cleaning data files, conducting statistical routines, producing tabular output, and weighting data to provide an accurate measure of the population as a whole.

The following are the basic steps taken when cleaning data files:

1. Ensure that all coded questions have updated codes and multiple mentions do not have duplicate codes;
2. Create all new variables as a result of programming;
3. Confirm that all relevant variables are included in the data file;
4. Final frequency check (for out-of-range values) and recodes created, including those for outliers;
5. Verify that variable names and question numbers match the final version of the questionnaire; and
6. Create and verify new variable creations (against source variables) as outlined in the analysis plan and perform spell check on all variables.

In addition to these generic rules, project specific requirements are also taken into account.  It is also noteworthy that because the CATI software controls the questionnaire flow and data entry, data are typically quite clean from the outset.

Coding Procedures: The following details our coding procedures, which were performed on this study. The coding department takes the verbatim responses and creates a numeric code list of common answers.  Our head coder, in close conjunction with the consulting team, collapses lists of responses to open-ended variables into categories. A single coder is used to maximize consistency on this task. The rough frequencies obtained from this exercise are used to develop a code list. Once final approval is granted, the code list is annotated with specific examples so that accurate coding is assured.

The annotated code list is provided to our coding team, which attaches codes directly to the electronic coding file. This exercise can also be performed in a two-pass format, by two different coders. The head coder reconciles inconsistencies, guaranteeing consistent and accurate reporting of open-ended responses.  In general, Harris/Decima aims for less than 10% of responses remaining under a ‘other specify’ code category, creating codes for any mentions that add up to 1% or more of total responses.  The resulting data file is exported to the statistical package to quantify the responses for statistical analysis. The generated code lists are submitted to the client for approval and subsequently we use our internal quality assurance lists to verify that all approved codes have been coded correctly.

Weighting: At the conclusion of the data collection and cleaning, Harris/Decima weighted the data by each stratum (in this case, region, age, gender, and cell phone ownership) to reflect the actual proportions found in the population based on 2011 Census data.  This ensured the findings from the research can be extrapolated to the entire population with accuracy.  Harris/Decima uses a standard procedure for calculating weighting factors, based on established methodological standards and extensive experience in sample weighting over literally hundreds of projects (including many for the Government of Canada). 

This procedure involves calculating the actual population within each segment and the true proportion of the sample that would fall into each segment if the survey were conducted on strictly a random basis. Into this number is divided the actual segment sub-sample to produce a weighting factor that is then used to “weight” the data for that segment. While there are various ways of accomplishing this task, this procedure is the most straightforward and effective.

The stratum selected for the project were as follows:

1. Region (Atlantic, Quebec, Ontario, Manitoba/Saskatchewan, Alberta and British Columbia/Territories);
2. Gender (male and female);
3. Age (18 to 34, 35 to 54, and 55 plus); and
4. Cell phone (cell only, landline only, cell and landline).

The weights applied to the final data set are outlined in the table below. The highest weight used to bring the sample in line with demographic proportions seen in the general population was 2.92.

Data Analysis: Harris/Decima prepared an analysis plan that included key banner breaks as required.   Once the survey data had been collected and cleaned, Harris/Decima ran a series of data tables that provided results for all questions in the survey, both overall and broken down by selected “banners.” This permitted the comparison of results from various sub-group segments of interest; statistical significance testing at the 90% and 95% confidence level was shown between all banner points in the data tables. The analysis plan included banners for the key segments including region, age, gender, income, and education.

Appendix A – Questionnaire

Hello/Bonjour (pause), the Government of Canada is conducting a research survey on current issues of interest to Canadians. Would you prefer that I continue in English or French? Préférez-vous continuer en français ou en anglais?

My name is ___________ of Harris/Decima, the company hired to do the survey. The survey takes about 10 minutes to complete. It is registered with the Marketing Research and Intelligence Association. Your participation is voluntary and completely confidential. Your answers will remain anonymous.

• Would you be willing to take part in this survey? We can do it now or at a time more convenient for you.

[IF ASKED] The survey will take about 10 minutes to complete.

• Yes, now => CONTINUE
• Yes, call later => SCHEDULE CALLBACK DATE AND TIME
• REFUSED => THANK AND TERMINATE

Have I reached you on a cellular phone or a traditional telephone line?

• Cellular phone => CONTINUE to c)
• Traditional phone => SKIP to d)
• REFUSED => THANK AND TERMINATE

[if b=cellular phone, ask] Are you in a safe place to answer a survey?

• Yes => CONTINUE
• No => SCHEDULE CALLBACK DATE AND TIME
• REFUSED => THANK AND TERMINATE

Are you 18 years of age or older?

• Yes => CONTINUE
• No => THANK AND TERMINATE
• REFUSED => THANK AND TERMINATE

1. [if b= cellular phone, ask] At home, do you have a traditional telephone line other than a cell phone?

• Yes => CONTINUE [CODE AS “CELL/LANDLINE’]
• No => CONTINUE [CODE AS “CELL ONLY”]
• REFUSED => THANK AND TERMINATE

e) 2. [if b= landline, ask] Do you have a cell phone?

• Yes => CONTINUE [CODE AS “CELL/LANDLINE”]
• No => CONTINUE [CODE AS “LANDLINE ONLY”]
• REFUSED => THANK AND TERMINATE

In which province or territory do you live?

• Newfoundland 1
• Prince Edward Island 2
• Nova Scotia 3
• New Brunswick 4
• Quebec 5
• Ontario 6
• Manitoba 7
• Saskatchewan 8
• Alberta 9
• British Columbia 10
• Yukon 11
• Northwest Territories 12
• Nunavut 13

DO NOT READ – RECORD GENDER

D3:
In what year were you born?

• RECORD YEAR: __ __ __ __
• REFUSED [DO NOT READ]

[IF PREFERS NOT TO PROVIDE A PRECISE BIRTH YEAR, ASK:]

Would you be willing to tell me in which of the following age categories you belong?

• 18 to 34
• 35 to 49
• 50 to 54
• 55 to 64 OR 65 or older?
• Refused [DO NOT READ]

CORE QUESTIONS

ASK ALL RESPONDENTS

Now, I would like to ask you about Canada’s Temporary Foreign Worker Program.

Q2:

How familiar are you with the Temporary Foreign Worker Program? Would you say you are very familiar, somewhat familiar, not very familiar or not at all familiar with Canada’s Temporary Foreign Program?

• Very familiar
• Somewhat familiar
• Not very familiar
• Not at all familiar
• Never heard of it [DO NOT READ]
• Refused [DO NOT READ]

Q3:

As you may know, Canada’s Temporary Foreign Worker Program is a federal government program that allows employers to hire temporary workers from other countries for jobs that qualified Canadians don’t apply for.

From what you have seen, read or heard about the Temporary Foreign Worker Program, do you strongly support, somewhat support, somewhat oppose or strongly oppose the Temporary Foreign worker program?

• Strongly support
• Somewhat support
• Somewhat oppose
• Strongly oppose
• Don’t know [DO NOT READ]
• Refused [DO NOT READ]

Q4:

What is the main reason you [SUPPORT/OPPOSE – EXCL SOMEWHAT LANGUAGE] the Temporary

Foreign Worker program? [CAPTURE FIRST MENTION]

[NO PRE-CODED LIST - INTERVIEWER NOTE: TOP ANSWER MUST BE RECORDED FIRST]

• Don’t know [DO NOT READ]
• Refused [DO NOT READ]

Q5:

[AMONG THOSE OPPOSED TO THE PROGRAM IN Q3] Do you think the Temporary Foreign Worker Program should be reformed or just abolished all together?

• Reformed
• Abolished
• Don’t know [DO NOT READ]
• Refused [DO NOT READ]

Q6:

To the best of your knowledge, what percentage of all people working in Canada do you think are temporary foreign workers?

[RECORD PERCENTAGE]

• Don’t know [DO NOT READ]
• Refused [DO NOT READ]

Q7a:

In your opinion, do you think that some employers abuse the Temporary Foreign Worker Program by not doing enough to recruit Canadians for available jobs before hiring workers from others countries?

• Yes
• No
• Don’t know [DO NOT READ]
• Refused [DO NOT READ]

Q7b:

[ASK IF Q7a=yes] How often do you think this occurs:

• Never occurs
• Rarely occurs
• Occasionally occurs
• Frequently occurs
• Occurs all the time

• Don’t know [DO NOT READ]
• Refused [DO NOT READ]

Q8:

Now, I'd like to know how strongly you agree or disagree with the following statements. Please use a 10-point scale, where 10 means you strongly agree, and 1, means you strongly disagree.  [RANDOMIZE STATEMENTS]

1. There is a shortage of workers in certain regions in Canada/There is a shortage of workers for certain industries in Canada [SPLIT SAMPLE] 
2. There are some jobs that Canadians aren't willing to do these days.
3. Employers should be allowed to bring in workers from other countries if qualified Canadians don't apply for certain jobs
4. If employers have a hard time finding Canadians to fill certain jobs, they should raise wages for Canadians before being allowed to bring in workers from other countries
5. There is no need for temporary foreign workers in Canada / There is no need for temporary foreign workers in low wage jobs in Canada [SPLIT SAMPLE]

• Don’t know [DO NOT READ]
• Refused [DO NOT READ]

Q9:

Please indicate your level of agreement as to whether employers in the following sectors may sometimes need to hire temporary foreign workers because qualified Canadians are not available. Please use a 10-point scale, where 10 means you strongly agree, and 1, means you strongly disagree [ROTATE ITEMS]

1. Basic agricultural work on farms
2. Information technology firms who need people with specialized hi-tech skills
3. Major construction projects in remote areas such as mining and energy projects
4. The restaurant industry
5. Hotels and retail sector
6. Food processing employers such as meat packing plants

• Don’t know [DO NOT READ]
• Refused [DO NOT READ]

Q10:

Now, I'd like to know how strongly you agree or disagree with the following measures that could be applied to the Temporary Foreign Worker Program. Please use a 10-point scale, where 10 means you strongly agree, and 1, means you strongly disagree.  [RANDOMIZE STATEMENTS]

1. Putting a limit on the number of temporary foreign workers that any business can hire
2. Significantly increase the fees businesses pay to bring Temporary Foreign Workers to Canada
3. Stiffer fines and penalties for employers who break the rules of the temporary foreign program
4. Banning employers from hiring Temporary Foreign Workers in low-skill, low-wage jobs

• Don’t know [DO NOT READ]
• Refused [DO NOT READ]

[SPLIT SAMPLE – HALF Q11A HALF Q11B]

Q11a:

Canada has youth exchange agreements with countries such as Australia and Ireland that allow a certain number of their youth to live and work in Canada for about a year, and which in turn allow young Canadians to live and work overseas. Generally speaking, do you support or oppose such youth exchange programs? Please use a 10-point scale, where 10 means you strongly support, and 1, means you strongly oppose.

• Don’t know [DO NOT READ]
• Refused [DO NOT READ]

Q11b:

As you may know, each year Canada admits approximately 250,000 new immigrants who are allowed to stay permanently and apply for citizenship. In addition to admitting new immigrants, Canada also admits on average 200,000 temporary foreign workers each year who eventually return to their home countries.

 In your view, would you support or oppose the Government of Canada, in addition to new immigrants, allowing temporary foreign workers to stay permanently and apply for citizenship?  Please use a 10-point scale, where 10 means you strongly support, and 1, means you strongly oppose.

• Don’t know [DO NOT READ]
• Refused [DO NOT READ]

DEMOGRAPHIC QUESTIONS

ASK ALL RESPONDENTS

Finally, I’d like to ask you some questions for statistical purposes. I'd like to remind you that all your answers are completely confidential.

D1:

Which of the following categories best describes your current employment status? Are you...? [READ LIST, ACCEPT ONE RESPONSE]

• Working full-time (35 or more hours per week)
• Working part-time (less than 35 hours per week)
• Self-employed
• Student attending full time school (not working)
• Unemployed, but looking for work
• Not in the workforce (e.g. unemployed, but not looking for work, a full-time homemaker or parent)
• Retired
• Other
• REFUSED [DO NOT READ]

D2:

What is the highest level of formal education that you have completed to date? [READ LIST, ACCEPT ONE RESPONSE]

• Elementary school or less
• Secondary school
• Some post-secondary
• College, vocational or trade school
• Undergraduate university program
• Graduate or professional university program
• REFUSED [DO NOT READ]

D3:

Which of the following categories best describes your total household income? That is, the total income of all persons in your household, before taxes? [READ LIST, ACCEPT ONE RESPONSE]

• Under $20,000 
• $20,000 to just under $40,000 
• $40,000 to just under $60,000 
• $60,000 to just under $80,000 
• $80,000 to just under $100,000
• $100,000 to just under $120,000 
• $120,000 to just under $150,000
• $150,000 and above 
• [DO NOT READ] REFUSED

D4:

Were you born in Canada?

• Yes
• No
• REFUSED [DO NOT READ]            

D5.

Are you either the owner of your business or a manager with responsibility for hiring and other human resource matters?

• Yes
• No
• REFUSED [DO NOT READ]

D6.

[If Business Owner or Manager with Responsibility for Hiring] Have you ever filled positions using Canada’s Temporary Foreign Workers Program?

• Yes
• No
• REFUSED [DO NOT READ]

That concludes the survey. This survey was conducted on behalf of Employment and Skills Development Canada. In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to participate, it is greatly appreciated.