Data Collection for the 2024 Canadian Financial Capability Survey (CFCS)

October 2024

Methodological Report

Contract Information

For more information on this report, please contact Financial Consumer Agency of Canada (FCAC) at: info@fcac-acfc.gc.ca

Ce rapport est aussi disponible en français.

This report presents the methodological details for the 2024 Canadian Financial Capability Survey conducted by Advanis Inc. on behalf of the Financial Consumer Agency of Canada (FCAC). The survey was administered among 7,963 members of the adult Canadian general public, between February 14 and March 22, 2024.

Ce rapport est aussi disponible en français sous le titre: Collecte de données pour l'Enquête canadienne sur les capacités financières de 2024

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from the Financial Consumer Agency of Canada (FCAC). For more information on this report, please contact: info@fcac-acfc.gc.ca

• Catalogue Number: FC5-42/2-2024E-PDF
• International Standard Book Number (ISBN): 978-0-660-73840-6

• Related publications:

  Catalogue number: FC5-42/2-2024F-PDF (Methodological Report, French)

  International Standard Book Number (ISBN): 978-0-660-73841-3

© His Majesty the King in right of Canada, as represented by the Financial Consumer Agency of Canada, 2024

1. Executive Summary

1.1 Background

The CFCS was first fielded in 2009, then in 2014 and 2019. Statistics Canada designed the survey to gather in-depth and comprehensive information about Canadians’ financial capability. The original CFCS, launched in 2009, conceptualized financial capability as consisting of “the skills and confidence [that allow one] to be aware of financial opportunities, to know where to go for help, to make informed choices, and to take effective action to improve their financial well-being.” In other words, the CFCS sheds light on how Canadians understand their financial situation, the financial services available to them and their plans for the future.

Statistics Canada developed the CFCS as a telephone survey that took, on average, more than 35 minutes to complete. In 2019, FCAC divided the CFCS into two parts: a core survey, which was administered to all respondents and took approximately 25 minutes to complete, and an extended ^[1] survey, which was administered a few weeks later to approximately 40% of the respondents and took 10 minutes to complete. This extended component was introduced primarily to cover new content on psychological factors, fraud, debt management, and financial well-being. A hybrid approach (online and telephone) was adopted in 2019 to adapt the survey methodology to technological advances (i.e., probability research panels). The new approach allowed respondents to complete the survey either online or by telephone. Further, FCAC augmented the sample frame of the 2019 CFCS with random digit dialling (RDD).

The CFCS is a critical tool to identify trends and emerging needs to support the development of evidence-based recommendations and guidance, and to inform policy discussions. The CFCS was last fielded in 2019. Since then, the economic landscape has shifted for both financial consumers and FCAC. For example, Canada’s inflation rate hit a 40-year high of 8.1% in June 2022. Consequently, the Bank of Canada, after slashing its benchmark lending rate in the early days of the pandemic, began an aggressive campaign of rate hikes in early 2022. Since then, Canadians have been faced with difficult economic conditions, which include higher-than-usual prices for necessities like food, transportation, and shelter, and higher mortgage costs for Canadian homeowners with a mortgage. Although inflation is down from its peak, it rose again in April 2023, largely due to the cost of gasoline and rent,^[2] causing economic hardships for Canadians with vulnerabilities.

Moreover, as outlined in the Financial Consumer Agency of Canada Business Plan 2022–2023 to 2024–2025 , ^[3] FCAC plans to increase the frequency of foundational data gathering. This involves transitioning from the historical 5-year cycle of the Canadian Financial Capability Survey to a more frequent schedule. FCAC has made considerable progress regarding its understanding of financial literacy. For instance, Make Change that counts: National Financial Literacy Strategy 2021–2026 highlights five key consumer building blocks to help consumers develop the skills, capacity, and behaviours that lead to financial resilience and are relevant for all Canadians, regardless of income, personal context, or circumstances:

1. Skills to navigate the financial marketplace;
2. Just-in-time knowledge and confidence;
3. Managing expenses;
4. Managing debt;
5. Managing saving.

Fielding the CFCS contributes to the ongoing evolution of financial literacy research. FCAC shares its data with researchers who want to use them to further their research. Collaboration with academics enhances the collective knowledge base and strengthens the impact of research outcomes on policy and financial well-being of consumers.

The intent of the 2024 CFCS is to continue to gather in-depth and comprehensive information about Canadians’ financial capability and financial well-being. This intent stems from the recognition of the crucial role that such information plays in shaping policy, guiding financial education initiatives, and addressing the evolving needs of Canadian consumers.

1.2 Objectives

The primary objective of this research is to gather in-depth and comprehensive information about Canadians’ financial capability. The financial capability was defined in the first CFCS in 2009 as “the skills and confidence [that allow one] to be aware of financial opportunities, to know where to go for help, to make informed choices, and to take effective action to improve their financial well-being.” The objective of the CFCS is to shed lights on how Canadians understand their financial situation, the financial services available to them and their plans for the future.

The 2024 CFCS aims to achieve the objectives described below.

i) Provide a reliable estimate of the level of financial literacy among Canadians that allows for comparability across time while strengthening linkages to FCAC business priorities and NSFL goals.

A key aim for the 2024 CFCS is to continue to collect information that provides a reliable measure of financial literacy in Canada—particularly related to financial knowledge, financial confidence and financial behaviours. The questions are comparable to previous waves of the CFCS, particularly for the key outcome measures. The 2024 CFCS includes measures of financial literacy most closely connected to FCAC business priorities and NFLS goals such as measures related to navigating the financial marketplace, financial knowledge and confidence, managing expenses, managing debt and managing savings, digital access and digital literacy.

ii) Enhance the quality and the efficiency of data collected to improve our understanding of issues related to financial literacy.

The 2024 CFCS aims to optimize the flow of the survey questionnaire and enhance question wording while ensuring comparability across time. Revisions include eliminating questions with low response rates, consolidating or simplifying response categories, or rewording questions to facilitate responses. The survey methodology prioritizes cost efficiency, data quality, response rates and coverage of the target population.

iii) Encourage the dissemination of information on financial literacy to strengthen the knowledge of financial consumers.

Finally, a key objective is to optimize the 2024 CFCS as a tool to disseminate relevant and useful information about financial consumers. This includes collecting useful data and reporting on it in a way that provides relevant and useful information to help better inform financial consumers about key issues related to current NFLS priorities and other emerging issues. FCAC plans to make the 2024 CFCS data readily available and accessible for use by other researchers. ^[4]

1.3 Methodology

Data collection started February 14, 2024, and ended March 22, 2024, and was conducted by Advanis. Advanis sought a probability-based sample of 7,963 Canadian aged 18 or older through the use of Advanis’ General Population Representative Sample (GPRS) sample and through Random digit dialling where phone numbers are randomly generated using a software system.

Data was collected using a multimodal approach, collecting survey responses online and on the phone, to obtain a nationally representative sample. First, participants were randomly pulled from our GPRS sample, recruited by phone and were invited to participate in a Web survey. Those who agreed to participate received an email or SMS inviting them to take part in the survey. In hard-to-reach populations, the survey was also offered by phone, using a Computer Assisted Telephone Interviewing (CATI) methodology.

Survey results were weighted by region, gender+, income, education and age group. The results for 2024 are based on responses from 7,963. Recruitment ensured quotas were reached for key sub-populations to ensure statistical power and representativeness.

Weighted results can be extrapolated to the broader population. Cross tabulations must align with the weighted categories to be extrapolated to the broader population. Failing to do so can create data distortions.

1.4 Contract value

The contract value for this survey was $259,763.33 (including HST).

1.5 Political Neutrality Certification

I hereby certify as a Senior Officer of Advanis that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Communications Policy of the Government of Canada and Procedures for Planning and Contracting Public Opinion Research.

Specifically, the deliverables do not contain any reference to electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leader.

Signed:

Nicolas Toutant

Vice-President, Research and Evaluation

ntoutant@advanis.net

2. Methodology

FCAC contracted Advanis to conduct the 2024 Canadian Financial Capability Survey with Canadian adults from the general public, between February 14 and March 22, 2024.

The project used Advanis’ proprietary General Population Random Sample (GPRS). This is a two-step approach where people who are part of our GPRS sample were recruited by telephone to participate in an online web survey.

The first step consists of using an IVR-to-Web and CATI-to-Web methodology to contact potential respondents. We used our proprietary interactive voice response (IVR) system and our in-house CATI call center to conduct random digit dialling (RDD) to recruit respondents to be part of the GPRS sample. For the second step, people who are part of our GPRS sample were recruited by telephone to participate in an online web survey.

Respondents from the GPRS sample were invited to participate in the online survey by either email or SMS (text message), based on their preference at the time of recruitment. After the initial invitation, if respondents had not yet completed the survey, they were sent up to two (2) reminder messages. Reminder messages were sent 3 and 6 days after the initial recruitment.

Some respondents were also contacted by using a random digit dialling (RDD) approach and completed the survey on the phone. This was done in order to make the survey available for harder to reach populations. For example, this includes individuals with lower income or lower education, which are underrepresented in online samples because of the required literacy skills. However, if they expressed the desire to complete the survey online, they were sent an email or SMS invitation.

These methods are probability-based; that is, every recruit has an equal and known chance of being invited.

There can be an unknown bias since not everyone agrees to participate in studies. The inherent potential bias of our GPRS sample is similar to other random sampling approaches.

2.1 Survey Design

The questions for this survey were designed by the FCAC and supplied to Advanis. Advanis was responsible for the French survey translation. The questionnaire contained questions inquiring about the financial capability of participants. The survey addressed different themes like day-to-day financial management (e.g., budgeting), long-term financial planning (e.g., planning for retirement), assets, debts, financial education, etc.

The Government of Canada’s standards for pre-testing were adhered to. The pretest was conducted in both English and French. The pretest was conducted from February 14 to 16, 2024. Advanis conducted surveys with 69 participants from across Canada (41 English and 28 French). The interviews of the pretest were not kept since many adjustments to the survey were required.

During the data collection phase, the phone survey length averaged 30.6 minutes (expected was 24) and the web survey averaged 15.5 minutes (expected was 16).

2.2 Sampling and Administration

The target population for this project was Canadian public aged 18 and over living in the 10 provinces and three territories. We used survey quotas for key socio-demographic groups to ensure that the sample reflects the general Canadian population as closely as possible. The targeted number of completed surveys was 8,000 Canadian adults. Key sub-population that required quotas were:

• Canadians with a high school diploma or less (resulting MOE from the data: 2%)
• Canadians with low income (60K or less) (resulting MOE from the data: 2%)
• Canadians under the age of 35 (resulting MOE from the data: 2%)
• Indigenous Canadians (resulting MOE from the data: 5%)
• Recent immigrants (MOE unavailable due to data availability limitation – 2021 census)

Canadians 18 years of age or older participated in the survey. Results from the final survey sample can be extrapolated to the broader general public of Canadians 18 and over, with a margin of 1.2 percent.

Respondents needed to be aged 18 and living in Canada to be eligible for the survey. As mentioned previously, participants were taken from our GPRS sample and others were reached through Random digit dialling (RDD). Participants from the GPRS sample were sent an email or SMS invitation. RDD participants completed the survey on the phone, but if they expressed the desire to complete the survey online, they were sent an email or SMS invitation.

To target Canadians with a high school diploma who are often more difficult to reach online (lower literacy level), prospective participants were reached on the phone (through RDD) and were asked if they wanted to complete the survey with an interviewer. Those who agreed to participate in the survey were interviewed with the use of Computer Assisted Telephone Interviewing (CATI) methodology. A total of 7,963 responses were obtained (6,347 in English and 1,616 in French) to reach quotas (Table 2). All quotas were achieved.

The overall response rate for the survey was 14.1 percent.

2.3 Web methodology

Invitations were sent by SMS/email and grouped by province, to ensure that they were sent out during appropriate hours within each time zone. Invitations were sent to a targeted sample that matched the target audience characteristics based on already compiled information in our GPRS sample. Overall, 24,166 people were recruited to the web survey, with 5,991 participating, for a response rate of 24.8%.

After sending the initial invitation, a reminder message was sent three days later to applicants who did not complete a survey. The majority of respondents were sent two reminder messages. Overall, 56,280 SMS and email messages were sent during data collection for this study.

Each survey had a unique number embedded in the hyperlink to eliminate the possibility of duplicate responses from any participant.

2.4 CATI methodology

Participants who completed the survey on the phone were reached between February 15, 2024 and March 22, 2024. Interviews were offered in French or English based on respondent preference. In total, 48,256 potential respondents were called and 2,840 agreed to participate in the survey. The response rate for the CATI portion of the study was 7.5%. Of those who agreed to participate, 898 were ineligible. In total, 1,972 respondents completed the survey on the phone. More detailed data collection statistics can be found in Appendix B.

2.5 Weighting and Data Cleaning

A direct weighting approach was conducted using five variables: region, gender+, income, education, and age. The population sizes are based on the latest Statistics Canada census results published for the 2021 census. This weight and its corresponding weighting scheme identifier are provided in the final clean data. For missing data and for gender diverse individuals, values were randomly imputed for weighting purposes. This applies to the gender and education variables used to compute the weights (not the original data collection variables, new variables were created for the weighting scheme to avoid modifying data collection variables and allow transparency). For the missing data of the income variable, data was imputed based on available survey responses when possible and randomized when no probable category was determined. The missing data were redistributed in a way that kept the same proportions as the sample. Detailed weight values can be found in Appendix A.

The database was cleaned to remove any errors at the end of the data-collection phase, and all unique identifiers in the client profiles were removed in the final data set provided to FCAC. The verbatim responses were reviewed, and no unique identifiers were found. In other words, no respondent entered personally identifiable information, meaning all survey responses are anonymous. For verbatim responses, if the verbatim response aligned with an existing question level the response was recoded to match the existing level. New levels were created when needed. When open-ended questions were recoded into existing categories and a survey filter was affected, the respondent was removed from the dataset. This was done to avoid any inconsistencies in base calculation.

2.6 Quality Control

Advanis employs several quality control measures to ensure success across the entire life cycle of the project. These measures were used on this study and are detailed below.

Survey Programming:

Advanis utilizes technology to maximize quality control in survey programming. Having developed a proprietary survey engine tool, Advanis professionals are able to design and program a survey in a browser-based environment, eliminating the need to involve a programmer who is less familiar with the survey subject matter. The survey was thoroughly pre-tested by Advanis’ project team members, as well as by non-team members (non-team members provide “fresh eyes” for catching potential errors).

CATI Methodology:

The CATI recruit script was programmed on Advanis’ proprietary CATI platform with no unforeseen challenges. Advanis was able to leverage its experience for the survey programming and the reminder process to achieve high quality standards. Advanis implemented the following to ensure the highest quality data collection:

• Trained the interviewers to best understand the survey’s objectives and to ensure that they were able to pronounce and understand the survey wording.
• Detailed call records were kept by the automated CATI system, and were monitored for productivity analysis (i.e., not subject to human error).
• The recruit scripts were pre-tested for best possible flow.
• Our average interviewer employment tenure is very high compared to industry standards, resulting in a team of interviewers who are more experienced and knowledgeable regarding the target audience.
• Advanis’ Quality Assurance team listened to the actual recordings of ten percent of completed surveys and compared the responses to those entered by the interviewer, to ensure that responses were properly recorded. This is in addition to the live monitoring done by field supervisors.
• Team Supervisors conduct regular, more formal evaluations with each interviewer, in addition to nightly monitoring of each interviewer on their team.

To ensure high interview quality, our interviewers are trained to use various interviewing techniques. As well as maintaining a professional attitude, our interviewers must also be convincing, read word-for-word, take notes, systematically confirm the information given and listen to the respondent. Advanis has also created internal tools within the survey script for interviewers allowing them to use the phonetic alphabet to confirm email address spelling (e.g., a for alpha, b for bravo, etc.) to help reduce the amount of bounced email addresses. However, should bounce emails occur, Advanis has also developed additional tools that allow for someone to re-listen to the recording and easily adjust to correct the email address.

Web Methodology:

The web survey for this study, like for all Advanis web surveys, was hosted internally by Advanis, and employed a rigorous and stringent set of data collection control mechanisms to ensure the highest quality for the data collected, including:

• Respondents had a unique access code to ensure that only that respondent can complete the online survey.
• Extensive internal logic checks were programmed directly into the survey to ensure logical responses.
• The web survey was implemented using Advanis’ proprietary software (which is designed to handle complicated survey formats).
• Advanis administered a detailed internal test and an external pretest to ensure that the survey instrument was working as planned.
• Advanis tested the questionnaire in multiple browsers and provided FCAC with a link so they could do internal testing.

Data Handling and Reporting:

For the data collected, Advanis develops rules to check the validity of the data. These rules were used for this study and include items such as:

• Time taken to complete the survey.
• Checking for verbatims that are gibberish or don’t make sense.
• Rigorous checks are completed to ensure the data is accurate and error-free according to the questionnaire logic (skip patterns).

All data cleaning performed on projects are outlined and tracked in an internal spec document so they can be QA’d and signed off on. The original raw data file is never overwritten, so that if an error is discovered in our code, we can quickly and easily rerun things to produce a new data file. Individuals developing code incorporate internal checks in their code (e.g., crosstabs) to ensure the adjustment had the desired effect. In addition, all recoding is reviewed by another team member or technical specialist for accuracy.

3. Non-response Bias and Limitations

Non-response bias occurs when non-responders differ in a meaningful way from respondents and this difference impacts the information gathered. It is difficult to assess the presence of non-response bias since information about why non-responders did not participate is usually unavailable. One way to gauge the potential impacts of non-response bias is to evaluate if the sample is representative by comparing the respondents' characteristics and gauge if they reflect known population characteristics. Where possible, we can check the distribution of respondents across various demographic (e.g., age and gender) and geographic categories and compare those distributions against known population characteristics. If the variation is fairly small and we have no reason to believe there are other factors impacting respondents’ willingness to participate, we can conclude that the likelihood of non-response bias impacting the information gathered in the survey is minimal. This is the case with the current survey. More educated individuals are more likely to participate in surveys, which leads to an underrepresentation of less educated, often lower-income groups.

Several strategies were employed to increase response rates and reduce the effects of non-response bias:

• We recruited respondents by telephone (cellular).
• We outpulsed a local phone number (rather than a toll-free number) which increases pick-up rates (reducing call screening).
• We systematically set the next call date and time based on the outcome of the current call, which ensured that each respondent was called methodically across days of the week and times of the day. Especially for respondents that were difficult to reach, this maximizes the likelihood of reaching them.
• We sent an SMS text message to recruits, which assures a seamless transition from the telephone survey to the online survey, as receipt can be confirmed in real time and encourages respondents to complete the survey as soon as the call ends. 
• We collected both email address and telephone number for recruitment so that if the email address bounced, we could contact them via SMS message if they agreed.
• We offered the survey in both official languages to maximize ease of completion.

4. Guidelines for Analysis and Release

When doing any analyses, it is important to align the analysis plan with the weighting scheme. The weights adjust the data to better reflect the population based on parameters that have been chosen to maximize the level of detail without creating distortions due to extreme weights (an extreme weight will occur when a population group is represented by a proportionally smaller subset of respondents compared to other population groups, thus introducing an important risk of bias due to their specific profile).

For this survey, the basic sociodemographic information that should be used in the analysis of results are:

• Regions (Atlantic, Québec, Ontario, Prairies or British Columbia and Territories)
• Gender+ (Men+ or women+)
• Income (under 60k or 60k or more)
• Education (High school or less, or More than high school)
• Age group (18 to 34 years old, 35 to 54 years old or 55 or more)

Using age groupings other than the ones described above could potentially produce distorted data. As these results could be inaccurate based on how the weights were calculated, we strongly advise not to report any results that are not aligned with these specified categories.

Any results with an unweighted base size (denominator) of less than 30 should not be reported, due to statistical robustness. ^[5]-[6] This is due to the increased coefficient of variation and, hence, there are larger confidence intervals around results with smaller bases. Furthermore, for confidentiality purposes, any estimates with an unweighted numerator of less than 5 (i.e., 1 to 4) should be suppressed.

For all estimates based on a denominator size of 30 or more, the following guidelines for data suppression related to coefficient of variations (CV) should be used when reporting estimates:

Examining the confidence interval of the estimate will provide further indication of the quality of the estimate in terms of the variability. Long confidence intervals indicate less precision in the estimate while smaller confidence intervals indicate greater precision. When assessing the trustworthiness of sample proportions, the confidence intervals of estimates should be taken into account.

4.1 Rounding Guidelines

Users are urged to adhere to the following rounding guidelines for estimates.

• Estimates in the main body of a statistical table are to be rounded to the nearest hundred units using the normal rounding technique. In normal rounding, if the first or only digit to be dropped is 0 to 4, the last digit to be retained is not changed. If the first or only digit to be dropped is 5 to 9, the last digit to be retained is raised by one. For example, in normal rounding to the nearest 100, if the last two digits are between 00 and 49, they are changed to 00 and the preceding digit (the hundreds digits) is left unchanged. If the last digits are between 50 and 99, they are changed to 00 and the preceding digit is increased by 1.
• Marginal sub-totals and totals in statistical tables are to be derived from their corresponding unrounded components and then are to be rounded themselves to the nearest 100 units using normal rounding.
• Averages, rates and percentages are to be computed from unrounded components (i.e., numerators and/or denominators) and then are to be rounded themselves to one decimal using normal rounding. In normal rounding to a single digit, if the final or only digit to be dropped is 0 to 4, the last digit to be retained is not changed. If the first or only digit to be dropped is 5 to 9, the last digit to be retained is increased by 1.
• Under no circumstances are unrounded estimates to be published or otherwise released by users. Unrounded estimates imply greater precision than actually exists.

5. Appendix

Appendix A: Weights for the direct weighting method

Appendix B: Response rate calculation