Service Canada
Grants And Contributions Applicants Client
Experience Research (Year Three)

Detailed Methodology Report

DATE – OCTOBER 25, 2023

Prepared for Employment and Social Development Canada
POR Registration #: POR 099-22
SUPPLIER: Ipsos Limited Partnership
CONTRACT AWARD DATE: 2022-12-19
CONTRACT #: CW2266044 (G9292-24-2550)
CONTRACT VALUE: $149,885.85 (tax included)

For more information on this report, please contact:
NC-POR-ROP-GD@HRSDC-RHDCC.GC.CA 

Grants and Contributions Applicants Client
Experience Research (Year Three)
Detailed Methodology

This detailed methodology report outlines the approach used for Year Three of the Program Operations Branch's (POB) Grants and Contributions (Gs&Cs) Applicants Client Experience (CX) Research Program at Employment and Social Development Canada (ESDC). Year Three will build on previous years of research to support the systematic and integrated approach to measure and improve CX in Gs&Cs program design and service delivery. This work allows the department to monitor CX trends over time and track progress in ongoing efforts to improve the experience of Gs&Cs applicants.

A mixed-methods research approach was used in Year Three, which included a quantitative online survey and qualitative online focus groups and in-depth interviews. The online survey was conducted with 3,041 Gs&Cs clients* across 11 programs between April 19 and June 9, 2023. The qualitative phase consisted of four online focus groups and 26 in-depth telephone interviews. The qualitative fieldwork was conducted between July 26 and August 10, 2023. The quantitative survey served as the recruitment tool for the qualitative research, by asking if organizations would be interested in voluntarily opting-in to be contacted to participate in focus groups or in-depth interviews at a later date.

The detailed methodology below provides a rationale for using an online survey and online focus groups/interviews, describes the target population for the study, describes the reference period of the study, explains the sampling strategy for the study, and finally, it identifies the study limitations. A copy of the online survey and the online focus group/online interview discussion guide are also provided in the annexes of this report. The initial recruitment text is also provided in an annex. Detailed findings are available under a separate cover.

*A client for the purpose of this Gs&Cs CX research is an organization that applied to a grants and contributions funding program within the last two intake years (2020/2021 and 2021/2022). Clients include both funded and unfunded organizations.

Rationale for Online Survey Methodology

The objective of the Year Three Gs&Cs CX Survey was to explore and understand client needs and expectations on the quality of service they received when applying to a Service Canada grant or contribution. The survey also aimed to understand what is most important for clients and identify opportunities for changes in services to improve client satisfaction or experience. To do this, information was collected from clients on how easily and effectively they completed the stages of the client journey, taking into account their experience with the service channels they used, and their satisfaction with the overall process of applying for a Gs&Cs program.

The primary focus of Year Three was to monitor selected programs that were previously studied in Year One and Year Two, to capture CX insights from programs that have not previously been studied, and to increase awareness of organizational characteristics and diversity considerations. Client organizations, both funded and unfunded, were recruited by ESDC's POB team to participate in the research. Clients were defined as organizations that applied for grants and contributions funding (including both funded and unfunded) within the last two intake years (FY 2020/21 and 2021/22). A random sampling of organizations that applied to CSJ or NHSP were included, while all organizations for remaining programs were invited to complete the survey. ESDC, specifically ESDC's POB team, distributed the survey links to participating organizations.

Clients were recruited to the survey by e-mail using information from administrative databases. More precisely, clients were contacted by ESDC's POB team using the GCNotify platform. Clients' contact information was not shared with Ipsos. Since most clients provided their email addresses as a form of contact, an online survey was chosen as the most efficient way to invite clients to participate.

The survey data was quantitatively analyzed using descriptive and inferential statistical analysis procedures. Results were analyzed at the overall level and by program to identify general trends and patterns relevant to the research objectives. In addition, analysis was also conducted among key subgroups (i.e., region, program complexity, sector, program streams etc.) to highlight notable differences. Lastly, a multivariate analysis was conducted through regression among all clients to identify the aspects of service that have the greatest impact on driving overall satisfaction. This information was used to identify key strengths and opportunities for improvement.

Rationale for Focus Groups and Interviews

A qualitative phase of research followed the survey. The qualitative phase included online focus groups and interviews. The qualitative phase allowed the research team to both further explore quantitative trends in more detail and focus on the lived experiences of the Gs&Cs processes. Ipsos worked in close consultation with ESDC's POB team to ensure the qualitative research objectives were thoroughly explored. A qualitative interview/focus group guide was drafted to direct the qualitative phase. With qualitative data collection and analysis methods and through rich description, the qualitative phase of the Year Three project was able to add fresh and unique insights to the CX project.

The Year Three qualitative research explored:

• Organizational capacity to effectively and efficiently complete the application process and the extent to which the process is inclusive;
• What impressed and what frustrated applicants;
• A deep dive on all aspects of the application journey, including GCOS;
• The ideal application journey and opportunities for future improvements;
• Impact of funding and non-funding decisions;
• Future interest in funding opportunities.

The value of qualitative research is in exploring the issues and experiences of research participants in depth, free from the constraints of a structured quantitative questionnaire. Qualitative evidence is rich and allows researchers to hear first-hand the underlying factors shaping experiences and opinions, as well as the interplay between factors.

Qualitative findings should not be extrapolated to the broader population, as they are not statistically projectable. Notable nuances that emerged in the interviews have been highlighted where relevant and these should be treated as strictly directional.

The qualitative findings should thus be viewed as complementary to the quantitative survey findings in terms of building a more complete understanding of the Gs&Cs client experience. Based on findings from the quantitative research where certain programs had lower levels of overall client satisfaction (i.e., % rated 4 or 5 out of 5), it was determined that clients from Apprenticeship Service (AS), Workplace Opportunities: Removing Barriers to Equity (WORBE), Social Development Partnerships Program - Disability (SDPP-D), Supports for Student Learning Program (SSLP), and Sectoral Initiatives Program (SIP) would be targeted for the in-depth interviews. The focus groups targeted clients from all programs and were broken out by funded/unfunded and English/ French.

The quantitative survey served as the recruitment tool for the qualitative research, by asking if organizations would be interested in voluntarily opting-in to be contacted to participate in focus groups or in-depth interviews at a later date. Within the confines of those who opt-in, an analysis was conducted to ensure a mix of programs, regions, and to ensure inclusion of participants in both official languages, were included. The potential participants were contacted randomly and asked if they would like to be taken through a screening questionnaire to confirm their eligibility for an in-depth interview or focus group.

The online focus groups and in-depth interviews were conducted in English and French for both funded (Funded: 8 participants in English and 7 participants in French) and unfunded clients (Unfunded: 6 participants in English and 7 participants in French). In addition to the four online focus groups, 26 in-depth interviews were conducted (19 participants in English and 7 participants in French, including both funded and unfunded clients). Fieldwork took place between July 26 and August 10, 2023.

Following the focus groups and interviews, transcripts of the conversations were produced. Those transcripts were then qualitatively analyzed using thematic content analysis techniques. The basic elements of the qualitative analysis included analyzing the results by:

• Universal agreement where participants all agree, or there is agreement across different groups of stakeholders;
• Consensus perspectives that reflect the view of most participants; areas of wide agreement without much counter point (Many, most, several);
• Conflicting or polarized perspectives where views are much more divided, or if there is a spectrum or variety of views (Some vs others);
• Minority perspectives, often expressed by one or two participants as a counterpoint to a consensus viewpoint, or if they have an individual take or example/story (a few, a couple, mention);
• Verbatim commentary, providing examples of what participants actually said during a discussion (direct unattributed quotes);
• External context, for this project it is the results of quantitative research that provided a foundation for the qualitative research conducted and the discussion questions posed.

Definition of "Client" for Sample Selection (Online survey)

The target audience of the survey was clients of Service Canada's grants and contributions programs. Clients were defined as organizations that have applied for grants and contributions funding (including both funded and unfunded) within the last two intake years (FY 2020/21 and 2021/22).

Clients of 11 programs were included in the target audience for this research as they represent the large majority of Gs&Cs clients accessing federal programs. Where applicable, different program streams were included and clearly identified for participants. The programs included were as follows:

• Enabling Accessibility Fund (EAF)
• Enabling Accessibility Fund: Mid-sized (Contribution)
• Enabling Accessibility Fund: Early Learning and Child Care (Grant)
• Enabling Accessibility Fund: Youth Innovation (Grant)
• New Horizons for Seniors Program (NHSP)
• Community Based stream
• Canada Summer Jobs (CSJ)
• Apprenticeship Service (AS)
• Workplace Opportunities: Removing Barriers to Equity (WORBE) 
• Social Development Partnerships Program – Children and Families (SDPP-C&F)
• Financial Empowerment of Low-Income People
• Social Inclusion of Vulnerable Children and Youth
• Social Development Partnerships Program – Disability (SDPP-D) – Disability Inclusion
• National AccessAbility Week (Grant)
• Phase 1 – Partnerships (Contribution)
• Supports for Student Learning Program (SSLP)
• Women's Employment Readiness (WER) Pilot Program
• Skilled Trades Awareness and Readiness Program (STAR)
• Sectoral Initiatives Program (SIP)

Four of the 11 programs were included in the research conducted in Year Two (EAF, NHSP, CSJ, and SDPP), while seven of the programs were new and have not been previously studied. The clients for these programs were included in the target audience as they represent key program intakes during the reference period and represent the majority of Gs&Cs clients accessing federal programs and are accessible to ESDC for sampling purposes.

For the purpose of this study, program complexity has been defined as low, moderate, and high as outlined in the following table. These service standard clusters are informed by departmental reporting in the Performance Measurement and Management Framework.

Note: Canada Summer Jobs does not fit into these distinct clusters and has been analyzed as a separate group.

Program Selection for Year Three

Program selection for the Year Three of the study was based on consultation, by ESDC's POB team, with program areas and analysis of the program intakes during the 2020/21 and 2021/22 fiscal years. The sampling of programs was designed to include yearly participation of key programs that represent the majority of clients to Gs&Cs (e.g., CSJ, EAF, NHSP) along with a staggered approach from smaller programs to participate in client experience research at regular intervals. The staggered selection of programs is based on considerations like the volume of clients, the regional/national and geographical considerations, the type of funding mechanism (e.g., grant vs. contribution), and the clients that participate in just the application process and those who make it to a funding agreement and close out. This two-pronged approach was taken as it would be unfeasible to have all programs participate annually in the Gs&Cs CX research program. The sampling approach has the benefit of systematically considering a variety of clients while still managing the size and scope of the overall research project.

Within the parameters of a 15-minute questionnaire, the examination of the client journey included the stages at which clients gathered information about the program, completed, and submitted an application. In addition, among those who received funding approval, the client journey included the monitoring, follow-up, and close-out phases of the project.

The survey was also used to track progress on key client experience service performance measures, primarily overall satisfaction and ease, effectiveness, and confidence with the service experience by program, and service channels used. Finally, the survey was used to provide diagnostic insights regarding the opportunities for improvement. Below is graphic that provides an overview of the Gs&Cs client journey.

Reference Period (Program Intake Period)

Fiscal Year 2021-22:

• Enabling Accessibility Fund (EAF)
• Enabling Accessibility Fund: Mid-sized (Contributions)
• Enabling Accessibility Fund: Early Learning and Child Care (Grants)
• Enabling Accessibility Fund: Youth Innovation (Grants)
• New Horizons for Seniors Program (NHSP) (Grants)
• Community Based stream
• Canada Summer Jobs (CSJ)
• Apprenticeship Service (AS) (Contributions)
• Workplace Opportunities: Removing Barriers to Equity (WORBE) (Contributions)
• Social Development Partnerships Program – Children and Families (SDPP-C&F)
• Financial Empowerment of Low-Income People
• Social Inclusion of Vulnerable Children and Youth
• Social Development Partnerships Program – Disability (SDPP-D) – Disability Inclusion
• National AccessAbility Week (Grants)
• Phase 1 – Partnerships (Contributions)
• Supports for Student Learning Program (SSLP) (Contributions)
• Women's Employment Readiness (WER) Pilot Program (Contributions)
• Skilled Trades Awareness and Readiness Program (STAR) (Contributions)

 Fiscal Year 2020-21:

• Sectoral Initiatives Program (SIP) (Contributions)

Sampling Strategy for the Survey

Program client data was extracted by ESDC's POB team from the Common System for Grants and Contributions (CSGC) used by Service Canada to manage Gs&Cs data. Once client contact information was extracted from CSGC, they were invited by ESDC's POB team using the GCNotify platform to respond to the online survey, hosted by Ipsos. The quantitative survey also served as a recruitment tool for the qualitative research, by asking if organizations would be interested in voluntarily opting-in to be contacted to participate in focus groups or in-depth interviews at a later date.

A random sampling of organizations that applied to CSJ or NHSP were included in the survey, while all organizations for remaining programs were invited to complete the survey. The goal was to obtain as many completed surveys as possible among the target audience during the fieldwork period.

ESDC, specifically ESDC's POB team, managed the distribution of unique links for the online survey to participating organizations. Of the 9,862 organizations that were invited to participate, a total of 3,041 organizations completed the survey. The survey sample size has a margin of error of +/-1.75%. The response rate for the survey was 31% which is considered strong compared to industry standards for a survey of this nature.

Weighting / Risk of Non-response Bias of the Survey Data

When assessing non-response bias, typically comparisons are made between the profile of survey participants and the broader client population profile on key variables relevant to the target population. If there is no systematic bias in response to the survey, the profile of the survey participants for each sample source would be very similar to the population profile within normal sampling error.

As in Year Two, there were limitations on available information of clients which mean that non-response bias cannot be assessed.

To correct for any differences between the survey sample and the actual client universe, the survey data were weighted. Information was available on the true proportion of the volume of clients by program and was used for weighting purposes. Disproportionate sampling was used and it was not anticipated that the final composition of the sample by program would match the overall proportion by volume of clients. Weighting adjustments were made to bring the sample into proportion with the universe by program volume. The final data was weighted by the number of respondents in each program in proportion to the total number of clients as detailed below.

Limitations

Contacting clients by email

Only organizations who received an email invitation could participate in the research. Client email addresses were extracted, by ESDC's POB team, from CSGC database. Extracting email addresses from the administrative database system uncovered duplicate, missing, and invalid email addresses. Missing or incomplete email addresses means that some clients may not have been captured in the sampling strategy and were not invited to participate.

Lag time

Lag time is the amount of time between when the client received a decision and was invited to respond to the survey. The longer the lag time the more difficult it can be for clients to accurately recall the specific aspects of their experience. The average lag ranged between one to two years depending on the program, its intake period, and when decisions where communicated. In Year Two and Year Three, it was possible to reduce the lag time between receiving a decision and answering the questionnaire which was up to three years for some programs in Year One of the research.

Response Rate

Of the 9,862 organizations that were invited by ESDC's POB team to participate in the survey, a total of 3,041 organizations completed the survey. The response rate for the survey was 31% which is considered strong compared to industry standards for a survey of this nature (i.e., online surveys utilizing client provided sample lists).

ANNEX 1:
QUESTIONNAIRE - GRANTS AND CONTRIBUTIONS CLIENT EXPERIENCE RESEARCH
(YEAR THREE)

Questionnaire 

Would you prefer to continue the survey in English or French? Please select one. 

• English
 
• French

Introduction

Ipsos, a market research firm, is conducting a survey on behalf of the Government of Canada.

Through this survey, we are reaching out to applicants who are currently or have recently applied for funding to a program delivered by Service Canada (part of the Department of Employment and Social Development Canada).

The purpose of the survey is for us to better understand your needs and expectations around the quality of service you received. We also want to understand what is most important to you and identify potential changes to our services that would improve your satisfaction or experience. The survey does not concern the amount of funding you received, but rather it explores your experience during the application process from beginning to end.

Your participation in the survey is voluntary. Your feedback is anonymous and confidential and will not affect any dealings you may have with Service Canada / Government of Canada. The information you provide will be used for research purposes only and will be administered according to the requirements of the Privacy Act, the Access to Information Act, and any other pertinent legislation. Click to view Ipsos' privacy policy.

The survey results will allow Service Canada to have up-to-date information on the experiences of grants and contributions applicants, identify areas of strength and opportunities for improvement, and make appropriate program design and service delivery improvements.

The survey should take approximately 15 minutes to complete. By completing the survey, you agree that the information and feedback you provide will be used for service delivery improvements and research purposes. Should you wish to verify the credibility of this survey, you can utilize the Canadian Research Insights Council (CRIC) Research Verification Service by clicking here and entering the following project code 20230328-IP537.

Should you have any questions about the survey or need an alternative means of accessing the survey, please contact Service Canada at pob.ri.dgop@hrsdc-rhdcc.gc.ca

[PROGRAM VARIABLE]

[PULLED FROM SAMPLE FILE AND USED FOR PROGRAM INSERTS THROUGHOUT]

[PROGRAM STEAM VARIABLE]

[PULLED FROM SAMPLE FILE]

[FUNDING APPROVAL VARIABLE]

[PULLED FROM SAMPLE FILE]

2. Which of the following did you use to find out about [INSERT PROGRAM] before you applied? Consider all the methods you used to learn about the program before filling out the application. Please select all that apply.

[RANDOMIZE. ALWAYS KEEP 'Went online to the Government of Canada website', 'Went online to the Government of Canada website for the [INSERT PROGRAM]', 'Went online to websites for other levels of government (provincial, territorial or municipal)', AND 'Went online to other websites' GROUPED]

Went online to the Government of Canada websiteWent online to the Government of Canada website for the [INSERT PROGRAM]
Went online to websites for other levels of government (provincial, territorial or municipal)
Went online to other websites
Used social media to get information
Called a Service Canada office directly
Called 1 800 O-Canada phone line
Emailed a Service Canada office
Emailed a Program Officer for [INSERT PROGRAM] directly
Went to a Service Canada office
Participated in a Government of Canada information session or webinar
Talked to my local Member of Parliament (MP)
Talked to my peers/community network
Received an email from the Government of Canada, ESDC or [INSERT PROGRAM] directly
Used the applicant guide for [INSERT PROGRAM]
NONE OF THESE [EXCLUSIVE; ANCHOR]

[IF Q2= ''NONE OF THESE'/DK/REF SKIP TO Q6]
[IF Q2= 'WENT ONLINE TO THE GOVERNMENT WEBSITE' OR 'WENT ONLINE TO THE GOVERNMENT OF CANADA WEBSITE FOR THE [INSERT PROGRAM]' ASK Q5, OTHERWISE SKIP]

5. On a scale from 1 to 5, where 1 is "very difficult" and 5 is "very easy", how difficult or easy was it to find the following information about [INSERT PROGRAM] on the Government of Canada website? Select one response per item.

[ROWS. RANDOMIZE]

Find general information about [INSERT PROGRAM]
Understand the information about [INSERT PROGRAM]
Determine if your organization is eligible for [INSERT PROGRAM] funding
Determine the steps to apply for funding
Find out what information you need to provide when applying for [INSERT PROGRAM]
Determine the amount of time each phase of the application process is anticipated to take (i.e. service standard for time to acknowledge proposal, time to notify of funding decision, and time to issue payment)
Determine when the application period for [INSERT PROGRAM]  takes place (i.e. opening and closing of call for proposals)

[COLUMNS]

1 – Very difficult
2
3
4
5 – Very easy
Don't know

For Q5b, this should be asked of all except those who responded NONE OF THESE'/DK/REF at Q2.  For Q5c, yes that should only be asked IF Q2= 'WENT ONLINE TO THE GOVERNMENT WEBSITE' OR 'WENT ONLINE TO THE GOVERNMENT OF CANADA WEBSITE FOR THE [INSERT PROGRAM]'

5b. How much do you agree or disagree that you were able to find the information you needed (online, in person, or by phone) within a reasonable amount of time? Please use a scale of 1 to 5, where 1 is "strongly disagree" and 5 is "strongly agree".

[COLUMNS]

1 – Strongly disagree
2
3
4
5 – Strongly agree
Don't know

IF Q2= 'WENT ONLINE TO THE GOVERNMENT WEBSITE' OR 'WENT ONLINE TO THE GOVERNMENT OF CANADA WEBSITE FOR THE [INSERT PROGRAM]

5c. When learning about [INSERT PROGRAM] on the Government of Canada website, which of the following changes would have improved your experience the most? SINGLE SELECT

If it was easier to…

[RANDOMIZE]

Find general information about [INSERT PROGRAM]
Understand the information about [INSERT PROGRAM]
Determine if your organization is eligible for [INSERT PROGRAM] funding
IDetermine the steps to apply for funding
Find out what information you need to provide when applying for [INSERT PROGRAM]
Determine the amount of time each phase of the application process is anticipated to take (i.e., service standard for time to acknowledge proposal, time to notify of funding decision, and time to issue payment)
Determine when the application period for [INSERT PROGRAM]  takes place (i.e. opening and closing of call for proposals)

6. To prepare and complete your application (up until when you submitted) did you consult with any of the following? Please select all that apply.

[RANDOMIZE][MULTIPUNCH]

Went online to the Government of Canada website
Went online to the Government of Canada website for [INSERT PROGRAM NAME]
Went to a Service Canada office
Called a Service Canada office directly
Called 1 800 O-Canada phone line
Used social media to get information
Went online to other websites for information
Talked to my peers/community network
Emailed a Service Canada office
Emailed a Program Officer for [INSERT PROGRAM] directly
Participated in a Government of Canada information session or webinar
Talked to my local Member of Parliament (MP)
Worked with a private consultant
Used the applicant guide for [INSERT PROGRAM]
NONE OF THESE [EXCLUSIVE; ANCHOR]

7. On a scale of 1 to 5, where 1 is "very difficult" and 5 is "very easy", how would you rate the following elements of the application for [INSERT PROGRAM]? Select one response per item.

[ROWS. RANDOMIZE]

Understanding the requirements of the application
Putting together the information you needed to apply for [INSERT PROGRAM]
Completing the narrative questions (i.e., funding objectives, description of project, scope of project, etc.)
Completing the budget document
Completing the project timeline
Meeting the requirements of the application process

[COLUMNS]

1 – Very difficult
2
3
4
5 – Very easy
Don't know

[IF EAF Mid-Sized -OR- EAF Early Learning and Child Care ASK Q7B, OTHERWISE SKIP]
This is based on the Steam Name variable and specifically: Enabling Accessibility Fund: Mid-Sized or Enabling Accessibility Fund: Early Learning and Child Care (Grant)

7b. The application process for Enabling Accessibility Fund involved the use of a budget calculator. On a scale from 1 to 5, where 1 is "strongly disagree" and 5 is "strongly agree", how much do you agree or disagree with the following statement?

The budget calculator…Was easy to use
Made it easy to complete the budget for the application
Helped to ensure I requested sufficient funding for my project

1 – Strongly disagree
2
3
4
5 – Strongly agree
Don't know

IF EAF Mid-Sized -OR- EAF Early Learning and Child AND FUNDING STATUS = APPROVED ASK, OTHERWISE SKIP

7c. Thinking about the amount of funding determined using the budget calculator, would you say this was too little, too much or the right amount for your project?

1- Too little
2
3- Right amount
4
5- Too much

9. On a scale from 1 to 5, where 1 is "strongly disagree" and 5 is "strongly agree", please rate the following statement:

The application took a reasonable amount of time to complete.

1 – Strongly disagree
2
3
4
5 – Strongly agree
Don't know

10. Which of the following methods did you use to submit your application? Please select only one.

Submitted an application using the online fillable form
Submitted an application using the Grants and Contributions Online Services (GCOS) account/web portal
Downloaded the application documents and then submitted by email
Downloaded the application documents and then submitted by mail
Submitted application documents to a Service Canada office
Submitted on my behalf by my local Member of Parliament
Other
NONE OF THESE [EXCLUSIVE; ANCHOR]

[IF NONE OF THESE SKIP TO Q13 OTHERWISE CONTINUE.]

11. Why did you choose this method to submit your application? Please select one reason only.

[RANDONMIZE]

It was the only method available
It was the easiest/most familiar way to apply
I felt more confident my application would be submitted properly 
It was the method I was directed to use
I did not know any other way to apply
Other

[IF USED 'Submitted an application using the online fillable form' OR 'Submitted an application using the Grants and Contributions Online Services (GCOS)' AT Q10 ASK Q12, OTHERWISE SKIP TO Q13]

12. On a scale from 1 to 5, where 1 means "very difficult" and 5 means "very easy", how difficult or easy was it to......

Submit your application online for [INSERT PROGRAM]
[IF USED Submitted an application using the Grants and Contributions Online Services (GCOS) AT Q10 ASK] Register for a Grants and Contributions Online Services (GCOS) account
Complete the steps of the application process for [INSERT PROGRAM] using the GCOS portal]
[IF FUNDING STATUS = APPROVED -AND- USED Submitted an application using the Grants and Contributions Online Services (GCOS) AT Q10 ASK] Manage your active project using the GCOS portal (e.g., create, modify, and submit claim and activity reports, supporting documents, forecast of project expenditures, etc.)

Clarification for above below:
option 1) "Submit your application online for [INSERT PROGRAM]"
option 2) [IF USED Submitted an application using the Grants and Contributions Online Services (GCOS) AT Q10 ASK] Register for a Grants and Contributions Online Services (GCOS) account
option 3) [IF USED Submitted an application using the Grants and Contributions Online Services (GCOS) AT Q10 ASK]  Complete the steps of the application process for [INSERT PROGRAM] using the GCOS portal
option 4) [IF FUNDING STATUS = APPROVED -AND- USED Submitted an application using the Grants and Contributions Online Services (GCOS) AT Q10 ASK] Manage your active project using the GCOS portal (e.g., create, modify, and submit claim and activity reports, supporting documents, forecast of project expenditures, etc.)

1 – Very difficult
2
3
4
5 – Very easy
Don't know

13. After you submitted your application, were you contacted by Service Canada to provide additional information to support your application?

Yes
No
Don't know

[IF YES TO Q13 ASK Q14, OTHERWISE SKIP]

14. Why were you contacted by Service Canada? Select all that apply.

[RANDOMIZE]

Missing documents or information in my application
Clarify information in my application
Budget template needed modifications
My organization or project was not eligible
An outstanding issue with a previous application
Other reason
Don't know

15. Did you contact Service Canada for any of the following reasons before receiving your funding decision? Select all that apply.

To check the status of your application
To find out timelines for receiving a funding decision
To withdraw your application
To modify your application
Other reason
Don't know [exclusive]
Did not contact Service Canada [exclusive]

[IF "DON'T KNOW" OR 'DID NOT CONTACT SERVICE CANADA' WAS SELECTED FOR Q15, SKIP TO Q17, OTHERWISE CONTINUE]

16. On a scale from 1 to 5, where 1 is "very difficult" and 5 is "very easy", how was your experience following up with Service Canada about your application?

1 – Very difficult
2
3
4
5 – Very easy
Don't know

17. How were you notified of the funding decision about your application for [INSERT PROGRAM]? Please select all that apply.

[RANDOMIZE]

By mail
By e-mail
By receiving a direct deposit
Online through your Grants and Contributions Online Services (GCOS) account
By telephone
From my local Member of Parliament (MP)
I did not receive a funding decision

[IF FUNDING STATUS = DENIED ASK, OTHERWISE SKIP]

19. Did your organization receive an explanation why you did not receive an approval for funding?

Yes
No
Don't know

[IF YES ASK Q20, OTHERWISE SKIP]

20. On a scale from 1 to 5, where 1 is "very dissatisfied" and 5 is "very satisfied", how dissatisfied or satisfied were you with the explanation of the decision?

1 – Very dissatisfied
2
3
4
5 – Very satisfied
Don't know

[IF FUNDING STATUS = APPROVED ASK, OTHERWISE SKIP]

22. Once your project began and the details of the funding agreement were finalized with [INSERT PROGRAM], did you have to work with a Service Canada Program Officer to request changes to your project and/or submit an amendment to the funding agreement? Examples could include changes to project timelines, project description, budget, etc.

[ROWS]

Yes
No
Don't know

[COLUMNS]

Changes to your project scope
Changes to project timelines
Changes to project activities
Changes to project funding
Covid-19 related amendment
Other reason

[IF YES TO ANY ITEM AT Q22 ASK Q23, OTHERWISE SKIP]

23. How long did the following take to complete? If you are uncertain, please provide your best guess.

[ROWS]

1 day
2 to 3 days
4 to 7 days / one week
More than 7 days / more than one week

[COLUMNS]
[INSERT ITEMS FROM Q22 = YES]

[IF FUNDING STATUS = APPROVED ASK Q24, OTHERWISE SKIP]

24. On a scale of 1 to 5, where 1 is "very difficult" and 5 is "very easy", how would you rate the following tasks related to your funding agreement with [INSERT PROGRAM]? Select one response per item or select not applicable if you did not have to complete the task as part of your agreement.

[ROWS. RANDOMIZE]

Completing the final budget/final claim 
Submitting the final budget
Completing the final project report and/or program-specific requirement(s)
Submitting the final project report and/or program-specific requirement(s)
Resolving any outstanding issues with funding (i.e. dealing with remaining funds, overpayment, etc.)

[COLUMNS]

1 – Very difficult
2
3
4
5 – Very easy
Not applicable

[PROGRAMMING FOR SERVICE CHANNEL ASSESSMENTS SECTION: EACH RESPONDENT WILL BE ASKED TO COMPLETE A SERVICE CHANNEL ASSESSMENT FOR EACH SOURCE USED BY THE RESPONDENT.]

[TO DETERMINE SOURCES:]

[CALLED SC OFFICE = Q2 or Q4 or Q6  'Called a Service Canada office directly' OR Q17 'By telephone']
[WENT TO A SC OFFICE = Q2 or Q4 or Q6 'Went to a Service Canada office' OR Q10 'Submitted to a Service Canada office']
[GOVERNMENT WEBSITE = IF Q2 OR Q7 'Went online to the Government of Canada website']
[SC EMAIL = IF Q2 or Q4 or Q6 'Emailed a Service Canada office' or Q10 'Downloaded application documents and submitted by email' or Q17 'By email']
[PROGRAM OFFICER EMAIL = IF Q2 or Q4 or Q6 'Emailed a Program Officer directly']
[1-800 O'CANADA = IF Q2 or Q4 or Q6 'Called 1 800 O-Canada phone line']
[ONLINE PORTAL = IF Q10 'Submitted an application using the Grants and Contributions Online Services (GCOS) web portal' OR 17 'Online through your Grants and Contributions Online Services (GCOS) account']
[MAIL = IF Q10 'Downloaded application documents and submitted by mail' OR Q17 'By mail']

25. Thinking about your overall experience, how many times did you [IF MULTIPLE SOURCES 'use each of the following' IF ONLY ONE SOURCE 'use the following']? Please provide one response per item.

[INSERT ITEMS BASED ON SOURCE VARIABLE(S) SELECTED]

[ROWS][RANDOMIZE ITEMS]
[IF VISIT A SERVICE CANADA OFFICE SELECTED] Go to a Service Canada office
[IF GOC WEBSITE SELECTED] Go online to the Government of Canada website
[IF SC EMAIL SELECTED] Email a Service Canada Office
[IF PROGRAM OFFICER EMAIL SELECTED] Email a Program Officer directly
[IF 1-800 O'CANADA SELECTED] Call 1 800 O-Canada
[IF CALLED A SERVICE CANADA OFFICE SELECTED] Call a Service Canada office directly
IF MAIL SELECTED] Communicate by mail with the Government of Canada
[IF ONLINE PORTAL SELECTED] Go online to the Grants and Contributions Online Services (GCOS) web portal
[COLUMNS]
[INSERT NUMERIC TEXT BOX BESIDE EACH ITEM]
Don't know

26. On a scale from 1 to 5, where 1 means "very dissatisfied", and 5 means "very satisfied". How dissatisfied or satisfied were you with the overall quality of service you received from each of the following?

[INSERT ITEMS BASED ON SOURCE VARIABLE(S) SELECTED]

[ROWS] [RANDOMIZE ITEMS]
[IF VISIT A SERVICE CANADA OFFICE SELECTED] Service Canada office 
[IF GOC WEBSITE SELECTED] Government of Canada website
[IF SC EMAIL SELECTED] Email support from a Service Canada office
[IF PROGRAM OFFICER EMAIL SELECTED] Email support from a Program Officer
[IF 1-800 O'CANADA SELECTED] 1 800 O-Canada phone line
[IF CALLED A SERVICE CANADA OFFICE SELECTED] Telephone support from a Service Canada office
[IF MAIL SELECTED] Mail
[IF ONLINE PORTAL SELECTED] Grants and Contributions Online Services (GCOS) web portal

[COLUMNS]

1 – Very dissatisfied
2
3
4
5 – Very satisfied
Don't know

27. Thinking about your overall experience getting information and applying

Yes
No

[IF YES ASK Q28 AND Q29. OTHERWISE SKIP]

28. How would you describe the problem or issue you experienced? Select all that apply. [MULTIPUNCH]

[RANDOMIZE]

Application form was too long
Application form was complicated
Application requirements were difficult to understand
Information on the program was difficult to understand
Took too long to receive a funding decision
Took too long to receive an update on my application
Telephone lines were busy
I received different answers from different Program Officers
Staff were not knowledgeable/could not answer my questions
Government of Canada website information was confusing
[INSERT PROGRAM] website information was confusing
Online account creation was confusing
Online application process was confusing
The information session was confusing
Technical difficulties
Other, please specify [TEXT BOX]

29. On a scale from 1 to 5, where 1 is "strongly disagree" and 5 is "strongly agree", how much would you agree or disagree that the problem or issue was easily resolved?

1 – Strongly disagree
2
3
4
5 – Strongly agree
Don't know

30. Thinking about the overall service you received, from getting information about [INSERT PROGRAM] to receiving a funding decision, how much do you agree or disagree with the following statements, using a 5-point scale where 1 means "strongly disagree", and 5 means "strongly agree".

[ROWS. RANDOMIZE]

I was able to move smoothly through all of the steps related to the [INSERT PROGRAM] application.
[IF Q10 'Submitted an application using the Grants and Contributions Online Services (GCOS) web portal' OR 17 'Online through your Grants and Contributions Online Services (GCOS) account']
Being able to complete steps online made the process easier for me.
It was clear what to do if I had a problem or question.
Throughout the process it was clear what would happen next and when it would happen.
I was confident that any issues or problems would have been easily resolved.
I needed to explain my situation only once.
It was easy to get help when I needed it.
Overall, it was easy for me to apply for [INSERT PROGRAM]]
I was provided with service in my choice of English or French.
I was confident that my personal information was protected.
[IF SOURCE = CALLED A GOVERNMENT OFFICE OR 1 800 O-CANADA] Service Canada phone representatives were helpful
[IF SOURCE = WENT TO GOVERNMENT OFFICE] Service Canada representatives that I dealt with in person were helpful 
[IF SOURCE = WENT TO GOVERNMENT OFFICE] I travelled a reasonable distance to access the Service Canada Office
I received consistent information
It was easy to access service in a language I could speak and understand well
[ALWAYS LAST] The amount of time it took, from when I started gathering information to when I got a decision on my application, was reasonable.

[COLUMNS]

1 – Strongly disagree
2
3
4
5 – Strongly agree
Don't know

31. On a scale from 1 to 5, where 1 means very dissatisfied, and 5 means very satisfied, how dissatisfied or satisfied were you with the overall service you received from Service Canada from getting information about [INSERT PROGRAM] to receiving a funding decision?

1 – Very dissatisfied
2
3
4
5 – Very satisfied
Don't know

32. On a scale from 1 to 5, where 1 means do not trust at all, and 5 means trust a great deal, how much do you trust Service Canada to deliver services effectively to Canadians?

1 – Do not trust at all
2
3
4
5 – Trust a great deal
Don't know

26a. And thinking about the entire process applying for [INSERT PROGRAM], which of the following changes would have improved your overall experience the most? Single punch

[RANDOMIZE]

Being able to move more smoothly through all of the steps
It was easier to get help when I needed it
It was easier to resolve any issues or problems
It is clearer what to do if I had a problem or question
The amount of time, from when I start gathering information to when I get a decision on my application, was more reasonable
The amount of time to prepare the application was more reasonable
It was easier to access service in a language I speak and understand well
It was easier to complete steps online
It was more clear what will happen next in the process and when it would happen

[SHOW DESCRIPTION ABOVE Q36]

A service standard is a public commitment to a measurable level of performance applicants can expect under normal circumstances.  Currently, Service Canada has service standards for all funding programs for the following:

Time to acknowledge a proposal: within 14 calendar days of receiving your application package.

Time to notify of funding decision: within 12 to 22 weeks (84 to 154 calendar days) from the date it was received or the end date of the intake process, depending on the characteristics of the intake method and program stream.

Time to issue payment once payment claim is submitted: For contributions, within 14 calendar days of receiving your completed claim package. For grants, within 14 calendar days of the approved project start date.

Service Canada commits to meeting three (3) service standards 80% of the time (under normal circumstances).

33. Before today, were you aware of these service standards?

[ROWS]

Time to acknowledge the submission of a funding application
Time to issue a finding decision notification
Time to issue payment once payment claim is submitted

[COLUMNS]

Yes
No

Service Canada and the Department of Employment and Social Development Canada are committed to better understanding how diversity factors such as personal and social identities (e.g. gender, sexual orientation, ability, ethnicity, religious identity, etc.) may shape experiences of people or organizations applying for or obtaining federal funding.

34. Would the funding you applied for assist any of the following communities, clients or people:

Those who identify as women
Those who identify as belonging to a minority racial or ethnic background
Those who identify as Black Canadians
Those who identify as Indigenous (First Nations, Inuit, Métis, urban non-affiliated)
Those who identify as having a mental or physical disability
Those who identify as belonging to a religious group
Those who identify as youth
Those who identify as senior
Those who identify as veterans
Those who identify as trans, non-binary, other gender, gender-diverse or queer people
Those who identify as lesbians, gays, bisexuals, queers or other sexual minorities
Those who identify as Two-Spirit or Indigenous LGBTQQIA+ people
English or French-language minority community
Those who identify as a low socioeconomic status
Those who identify as an immigrant or a non-permanent resident
Those who identify as newcomers to Canada
Those who are experiencing homelessness and/or currently unhoused
None of the above [MUTUALLY EXCLUSIVE]

The last few questions are for statistical purposes only. All responses will remain confidential.

36. Which province/territory does your organization operate in? Please select all that apply.

Alberta
British Columbia
Manitoba
New Brunswick
Newfoundland and Labrador
Northwest Territories
Nova Scotia
Nunavut
Ontario
Prince Edward Island
Quebec
Saskatchewan
Yukon

36b. Which province/territory will your organization deliver project activities for [INSERT PROGRAM]?

Alberta
British Columbia
Manitoba
New Brunswick
Newfoundland and Labrador
Northwest Territories
Nova Scotia
Nunavut
Ontario
Prince Edward Island
Quebec
Saskatchewan
Yukon

37. Which statement best describes your organization as it relates to completing the application for funding.  Select one response.

I am solely responsible for completing the funding application
A team of employees are dedicated to completing the funding application
A dedicated in-house proposal writer completes the funding application
A team of both employees and volunteers completes the funding application
A team of volunteers complete the funding application
We hire a consultant(s) to complete the funding application
I am not personally involved although I oversee this, or have some awareness

38. Was this the first application your organization submitted to [INSERT PROGRAM], or have you applied to [INSERT PROGRAM] in the past? Select one response.

First application
Applied once or twice before
Applied several times before
Apply for the same program on an annual basis
Don't know

[IF 'FIRST APPLICATION SELECTED' AT Q41, ASK Q41B OTHERWISE SKIP]

38b. Has your organization submitted an application to a different Grant or Contribution program from Service Canada program in the past five years?

Yes
No
Unsure / don't know

38c. Thinking about the last five years, how often does your organization apply for international, federal, provincial/territorial, and/or municipal or local funding of any kind?

[ROWS]

International
Federal
Provincial/territorial
Municipal/local

[COLUMNS]

Monthly
Quarterly
Bi-annually
Annually
Less often than annually
Never

38d. How many years has your organization been in operation?

Less than one year
One year to less than three years
Three years to less than five years
Five or more years

39. What sector does your organization operate in? Please select all that apply.

Not-for-Profit [HEADER]
Community, charitable or voluntary organizations, including faith-based organizations (for example, churches, synagogues, temples, mosques)
Associations of workers or employers as well as professional and industrial organizations
Indigenous not-for-profit organizations
Non-governmental organizations
Unions
Sector councils
Not-for-profit Band Councils
Other

Public Sector [HEADER]
Public community colleges and vocational schools
Public health, including public hospitals, nursing homes, senior citizen homes, rehabilitation homes
Public degree-granting universities and colleges
Municipal governments and agencies, including regional legislative bodies and departments
School boards and elementary and secondary institutions
Territorial governments
Other

Private Sector [HEADER]
Financial Institutions
Business, incorporated or unincorporated bodies including partnerships and sole proprietorships
Indian Band corporations
Private Band Councils
Private universities or colleges
Other

40. Approximately how many employees work (full-time or part-time) for your organization? Please select one.

None
1-4 employees
5-9 employees
10-19 employees
20-49 employees
50 or more employees
Don't know

41. Approximately how many volunteers work (full-time or part-time) for your organization? Please select one.

None
1-4 volunteers
5-9 volunteers
10-19 volunteers
20-49 volunteers
50 or more volunteers
Don't know

42. In which of the official languages does….? Please select one response for each item.

[ROW]

Your organization prefer to receive service in
Your organization provide service in
Your client population speak

[COLUMN]

English
French
Both/Either
Other

[END OF CORE SURVEY, ALL REMAINING MEASURES OPTIONAL]

43. Thinking about your experience with Service Canada, throughout the entire application process, have you ever felt discriminated against on the basis of your identity?

Yes
No

[IF YES ASK Q44, OTHERWISE SKIP]

44. On which grounds did you feel discriminated against? Select all that apply

Race
National or ethnic origin
Colour
Religion or Religious identity
Age
Sex
Sexual Orientation
Gender identity or expression
Marital status
Family status
Ability/disability
Genetic characteristics
A conviction for which a pardon or record suspension has been granted
Language
Other

[IF 'FIRST APPLICATION' OR 'DON'T KNOW' SELECTED AT Q38 SKIP Q44a, OTHERWISE CONTINUE]

44a.  You indicated your organization has applied for [INSERT PROGRAM] before. Comparing the service you received for [INSERT PROGRAM] in the past with your most recent experience, from getting information about the program to receiving a funding decision, do you feel the quality of service has improved, declined or stayed about the same?

[ROWS. RANDOMIZE]

Overall level of satisfaction with service received
Ease of completing application
Ease of submitting application
Clarity of information on [INSERT PROGRAM] website
Ease of getting assistance when needed
The amount of time it took from when I started gathering information to when I got a decision on my application

[COLUMNS]

Declined significantly
Declined somewhat
About the same
Improved somewhat
Improved significantly
Don't know

45. Thank you for participating in the Service Canada's Grants and Contributions client experience survey. This survey collects valuable insight about your satisfaction, how services are delivered, and areas for improvement. The research also involves a qualitative phase of focus groups and interviews expected to take place April-May 2023.

Are you interested in having a representative from Ipsos contact you to participate in focus groups or interviews about the service you received?
Yes
No

[IF YES ASK CONTACT INFO, OTHERWISE END SURVEY]

46. You indicated you would like to participate in the qualitative phase of focus groups and interviews. Your feedback is important to us and your answers will always remain confidential, meaning that your contact information will not be shared with Service Canada and will not be attached to your survey responses. Please provide an email address and phone number where Ipsos may contact you. [mandatory if they indicate 'yes']

Full Name
Email [VALID FOR PROPER EMAIL]
Phone Number [VALIDATE FOR PROPER PHONE NUMBER]

The survey is complete. This survey is part of a larger research study. Your anonymous responses will be combined with all responses gathered during the survey, then aggregated, analyzed and published as public information.

The full detailed survey report, along with an executive summary and data tables will be posted on the Library and Archives Canada website within six months of the conclusion of this study.

Should you have any question, you can reach those responsible for the survey at Service Canada (Employment and Social Development Canada) directly by emailing pob.ri.dgop@hrsdc-rhdcc.gc.ca. 

Thank you for your participation.

ANNEX 2:
Discussion Guide for Online Focus Groups and In-depth Interviews - Grants and Contributions Applicants Client Experience Research
(Year Three)

AUDIENCE BREAKDOWN

SESSION BREAKDOWN

ANNEX 3:
Recruitment Text

At a glance: The recommended recruitment approach will take place as follows:

1. Information Email
   Context: An information email about the CX research with general background WITHOUT the link to participate
   Audience: All organizations included in Year Three
   Timing: Thursday AM (TBD)
2. Email with invitation to participate
   An invitation email with background information AND the link to participate in the survey, sent 3-5 days after the information email
   Audience: All organizations included in Year Three
   Timing: Tuesday AM, following the Thursday AM message
3. Reminders
   Context: Reminders to participate in the survey. The number align with POR standards for follow-ups
   Audience: All organizations included in Year Three
   Timing: Varied
4. Final thank you WITH survey link
   Context: Final reminder and thank you to participants.
   Audience: All organizations included in Year Three
   Timing: Two days before closing

ESDC will be responsible for recruitment via GCNotify.

Summary of changes in Year Three:

• Updated the year of the study
• Inclusion of program name in the text of the invite (Note: this is subject to some final testing that we are doing with GC Notify)
• Added language of: "Your participation is important and easy"
• Note: Special characters (* ^ etc.) are used for coding font differences in GCNotify

Next steps:

• Approval of the recruitment text
• Seek final input from IPSOS
• Once ready to recruit, ESDC to send invitation via GCNotify