Grants and Contributions Applicants Client Experience Research (Year Two)

Detailed Methodology

This detailed methodology report outlines the approach used for Year Two of the Program Operations Branch’s Grants and Contributions (Gs&Cs) Applicants Client Experience (CX) Research Project at Employment and Social Development Canada (ESDC). Year Two builds on the first year of research by continuing to use a systematic approach to measuring CX in Gs&Cs service delivery and allowing the department to track progress on CX indicators over time.

A mixed-method approach was used in Year Two, which included a quantitative online survey and qualitative online focus groups and in-depth interviews. The online survey was conducted with 1,942 Gs&Cs clients across 12 programs between February 16, 2022 to March 15, 2022. The qualitative phase consisted of four online focus groups and 26 telephone in-depth interviews. The qualitative fieldwork was conducted between May 25, 2022 and May 31, 2022. The quantitative survey served as a recruitment tool for the qualitative research, by asking if organizations would be interested in voluntarily participating in focus groups or in-depth interviews at a later date.

The detailed methodology below provides a rationale for using an online survey and online focus groups/interviews, describes the target population for the study, the reference period of the study, the sampling strategy for the study, and finally, it identifies study limitations. A copy of the online survey and the online focus group/online interview discussion guide are also provided.

Rationale for Online Survey Methodology

The objective of the Grants and Contributions (Gs&Cs) Client Experience Survey was to explore and understand client needs and expectations on the quality of service they received when applying to a Service Canada grant or contribution. The survey also aimed to understand what is most important for clients and identify opportunities for changes in services to improve client satisfaction or experience. To do this, the Program Operations Branch (POB) collected information from clients on how easily and effectively they completed the stages of the client journey, taking into account their experience with the service channels they used, and their satisfaction with the overall process of applying for a Gs&Cs program.

The primary focus of Year Two of this project was to monitor selected programs that were previously studied in Year One and to capture new CX insights from programs that have not previously been studied. Client organizations, both funded and unfunded, were recruited to participate in the research. Clients were defined as organizations that applied for grants and contributions funding (including both funded and unfunded) within the last two intake years (2019/20 or 2020/21). A random sampling of organizations that applied to Canada Summer Jobs (CSJ), Enabling Accessibility Fund (EAF) or New Horizons for Seniors Program (NHSP) were included, while all organizations for remaining programs were invited to complete the survey.

Clients were recruited by e-mail using information from administrative databases. Since most clients provided their email addresses as a form of contact, an online survey was chosen as the most efficient way to invite clients to participate. Clients were contacted by email by ESDC’s POB team from the GCNotify platform.

For Year Two, fieldwork launch was executed using two approaches in order to better understand the impact of sharing information emails and small variations in the recruitment approach on response rates. Half of the sampling of organizations that applied to CSJ, EAF and NHSP were sent an information email prior to receiving the survey invitation email containing the survey link, while the other half (and those who applied to all other programs) were only sent the survey invitation email.

The online survey was fielded from February 16, 2022 to March 15, 2022.

The survey data was quantitatively analyzed using descriptive and inferential statistical analysis procedures. Results were analyzed at the overall level and by program to identify general trends and patterns relevant to the research objectives. In addition, analysis was also conducted among key subgroups (i.e., region, program complexity, sector, etc.) to highlight notable differences. Lastly, a multivariate analysis was conducted through regression among all clients to identify the aspects of service that have the greatest impact on driving overall satisfaction. This information was used to identify key strengths and opportunities for improvement.

Rationale for Focus Groups and Interviews

The Gs&Cs CX Research Project was carried out in two phases. A qualitative phase of online focus groups and interviews followed the survey and allowed the research team to both further explore quantitative trends in more detail and focus on the lived experiences of the Gs&Cs processes. With qualitative data collection and analysis methods and through rich description, the qualitative phase of the Year Two project was able to add fresh and unique insights to the CX project.

For the focus groups and interviews, the goal of POB was to explore experiences of clients who recently went through the application process for a grant or contribution at Service Canada. The discussion with clients explored their needs and expectations around service delivery (e.g., the client experience on aspects that make it easy for clients and the obstacles/barriers they face when going through the application process), their satisfaction or experiences with the service (e.g., assessing which service dimensions hold greater or lesser value for clients with respect to accessing service), and organizational characteristics (e.g., characteristics, qualities, and experiences impacting how the organization experiences service from POB), and unique quantitative findings (e.g., new or emerging themes from the survey that warrant further exploration and explanation, paying particular attention to prioritize the lived experiences of clients).

Based on findings from the quantitative research where certain programs had lower levels of client experience satisfaction, it was determined that clients from Early Learning and Child Care Innovation (ELCC), and Sustainable Development Goals (SDG), would be targeted for the in-depth interviews. The focus groups shifted to target programs other than ELCC and SDG. Additional questions that were explored based on quantitative findings included understanding perceptions on the length and complexity of the application and technical issues with the application process.

The quantitative survey served as the recruitment tool for the qualitative research, by asking if organizations would be interested in voluntarily participating in focus groups or in-depth interviews at a later date. After conducting an analysis of those interested in participating to ensure a mix of programs, regions, and to ensure inclusion of participants in both official languages, potential participants were contacted randomly and asked if they would like to be taken through the screening questionnaire to confirm their eligibility for an in-depth interview or focus group.

The online focus groups and in-depth interviews took place between May 25, 2022 and May 31, 2022 were conducted in English and French for both funded (Funded: 6 participants in English and 5 participants in French) and unfunded clients (Unfunded: 5 participants in English and 5 participants in French). In addition to the four online focus groups, 26 in-depth interviews were conducted (18 participants in English and 8 participants in French, including both funded and unfunded clients).

The focus group and interview materials were qualitatively analyzed using thematic content analysis techniques. The basic elements of the qualitative analysis included analyzing the results by:

· Universal agreement where participants all agree, or there is agreement across different groups of stakeholders;

· Consensus perspectives that reflect the view of most participants; areas of wide agreement without much counter point (Many, most, several);

· Conflicting or polarized perspectives where views are much more divided, or if there is a spectrum or variety of views (Some vs others);

· Minority perspectives, often expressed by one or two participants as a counterpoint to a consensus viewpoint, or if they have an individual take or example/story (a few, a couple, mention);

· Verbatim commentary, providing examples of what participants actually said during a discussion (direct unattributed quotes);

· External context, for this project it is the results of quantitative research that provided a foundation for the qualitative research conducted and the discussion questions posed.

Definition of “Client” for Sample Selection

The target audience of the survey were clients of Service Canada’s grants and contributions programs. Clients were defined as organizations that have applied for grants and contributions funding (including both funded and unfunded) within the last two intake years (2019/20 or 2020/21).

Clients of 12 programs were included in the target audience for this research as they represent the large majority of Gs&Cs clients accessing federal programs. Where applicable, different program streams were included and clearly identified for participants. The programs included were as follows:

· Canada Summer Jobs (CSJ)

· Early Learning and Child Care – Innovation (ELCCI)

· Enabling Accessibility Fund (EAF)

· Foreign Credential Recognition Program (FCRP)

· Indigenous Early Learning and Childcare (IELCC)

· Innovative Work Integrated Learning Initiative (IWILI)

· New Horizons for Seniors Program (NHSP)

· Social Development Partnerships Program (SDPP)

· Student Work Placement (SWP)

· Sustainable Development Goals (SDG)

· Union Training and Innovation Program (UTIP)

· Youth Employment and Skills Strategy (YESS)

Seven of the twelve programs were included in the research conducted in Year One (EAF, NHSP, CSJ, YESS+, UT&IP, EL&CCI and SDPP), while five of the programs were new and have not been previously studied (FCRP, IELCC, IWILI, SWP and SDG). The clients for these programs were included in the target audience as they represent key program intakes during the reference period and represent the majority of Gs&Cs clients accessing federal programs and are accessible to ESDC for sampling purposes.

Program Selection

Program selection for the study was based on extensive consultation with program areas and analysis of the program intakes during the 2019/20 and 2021/21 fiscal years. The sampling of programs was informed by yearly participation of key programs that represent the majority of clients to Gs&Cs (e.g., CSJ, EAF, NHSP) along with a staggered approach from smaller programs to participate in client experience research at regular intervals. The staggered selection of programs is based on considerations like the volume of clients, the regional/national and geographical considerations, the type of funding mechanism (e.g., grant vs. contribution), and the clients that participate in just the application process and those who make it to a funding agreement and close out. This two-pronged approach was taken as it would be unfeasible to have all programs participate yearly. The sampling approach has the benefit of systematically considering a variety of clients while still managing the size and scope of the overall research project.

Within the parameters of a 15-minute questionnaire, the examination of the client journey included the stages at which clients gathered information about the program, completed, and submitted an application. In addition, among those who received funding approval, the client journey included the monitoring, follow-up, and close-out phases of the project.

The survey was also used to track progress on key client experience service performance measures, primarily overall satisfaction and ease, effectiveness and confidence with the service experience by program, and service channels used. Finally, the survey was used to provide diagnostic insights regarding the opportunities for improvement. Below is graphic that provides an overview of the Gs&Cs client journey.

Reference Period

The exact intake periods referred to in this study are as follows:

· Canada Summer Jobs (CSJ): January to February 2020; December 2020 to February 2021

· Early Learning and Child Care – Innovation (ELCCI): October 2020 to January 2021

· Enabling Accessibility Fund (EAF): June 2020 to November 2020

· Foreign Credential Recognition Program (FCRP): March to April 2019; February 2020 to June 2020

· Indigenous Early Learning and Childcare (IELCC): February 2021 to April 2021

· Innovative Work Integrated Learning Initiative (IWILI): September 2020 to November 2020

· New Horizons for Seniors Program (NHSP): September 2020 to Oct. 2020

· Social Development Partnerships Program (SDPP): June 2020 to July 2020; March 2021 to April 2021; December 2020 to January 2021

· Student Work Placement (SWP): November 2020 to December 2020

· Sustainable Development Goals (SDG): Grants: May 2019 to November 2019; Contributions: June 2019 to Sept. 2019

· Union Training and Innovation Program (UTIP): July to August 2020

· Youth Employment and Skills Strategy (YESS): June 2019 to July 2019; March 2021 to April 2021

Sampling Strategy

Program client data was extracted from the Common System for Grants and Contributions (CSGC) used by Service Canada to manage Gs&Cs data. Once client contact information was extracted from CSGC, they were invited by Service Canada using the GCNotify platform to respond to the online survey. The quantitative survey also served as a recruitment tool for the qualitative research, by asking if organizations would be interested in voluntarily participating in focus groups or in-depth interviews at a later date.

A random sampling of organizations that applied to CSJ, EAF or NHSP were included, while all organizations for remaining programs were invited to complete the survey. The goal was to obtain as many completed surveys as possible among the target audience during the fieldwork period.

Fieldwork launch was executed using two approaches in order to better understand the impact on response rates. Half of the sampling of organizations that applied to CSJ, EAF and NHSP were sent an information email in advance of receiving the survey invitation email containing the survey link, while the other half (and those who applied to all other programs) were only sent the survey invitation email.

In total, 2250 clients were sent an information email in advance of receiving the survey invitation email. The remaining 2250 clients only received the survey invitation email. The response rate among those who received the information email was 25% versus 23% among those who only received the survey invitation email. Although higher, this difference is not statistically significant at the 95% level.

In addition, there was no statistically significant difference in response rates by distribution method by program or whether the client was approved or denied funding. Directionally speaking, EAF clients and those who were approved for funding who have received the informational email were somewhat more likely to respond, but results are not considered statistically significant.

Weighting / Risk of Non-response Bias

When assessing non-response bias, typically comparisons are made between the profile of survey participants and the broader client population profile on key variables relevant to the target population. If there is no systematic bias in response to the survey, the profile of the survey participants for each sample source would be very similar to the population profile within normal sampling error.

As in Year One, there were limitations on available information of clients which mean that non-response bias cannot be assessed.

To correct for any differences between the survey sample and the actual client universe, the survey data were weighted. Information was available on the true proportion of the volume of clients by program and was used for weighting purposes. Disproportionate sampling was used and it was not anticipated that the final composition of the sample by program would match the overall proportion by volume of clients. Weighting adjustments were made to bring the sample into proportion with the universe by program volume. The final data was weighted by the number of respondents in each program in proportion to the total number of clients as detailed below.

ABBREVIATION	Program	# of completed surveys	% of completes	Volume of clients	% of total
CSJ	Canada Summer Jobs	865	44.5%	39202	74.13%
EAF	Enabling Accessibility Fund	207	10.7%	2173	4.11%
NHSP	New Horizons for Seniors Program	384	19.8%	7194	13.60%
FCRP	Foreign Credential Recognition Program	20	1.0%	127	0.24%
ELCC	Early Learning and Child Care	65	3.3%	503	0.95%
IELCC	Indigenous Early Learning and Childcare	8	0.4%	68	0.13%
IWIL	Innovative Work Integrated Learning Initiative	13	0.7%	10	0.02%
SWPP	Student Work Placement	4	0.2%	30	0.06%
SDG	Sustainable Development Goals	39	2.0%	722	1.37%
YESS	Youth Employment and Skills Strategy	152	7.8%	971	1.84%
SDPP	Social Development Partnerships Program	153	7.9%	1755	3.32%
UTIP	Union Training and Innovation Program	32	1.6%	126	0.24%
	Total	1942	100%	52811	100%

Limitations

Contacting clients by email

Only organizations who received an email invitation could participate in the research. Client email addresses were extracted from CSGC databases. Extracting email addresses from the administrative database system uncovered duplicate, missing, and invalid email addresses. Missing or incomplete email addresses means that some clients may not have been captured in the sampling strategy and were not invited to participate. Also, when inviting clients to participate in research by email, there may be distrust in email communications and concerns about email-related fraud during the COVID-19 pandemic. Consequently, some clients may have overlooked the research invitation or questioned the authenticity of the invitation leading to non-participation.

Lag time

Lag time is the amount of time between when the client received a decision and was invited to respond to the survey. The longer the lag time the more difficult it can be for clients to accurately recall the specific aspects of their experience. The average lag ranged between one to two years depending on the program, its intake period, and when decisions where communicated. In Year Two, it was possible to reduce the lag time between receiving a decision and answering the questionnaire which was up to three years for some programs.

COVID-19

While restrictions on in-person gatherings have lifted for the most part when research was conducted, the lasting effects of the COVID-19 pandemic on program clients may have impacted the ability for organizations to participate in the research study (e.g., working remotely, working at reduced capacity, etc.).

Response Rate

Of the 8,704 organizations that were invited to participate, a total of 1,942 organizations completed the survey. The response rate for the survey was 22% which is consistent with industry standards for a survey of this nature.

	TOTAL
Invited to participate	8704
Click-Through	2941
Partial Completes	999
Terminates	0
Over Quota	0
Completed Surveys	1942
Response Rate	22%

ANNEX 1:
Questionnaire - Grants and Contributions Client Experience Research (Year Two)

Questionnaire

Would you prefer to continue the survey in English or French? Please select one.

English

French

Introduction

Ipsos, a market research firm, is conducting a survey on behalf of the Government of Canada.

Through this survey, we are reaching out to applicants who are currently or have recently applied for funding to a program delivered by Service Canada (part of the Department of Employment and Social Development Canada).

The purpose of the survey is for us to better understand your needs and expectations around the quality of service you received. We also want to understand what is most important to you and identify potential changes to our services that would improve your satisfaction or experience. The survey does not concern the amount of funding you received, but rather it explores your experience during the application process from beginning to end.

Your participation in the survey is voluntary. Your feedback is anonymous and confidential, and will not affect any dealings you may have with Service Canada / Government of Canada. The information you provide will be used for research purposes only and will be administered according to the requirements of the Privacy Act, the Access to Information Act, and any other pertinent legislation. Click to view Ipsos’ privacy policy.

The survey should take approximately 15 minutes to complete. By completing the survey, you agree that the information and feedback you provide will be used for service delivery improvements and research purposes. Should you wish to verify the credibility of this survey, you can utilize the Canadian Research Insights Council (CRIC) Research Verification Service by clicking here and entering the following project code [insert Ipsos’ link to reference number for research registration system.]

Should you have any questions about the survey or need an alternative means of accessing the survey, please contact Service Canada at pob.ri.dgop@canada.gc.ca

Programs [for IPSOS reference only]

English	French
PROGRAM	PROGRAMME
Canada Summer Jobs (CSJ)	Emplois d'été Canada (EÉC)
Enabling Accessibility Fund (EAF)	Fonds pour l'accessibilité (FA)
New Horizons for Seniors Program (NHSP)	Programme Nouveaux Horizons pour les aînés (PHHA)
Foreign Credential Recognition Program (FCRP)	Programme de reconnaissance des titres de compétences étrangers (PRTCE)
Early Learning and Child Care Innovation (ELCC)	Innovation en matière d'apprentissage et de garde des jeunes enfants (AGJE)
Indigenous Early Learning and Childcare (IELCC)	Innovation en matiere d’apprentissage et garde de jeunes enfants autochtones (AGJEA)
Innovative Work Integrated Learning Initiative (IWIL)	Innovation de l’apprentissage intégré au travail (IAIT)
Student work placement program (SWPP)	Programme de stages pratiques pour étudiants (PSPE)
Sustainable development goals (SDG)	Programme de financement des objectifs de développement durable (ODD)
Youth Employment and Skills Strategy (YESS))	La Stratégie emploi et compétences jeunesse (SECJ)
Social Development Partnerships Program (SDPP)	Programme de partenariats pour le développement social (PPDS)
Union Training and Innovation Program (UTIP)	Programme pour la formation et l'innovation en milieu syndical (PFIMS)

1. To start, please confirm which funding program your organization applied to and received a funding decision for in 2019, 2020 or 2021 (either funded or not funded). The remainder of the survey will refer to the selected funding program. Please select one. If you applied to multiple programs, select the most recent one.

Canada Summer Jobs (Jan. – Feb. 2020; Dec. 2020 – Feb. 2021)

Enabling Accessibility Fund (June – Nov. 2020)

New Horizons for Seniors Program (Sept. – Oct. 2020)

Foreign Credential Recognition Program (Mar. – April 2019; Feb. – June 2020)

Early Learning and Child Care – Innovation (Oct. 2020 – Jan. 2021)

Indigenous Early Learning and Childcare (Feb. – April 2021)

Innovative Work Integrated Learning Initiative (Sept. – Nov. 2020)

Student Work Placement (Nov. – Dec. 2020)

Sustainable Development Goals (May – Nov. 2019; June – Sept. 2019)

Youth Employment and Skills Strategy (June – July 2019; Mar.– April 2021)

Social Development Partnerships Program (June – July 2020; Mar.– April 2021; Dec. 2020 – January 2021)

Union Training and Innovation Program (July– Aug. 2020)

[ASK Q1B IF SELECTED ‘Sustainable Development Goals’ OR ‘Enabling Accessibility Fund’ OR ‘New Horizons for Seniors Program’ OR ‘Social Development Partnerships Program’ OR ‘Union Training and Innovation Program (UTIP – 2020/21)’ AT Q1, OTHERWISE SKIP]

1b. Which stream of the [INSERT PROGRAM] did your organization apply for?

Please select one.

[IF SELECTED ‘Sustainable Development Goals’ SHOW THE FOLLOWING ITEMS]

Sustainable Development Goals: Grants (May – Nov. 2019)

Sustainable Development Goals: Contributions (June – Sept. 2019)

[IF SELECTED ‘Enabling Accessibility Fund’ SHOW THE FOLLOWING ITEMS]

Small Projects (June 2020 – July 2020)

Youth Innovation (June 2020 – Nov. 2020)

[IF SELECTED ‘New Horizons for Seniors Program’ SHOW THE FOLLOWING ITEMS]

Small Grant (up to $5000) (Sept. 2020 – Oct. 2020)

Community-Based Projects (up to $25,000) (Sept. 2020 – Oct. 2020)

[IF SELECTED ‘Social Development Partnerships Program’ SHOW THE FOLLOWING ITEMS]

Supporting Black Canadian Communities (June 2020 – July 2020)

Supporting Black Canadian Communities – West Intermediaries (Mar. 2021 – April 2021)

Disability – Community Inclusion Initiative (Dec. 2020 – Jan. 2021)

[IF SELECTED ‘Union Training and Innovation Program’ SHOW THE FOLLOWING ITEMS]

Investments in Training Equipment (July 2020 – Aug. 2020)

Innovation and Apprenticeship (July 2020 – Aug. 2020)

FOR PIPE-INS THROUGHOUT WHERE [INSERT PROGRAM] IS INDICATED:

If at Q1b “Sustainable Development Goals: Grants (May – Nov. 2019)”, insert “Sustainable Development Goals: Grants”

If at Q1b “Sustainable Development Goals: Contributions (June – Sept. 2019)”, insert “Sustainable Development Goals: Contributions”

If at Q1b “Small Projects (June 2020 – July 2020)”, insert “Enabling Accessibility Fund – Small Projects”

If at Q1b “Youth Innovation (June 2020 – Nov. 2020)”, insert “Enabling Accessibility Fund – Youth Innovation”

If at Q1b “Small Grant (up to $5000) (Sept. 2020 – Oct. 2020)”, insert “New Horizon for Seniors Program – Small Grant”

If at Q1b “Community-based projects (up to $25,000) (Sept. 2020 – Oct. 2020)”, insert “New Horizon for Seniors Program – Community-Based Projects”

If at Q1b “Supporting Black Canadian Communities (June 2020 – July 2020)”, insert “Social Development Partnerships Program – Supporting Black Canadian Communities”

If at Q1b “Supporting Black Canadian Communities – West Intermediaries (Mar. 2021 – April 2021)”, insert “Social Development Partnerships Program – Supporting Black Canadian Communities – West Intermediaries”

If at Q1b “Disability – Community Inclusion Initiative (Dec. 2020 – Jan. 2021)”, insert “Social Development Partnerships Program – Disability–Community Inclusion Initiative”

If at Q1b “Investments in Training Equipment (July 2020 – Aug. 2020)”, insert “Union Training and Innovation Program – Investments in Training Equipment”

If at Q1b “Innovation and Apprenticeship (July 2020 – Aug. 2020)”, insert “Union Training and Innovation Program – Innovation and Apprenticeship”

If at Q1 “Canada Summer Jobs (Jan. – Feb. 2020; Dec. 2020 – Feb. 2021)”, insert “Canada Summer Jobs”

If at Q1 “Foreign Credential Recognition Program (Mar. – April 2019; Feb. – June 2020)”, insert “Foreign Credential Recognition Program”

If at Q1 “Early Learning and Child Care – Innovation (Oct. 2020 – Jan. 2021)”, insert “Early Learning and Child Care – Innovation”

If at Q1 “Indigenous Early Learning and Childcare (Feb. – April 2021)”, insert “Indigenous Early Learning and Childcare”

If at Q1 “Innovative Work Integrated Learning Initiative (Sept. – Nov. 2020)”, insert “Innovative Work Integrated Learning Initiative”

If at Q1 “Student Work Placement (Nov. – Dec. 2020)”, insert “Student Work Placement”

If at Q1 “Youth Employment and Skills Strategy (June – July 2019; Mar. – April 2021)”, insert “Youth Employment and Skills Strategy”

Pre-application—Information Gathering about Program

2. Which of the following did you use to find out about [INSERT PROGRAM] before you applied? Consider all the methods you used to learn about the program before filling out the application. Please select all that apply.

[RANDOMIZE. ALWAYS KEEP ‘Went online to the Government of Canada website’, ‘Went online to websites for other levels of government (provincial, territorial or municipal)’, AND ‘Went online to other websites’ GROUPED]

Went online to the Government of Canada website

Went online to the Government of Canada website for the [INSERT PROGRAM]

Went online to websites for other levels of government (provincial, territorial or municipal)

Went online to other websites

Used social media to get information

Called a Service Canada office directly

Called 1800 O Canada phone line

Emailed a Service Canada office

Emailed a Program Officer for [INSERT PROGRAM] directly

Went to a Service Canada office

Participated in a Government of Canada information session or webinar

Talked to my local Member of Parliament (MP)

Talked to my peers/community network

Received an email from the Government of Canada, ESDC or [INSERT PROGRAM] directly

NONE OF THESE [EXCLUSIVE; ANCHOR]

[IF Q2= ‘‘NONE OF THESE’/DK/REF SKIP TO Q6]

[IF Q2 ‘WENT ONLINE TO THE GOVERNMENT WEBSITE’ OR ‘WENT ONLINE TO THE GOVERNMENT OF CANADA WEBSITE FOR THE [INSERT PROGRAM]‘, ASK Q5, OTHERWISE SKIP]

5. On a scale from 1 to 5, where 1 is very difficult and 5 is very easy, how difficult or easy was it to find the following information about [INSERT PROGRAM] on the Government of Canada website? Select one response per item.

[ROWS. RANDOMIZE]

Find general information about [INSERT PROGRAM]

Understand the information about [INSERT PROGRAM]

Determine if your organization is eligible for [INSERT PROGRAM] funding

Determine the steps to apply for funding

Find out what information you need to provide when applying for [INSERT PROGRAM]

Determine the amount of time each phase of the application process is anticipated to take (i.e. service standard for time to acknowledge proposal, time to notify of funding decision, and time to issue payment)

Determine when the application period for [INSERT PROGRAM] takes place (i.e. opening and closing of call for proposals)

[COLUMNS]

1 – Very difficult

5 – Very easy

Don’t know

Application Process—Applying for Funding

6. To prepare and complete your application (up until when you submitted) did you consult with any of the following? Please select all that apply.

[RANDOMIZE][MULTIPUNCH]

Went online to the Government of Canada website

Went to a Service Canada office

Called a Service Canada office directly

Called 1800 OCanada phone line

Used social media to get information

Went online to other websites for information

Talked to my peers/ community network

Emailed a Service Canada office

Emailed a Program Officer for [INSERT PROGRAM] directly

Participated in a Government of Canada information session or webinar

Talked to my local Member of Parliament (MP)

Worked with a private consultant

NONE OF THESE [EXCLUSIVE; ANCHOR]

7. On a scale of 1 to 5, where 1 is very difficult and 5 is very easy, how would you rate the following elements of the application for [INSERT PROGRAM]? Select one response per item.

[ROWS. RANDOMIZE]

Understanding the requirements of the application

Putting together the information you needed to apply for [INSERT PROGRAM]

Completing the narrative questions (i.e. funding objectives, description of project, scope of project, etc.)

Completing the budget document

Completing the project timeline

Meeting the requirements of the application

[COLUMNS]

1 – Very difficult

5 – Very easy

Don’t know

9. On a scale from 1 to 5, where 1 is strongly disagree and 5 is strongly agree, please rate the following statement:

The application took a reasonable amount of time to complete.

1 – Strongly disagree

5 – Strongly agree

Don’t know

10. Which of the following methods did you use to submit your application? Please select only one.

Submitted an application using the online fillable form

Submitted an application using the Grants and Contributions Online Services (GCOS) account/ web portal

Downloaded the application documents and then submitted by email

Downloaded the application documents and then submitted by mail

Submitted application documents to a Service Canada office

Submitted on my behalf by my local Member of Parliament

Other

NONE OF THESE [EXCLUSIVE; ANCHOR]

[IF NONE OF THESE SKIP TO Q13 OTHERWISE CONTINUE.]

11. Why did you choose this method to submit your application? Please select one reason only.

[RANDONMIZE]

It was the only method available

It was the easiest/most familiar way to apply

I felt more confident my application would be submitted properly

It was the method I was directed to use

I did not know any other way to apply

Other

[IF USED ‘Submitted an application using the online fillable form’ OR ‘Submitted an application using the Grants and Contributions Online Services (GCOS)’ AT Q10 ASK Q12, OTHERWISE SKIP TO Q13]

12. On a scale from 1 to 5, where 1 means very difficult and 5 means very easy, how difficult or easy was it to submit your application online?

1 – Very difficult

5 – Very easy

Don’t know

13. After you submitted your application, were you contacted by Service Canada to provide additional information to support your application?

Yes

Don’t know

[IF YES TO Q13 ASK Q14, OTHERWISE SKIP]

14. Why were you contacted by Service Canada? Select all that apply.

[RANDOMIZE]

Missing documents or information in my application

Clarify information in my application

Budget template needed modifications

My organization or project was not eligible

An outstanding issue with a previous application

Other reason

Don’t know

Post-application—Decision

15. Did you contact Service Canada for any of the following reasons before receiving your funding decision? Select all that apply.

To check the status of your application

To find out timelines for receiving a funding decision

To withdraw your application

To modify your application

Other reason

Don’t know

[IF “DON’T KNOW” WAS SELECTED FOR Q15, SKIP TO Q17, OTHEWISE CONTINUE]

16. On a scale from 1 to 5, where 1 is very difficult and 5 is very easy, how was your experience following up with Service Canada about your application?

1 – Very difficult

5 – Very easy

Don’t know

17. How were you notified of the funding decision about your application for [INSERT PROGRAM]? Please select all that apply.

[RANDOMIZE]

By mail

By e-mail

By receiving a direct deposit

Online through your Grants and Contributions Online Services (GCOS) account

By telephone

From my local Member of Parliament (MP)

I did not receive a funding decision

[IF I DID NOT RECEIVE A FUNDING DECISION, SKIP TO Q25]

18. After you submitted your application to [INSERT PROGRAM], did your organization receive approval for funding?

Yes

[IF NO TO Q18, ASK Q19 AND Q20, AND SKIP Q21-24]

[IF YES TO Q18, SKIP Q19 AND Q20 AND ASK Q21 AND Q22]

19. You indicated that your organization did not receive an approval for funding. Did you receive an explanation why?

Yes

Don’t know

[IF YES ASK Q20, OTHERWISE SKIP]

20. On a scale from 1 to 5, where 1 is very dissatisfied and 5 is very satisfied, how dissatisfied or satisfied were you with the explanation of the decision?

1 – Very dissatisfied

5 – Very satisfied

Don’t know

[IF YES AT Q18 ASK Q22, OTHERWISE SKIP]

22. Once your project began and the details of the funding agreement were finalized with [INSERT PROGRAM], did you have to work with a Service Canada Program Officer to make any of the following changes to your project and/or submit an amendment to the funding agreement?

[ROWS]

Yes

Don’t know

[COLUMNS]

Changes to your project scope

Changes to project timelines

Changes to project activities

Changes to project funding

Covid-19 related changes

Other reason

[IF YES TO EITHER ITEM AT Q22 ASK Q23, OTHERWISE SKIP]

23. How long did the following take to complete? If you are uncertain, please provide your best guess.

[ROWS]

1 day

2 to 3 days

4 to 7 days / one week

More than 7 days / more than one week

[COLUMNS]

[INSERT ITEMS FROM Q22 = YES]

Post-agreement—Monitoring, Follow-up, and Close-out

[IF YES AT Q18 ASK Q24, OTHERWISE SKIP]

24. On a scale of 1 to 5, where 1 is very difficult and 5 is very easy, how would you rate the following tasks related to your funding agreement with [INSERT PROGRAM]? Select one response per item.

[ROWS. RANDOMIZE]

Completing the final budget/ final claim

Submitting the final budget

Completing the final project report

Submitting the final project report

Resolving any outstanding issues with funding (i.e. dealing with remaining funds, overpayment, etc.)

[COLUMNS]

1 – Very difficult

5 – Very easy

Not applicable

Tracking Service Channel Assessments

[PROGRAMMING FOR SERVICE CHANNEL ASSESSMENTS SECTION: EACH RESPONDENT WILL BE ASKED TO COMPLETE A SERVICE CHANNEL ASSESSMENT FOR EACH SOURCE USED BY THE RESPONDENT.]

[TO DETERMINE SOURCES:]

[CALLED SC OFFICE = Q2 or Q4 or Q6 ‘Called a Service Canada office directly’ OR Q17 ‘By telephone’]

[WENT TO A SC OFFICE = Q2 or Q4 or Q6 ‘Went to a Service Canada office’ OR Q10 ‘Submitted to a Service Canada office’]

[GOVERNMENT WEBSITE = IF Q2 OR Q6 ‘Went online to the Government of Canada website’]

[SC EMAIL = IF Q2 or Q4 or Q6 ‘Emailed a Service Canada office’ or Q10 ‘Downloaded application documents and submitted by email’ or Q17 ‘By email’]

[PROGRAM OFFICER EMAIL = IF Q2 or Q4 or Q6 ‘Emailed a Program Officer directly’]

[1-800 O’CANADA = IF Q2 or Q4 or Q6 ‘Called 1800 OCanada phone line’]

[ONLINE PORTAL = IF Q10 ‘Submitted an application using the Grants and Contributions Online Services (GCOS) web portal’ OR 17 ‘Online through your Grants and Contributions Online Services (GCOS) account’]

[MAIL = IF Q10 ‘Downloaded application documents and submitted by mail’ OR Q17 ‘By mail’]

25. Thinking about your overall experience, how many times did you [IF MULTIPLE SOURCES ‘use each of the following’ IF ONLY ONE SOURCE ‘use the following’]?

Please provide one response per item. If you are uncertain, please provide your best guess.

[INSERT ITEMS BASED ON SOURCE VARIABLE(S) SELECTED]

[ROWS][RANDOMIZE ITEMS]

[IF VISIT A SERVICE CANADA OFFICE SELECTED] Go to a Service Canada office

[IF GOC WEBSITE SELECTED] Go online to the Government of Canada website

[IF SC EMAIL SELECTED] Email a Service Canada office

[IF PROGRAM OFFICER EMAIL SELECTED] Email a Program Officer directly

[IF 1-800 O’CANADA SELECTED] Call 1800 O-Canada

[IF CALLED A SERVICE CANADA OFFICE SELECTED] Call a Service Canada office directly

IF MAIL SELECTED] Communicate by mail with the Government of Canada

[IF ONLINE PORTAL SELECTED] Go online to the the Grants and Contributions Online Services (GCOS) web portal

[COLUMNS]

[INSERT NUMERIC TEXT BOX BESIDE EACH ITEM]

Don’t know

26. On a scale from 1 to 5, where 1 means “very dissatisfied”, and 5 means “very satisfied”, how satisfied or dissatisfied were you with the overall quality of service you received from each of the following?

[INSERT ITEMS BASED ON SOURCE VARIABLE(S) SELECTED]

[ROWS][RANDOMIZE ITEMS]

[IF VISIT A SERVICE CANADAOFFICE SELECTED] Service Canada office

[IF GOC WEBSITE SELECTED] Government of Canada website

[IF SC EMAIL SELECTED] Email support from a Service Canada office

[IF PROGRAM OFFICER EMAIL SELECTED] Email support from a Program Officer

[IF 1-800 O’CANADA SELECTED] 1800 O-Canada phone line

[IF CALLED A SERVICE CANADA OFFICE SELECTED] Telephone support from a Service Canada office

IF MAIL SELECTED] Mail

[IF ONLINE PORTAL SELECTED] Grants and Contributions Online Services (GCOS) web portal

[COLUMNS]

1 – Very dissatisfied

5 – Very satisfied

Don’t know

Barriers and Issue Resolution

27. Thinking about your overall experience getting information and applying for [INSERT PROGRAM], did you experience any problems or issues during this process?

Yes

[IF YES ASK Q28 AND Q29. OTHERWISE SKIP]

28. How would you describe the problem or issue you experienced? [MULTIPUNCH]

[RANDOMIZE]

Application form was too long

Application form was complicated

Application requirements were difficult to understand

Information on the program was difficult to understand

Took too long to receive a funding decision

Took too long to receive an update on my application

Telephone lines were busy

I received different answers from different Program Officers

Staff were not knowledgeable/could not answer my questions

Government of Canada website information was confusing

[INSERT PROGRAM NAME] website information was confusing

Online account creation was confusing

Online application process was confusing

Technical difficulties

Other, please specify [TEXT BOX]

29. On a scale from 1 to 5, where 1 is “strongly disagree” and 5 is “strongly agree”, please rate the following statement:

The problem or issue was easily resolved.

1 – Strongly disagree

5 – Strongly agree

Don’t know

Overall Satisfaction

30. Thinking about the overall service you received, from getting information about [INSERT PROGRAM] to receiving a funding decision, how much do you agree or disagree with the following statements, using a 5-point scale where 1 means strongly disagree, and 5 means strongly agree.

[ROWS. RANDOMIZE]

I was able to move smoothly through all of the steps related to the [INSERT PROGRAM] application.

[F Q10 ‘Submitted an application using the Grants and Contributions Online Services (GCOS) web portal’ OR 17 ‘Online through your Grants and Contributions Online Services (GCOS) account’] Being able to complete steps online made the process easier for me.

It was clear what to do if I had a problem or question.

Throughout the process it was clear what would happen next and when it would happen.

I was confident that any issues or problems would have been easily resolved.

I needed to explain my situation only once. It was easy to get help when I needed it.

Overall, it was easy for me to apply for [INSERT PROGRAM]]

I was provided with service in my choice of English or French.

I was confident that my personal information was protected.

[IF SOURCE = CALLED A GOVERNMENT OFFICE OR 1800 OCANADA] Service Canada phone representatives were helpful

[IF SOURCE = WENT TO GOVERNMENT OFFICE] Service Canada representatives that I dealt with in person were helpful

[IF SOURCE = WENT TO GOVERNMENT OFFICE] I travelled a reasonable distance to access the Service Canada Office

I received consistent information

It was easy to access service in a language I could speak and understand well

[ALWAYS LAST] The amount of time it took, from when I started gathering information to when I got a decision on my application, was reasonable.

[COLUMNS]

1 – Strongly disagree

5 – Strongly agree

Don’t know

31. On a scale from 1 to 5, where 1 means very dissatisfied, and 5 means very satisfied, how satisfied or dissatisfied were you with the overall service you received from Service Canada from getting information about [INSERT PROGRAM] to receiving a funding decision?

1 – Very dissatisfied

5 – Very satisfied

Don’t know

32. On a scale from 1 to 5, where 1 means do not trust at all, and 5 means trust a great deal, how much do you trust Service Canada to deliver services effectively to Canadians?

1 – Do not trust at all

5 – Trust a great deal

Don’t know

Service Standards

[SHOW DESCRIPTION ABOVE Q36]

A service standard is a public commitment to a measurable level of performance applicants can expect under normal circumstances. Currently, Service Canada has service standards for all funding programs for the following:

Time to acknowledge a proposal: within 14 calendar days of receiving your application package.

Time to notify of funding decision: within 12 to 22 weeks (84 to 154 calendar days) from the date it was received or the end date of the intake process, depending on the characteristics of the intake method and program stream.

Time to issue payment once payment claim is submitted: For contributions, within 14 calendar days of receiving your completed claim package. For grants, within 14 calendar days of the approved project start date.

33. Before today, were you aware of each of these service standards?

[ROWS]

Time to acknowledge the submission of a funding application (within 14 calendar days of receiving your application package)

Time to issue a finding decision notification (within 84 to 154 calendar days from the date it was received or the end date of the intake process, depending on the intake method and program stream)

Time to issue payment once payment claim is submitted (For contributions, within 14 calendar days of receiving your completed claim package. For grants, within 14 calendar days of the approved project start date.)

[COLUMNS]

Yes

GBA+

Service Canada and the Department of Employment and Social Development Canada are committed to better understanding how diversity factors such as personal and social identities (e.g. gender, sexual orientation, ability, ethnicity, religious identity, etc.) may shape experiences of people or organizations applying for or obtaining federal funding.

Would the funding you applied for assist any of the following communities, clients or people:

Those who identify as women

Those who identify as belonging to a minority racial or ethnic background

Those who identify as Black Canadians

Those who identify as Indigenous (First Nations, Inuit, Metis, urban non-affiliated)

Those who identify as having a mental or physical disability

Those who identify as belonging to a religious group

Those who identify as youth

Those who identify as senior

Those who identify as veterans

Those who identify as trans, non-binary, other gender, gender-diverse or queer people

Those who identify as lesbians, gays, bisexuals, queers or other sexual minorities

Those who identify as Two-Spirit or Indigenous LGBTQQIA+ people

English or French-language minority community

Those who identify as a low socioeconomic status

Those who identify as an immigrant or a non-permanent resident

Those who identify as newcomers to Canada

Those who are experiencing homelessness and/or currently unhoused

None of the above [MUTUALLY EXCLUSIVE]

Respondent / Organization Profile

The last few questions are for statistical purposes only. All responses will remain confidential.

36. Which province/territory does your organization operate in? Please select all that apply.

Alberta

British Columbia

Manitoba

New Brunswick

Newfoundland and Labrador

Northwest Territories

Nova Scotia

Nunavut

Ontario

Prince Edward Island

Quebec

Saskatchewan

Yukon

36b. Which province/territory will your organization deliver project activities for [INSERT PROGRAM]?

Alberta

British Columbia

Manitoba

New Brunswick

Newfoundland and Labrador

Northwest Territories

Nova Scotia

Nunavut

Ontario

Prince Edward Island

Quebec

Saskatchewan

Yukon

37. Which statement best describes your organization as it relates to completing the application for funding. Select one response.

I am solely responsible for completing the funding application

A team of employees are dedicated to completing the funding application

A dedicated in-house proposal writer completes the funding application

A team of both employees and volunteers completes the funding application

A team of volunteers complete the funding application

We hire a consultant(s) to complete the funding application

I am not personally involved although I oversee this, or have some awareness

38. Was this the first application your organization submitted to [INSERT PROGRAM], or have you applied to [INSERT PROGRAM] in the past? Select one response.

First application

Applied once or twice before

Applied several times before

Apply for the same program on an annual basis

Don’t know

[IF ‘FIRST APPLICATION SELECTED’ AT Q38, ASK Q38b OTHERWISE SKIP]

38b. Has your organization submitted an application to a different Grant or Contribution program from Service Canada program in the past five years?

Yes

Unsure / don’t know

38c. Thinking about the last five years, how often does your organization apply for international, federal, provincial/territorial, and/or municipal or local funding of any kind?

[ROWS]

International

Federal

Provincial/territorial

Municipal/local

[COLUMNS]

Monthly

Quarterly

Bi-annually

Annually

Less often than annually

Never

I don’t know

38d. How many years has your organization been in operation?

Less than one year

One year to less than three years

Three years to less than five years

Five or more years

What sector does your organization operate in? Please select all that apply.

Not-for-Profit [HEADER]

Community, charitable or voluntary organizations, including faith-based organizations (for example, churches, synagogues, temples, mosques)

Associations of workers or employers as well as professional and industrial organizations

Indigenous not-for-profit organizations

Non-governmental organizations

Unions

Sector councils

Not-for-profit Band Councils

Other

Public Sector [HEADER]

Public community colleges and vocational schools

Public health, including public hospitals, nursing homes, senior citizen homes, rehabilitation homes

Public degree-granting universities and colleges

Municipal governments and agencies, including regional legislative bodies and departments

School boards and elementary and secondary institutions

Territorial governments

Other

Private Sector [HEADER]

Financial Institutions

Business, incorporated or unincorporated bodies including partnerships and sole proprietorships

Indian Band corporations

Private Band Councils

Private universities or colleges

Other

40. Approximately how many employees work (full-time or part-time) for your organization? Please select one.

None

1-4 employees

5-9 employees

10-19 employees

20-49 employees

50 or more employees

Don’t know

Approximately how many volunteers work (full-time or part-time) for your organization? Please select one.

None

1-4 volunteers

5-9 volunteers

10-19 volunteers

20-49 volunteers

50 or more volunteers

Don’t know

42. In which of the official languages does….? Please select one response for each item.

[ROW]

Your organization prefer to receive service in

Your organization provide service in

Your client population speak

[COLUMN]

English

French

Both/ Either

[END OF CORE SURVEY, ALL REMAINING MEASURES OPTIONAL]

43. Thinking about your experience with Service Canada, throughout the entire application process, have you ever felt discriminated against on the basis of your identity?

Yes

[IF YES ASK Q44, OTHERWISE SKIP]

44. On which grounds did you feel discriminated against? Select all that apply

Race

National or ethnic origin

Colour

Religion or Religious identity

Age

Sex

Sexual Orientation

Gender identity or expression

Marital status

Family status

Ability/disability

Genetic characteristics

A conviction for which a pardon or record suspension has been granted

Language

Other

[IF ‘FIRST APPLICATION’ OR ‘DON’T KNOW’ SELECTED AT Q38 SKIP Q44a, OTHERWISE CONTINUE]

44a. Comparing the service you received for [INSERT PROGRAM] in the past with your most recent experience, do you feel the quality of service has improved, declined or stayed about the same?

[ROWS. RANDOMIZE]

Overall level of satisfaction with service received

Ease of completing application

Ease of submitting application

Clarity of information on [INSERT PROGRAM] website

Ease of getting assistance when needed

The amount of time it took from when I started gathering information to when I got a decision on my application

[COLUMNS]

Improved significantly

Improved somewhat

About the same

Declined somewhat

Declined significantly

Don’t know

45. Thank you for participating in Service Canada’s Grants and Contributions client experience survey. This survey collects valuable insight about your satisfaction, how services are delivered, and areas for improvement. The research also involves a qualitative phase of focus groups and interviews expected to take place in March-April 2022.

Are you interested in having a representative from Ipsos contact you to participate in focus groups or interviews about the service you received?

Yes

[IF YES ASK CONTACT INFO, OTHERWISE END SURVEY]

46. You indicated you would like to participate in the qualitative phase of focus groups and interviews. Your feedback is important to us and your answers will always remain confidential, meaning that your contact information will not be shared with Service Canada and will not be attached to your survey responses. Please provide an email address and phone number where Ipsos may contact you. [mandatory if they indicate ‘yes’]

Full Name

Email [VALID FOR PROPER EMAIL]

Phone Number [VALIDATE FOR PROPER PHONE NUMBER]

The survey is complete. This survey is part of a larger research study. Your anonymous responses will be combined with all responses gathered during the survey, then aggregated, analyzed and published as public information.

The full detailed survey report, along with an executive summary and data tables will be posted on the Library and Archives Canada website within six months of the conclusion of this study.

Should you have any question, you can reach those responsible for the survey at Service Canada (Employment and Social Development Canada) directly by emailing pob.ri.dgop@canada.ca.

Thank you for your participation.

ANNEX 2:
Discussion Guide for Online Focus Groups and In-depth Interviews - Grants and Contributions Applicants Client Experience Research (Year Two)

AUDIENCE BREAKDOWN

Group Composition

Group 1: Unfunded applicants to any program other than ELCC and SDG, or those who are unsure

NATIONAL - ENGLISH

Group 2: Funded applicants to any program other than ELCC and SDG

NATIONAL - ENGLISH

Group 3: Unfunded applicants to any program other than ELCC and SDG, or those who are unsure QUEBEC - FRENCH

Group 4: Funded applicants to any program

QUEBEC - FRENCH

In-depth interviews were focused on applicants to ELCC and SDG programs

SESSION BREAKDOWN

	Focus Group	IDI
Welcome and Introduction	10 Minutes	5-10 Minutes
Section 1: Organizational Characteristics	20 -25 Minutes	10 Minutes
Section 2: Service Dimensions	20 -25 Minutes	10 Minutes
Section 3: Client Needs and Expectations	20 -25 Minutes	10 Minutes
Section 4: New or Unique Quantitative Findings	20 -25 Minutes	10 Minutes
Wrap Up	10 Minutes	5-10 Minutes
SESSION TOTAL	90-120 Minutes	50 -60 Minutes

DETAILED SESSION AGENDA

MODERATOR WELCOME (10 MINUTES)

· Welcome & thanks for attending

· My name is [FULL NAME]

· Ipsos - research company and neutral third party

· Expected length of this session is approximately 2 hours (FOCUS GROUP) / 1 hour (IN-DEPTH INTERVIEW)

· The role of the moderator is to ask questions, act as a timekeeper, and remain objective/no vested interest

· Anonymity of your participation - remarks are not attributed and your privacy will be protected, results are confidential and reported in aggregate

· Recording will be used by the researchers to inform our report writing. As well, transcripts will be provided to our direct client at Service Canada. Please note, before the transcripts are sent, we will delete any references that might disclose the identity of any participant (e.g. use of full name, reference to employer or position).

· We will be obtaining written confirmation from the client that the recordings or transcripts provided will be used for research purposes only.

· The confirmation must also state that the recordings or transcripts will not be shared with a third party, except in those cases where participants have given explicit informed consent for such sharing.

· Please let me know at this time if you have any questions about the recordings or transcripts.

· There may be one or more Service Canada staff members observing the focus group (or interview). They will be a neutral researcher with no connection or involvement with the Grants and Contributions funding decision making.

· Your participation and responses today will not in any way affect your relationship with the Government of Canada, or your application / funding

· Rules of engagement for participants – you are not expected to be experts, speak openly and frankly about opinions. There are no right or wrong answers.

· Open and respectful dialogue, don’t all need to agree with each other

· Technical considerations in using online video meeting / focus group platform

For today’s discussion: [VERBAL INTRODUCTION; NO RESPONSE REQUIRED]

We are speaking to clients who are currently or have recently applied for a Grant or Contribution through Service Canada, also known as Employment and Social Development Canada (ESDC).

I know that some of you may have applied to many programs in previous years, so I would like to clarify the timeframe of today’s discussion.

We will be discussing the program that you applied to between April 1, 2019, and March 31, 2020, or between April 1, 2020, and March 31, 2021.

Please keep this timeframe in mind in your responses.

· For today’s discussion, we would like:

o To understand how the characteristics of your organization either help or present challenges when applying for funding. This may include organizational characteristics for those serving diverse populations, those organizations that were successful or unsuccessful in obtaining funding, and/or organizations that did or did not re-apply for funding.

o To understand different aspects of service that are important

o To determine your needs and expectations around service delivery

o To identify potential changes to service delivery that would improve your experience

· For those of you who may have applied for funding or had your project impacted at any point during COVID-19 – we also welcome the opportunity to hear about these experiences. However, we would like to hear about your experience throughout the whole life cycle of applying including before COVID-19. Please be mindful of this in your feedback.

I would also like you to consider diversity in our discussion. This includes both your diversity and the diversity of the population(s) your organization serves. For example, we are interested in how diversity factors may shape and intersect with your experiences with the funding process.

· Service Canada/Employment and Social Development Canada is committed to better understanding how diversity factors such as ethnicity, religion, ability, and age shape people’s experience with government policies, programs, and services.

· If at any point in the discussion, you feel that this is a consideration in one of your responses, please feel free to bring this up proactively.

SECTION 1: ORGANIZATIONAL CHARACTERISTICS

(20-25 MINUTES)

Question 1.

The first item I’d like to discuss today is the organization that you lead, or work/volunteer for. I’d like to get to know each of your organizations in more detail.

I’d like to start with a round of introductions – please share your first name, the type of organization you work for, your role at your organization, and which federal government grant or contribution program you applied for. I’d also like to understand how you applied.

MODERATOR TO ASK EACH PARTICIPANT TO INTRODUCE THEMSELVES AND THEIR ORGANIZATION IN DETAIL

DO NOT READ – FOR MODERATOR REFERENCE ONLY:

· Submitted an application using the online fillable form

· Submitted an application using the Grants and Contributions Online Services (GCOS) account/ web portal

· Downloaded the application documents and then submitted by email

· Downloaded the application documents and then submitted by mail

· Submitted application documents to a Service Canada office

· Submitted on my behalf by my local Member of Parliament

Probes:

Interview participant role:

First, I’d like to understand / hear more about you.

· Are you an employee, or is this a volunteer role?

· What is your role in applying for a grant or a contribution (e.g., leader/management, decision-maker, writer, gathering information and supporting documentation, finding and connecting with resources, submitting and/or following up on the application)?

· How long / how many years have you been with this organization, and how long have you been involved in applications for funding?

Organizational characteristics:

Now, I’d like to hear more about the organization you work or volunteer for.

· Can you tell me more about your organization?

· What population(s) are served by your organization?

· Can you describe the community/area where you are located? What about where you provide services? For example, whether you are located in, and/or serve a small community or if you are in a large metropolitan / urban area.

· Is your organization a new client/applicant to this program?

· Have you successfully received federal funding in the past?

o Specifically, what about funding from ESDC/Service Canada?

o And more generally, what other funding has been received?

Question 2.

I would like to hear from each of you about your experience in applying for funding, and how you would characterize it in your own words.

SUBTITLE FOR REFERENCE ONLY: Questions on organization characteristics and success

· Think about your organization. What is it about your organization that may have helped/supported you with your positive/satisfactory/easy experiences?

o For example, applying repeatedly and successfully each year, having internal resources, expertise, or supports, feeling confident about your application….

o Please think about items such as experiences, skills, expertise, resources, relationships, external consultants, etc.

· Thinking again about your organization, what are some of the qualities of the application process that made your organization well-equipped to apply for funding?

o For example, it was easy to apply, clear steps, clear language was used, process is easy to understand, there was an information session, applying online, etc.

o Is there anything else?

SUBTITLE FOR REFERENCE ONLY: Questions on organization characteristics and challenges

· Thinking about your organization, help me understand any barriers or challenges you faced.

· I would like to focus first on the characteristics of your organization and the challenges you experienced. For example, tell me about how you may not have been well-equipped, or not well supported for the application process.

o For example, organization structure: too small, membership, years in operation, (no) resources, (no) dedicated person (employee or volunteer), too busy serving clients, etc.

o Individuals: lack of training, lack of time, no expertise, front line staff without expertise in grant process, etc.

o Access to resources/expertise: no previous experience, new process, no expertise in the organization, unable to retain consultant, etc.

o Barriers related to identity, accessibility, complexity, time commitment to complete, etc.

· Thinking again about your organization, what are some of the characteristics of the application process that made your organization ill-equipped or created struggles when applying for funding?

o For example, not sure how to fill out the application, challenging /inaccessible language, instructions not clear, not clear on who to contact for support, process was confusing or difficult, etc.

o Is there anything else?

Question 3.

Let’s continue thinking about barriers or challenges you have faced.

Some of you may identify as belonging to diverse populations. And, others may be working with / serving diverse populations.

In both instances, I would like to understand if and how belonging to or serving diverse populations affect the process of applying for a federal grant or contribution.

· Please tell me, what, if any, barriers or challenges did you face?

· Tell me whatever you are comfortable to share – first about the diverse population you belong to or serve, and then tell me about the barriers or challenges. Please share whatever you are comfortable to tell me.

LISTEN FOR / PROBE BUT DO NOT READ - FOR MODERATOR REFERENCE:

· Those who identify as women

· Those who identify as belonging to a minority racial or ethnic background

· Those who identify as Black Canadians

· Those who identify as Indigenous (First Nations, Inuit, Metis, urban non-affiliated)

· Those who identify as having a mental or physical disability

· Those who identify as belonging to a religious group

· Those who identify as youth

· Those who identify as senior

· Those who identify as veterans

· Those who identify as trans, non-binary, other gender, gender-diverse or queer people

· Those who identify as lesbians, gays, bisexuals, queers or other sexual minorities

· Those who identify as Two-Spirit or Indigenous LGBTQQIA+ people

· English or French-language minority community

· Those who identify as a low socioeconomic status

· Those who identify as an immigrant or a non-permanent resident

· Those who identify as newcomers to Canada

· Those who are experiencing homelessness and/or currently unhoused

Question 4.

ASK FUNDED GROUPS:

· Even upon being successful, what did you wish you knew about/during the application process?

· Can you tell me about any lessons that you learned?

· Will you apply again in the future for the Service Canada/ESDC grant or contribution?

· Tell me the considerations that will go into whether you apply again.

ASK UNFUNDED GROUPS:

· Can you tell me the things that you wish you knew while you were applying? Or, what would you tell someone who was applying?

· Can you tell me about any lessons that you have learned?

· Now that you have been through the process of applying, where do you stand on applying in future years to a Service Canada/ESDC grant or a contribution? Please tell me in detail the reason for your response and what considerations go into whether you apply again.

ASK IF ORGANIZATIONS MENTION BEING FUNDED OR UNFUNDED IN PAST YEARS:

· Do you apply whenever the program becomes available? Are there any considerations that you take into account each year?

· Is the criteria to be successful in the funding process clearly communicated to you?

· When there was a decision for your application, did Service Canada/ESDC clearly explain the decision to you? Did Service Canada/ESDC clearly explain why you were funded/unfunded?

SECTION 2: SERVICE DIMENSIONS

(20-25 MINUTES)

Question 5.

Thank you for that discussion.

I would now like to discuss the service you received from Service Canada. When Service Canada looks at quality of service, they evaluate it using three areas:

MODERATOR TO DISPLAY ON SCREEN QUIETLY, THEN READ ALOUD:

· Ease: simplicity, clarity, and convenience of the information and service

· Effectiveness: availability, timeliness, and consistency of help and information; effectiveness of service

· Emotion: respectful treatment and confidence in service

I would like to discuss each of these in detail as it relates to your whole experience with Service Canada. This includes getting information about the program you applied for to receiving a funding decision.

· First, let’s start with EASE. Service Canada defines EASE as:

o Information was easy to find when needed

o You only had to input your information once

o The information was easy to complete and understand

o The process was easy to determine and it was easy to know the steps, the information needed, how to get help, etc.

o You could get information easily

Given these elements of ease, tell me about your experiences from finding information about a grant or contribution to receiving a funding decision.

· Next, let’s discuss EFFECTIVENESS during your grant or contribution experience. Tell me about how:

o You received the information you needed

o You were able to get help when you needed it

o You received service in the official language of your choice (documents or in-person service)

o You were provided and/or provided feedback easily

o The process was transparent (including the process, stages, status)

o There was a reasonable amount of time to access the service, complete task(s), receive information, resolve an issue, or receive a decision

o You received consistent information throughout the process

o The process was easy to follow to complete the tasks

o You were able to complete tasks/resolve issues

o You knew what to do if you had a problem

o You felt as though you were always advancing/moving forward in the process

With these elements of effectiveness, tell me about your experiences with effectiveness.

Probe as relevant: Timing is from finding information about a grant or contribution to receiving a funding decision.

· Finally, we will discuss EMOTION. This relates to how you experienced respectful treatment and had confidence in the services offered to you/your organization. Tell me about how:

o (if applicable) Your interaction Service Canada agent(s) was respectful, courteous, and helpful

o Service Canada demonstrated an understanding and ability to address your concerns

o Your personal information was protected

o You were confident that you were following the correct steps

o And you knew that the information/decision will be received and know what steps to take next.

Given these elements of emotion, what else can you share about your experiences?

· In the previous discussion, we discussed the link between identifying as/serving a diverse population and your experiences.

Thinking about your personal identity, or those of the community that your organization serves, do you feel that you faced challenges or barriers that may relate to this service area?

Please tell me whatever you are comfortable to share.

AFTER ALL 3 SERVICE AREAS HAVE BEEN DISCUSSED:

Question 6.

· Of these 3 service areas, is there one that is more important to you than the others, or do you consider them all important? Please tell me in detail the reason for your response.

Question 7.

For those of you who applied online using the Grants and Contributions Online Services (GCOS) account/ web portal for your grant or contribution, I would like to understand your experiences with the online environment.

· How would you characterize the experience in your own words?

· Was it easy to use and navigate?

· Were there any difficulties or pain points? What were these challenges?

· What could be improved (navigation, saving information, layout, functionality, speed, information provided on how to apply) that were challenging or could be improved?

ASK EAF PROGRAM PARTICIPANTS ONLY:

Did you use the Enabling Accessibility Fund (EAF) calculator (If needed: this is the online tool that was available to you, that allows you to estimate your budget)? How easy to use and helpful was this tool? Please tell me in detail the reason for your response.

Question 8.

· For those of you who did not apply online, what was the reason for that?

· What would you need to hear or have in place in order to encourage you to apply online?

· What could Service Canada do for you to help you apply online?

· What would you or your organization need to do to apply online? Or, is it something that Service Canada would need to do for you?

SECTION 3: CLIENT NEEDS AND EXPECTATIONS

(20-25 MINUTES)

Question 9. [IF NEEDED BY MODERATOR]

I would like to have a discussion about the channel that you used to apply for the Service Canada/ESDC grant or contribution, that you told me about previously. Could you please remind me how you applied?

PRESENT ONLY IF ASKED: FOR MODERATOR REFERENCE ONLY:

Some of the channels to apply include:

· Submitted an application using the online fillable form

· Submitted an application using the Grants and Contributions Online Services (GCOS) account/ web portal

· Downloaded the application documents and then submitted by email

· Downloaded the application documents and then submitted by mail

· Submitted application documents to a Service Canada office

· Submitted on my behalf by my local Member of Parliament

Thinking about your application:

· How could your experience with this channel have been improved? What were the positives, what were the drawbacks?

· Can you tell us where/if you struggled?

· Do you have a preference in how you apply? Is this normally how you would apply? What is your preferred channel? Why did you use it?

Question 10.

Regardless of the channel that you used, I’d like to understand what your ideal experience would be in applying for a grant or contribution at Service Canada.

[Probe, you may wish to draw on the current experience or experiences with other funding programs or departments to inform you ideal experience]

· Please walk me through from start to finish, what your organization would do, and anything Service Canada could do to create an ideal experience.

· Let’s start with what you and your organization would do in this ideal experience.

· Now, let’s consider what Service Canada could do to create an ideal experience.

Question 11.

I would like to know and understand your experience with getting help during the application process – be it internally within your organization, from Service Canada, or from any other source (e.g., consultant).

· How easy or difficult was it to get help on the application? What were the factors that contributed to this experience?

· Regardless of your previous experience, what is the best way to get support in applying for a federal grant or contribution?

Question 12 (Time Permitting)

I would like to understand if and how the COVID-19 impacted the application process.

· Did COVID have an impact on your application? If so, what was it?

· What could have mitigated these impacts?

SECTION 4: NEW OR UNIQUE QUANTITATIVE FINDINGS

(20-25 MINUTES)

Question 13.

If you apply for a Service Canada/ESDC Grant or Contribution program regularly, how familiar are you with the program / process? Is there a learning curve each time you apply?

Question 14.

I would like to understand your views on the length and complexity of the application.

· What was your experience with the length of the application?

· Was there anything you found confusing or didn’t understand in the application? If so, how could it have been better explained, or how could you have been supported?

· How does it compare to other funding programs you apply to? Those at Service Canada/ESDC? Can you compare and contrast with other federal, provincial, or municipal programs?

Question 15.

I would like to know if you experienced technical issues when applying.

If you experienced a technical issue, please let me know what it was, and what if anything could be done to improve this in future.

· Probes: portal issues, navigation, system glitches, usability

Question 16.

How about other issues – did anyone experience those? If so, what were they?

· Probes: application, budget, narrative questions, timeline, impressions of forms, documentation required

· Ask FUNDED: funding agreement, reporting

Question 17.

In terms of receiving a decision, how long did that take?

· Do you feel this was a reasonable wait time?

· Did you attempt to follow up with Service Canada/ESDC? If so, did you receive a response?

· What is the impact, if any, of not receiving a decision when you hope / expect?

Final question/Wrap up

· Is there anything else that you would like to share about your experience with Service Canada/ ESDC’s grant or contribution?

· Are there any reflections or experiences that you have already discussed that you would like to revisit?

MODERATOR TO CHECK WITH BACKROOM / OBSERVERS FOR ANY ADDITIONAL QUESTIONS

SESSION CONCLUSION AND WRAP-UP

ANNEX 1: Questionnaire - Grants and Contributions Client Experience Research (Year Two)

ANNEX 2: Discussion Guide for Online Focus Groups and In-depth Interviews - Grants and Contributions Applicants Client Experience Research (Year Two)

ANNEX 1:
Questionnaire - Grants and Contributions Client Experience Research (Year Two)

ANNEX 2:
Discussion Guide for Online Focus Groups and In-depth Interviews - Grants and Contributions Applicants Client Experience Research (Year Two)