Calculation of Relative Impact of Service Dimensions on Satisfaction

1. Creation of Composite Measures

The first task was to create composite measures for the following service dimensions: ease, effectiveness and overall experience. The objectives followed were:

• To maintain as many cases as possible in each composite measure;
• To keep the meaning and interpretation of the composite measures intuitive;
• To adjust the approach when necessary to accommodate different scales; and
• To minimize the use of composite measures that resulted in analyses relevant only for specific sub-groups (e.g., clients who needed to provide missing information regarding their application (n=285). 

Keeping as many cases as possible

A key consideration was to optimize the number of cases in the analysis, given that certain service dimensions were used to examine the client’s experience of individual service channels rather than the experience of completing a service task, which can involve multiple channels, and some service tasks were experienced by only a portion of the client population, such as providing information missing from an application, getting assistance, etc., which resulted in lower sample sizes for these service dimensions.  To minimize the loss of cases, the SPSS mean function was used to compute composites (rather than a summation of variables which results in cases being excluded any time there is a missing value on one of the variables).¹ The mean function used a single valid value if there was only one valid variable value and the mean if there were two or more valid variables. In the case of combining an in-person and telephone measure, it used the in-person value if an individual only used the in-person channel, the telephone value if they only used the telephone, and the mean of the two values if they used both channels. This resulted in a composite score for all cases except those surveyed clients who did not use either of these service channels.

Keeping the measures intuitive

Another advantage of this approach to creating the composite measures was that it retained the original scales. If there were five scales from 1 to 4, for example, the composite score has a potential range of 1 to 4 because the mean can never be lower than the lowest value and never higher than the highest value. This helped to keep the meaning and interpretation of the composite measures intuitive. This approach could have been used for the yes/no (binary) variables as well, but the mean of the yes/no variables would not be very intuitive when it comes to interpretation. Instead, yes/no variables were converted to 0 or 1 values and a simple addition was used to create a measure that counted the number of problems (e.g., yes: needed assistance) or successes (e.g., yes: got the information in a reasonable amount of time). While this is relatively intuitive for most people to interpret, it does suffer from the problem mentioned previously that cases will be lost if there is a missing value for any one of the variables. 

Accommodating different scales

This approach also accommodated the different scales found in the variables to be incorporated in the composite measures (e.g., yes/no variables, 4-point scales or 5-point scales).² To combine the yes/no and scale variables, scale variables were converted to a yes/no, hi/low³ format and then recoded to 1 or 0 values. The variables were then summed to obtain counts as described above. To combine a 4-point and 5-point scale, one of the scales was converted to the scale range of the second variable by examining the pattern and relationships of the variables. Once all variables were converted to the same scale, then the SPSS mean function was used to create the composite measures.  

Minimizing the creation of analysis only relevant to unique client groups

A consideration linked to the objective of avoiding loss of cases was to avoid unintentionally creating analysis relevant only for specific sub-groups of EI clients. Regression modelling is very sensitive to the inclusion of variables with smaller cases. The inclusion of variables with a relatively small number of cases (e.g., 200 or 500 out of 1,528 cases) may result in a specific group of cases being systematically excluded from the analysis (e.g., clients who had to provide additional information after submitting their application). This approach to creating the composite measures reduced the likelihood of case loss and the creation of analysis specific to small sub-groups. Exclusions of entire sub-groups of respondents would have limited the accuracy of the results and the extent to which the results could be generalizable to the entire EI population.   

2. Satisfaction Measures

The survey contained two client satisfaction measures of interest:

• Q39: How satisfied were you with the overall quality of service you received from Service Canada related to your EI benefits during the last 7-8 months? (5-point scale)
• Q40: If someone were to ask you, would you speak positively about the service you received? (4-point scale)

An objective of this analysis was to assess which measure serves as a better dependent variable. For this reason, a composite satisfaction measure was created (Q39 and Q40), and then the three satisfaction measures (the composite measure, Q39 and Q40) were examined in a bivariate correlation analysis. The regression analysis focussed only on the overall satisfaction measure Q39 and the composite satisfaction variable (Q39 and Q40 combined).

To create the composite variable, the reverse scale of Q40 needed to be addressed, as did the different scales (Q39 uses a 5-point scale and Q40 uses a 4-point-scale). To make the direction of the Q40 scale the same as other variables, the values were reverse coded so that 1 was a very negative response and 4 was very positive. One of the variables then needed to have its scale converted to the same scale as the other variable. After examining a crosstab between Q40 and Q39 to see where values seemed to correspond the best, Q40 was converted to a 5-point scale by making the end points the same as Q39, 1=1 and 4=5. Based on the analysis 3 was most similar in distribution to 4 on Q39’s 5-point scale and 2 was recoded to 2.5.  A composite overall satisfaction variable was created using the SPSS mean function.

3. Ease Measures and Client Satisfaction

To the extent possible, composite ease variables were created as follows:

Phase 1 of client journey: Information Gathering

• Q6a: Easy to find the information you were looking for.
• Q6b: Easy to determine if you were eligible for EI benefits.
• COMPOSITE: Ease of finding information

Phase 2 of client journey: Apply for Benefits

• Q14a: Understand the requirements of the application.
• Q14b: Put together the information you needed to apply for EI.
• COMPOSITE: Ease of application process

Phase 3 of client journey: Follow-up

• Q20a: Understand the information in the letter you received
• Q20b: Understand the next steps.
• Q20c: Understand what information was missing.
• COMPOSITE: Ease of understanding follow-up information

COMPOSITE: Ease of overall process

Table 1 shows the simple correlation coefficients between the overall satisfaction measures and the individual and composite ease variables, the significance levels, and the number of cases. Some observations from this simple bivariate analysis:

• The correlations between the composite satisfaction measure (Q39 and Q40 merged) and the ease variables were typically larger than the individual satisfaction measures.
• Some ease variables (individual and composite) excluded approximately 400 cases or more (Q20 variables) and represent a sub-group of EI clients that had follow-up communications related to their EI application.
• Although the combined measure may be slightly more robust, the simple overall satisfaction variable (Q39) yields similar results in most cases.
• The simple overall satisfaction variable had a stronger relationship to the ease variables than the variable, “would speak positively about the service received” (Q40).
• Among the ease variables, the best predictor of the composite variable, and the single satisfaction variables, was the overall ease measure (i.e., the index of all composite ease measures across the entire client journey).
• Regarding the pre-application ease composite of ease of finding information (Q6a) and ease of determining eligibility (Q6b), the correlation was almost as strong as the overall ease index, suggesting that the addition of the other ease variables (Q14a, Q14b, Q20a, Q20b and Q20c) to the index adds no predictive value. 

• *p < .05
• **p < .001

4. Effectiveness Measures and Client Satisfaction

Composite measures for the effectiveness variables were created as follows:

Phase 1 of client journey: Information Gathering

• Q5: Did you get the information you needed in a reasonable amount of time?
• Q6c: Easy to determine the steps in the application process.
• Q6d: Easy to know what documents you needed to apply for EI benefits.
• COMPOSITE: Effectiveness of process

Phase 2 of client journey: Apply for Benefits

• Q12: Was the assistance helpful in getting you what you needed?
• Q13: Were you able to complete the application in a reasonable amount of time?
• Q14c: Complete the online application form.
• COMPOSITE: Effectiveness of application process

Phase 3 of client journey: Follow-up

• Q17: Was this a reasonable amount of time to wait (for a decision)?
• Q20d: Submit the missing information to Service Canada.
• Q20e: Know what to do if you had a problem in submitting the information
• Q20f: Get information on the status of your application.
• COMPOSITE: Effectiveness of follow-up process

COMPOSITE: Effectiveness of overall process

Table 2 shows the simple correlation coefficients between the overall satisfaction measures and the individual and composite effectiveness variables, the significance levels, and the number of cases. Some observations from this simple bivariate analysis:

• As was the case with the ease measures, the correlations between the composite satisfaction measure and the effectiveness variables were typically larger than the individual satisfaction measures.
• Again, as was the case with the ease measures, some effectiveness variables (individual and composite) excluded a large number of cases.
• The combined satisfaction measure may be slightly more robust (as was the case with the ease variables), but the simple overall satisfaction variable (Q39) yields similar results in most cases.
• As was the case with “ease”, for many variables, the simple overall satisfaction variable had a stronger relationship to the effectiveness variables than the variable would speak positively about the service received (Q40).
• The largest correlation with the composite overall satisfaction measure and the single variable overall satisfaction measures was the composite effectiveness variable for the 5-point scales (Mean of Q27A, Q32A, Q27E, Q37A).  One of the variables in this index had a similarly high correlation, Q37A “You were able to move smoothly through all of the steps related to your EI claim”. 
• The composite effectiveness variable with the 4-point scales had a stronger correlation with the composite satisfaction variable than the individual 4-point scales. 

• *p < .05
• **p < .001

5. Emotion Measures and Client Satisfaction

The emotion measures were not included in this analysis because all but one variable was specific to in-person service.

• Q27c: [PHONE] Staff were helpful.
• Q32c: [IN-PERSON] Staff were helpful.
• COMPOSITE: Staff were helpful
• Q27d: [PHONE] You were treated respectfully.
• Q32d: [IN-PERSON] You were treated respectfully.
• COMPOSITE: You were treated respectfully
• Q37c: You are confident that any personal information you provided will remain confidential. 
• COMPOSITE: Overall experience

These measures would be more appropriate for segmented analysis by channel. 

6. Regression Models

Table 3 shows the results for the regression models for overall satisfaction measures and overall composite ease and effectiveness measures. Key findings were:

• The regression models show the single composite measure for ease and the two effectiveness composite measures could explain nearly 45% of the variance in the composite satisfaction measure (mean of Q39 and Q40) and a similar percentage for the simple overall satisfaction measure (Q39). 
• The variance explained by the ease and effectiveness measures for “Would you speak positively about the service you received” (Q39) was lower, 33%, but still highly statistically significant. 
• The best predictor for the composite satisfaction measure (mean of Q39 and Q40) was the composite effectiveness measure that used the 5-point scales (in Table 3, this is “3: COMPOSITE: Effectiveness of overall process”⁴). The coefficient was twice the size of the other variables. On its own, this variable accounted for nearly 39% of the variance in the composite satisfaction measure.
• Although not shown in Table 3, when the single best 5-point effectiveness measure, Q37A “You were able to move smoothly through all of the steps related to your EI claim”, was entered into the equation first, the effectiveness index using all of the 5-point scales added additional predictive value. This means that smooth progression through all steps is an important predictor of overall satisfaction, but that a reasonable wait time for a decision, complete responses, and consistent information add additional predictive power above this single measure.
• The same was not true of the ease variables. When the single best sub-index for ease was added to the equation first, “Ease of finding information during the pre-application phase of the client journey”⁵, the overall ease index using all the ease indices did not enter the equation. This means that a simpler composite measure for ease of finding information during the pre-application phase could account for virtually all of the variance explained in the composite satisfaction variable on its own. This does not mean these other measures are not important. An adverse change in these ratings would be expected to be associated with a decrease in overall client satisfaction.  However, the results do suggest that a focus on improving the ability of clients to find the information they are looking for and figure out if they are eligible should be a priority.

7. Summary

• Regarding the satisfaction measures, the combined variable (Q39 and Q40) performed slightly better than the single measures. However, the simple overall satisfaction variable (Q39) yielded similar results in most cases; this was not the case for Q40. 
• Ease and effectiveness measures explain nearly 45% of the variance in the composite satisfaction measure (mean of Q39 and Q40) and a similar percentage for the simple overall satisfaction measure (Q39). This means that changes in the average rating of any one of these composite variables would be expected to produce a change in clients’ overall satisfaction rating.
• With respect to individual measures, the smooth progression through all steps is the most important single predictor of overall satisfaction.
• Though not as strong, the ability of clients to find the information they are looking for and figure out whether they are eligible, the wait time for a decision, and receiving complete and consistent information, also have a notable impact on overall satisfaction.
• Finally, note that it would be reasonable to expect that factors not included in the model due to smaller sample sizes, such as respectful treatment, drives satisfaction among EI clients in light of the previous research (2006, 2008 and 2011 Service Canada Client Satisfaction Surveys). However, we do not know its relative strength compared to the other service dimensions.

Footnotes

¹ For example, if creating a composite measure that combines an in-person variable (survey question) with a telephone variable, adding the values will lose all cases where only one of the channels was used. To minimize response burden, surveyed clients were asked about their experience with one service channel.

² Another approach was to use standardized variables, but after testing this approach it was discarded because the composite measures resulting from standardized scores would have had limited intuitive meaning.

³ For example: High: agree, Low: disagree.

⁴ This includes (Q27A+Q32A) Questions were answered completely on phone and in-person, (Q27E) You received conflicting information from different phone agents, and (Q37A) You were able to move smoothly through all of the steps related to your EI claim.

⁵ This includes Q6A and Q6B: Would you say it was very difficult, somewhat difficult, somewhat easy, or very easy to: Find the information you were looking for. Would you say it was very difficult, somewhat difficult, somewhat easy, or very easy to: Determine if you were eligible for EI benefits.

⁶ This includes (Q27A+Q32A) Questions were answered completely on phone and in-person, (Q27E) You received conflicting information from different phone agents, and (Q37A) You were able to move smoothly through all of the steps related to your EI claim.