2012-603 Evaluation of Translation Bureau programs - Volume 3: Conference Interpretation
Program (Final report)

2012-603 - Volume 3 Final report (PDF, 439KB)

January 21, 2014

Introduction

1. The following presents the results of the Evaluation of the Conference Interpretation Program of the Translation Bureau. This Program is situated in the Public Works and Government Services Canada's (PWGSC) 2012-13 Program Alignment Architecture under sub-program 1.6.5 (Conference Interpretation). The Deputy Minister for PWGSC approved the conduct of this Evaluation, on recommendation of the Audit and Evaluation Committee, as part of the  2012-17 Risk-Based Audit and Evaluation Plan.
2. The Evaluation provides an assessment of the relevance and the performance of the Program. Planning and research took place between May 2012 and March 2013, in accordance with the evaluation standards of the Government of Canada and the Office of Audit and Evaluation at PWGSC. More information on the approach, methodologies and limitations of this evaluation can be found in Appendix A.
3. The research for this evaluation was conducted in conjunction with the research for the Evaluation of the Translation Bureau's related Programs (including the Translation and Other Linguistic Services Program and the Terminology Standardization Program). An analysis was conducted to determine the appropriate level of research effort to be dedicated to the Evaluation of the Conference Interpretation Program. Compared to the other Translation Bureau programs which underwent simultaneous evaluation, the Conference Interpretation Program's materiality is relatively low and the overall risks of the Program are low. Program spending accounts for less than 1% of the total PWGSC budget and 7.98% of the Translation Bureau budget for 2012-13, and the Program has a clear and unambiguous mandate. As a result of these factors, the evaluation team chose a compressed format to report on the core areas of evaluation.

Why it is important

4. The Conference Interpretation Program ensures the provision of conference interpretation services in both official languages and other languages to the judiciary and federal departments and agencies in support of government's operations, official events and missions abroad. Interpretation services are essential to the ability of the federal government to communicate with Canadians, and for the public to access the services of the federal government, in their official language of choice. For these reasons, as well as to meet the PWGSC's obligations under the Treasury Board Policy on Evaluation, it is important to periodically evaluate the Program's continued relevance and its performance to support decision-making.

About the program

5. The Translation Bureau, within PWGSC, is the federal organization that provides translation, interpretation and terminology services to the judiciary, federal departments and agencies and to Parliament. It supports the Government of Canada in its efforts to communicate with and provide services for Canadians in the official language of their choice. It was established in 1934.
6. The Conference Interpretation Program provides interpretation services in both official languages, Canada's Aboriginal languages, foreign languages and in visual languages to all federal government departments and agencies. Conference interpreters provide interpretation services at events such as international summits, bilateral or multilateral discussions between heads of state/government, intra- or inter-departmental conferences and meetings between federal ministers and their provincial or territorial counterparts. In addition, the Program also provides "elbow" interpretation services where the interpreter follows the proceedings in the same room and in a low voice translates what is being said into the client's ear.
7. The provision of official languages interpretation for the debates in the House of Commons, the Senate, Cabinet and their committees, press conferences and the proceedings of parliamentary associations are captured under sub-program 1.6.4 (Translation and Interpretation to Parliament) and were therefore excluded from the scope of this evaluation.
8. The Translation Bureau became a Special Operating Agency, reporting to PWGSC, in 1995 by decision of the Treasury Board. At that time, both the Conference Interpretation Program and the Translation and Other Linguistic Services Program became optional services to federal departments and agencies (except for Parliamentary Interpretation). Federal organizations have since been unrestricted in their choice to procure both translation and other linguistic services or conference interpretation services from the Translation Bureau or from the private sector directly.
9. Currently, conference interpretation services are paid for via transfers to the Translation Bureau Revolving Fund from PWGSC's Vote 1. This covers the costs in cases of interpretation services provided to federal departments and agencies listed under Schedules I, II, IV and V of the Financial Administration Act. These organizations also pay for travel, travel allowances (such as meals and incidentals^{Footnote 1}) and accommodation fees associated with interpretation services.
10. In fiscal year 2012-13, the total reported expenditures for the Program were $14.25M in direct and indirect costs. This included 44.75 full-time equivalent staff^{Footnote 2}, 15.69 of which were interpreters, 20.46 were administrative support staff^{Footnote 3} and 1.97 were management^{Footnote 4}. The Program represents 7.98% of the total costs of the Translation Bureau.
11. A logic model for the Translation Bureau was developed for the Evaluation and is provided in Appendix B. The activities, outputs and outcomes for the Conference Interpretation Program are bolded and italicized. The Conference Interpretation Program shares its intermediate outcomes with other Translation Bureau programming: The Terminology Standardization Program and the Translation and Other Linguistic Services Program.

Authority

12. The Translation Bureau Act, updated in 1985, established the Translation Bureau and outlined its duties and functions, including those related to the provision of interpretation services. The act also authorized the Minister of Public Works and Government Services (PWGS) to develop supporting regulations.
13. The Minister of PWGS is authorized to offer linguistic services to other government departments by the Department of Public Works and Government Services Act:

    "The powers, duties and functions of the Minister extend to and include all matters over which Parliament has jurisdiction, not by law assigned to any other department, board or agency of the Government of Canada, relating to […] (i) the provision to departments, boards and agencies of the Government of Canada of translation and related services." [Section 6 (i)]

14. The Translation Bureau Regulations outlines the responsibilities of the Bureau including the provision of "interpretation services requested by departments" (Section 3 (e)).
15. The Translation Bureau is managed in accordance with the powers and authorities conferred upon the Minister of PWGS. By order-in-council (1993-1459), the Minister of PWGS is responsible for implementing the Translation Bureau Act and the Translation Bureau Regulations.

Evaluation reporting

16. The results of the Evaluation research and analysis are presented in table format (see Sections 1 & 2). The tables present the findings, evidence, conclusions and recommendations by evaluation issue. The indicators and data sources used to assess the evaluation issues are included in the table as well. This reporting format meets the requirements for evaluation reporting set out in Section 6.4.1a ( i-ix) of the Treasury Board Standard on Evaluation for the Government of Canada. Two additional reports - Volume 1: Terminology Standardization Program - are submitted to the Audit and Evaluation Committee separately and Volume 2: Translation and Other Linguistic Services Program.

Summary of findings

17. The Evaluation found that the Conference Interpretation Program is relevant in that it addresses a continued need, is aligned with federal priorities and departmental strategic outcomes, and is an appropriate role and responsibility for the federal government.
18. The Program is achieving progress on its specific expected result (i.e. immediate outcome) as evidenced by the use of the Program's services among federal departments and agencies, who report them as being both high quality and timely.
19. Limitations in the data collected by the evaluation prevented the isolation and quantification of the Conference Interpretation Program's contribution to the achievement of shared intermediate outcomes. To the extent that it is meeting its immediate outcome and as evidenced by stable demand for the services, the Program is likely contributing to the intermediate outcomes which are shared with other Translation Bureau programs.
20. As the ultimate outcome for this Program is also shared with other Translation Bureau programs, and given that achievement of the ultimate outcome includes the contributions of multiple government departments and agencies involved in the promotion of Canada's official languages, it was not feasible for the Evaluation to assess the extent to which it has been achieved.
21. With regard to efficiency and economy, there may be potential for further improvements to efficiency and economy by increasing the utilization rate of internal interpreters. The Evaluation also found that the Program's use of external suppliers in recent years is economical. Current performance measures do not link financial information (i.e. program costs) with the Program's activities, outputs and outcomes.

Management response and action plan

Management response

The Translation Bureau Management concurs with the evaluation's findings as well as its recommendations, and is pleased to note that the evaluation team concluded that the Conference Interpretation Program is relevant, addresses a continued need and is aligned with the Departmental and Federal priorities.

Recommendations and management action plan

Recommendation 1: The Chief Executive Officer for the Translation Bureau should seek ways to maximize the utilization rate of internal interpreters.

Management action plan Recommendation 1

The Translation Bureau will seek ways to maximize internal interpreters. More specifically:

• Management action plan 1.1: Establish a consistent methodology of accounting for activities.
• Management action plan 1.2: Better document interpretation days, including essential related activities (e.g. travel and documentation).
• Management action plan 1.3: Consolidate interpretation services in order to optimize the use of resources.

Recommendation 2: The Chief Executive Officer of the Translation Bureau should develop and implement a performance measurement strategy for the measurement of expected outcomes which includes financial measures that link costing information to activities, outputs and outcomes.

Management action plan Recommendation 2

The Translation Bureau will develop and implement an ongoing performance measurement strategy to track the measurement of expected outcomes and related financial information:

• Management action plan 2.1: Conduct an analysis of available performance information related to the Program activities, which includes financial measures.
• Management action plan 2.2: Modify key performance indicators to ensure outcomes are matched to financial information (i.e. program costs/expenditures).
• Management action plan 2.3: Report annually on the Program's key performance indicators.

Section 1: Relevance

Continued need: The extent to which the program continues to address a demonstrable need and is responsive to its clients is assessed in this section.

Federal priorities and departmental strategic outcomes: The extent to which the program has linkages with federal government priorities and with departmental strategic outcomes is assessed in this section.

Appropriate role and responsibility for the federal government: Determined by examining decentralization to other federal departments and agencies; devolution of program responsibility to another level of government; and, devolution of responsibility for the program to the private sector.

Section 2: Performance

Outcome achievement: Immediate outcome - The extent to which federal departments and agencies and the judiciary have access to high-quality and timely linguistic services is assessed in this section.

Outcome achievement: Intermediate outcome - The Translation Bureau contributes to the capacity of federal departments and agencies and the judiciary to operate in both official languages and in other languages, as needed. ( This is a shared outcome and its achievement is dependent on all Translation Bureau programming.)

Outcome achievement: Intermediate outcome - The Translation Bureau supports federal departments and agencies and the judiciary in the fulfillment of their obligations under the Official Languages Act. (This is a shared outcome and its achievement is dependent on all Translation Bureau programming.)

Given the Program's indirect contribution to the ultimate outcome ( Canadians can communicate and have access to federal institutions services in the official language of their choice and in other languages, as needed) and that its achievement is dependent on multiple Translation Bureau programs as well other government departments and agencies involved in the promotion of Canada's official languages, it was not feasible for the Evaluation to assess achievement of the ultimate outcome.

Efficiency and economy: The extent to which resources are utilized in relation to the production of outputs and outcomes are assessed in this section. Efficiency refers to the extent to which resources are used such that a greater level of output is produced with the same level of input or, a lower level of input is used to produce the same level of output. Economy refers to minimizing the use of resources. A program has high demonstrable economy and efficiency when resources maximize outputs at least cost and when there is a high correlation between minimum resources and outcomes achieved.

Appendix A: About the evaluation

Authority

The Deputy Minister for Public Works and Government Services Canada approved the conduct of this evaluation, on recommendation by the Audit and Evaluation Committee, as part of the 2012-17 Risk-Based Audit and Evaluation Plan.

Evaluation objectives

The Evaluation examined the Translation Bureau Programs, including the Conference Interpretation Program, delivered by the Translation Bureau within PWGSC. This evaluation had two objectives:

• To determine the relevance of the Program: the continued need for the Program, its alignment with governmental priorities and departmental outcomes and its alignment with federal roles and responsibilities.
• To determine the performance of the Program: the achievement of its expected outcomes and a demonstration of the efficiency and economy by the Program.

Approach to the evaluation of Translation Bureau programs

The Evaluation was conducted in accordance with the Standard on Evaluation for the Government of Canada. The Evaluation of this program was done in conjunction with the evaluation of related Translation Bureau Programs: the Terminology Standardization Program and the Translation and Other Linguistic Services Program which will be reported on separately in the future. Planning and research took place between May 2012 and March 2013. To assess the evaluation issues and questions, the following lines of evidence were used.

Document review: An initial document review provided an understanding of the Program and its context to assist in both the planning phase and in providing background data for the Program. Documents reviewed included documents provided by the Program, as well as documents written about the Program. They included: legislative documents; Treasury Board submissions; business cases; annual and multi-year plans; and departmental documents such as annual Reports on Plans and Priorities and Departmental Performance Reports. Additionally, secondary documents were reviewed which provided data on the Program context.

Financial analysis: The Evaluation reviewed Program financial information and related comparative data as well as additional information provided by the Program on the cost of salaries and benefits for professional, administrative support and management staff. This information informed the analysis of Program efficiency and economy.

For the purposes of the cluster engagement to evaluating Translation Bureau programs, direct costs include salaries, operation and maintenance (O&M) and other costs that are incurred directly by the Program in the conduct of its activities and that are charged to the Program cost center. Indirect costs include the Program's share of the Translation Bureau overhead costs (e.g., the Chief Executive's Office, resource management, strategic planning) as well as other costs such as accommodation, human resources, information management/information technology (IM/IT) services, and finance provided to the Translation Bureau by PWGSC.

Interviews: Interviews with Program Stakeholders: Interviews were conducted with Managers and Senior Managers within the Translation Bureau (14), officials of federal organization translation coordination units (6), private sector translation and linguistic services providers (4). The interviewees provided information about the Program's activities, outputs, expected outcomes, stakeholders, environment/context, relevance, and performance. The qualitative analysis of the interviews provided information about the Program's activities, outputs, expected outcomes, stakeholders, relevance and performance from the perspective of Program managers, client departments, and other related stakeholders. Guides were developed and used to conduct these interviews.

Survey of Translation Bureau Clients: Clients of the Translation Bureau were asked to provide their feedback on their experience with the Program and its products to compare this experience with that of alternatives (such as internal linguistic services units and the private sector). A total of 1000 people were invited to complete the survey (from an estimated total population of 6,390), 291 people responded representing a response rate of 29.1%. 5 respondents had specific experience with the Conference Interpretation Services Program at PWGSC.

Review of Program Performance Data: The Translation Bureau generates performance data which was reviewed for this evaluation. This included data on Program activities, outputs and outcomes; examples include client satisfaction surveys and reports and studies conducted by external parties.

Limitations of the evaluation of the Conference Interpretation Program

The conduct of this evaluation was part of a larger engagement which piloted a "cluster" approach to evaluating Translation Bureau programs that had not yet been subject to an evaluation: the Terminology Standardization Program, the Translation and Other Linguistic Services Program, and the Conference Interpretation Program. A Logic model that depicted the outcomes that were common to all three programs was prepared by the Translation Bureau. The data collection methodologies employed for the cluster engagement did not support the isolation and quantification of a program's specific contribution to the achievement of the Translation Bureau's intermediate and ultimate results. Reference to this limitation is made within the Report.

The "cluster" engagement employed a risk-based approach to determining scope and level of effort, placing a greater emphasis on the assessment of the Translation and Other Linguistic Services Program, given its greater materiality (in terms of full time employees (FTEs) and $), sensitivity and delivery complexity. The Conference Interpretation Program's materiality is, by comparison, relatively low and the program has a less complex delivery model. Program spending accounts for less than 1% of the total PWGSC budget and approximately 8% of the Translation Bureau budget for 2012-13. As such, the evaluation of the Conference Interpretation Program focused on the program's achievement of its specific results at the immediate level. Furthermore, as a result of the risk-based approach to scope and level of effort, the evaluation team chose a compressed format to reporting the evaluation of the Conference Interpretation Program.

A number of challenges were experienced in the design, data collection and analysis of the Evaluation of the Conference Interpretation Program. These challenges, as well as related limitations and strategies used to mitigate their impact, are outlined here. It was difficult to compare the Program internationally, as the nature of Canada's language profile is unique. National comparisons are equally challenging as the Evaluation was unable to uncover equivalent organizations with similar mandates and activities in other provinces or territories. As a result of this challenge, benchmarks were established based on other Programs within the Translation Bureau, or with other service providers across the government.

The number of respondents to the client survey who had experience with the Program's conference interpretation services specifically was very low (5 respondents) and therefore the statistical significance was not robust. This was due to the fact that the survey was targeted to clients of the Translation and Linguistic Services Program, and not the Conference Interpretation Services Program specifically. Therefore, survey results from these clients were not included in this analysis or in the report. Instead, the evaluation team used data from the Translation Bureau's client survey (2011-12), which captured a greater number of clients that had experience with conference interpretation services.

At the time of the writing of this report, financial data was available up to and including period 11 for the 2012-13 fiscal year. Data from period 12 only became available during the final phases of the writing of this report.

The Evaluation of the Conference Interpretation Program focused on the use of existing data to inform reporting. Efforts were made to draw information on the specific contribution by the Program to the achievement of shared Translation Bureau outcomes from existing data however it was not possible to do so. Recommendation 2 of this Evaluation should help mitigate this risk in future evaluations.

Finally, efforts were made to draw linkages between the Program's outputs and outcomes but limitations in the existing data had an impact on the ability to assess the overall efficiency of the program. Recommendation 2 of this Evaluation should help mitigate this risk in future evaluations.

Reporting

Findings were documented in a Director's Draft Report, which was reviewed by the Office of Audit and Evaluation's Quality Assessment function. The Translation Bureau's Chief Executive Officer was provided with the Director's Draft Report and a request to validate facts and comment on the report. A Chief Audit and Evaluation Executive's Draft Report was prepared and provided to the Chief Executive Officer, Translation Bureau for acceptance as the Office of Primary Interest. The Office of Primary Interest was requested to respond with a Management action plan. The Draft Final report, including the Management action plan, was presented to PWGSC's Audit and Evaluation Committee for the Deputy Minister's approval in November 2013. The Final report will be submitted to the Treasury Board Secretariat and posted on the PWGSC website.

Project team

The Evaluation was conducted by employees of the Office of Audit and Evaluation, overseen by the Director of Evaluation and under the overall direction of the Chief Audit and Evaluation Executive. The evaluation was reviewed by the Quality Assessment function of the Office of Audit and Evaluation.

Appendix B: Logic model