2012-603 Evaluation of Translation Bureau programs - Volume 1: Terminology Standardization Program (Final report)

2012-603 Volume 1 Final report (PDF, 451KB)

January 21, 2014

Introduction

1. The following report presents the results of the Evaluation of the Terminology Standardization Program of the Translation Bureau. This Program is situated in the Public Works and Government Services Canada (PWGSC) Program Alignment Architecture under sub-program 1.6.2 (Terminology Standardization Program). The Deputy Minister for PWGSC approved the conduct of this evaluation, on recommendation of the Audit and Evaluation Committee, as part of the 2012-17 Risk-Based Audit and Evaluation Plan.
2. The Evaluation provides an assessment of the relevance and the performance of the Program. The Evaluation research was conducted from January, 2013 to March, 2013, in accordance with the evaluation standards of the Government of Canada and the Office of Audit and Evaluation at PWGSC. More information on the approach, methodologies and limitations associated with this Evaluation can be found in Appendix A.
3. The research for this evaluation was conducted in conjunction with the research for the evaluation of the Translation Bureau's related Programs including the Conference Interpretation Program and the Translation and Other Linguistic Services Program. An analysis was conducted to determine the appropriate level of research effort to be dedicated to the evaluation of this program. Compared to the other Translation Bureau programs which underwent simultaneous evaluation, the Terminology Standardization Program's materiality is relatively low and the overall risks of the Program are low. Program spending accounts for 0.4% of the total PWGSC budget and 7% of the Translation Bureau budget for 2012-13, and the Program has a clear and unambiguous mandate. As a result of these factors, the evaluation team chose a compressed format to report on the core areas of evaluation.

Why it is important

4. The Terminology Standardization Program promotes consistency and quality in the government's communication with Canadians and optimally manages the government's terminology expertise. The Program is also key enabler for helping federal departments and agencies to operate effectively in Canada's two official languages (as well as in other languages when required) and to meet their obligations under the Official Languages Act. For these reasons, as well as to meet the PWGSC's obligations under the Treasury Board Policy on Evaluation, it is important to periodically evaluate the Program's continued relevance and its performance to support decision-making.

About the program

5. The Translation Bureau, within PWGSC, is the federal organization that provides translation, interpretation and terminology services to the judiciary, Parliament and federal departments and agencies. It supports the Government of Canada in its efforts to communicate with and provide services for Canadians in the official language of their choice. It was established in 1934.
6. As part of the Translation Bureau, the Terminology Standardization Program originated with the establishment of the first terminology unit in Canada in 1953. It was designed to standardize vocabulary and to reduce the proliferation of different terms used to express the same concept within the federal government. In 1974, the Bureau acquired the standardization function, with respect to proper usage in the public service, in both administrative writing and terminology. Consistent with this function, it developed a terminology database in 1975. The Translation Bureau Act, updated in 1985, outlines the duties and functions of the Translation Bureau, including those related to the provision of terminology standardization. The Act also authorized the Minister of Public Works and Government Services (PWGS) to develop supporting regulations.
7. The Terminology Standardization Program is the terminology authority within the federal government. Its primary activities are identified in the program's logic model (see Appendix B) and are outlined below:
   • Developing, publishing and updating linguistic and terminological standardization tools.
     • The outputs of this work include writing tools, and a glossary of standardized terms stored on the Bureau's linguistic database, known as TERMIUM Plus^®, which serves as the repository for standardized terms. TERMIUM Plus^® is used by translators and helps federal employees to write and communicate in both official languages. It is available free of charge for the Canadian population and is accessible anywhere in the world. ^{Footnote 1}Other outputs include other glossaries and vocabularies, the Language Update, and Linguistic Recommendations and Reminders.
   • Establishing partnerships with federal, national and international organizations.
     • The primary outputs associated with this work includes terminological and linguistic advice.
   • Providing linguistic and terminological consulting services.
     • The primary outputs associated with this work includes partnerships with standardization organizations at the federal and national level as well as internationally.
   • Establishing professional and quality assurance standards for terminology.
     • The primary outputs associated with this work includes the applied linguistic standards.
8. The Terminology Standardization Program is funded by appropriation. In fiscal year 2012-13, the total reported expenditures for the Program were $12.2M in direct and indirect costs. This included 69.4 full-time equivalent staff, 55.4 of which were professional staff (including terminologists and non-terminologist translation professionals [TR classification], program administrators [PM classification] and general technicians [GT classification]) ^{Footnote 2}, 8.4 were administrative support staff ^{Footnote 3} and 5.6 were management ^{Footnote 4}. The Program represents 7% of the total costs for the Translation Bureau.
9. A logic model for the Translation Bureau was developed for the Evaluation and is provided in Appendix B. The activities, outputs and outcomes for the Terminology Standardization Program are bolded and italicized.
10. The Terminology Standardization Program shares its intermediate outcomes with other Translation Bureau programming: The Conference Interpretation Program and the Translation and Other Linguistic Services Program.

Authority

11. The Translation Bureau Act, updated in 1985, outlines the duties and functions of the Translation Bureau, including those related to the provision of terminology standardization. The Act also authorized the Minister of Public Works and Government Services (PWGS) to develop supporting regulations.
12. The Translation Bureau Regulations requires the maintenance of a terminology centre that coordinates terminology standardization activities for the federal government.
13. The Minister of PWGS is authorized to offer translation and related services to other government departments by the Department of Public Works and Government Services Act:

    "The powers, duties and functions of the Minister extend to and include all matters over which Parliament has jurisdiction, not by law assigned to any other department, board or agency of the Government of Canada, relating to […] (i) the provision to departments, boards and agencies of the Government of Canada of translation and related services." [Section 6(i)]

14. The Translation Bureau Regulations outline the responsibilities of the Bureau including the provision of "technical advice on questions relating to translation, terminology and style to employees of departments" (Section 3(f)). It also outlines the obligation of the Bureau to establish a "Terminology Centre" (Section 16).
15. The Translation Bureau is managed in accordance with the powers and authorities conferred upon the Minister of PWGS. By order-in-council (1993-1459), the Minister of PWGS is responsible for implementing the Translation Bureau Act and the Translation Bureau Regulations.

Evaluation reporting

16. The results of the Evaluation research and analysis are presented in table format (see Sections 1 & 2). The table presents the findings, evidence, conclusions and recommendations by evaluation issue. The indicators and data sources used to assess the Evaluation issues are included in the table as well. This reporting format meets the requirements for evaluation reporting set out in Section 6.4.1a ( i-ix) of the Treasury Board Secretariat Standard on Evaluation for the Government of Canada. Two additional evaluation reports - Volume 2: Translation and Other Linguistic Services Program and Volume 3: Conference Interpretation Program - are submitted to the Audit and Evaluation Committee separately.

Summary of findings

17. In general, the Evaluation found that the Terminology Standardization Program is relevant in that it addresses a continued need, is aligned with federal priorities and departmental strategic outcomes, and is an appropriate role and responsibility for the federal government.
18. The Program is achieving progress on its specific expected result (i.e. immediate outcome) as evidenced by the quantity and range of terminological and linguistic tools it makes available to the Government of Canada, as well as the number of users accessing its services. However, the Evaluation found that improvements are possible to enhance the user-friendliness of Program tools such as TERMIUM Plus^®.
19. Limitations in the data collected by evaluation prevented the isolation and quantification of the Terminology Standardization Program's contribution to the achievement of shared intermediate outcomes. To the extent that it is meeting its immediate outcome however, the Program is likely contributing to the achievement of its intermediate outcomes shared with other Translation Bureau Programs.
20. As the ultimate outcome for this Program is also shared with other Translation Bureau programs, and given that achievement of the ultimate outcome includes the contributions of multiple government departments and agencies involved in the promotion of Canada's official languages, it was not feasible for the Evaluation to assess the extent to which it has been achieved.
21. With regard to efficiency and economy, the centralized nature of the Program's design and delivery enables efficiencies and economies to be realized; and the evaluation did not identify any areas for potential increases to efficiency through adjustments to the complement of management and administrative support staff dedicated to the Program. Current performance measures do not link financial information (i.e. program costs) with the Program's outputs and outcomes.

Management response and action plan

Management response

The Translation Bureau Management concurs with the evaluation's findings as well as its recommendations, and is pleased to note that the evaluation team concluded that the Terminology Standardization Program is relevant, addresses a continued need and is aligned with Departmental and Government-wide policies and priorities.

Recommendations and management action plan

Recommendation 1: The Chief Executive Officer, Translation Bureau should take steps to improve the user-friendliness of TERMIUM Plus^®.

Management action plan Recommendation 1

The Translation Bureau will consider how best to improve the user-friendliness of TERMIUM Plus^®. More specifically:

• Management action plan 1.1: Conduct an analysis of the users' feedback on the user friendliness of TERMIUM Plus^®.
• Management action plan 1.2: Based on the feedback and best practices, develop a redesign plan with proposed improvements and obtain approval from the Translation Bureau's Senior Management Committee.
• Management action plan 1.3: Develop and implement an improved version of TERMIUM Plus^®.

Recommendation 2: The Chief Executive Officer of the Translation Bureau should develop and implement a performance measurement strategy for the measurement of expected outcomes which includes financial measures that link costing information to activities, outputs and outcomes.

Management action plan Recommendation 2

The Translation Bureau will develop and implement an ongoing performance measurement strategy to track the measurement of expected outcomes and related financial information:

• Management action plan 2.1: Conduct an analysis of available performance information related to the Program's activities, which includes financial information.
• Management action plan 2.2: Modify key performance indicators to ensure outcomes are matched to financial information (i.e. program costs/expenditures).
• Management action plan 2.3: Report annually on the Program's key performance indicators.

Section 1: Relevance

Continued need: Assesses the extent to which a program continues to address a demonstrable need and is responsive to the needs of its clients.

Federal priorities and departmental strategic outcomes: Determined by assessing program linkages with federal government priorities and with departmental strategic outcomes.

Appropriate role and responsibility for the federal government: Determined by examining decentralization to other federal departments and agencies; devolution of program responsibility to another level of government; and, devolution of responsibility for the Program to the private sector.

Section 2: Performance

Outcome achievement: Immediate outcome - The Government of Canada and national and international standardization organizations have access to, and use a standardized and widely accepted terminology.

Outcome achievement: Intermediate outcome - The Translation Bureau contributes to the capacity of federal departments and agencies and the judiciary to operate in both official languages and in other languages, as needed. (This is a shared outcome and its achievement is dependent on all Translation Bureau programming.)

Outcome achievement: Intermediate outcome - The Translation Bureau supports federal departments and agencies and the judiciary in the fulfillment of their obligations under the Official Languages Act (OLA). (This is a shared outcome and its achievement is dependent on all Translation Bureau programming.)

Given the Program's indirect contribution to the ultimate outcome (Canadians can communicate and have access to federal institutions services in the official language of their choice and in other languages, as needed) and that its achievement is dependent on multiple Translation Bureau programs as well as other government departments and agencies involved in the promotion of Canada's two official languages, it was not feasible for the Evaluation to assess achievement of the ultimate outcome.

Efficiency and economy: The extent to which resources are utilized in relation to the production of outputs and outcomes. Efficiency refers to the extent to which resources are used such that a greater level of output is produced with the same level of input or, a lower level of input is used to produce the same level of output. Economy refers to minimizing the use of resources. A program has high demonstrable economy and efficiency when resources maximize outputs at least cost and when there is a high correlation between minimum resources and outcomes achieved.

Appendix A: About the evaluation

Authority

The Deputy Minister for Public Works and Government Services Canada (PWGSC) approved the conduct of this Evaluation, on recommendation by the Audit and Evaluation Committee, as part of the 2012-17 Risk-Based Audit and Evaluation Plan.

Evaluation objectives

The Evaluation examined the Terminology Standardization Program, delivered by the Translation Bureau within PWGSC. This evaluation had two objectives:

• To determine the relevance of the Program: the continued need for the Program, its alignment with governmental priorities and departmental outcomes and its alignment with federal roles and responsibilities.
• To determine the performance of the Program: the achievement of its expected outcomes and a demonstration of the efficiency and economy by the Program.

Approach to the evaluation of Translation Bureau programs

The Evaluation was conducted in accordance with the Standard on Evaluation for the Government of Canada. The Evaluation of this Program was conducted in conjunction with the evaluation of related Translation Bureau Programs: the Conference Interpretation Program Translation and the Translation and Other Linguistic Services Program which will be reported on separately. Planning and research took place between May 2012 and March 2013. To assess the evaluation issues and questions, the following lines of evidence were used.

Document review: An initial document review provided an understanding of the Program and its context to assist in both the planning phase and in providing background data for the Program. Documents reviewed included documents provided by the Program, as well as documents written about the Program. They included: legislative documents; Treasury Board submissions; business cases; annual and multi-year plans; and departmental documents such as annual Reports on Plans and Priorities and Departmental Performance Reports. Additionally, secondary documents were reviewed which provided data on the program context.

Financial analysis: The Evaluation reviewed the Program's financial information and related comparative data as well as additional information provided by the Program on the cost of salaries and benefits for professional, administrative support and management staff. This information informed the analysis of Program efficiency and economy.

For the purpose of the cluster engagement to evaluating Translation Bureau programs, indirect costs include the Program's share of the Translation Bureau overhead costs (e.g., the Chief Executive's Office, resource management, strategic planning) as well as other costs such as accommodation, human resources, information management/information technology (IM/IT) services, and finance provided to the Translation Bureau by PWGSC. Direct costs include salaries, operation and maintenance (O&M) and other costs that are incurred directly by the Program in the conduct of its activities and that are charged to the Program cost center.

Interviews: Interviews with the Program's Stakeholders: Interviews were conducted with Managers and Senior Managers within the Translation Bureau (14), officials of federal organization translation coordination units (6), private sector translation and linguistic services providers (4). The interviewees provided information about the Program's activities, outputs, expected outcomes, stakeholders, environment/context, relevance, and performance. The qualitative analysis of the interviews provided information about the Program's activities, outputs, expected outcomes, stakeholders, relevance and performance from the perspective of the Program managers, client departments, and other related stakeholders. Interview guides were used.

Survey: Two online surveys were conducted for the Evaluation.

• Survey of Federal Government Employees. Clients of the Translation Bureau were asked to provide their feedback on their experience with the Program and its products to compare this experience with that of alternatives (such as internal linguistic services units and the private sector). A total of 1,000 people were invited to complete the survey (from an estimated total population of 6,390), 291 people responded representing a response rate of 29.1%. 135 respondents had specific experience with the Terminology Standardization Program's tools (including TERMIUM Plus^®).
• Survey of Private Sector Translation Providers. Private sector providers who are on the Translation Bureau's contractor database were invited to participate in a survey. The survey focused on providing a profile of the translation and linguistic services industry as well as assessing the Translation Bureau's quality, timeliness and cost of the services. A total of 500 people were invited to complete the survey (from an estimated total population of 2,219), 187 people responded representing a response rate of 37.2%. 94 respondents had specific experience with the Terminology Standardization Program's tools (including TERMIUM Plus^®).

Review of program performance data: The Translation Bureau generates performance data which was reviewed for this Evaluation. This included data on the Program's activities, outputs and outcomes; examples include data from TERMIUM Plus^®, client satisfaction surveys and from reports and studies conducted by external parties.

Limitations of the evaluation of the Terminology Standardization Program

The conduct of this evaluation was part of a larger engagement which piloted a 'cluster' approach to evaluating Translation Bureau programs that had not yet been subject to an evaluation: the Terminology Standardization Program, the Translation and Other Linguistic Services Program, and the Conference Interpretation Program. A Logic model that depicted the outcomes that were common to all three programs was prepared by the Translation Bureau. The data collection methodologies employed for the cluster engagement did not support the isolation and quantification of a program's specific contributions to the achievement of the Translation Bureau's intermediate and ultimate results. Reference to this limitation is made within the Report.

The 'cluster' engagement employed a risk-based approach to determining scope and level of effort, placing a greater emphasis on the assessment of the Translation and Other Linguistic Services Program, given its greater materiality (in terms of full time employees (FTEs) and $), sensitivity and delivery complexity. The Terminology Standardization Program's materiality is, by comparison, relatively low and the program has a less complex delivery model. Program spending accounts for less than 1% of the total PWGSC budget and 7% of the Translation Bureau budget for 2012-13. As such, the evaluation of the Terminology Standardization Program focused on the program's achievement of its specific results at the immediate level. Furthermore, as a result of a risk-based approach to scope and level of effort, the evaluation team chose a compressed format to reporting the evaluation of the Terminology Standardization Program.

A number of challenges were experienced in the design, data collection and analysis of the Evaluation of the Terminology Standardization Program. These challenges, as well as and strategies used to mitigate their impact, are outlined below.

It was difficult to compare the Terminology Standardization Program internationally, as the nature of Canada's language profile is unique. National comparisons are equally challenging. The Terminology Standardization Program is the largest terminology standardization body in Canada and, as such, this made comparisons with other service providers difficult. The most similar organization nationally, the Province of Quebec's Office de la Langue Française, has some similar components to the Program (such as Le grand dictionnaire terminologique (GDT), but the organizational mandate is substantially different to that of the Program. As a result of these challenges, benchmarks were established based on other Programs within the Translation Bureau, or with other service providers across the government. The number of respondents to the client survey who had experience with the Program's terminology standardization services specifically was somewhat low and therefore the statistical significance was not robust for some questions. This was due to the fact that the survey was targeted to clients of the Translation and Linguistic Services Program, and not the Terminology Standardization Program specifically.

At the time of the writing of this report, financial information was available up to and including period 11 for the  2012-13 fiscal year. Information from period 12 only became available during the final phases of the writing of this report.

The evaluation of Terminology Standardization Program focused on the use of existing data to inform reporting. Efforts were made to draw information on the specific contribution by the Program to the achievement of shared Translation Bureau outcomes from existing data however it was not possible to do so. Additionally, one particular line of evidence intended to support an assessment of the achievement of the intermediate outcome did not yield information that was useful to the evaluation. The evaluation team reviewed the Office of the Commissioner of Official Languages report card ratings of a select number of Translation Bureau clients to assess any linkages between use of the Program's services and positive rating outcomes. Unfortunately, the Evaluation was unable to attribute report card ratings to the Program's activities. Recommendation 2 of this evaluation should help mitigate this risk in future evaluations.

Finally, efforts were made to draw linkages between the Program's outputs and outcomes but limitations in existing data had an impact on our ability to assess the overall efficiency of the program. Recommendation 2 of this evaluation should help mitigate this risk in future evaluations.

Reporting

Findings were documented in a Director's Draft Report, which was reviewed by the Office of Audit and Evaluation's Quality Assessment function. The Translation Bureau's Chief Executive Officer was provided with the Director's Draft Report and a request to validate facts and comment on the report. A Chief Audit and Evaluation Executive's Draft Report was prepared and provided to the Chief Executive Officer, Translation Bureau for acceptance as the Office of Primary Interest. The Office of Primary Interest was requested to respond with a Management Response and Action Plan to address the evaluation's recommendations. The Draft Final report, including the Management action plan, was presented to PWGSC's Audit and Evaluation Committee for the Deputy Minister's approval in November 2013. The Final report will be submitted to the Treasury Board Secretariat and posted on the PWGSC website.

Project team

The Evaluation was conducted by employees of the Office of Audit and Evaluation, overseen by the Director of Evaluation and under the overall direction of the Chief Audit and Evaluation Executive. The evaluation was reviewed by the Quality Assessment function of the Office of Audit and Evaluation.

Appendix B: Logic model