2012-603 Evaluation of Translation Bureau programs - Volume 2: Translation and Other Linguistic Services Program (Final report)

2012-603 - Volume 2 Final report (PDF, 708KB)

January 21, 2014

Main points

What was examined

i. This evaluation examined the ongoing relevance and performance, for the period from April 1, 2008 to March 31, 2013, of the Translation and Other Linguistic Services Program of the Translation Bureau (a Special Operating Agency within Public Works and Government Services Canada (PWGSC)). The Program is located in sub-program 1.6.3 of the Department's 2012–13 Program Alignment Architecture.

ii. The Translation and Other Linguistic Services Program provides a variety of translation and related linguistic services, such as revising, editing and proofreading, to federal government departments and agencies, as well as to the judiciary. The Program was initially implemented under the Translation Bureau Act in 1934 as a mandatory service. In 1995, it was established as an optional service operating on a full cost recovery model under a revolving fund. Translation revenues in fiscal year 2012-13 were $131.5M.

Why it is important

iii. Access to affordable translation and related services is critical to the ability of federal departments and agencies to operate effectively in Canada's two official languages (as well as in other languages when required) and to meet their obligations under the Official Languages Act. It is also essential to the ability of Canadians to communicate with, and access the services of, the federal government in their official language of choice. The Translation and Other Linguistic Services Program accounted for an estimated 76% of the federal government's translation volume in fiscal year 2011–12 ^{Footnote 1}. For these reasons, as well as to meet the department's obligations under the Treasury Board Policy on Evaluation it is important to periodically evaluate the Program's continued relevance and its performance, to support decision-making.

What we found

iv. The Evaluation found that there is a demonstrable need for translation services which stems from the Official Languages Act. There is also a legislative requirement for the Translation and Other Linguistic Services Program to provide services to departments and agencies that request them. Currently 150 federal organizations use the Program for some or all of their translation needs.

v. The Evaluation was unable to document the rationale for the Program's establishment in 1934; however, three objectives set for the program at the time the Translation Bureau became a Special Operating Agency providing an optional service in 1995 continue to provide a valid rationale for the Program. The three objectives are to reduce overall costs by making departments responsible for their expenditures on translation; to improve the efficiency of the Bureau; and to promote the development of the private sector by allowing departments to use either the Translation Bureau or the private sector to meet their translation needs.

vi. The Evaluation found that the Canadian private sector for translation services is becoming increasingly more capable of providing the same range of services directly to federal government clients than it was in 1995; however, its capacity is not exactly equal in all respects to that of the public sector. The Evaluation found that the Program is aligned with federal priorities related to official languages and the centralization of administrative services. The Program is also aligned with the departmental priorities of PWGSC, as it provides a necessary service and supports the removal of barriers to accessing federal government outsources for small and medium sized enterprises within the Translation Sector.

vii. The Evaluation found that the Program is an appropriate role and responsibility for the federal government and for PWGSC. Decentralization of responsibility for the management and coordination of translation services is possible and several departments have recently taken on these responsibilities. However, it would be not possible to decentralize responsibility for the actual provision of translation services to individual departments as the Translation Bureau is the only authorized employer of in-house translators in the federal government. The provision of translation services could be decentralized to the private sector; however, this would involve transition costs. It is not appropriate to devolve the responsibility of this Program to other levels of government.

viii. Limitations in the data collected by the evaluation prevented the isolation and quantification of the Translation and Linguistic Services Program's contribution to the achievement of the shared intermediate outcomes. However, to the extent that the Program has met its immediate outcome (the provision of quality and timely translation services to federal departments and agencies and the judiciary), the Program is likely contributing to the achievement of the two intermediate outcomes.

ix. As the ultimate outcome is also shared with other Translation Bureau programs, and given that achievement of the ultimate outcome includes the contributions of multiple government departments and agencies involved in the promotion of Canada's official languages, it was not feasible for the Evaluation to assess the extent to which it has been achieved.

x. The Program's performance measurement system provides comprehensive information on internal operational and financial performance which is used to monitor performance and meet accountability requirements. There are some limits to the Program's ability to maximize efficiencies within its current design and delivery model; as well, the Program differs from the private sector in its per-hour versus per-word billing, making cost comparisons between the two difficult.

xi. The efficiency of operations of the Translation and Other Linguistic Services Program is not optimal. Proportions of administrative support staff to professional staff are higher than in comparable federal organizations and productivity rates of translators are below private sector norms.

xii. The Program is not maximizing its use of resources to the point where it can recover its full costs. There is potential to address this by improving translator productivity and utilization, and by increased leveraging of the private sector.

xiii. Alternative delivery mechanisms exist, but require further analysis due to challenges related to resource flexibility in the short term, client security concerns and the potentially high overhead costs to small departments and agencies when establishing in-house coordination units for the use of private sector translation services.

Management response

xiv. Overall, the Translation Bureau Management concurs with the Evaluation's findings as well as its recommendations, and is pleased to note that the evaluation team concluded that there is a need for translation services and that it is aligned with Departmental and Government-wide policies and priorities.

Recommendations and management action plan

Recommendation 1: The Chief Executive Officer for the Translation Bureau should, in the short term, improve the productivity of internal translation staff to levels closer to private sector norms.

• Management action plan 1: The Translation Bureau will establish a more transparent and result-oriented quantitative performance management system for its translators. More specifically:
  • 1.1 It will adopt a new productivity system focusing on concrete quantitative production; and
  • 1.2 It will increase the production rate by 10% by September 2014 to bring it closer to private sector norms.

Recommendation 2: The Chief Executive Officer for the Translation Bureau should, in the long term, continue to seek ways to reduce program delivery costs with consideration given to alternative delivery models.

• Management action plan 2: The Translation Bureau will take concrete action in order to reduce long-term program delivery costs. More specifically:
  • 2.1 It will reform its procurement processes in order to adopt a rationalized organization-wide strategy; and
  • 2.2 It will take every measure to maximize the use of its internal resources, reduce its fixed cost and increase the use of the private sector whenever possible:
    • 2.2.1 In the short term, the Translation Bureau's staffing freeze will continue, and only critical positions will be replaced;
    • 2.2.2 It will establish a human resources steering committee and will provide strict staffing guiding principles;
    • 2.2.3 It will centralize main functions such as procurement and other common tasks to achieve efficiencies; and
    • 2.2.4 It will expand telework and consolidate office space.

Recommendation 3: The Chief Executive Officer for the Translation Bureau should take steps to ensure that its billing practices are based on actual productivity rates (actual words per hour) rather than standardized rates.

• Management action plan 3: The Translation Bureau will modify its billing practices in order to adopt a more transparent approach based on actual volume by moving towards a word-based billing. More specifically:
  • 3.1 In the short-term:
    • 3.1.1 It will establish an hourly rate per client in order to recognize each client's unique needs in terms of demand/volume profile, instead of two rates for general and specialized text;
    • 3.1.2 It will have one "rush premium" rate of 35% instead of two (30% and 50%);
    • 3.1.3 It will establish a single rate for the on-site translator service; and
    • 3.1.4 It will introduce a fair and transparent discount policy to establish client loyalty, as a means of increasing business volume.
  • 3.2 In the long-term, it will move towards a cost per word to align with industry norms and client demand.

Introduction

1. This report presents the results of the Evaluation of the Translation and Other Linguistic Services Program of the Translation Bureau. This Program is situated in the 2012–13 Program Alignment Architecture as sub-program 1.6.3. The Deputy Minister for Public Works and Government Services Canada (PWGSC) approved the conduct of this evaluation, on recommendation of the Audit and Evaluation Committee, as part of the 2012–17 Risk-Based Audit and Evaluation Plan. The Evaluation was conducted as part of the Evaluation of Translation Bureau Programs and in accordance with the evaluation standards of the Government of Canada and the Office of Audit and Evaluation at PWGSC. Two additional evaluation reports - Volume 1: Terminology Standardization Program and Volume 3: Conference Interpretation Program - will be completed and submitted separately to the Audit and Evaluation Committee for recommendation for approval by the Deputy Minister.

Profile

Background

2. Established in 1934 as a mandatory service, the Translation Bureau is the federal organization that provides translation, interpretation and terminology services to federal departments and agencies as well as the judiciary and to Parliament. It supports the Government of Canada in its efforts to communicate with, and provide services for, Canadians in the official language of their choice.

3. As part of the Translation Bureau, the Translation and Other Linguistic Services Program provides translation and other linguistic services to the judiciary, and to federal government departments and agencies. Introduced in 1969, the Official Languages Act set out new requirements for federal departments regarding translation of documents in both official languages. This resulted in a substantial increase in demand for these services across the federal government. In the early 1970s, the Program grew and quickly became the largest provider of translation services to the federal government.

4. While previously operating as a common service organization, the Translation Bureau became a Special Operating Agency reporting to PWGSC in 1995, as a result of a Treasury Board decision. At that time, both the Conference Interpretation Program and the Translation and Other Linguistic Services Program became optional services to federal departments and agencies, relying on earned revenues rather than appropriation. Parliamentary Translation and Interpretation services, however, continued to be funded through an appropriation. Federal organizations have since been free to procure translation and other linguistic services from the Program (on a cost-recovery basis) or directly from the private sector.

5. The Program currently provides translation, revision, and editing services in both official languages and in over 100 Aboriginal and foreign languages. It also provides other linguistic services such as after-hours emergency service, writing assistance, and language advice. These services are provided to federal departments and agencies within a cost recovery model. In fiscal year 2011–12, based on Public Accounts of Canada data, the Program provided an estimated 76% of the Government of Canada's translation services.

Authority

6. The Translation Bureau Act, updated in 1985, outlines the duties and functions of the Translation Bureau.

7. The Minister of Public Works and Government Services is authorized to offer translation and related services to other government departments by the Department of Public Works and Government Services Act:

"The powers, duties and functions of the Minister extend to and include all matters over which Parliament has jurisdiction, not by law assigned to any other department, board or agency of the Government of Canada, relating to […] (i) the provision to departments, boards and agencies of the Government of Canada of translation and related services." [Section 6 (i)]

8. The Translation Bureau is managed in accordance with the powers and authorities conferred upon the Minister of Public Works and Government Services (PWGS). By order-in-council (1993-1459), the Minister of PWGS is responsible for implementing the Translation Bureau Act and the Translation Bureau Regulations.

Roles and responsibilities

9. The head of the Translation Bureau has the designation of Chief Executive Officer. The Chief Executive Officer is accountable to the Deputy Minister of PWGSC and is responsible for the overall performance of the Bureau, including establishing its strategic direction. As the head of the Translation Bureau, the Chief Executive Officer is responsible for the Translation and Other Linguistic Services Program.

10. The Translation Bureau Regulations outlines the responsibilities of the Bureau including the provision to "make all translations requested by departments" [(Section 3 (a)].

Resources

11. The Program's financial resources are managed through a revolving fund and services are offered within a full cost recovery model. The Translation Bureau had 1,409 employees in fiscal year 2012–13 (excluding its services to Parliament, which are funded by appropriation) and total revenues of $172.5 M. In that same year, the Translation and Other Linguistic Services Program, the largest Translation Bureau program, had 1,116 employees, revenues of $131.5 M and direct and indirect costs of $141.6 M, resulting in a loss of $5.0 M ^{Footnote 2}.

Logic model

12. A logic model is a visual representation that links a program's activities, outputs, and outcomes; provides a systematic and visual method of illustrating the program theory; and, shows the logic of how a program is expected to achieve its objectives. It also provides the basis for developing the performance measurement and evaluation strategies, including the evaluation matrix.

13. A logic model that integrated the Translation and Other Linguistic Services Program, the Terminology Standardization Program and the Conference Interpretation Program was developed for the Evaluation and is provided in Exhibit 1. The activities, outputs and outcomes for the Translation and Other Linguistic Services Program are bolded and italicized.

14. All other boxes represent activities, outputs and outcomes for the other two programs that underwent simultaneous evaluation (the Terminology Standardization Program and the Conference Interpretation Program) or are a part of the PWGSC initiatives that were funded through the Roadmap for Canada's Linguistic Duality 2008–13: Acting for the Future. These initiatives were evaluated by the PWGSC Office of Audit and Evaluation, the report of which was approved by the Deputy Minister in November 2012.

Exhibit 1: Logic model

Program activities and outputs

15. The primary activities of the Translation and Other Linguistic Services Program are the provision of translation and other linguistic services to federal departments and agencies as well as to the judiciary. These services are provided in both official languages, and in other languages on demand. They include translation, revision, editing and proofreading services; after-hour emergency service; and language advice. The main outputs of these activities are writing assistance and texts that have been translated, revised, or edited in both official languages or in other languages.

16. The Translation and Other Linguistic Services Program also includes activities related to the development of human and technological capacity to meet translation needs of federal departments, agencies, and the judiciary. This supports the development of the translation workforce through investing in post-university training and mentoring programs. There are three outputs that result from this activity: qualified translators in both official languages and in other languages; translation tools; and translation request management tools.

Focus of the evaluation

17. The objective of this Evaluation was to determine the Program's relevance and performance in achieving its expected outcomes in accordance with the Treasury Board Policy on Evaluation. The Evaluation also researched program design and delivery and alternative forms of delivery. The Evaluation assessed the Program for the period from April 1, 2008 to March 31, 2013. The results of the Evaluation are presented in this report by evaluation issue.

18. An evaluation matrix, including evaluation issues, questions, indicators, and data sources for the three Programs being simultaneously evaluated, was developed for the Translation Bureau Programs during the planning phase and was used to guide the conduct of the project and the preparation of the report. Information on the approach and methodologies used to conduct this Evaluation as well as on limitations encountered in the planning and conduct of the Evaluation and on risk mitigation measures taken by the Office of Audit and Evaluation, is located in the "About the Evaluation" section at the end of this report.

Approach and methodology

19. The research for this evaluation was conducted in conjunction with the research for the Evaluation of the Translation Bureau's related Programs (including the Terminology Standardization Program and the Conference Interpretation Program). An analysis was conducted to determine the appropriate level of research effort to be dedicated to the Evaluation of this Program. Compared to the other Translation Bureau Programs being evaluated, this Program's materiality is relatively high and it is currently undergoing a transformation and modernization. As such, the overall risk of this Program is the highest among the three that were simultaneously evaluated. In addition, the Translation and Other Linguistic Services Program accounts for 85.9% of Translation Bureau expenditures for fiscal year 2011–12 and 4.8% of direct program spending for PWGSC. As a result of these factors, the evaluation team chose to dedicate a higher level of effort to evaluating this Program.

20. Multiple lines of evidence were used to assess the Program ^{Footnote 3}. These include:

• Document review: The document review provided information on the Program's authorities, mandate, activities, and performance.
• Financial analysis: The financial analysis provided data on direct and indirect costs of the Translation and Other Linguistic Services Program. The evaluation team also calculated utilization and productivity rates and used this data to compare alternatives.
• Interviews with Program Stakeholders: Interviews were conducted with Program stakeholders. The interviewees provided information about the Program's activities, outputs, expected outcomes, stakeholders, environment/context, relevance, and performance.
• Surveys: Two online surveys were conducted as part of this evaluation:

  Survey of Translation Bureau Clients – Translation Bureau clients were asked to provide their feedback on their experience with the Bureau and its products to compare this experience with alternatives (such as in-house translation coordination units and directly with the private sector).

  Survey of Private Sector Translation Providers – This survey focused on compiling a profile of the translation and linguistic services industry as well as assessing the Translation Bureau Programs' quality, timeliness, and cost of services.

• Review of Program Performance Data: The Evaluation reviewed performance data collected from the Program which was previously gathered as a part of their performance measurement regime.
• Environmental Scan: This exercise provided industry best practice data and the data to inform a review of alternative forms of delivery.

Findings and conclusions

The findings and conclusions below are based on multiple lines of evidence collected during the evaluation in support of the indicators developed to assess questions based on the five core issues to be addressed in evaluations (as per Annex A of the Treasury Board Directive on the Evaluation Function) as well as two additional issues. They are presented in the following order: Relevance (Continuing Need for the Program; Alignment with Government Priorities; and Alignment with Federal Roles and Responsibilities) and Performance (Outcome Achievement; Design and Delivery; Efficiency and Economy; and Alternative Forms of Delivery).

Relevance

21. Relevance is the extent to which a program addresses a continuing need, is aligned with federal priorities and departmental strategic outcomes, and is an appropriate role and responsibility for the federal government.

Continuing need

22. Continuing need assesses the extent to which the Program continues to address a demonstrable need and is responsive to the needs of its clients. Continuing need was assessed through a review of: the federal government's current need for translation services; the continuing relevance of the original rationale for the program; and what capacity is available to meet existing demand. Analysis of these indicators identified a continuing need for translation services on the part of federal government departments and agencies and that objectives set in 1995 continue as a valid rationale for the Program.

23. Federal government departments and agencies require access to quality translation services in order to meet their obligations under the Official Languages Act relating to communications with the public and language of work for officers and employees of federal institutions. Relative demand for Program services by federal institutions has remained stable since fiscal year 2006–07. Public Accounts of Canada data shows that 74% ( ± 6%) of total volume of Federal government spending on translation services is with the Translation Bureau. However, the Evaluation's analysis of Public Accounts of Canada data since fiscal year 2006–07 has found that the total value of translation services obtained by the federal government has declined in recent years from a peak of $248 M in fiscal year 2009–10 to $205 M in fiscal year 2011–12, representing a decline of 17% ^{Footnote 4}. These findings suggest that, while the total federal government demand for translation services is declining, departments' choices of translation service providers appear to be relatively stable for the period reviewed.

24. The Translation Bureau was re-established in 1995 as a Special Operating Agency providing an optional service on the basis of three objectives: to reduce overall costs by making departments responsible for their expenditures on translation; to improve the efficiency of the Bureau; and to promote the development of the private sector by allowing departments to use either the Translation Bureau or the private sector to meet their translation needs. The objectives set in 1995 continue to provide a valid rationale for the Program.

25. Since 1995, departments now control their translation budgets and a number have opted to meet some or all of their translation needs through the private sector. The Evaluation, however, did not assess the extent to which overall costs to the Government of Canada may have been reduced as a result of departments becoming responsible for their own expenditures on translation.

26. As detailed in the Efficiency and Economy section of this report, the Translation Bureau has recently made investments in language technologies with the aim of improving its efficiency. The loss of several large clients to the private sector in recent years appears to have been a strong incentive to increase efficiencies, according to interviewees. The Evaluation has not assessed the extent to which the Translation Bureau may have improved its efficiency since becoming a Special Operating Agency.

27. The development of the private sector (by allowing departments to use either the Translation Bureau or the private sector to meet their translation needs) has been achieved to some extent. In 1991, federal government departments obtained all of their services from the Translation Bureau. In 2011–12, government departments outsourced directly to the private sector for 24% of their translation needs. While capacity in the private sector has increased (as illustrated by its provision of translation services to the Government of Canada), there is a need for further development. A discussion on the capacity of the private sector is provided in the following section of this report.

28. By 2012–13, the Translation Bureau had grown to about 1,116 employees in translation services. In the same period, the Canadian private sector translation industry had also grown.

29. The Canadian private sector translation industry is currently made up of small- and medium-sized firms that offer a broad range of translation and other linguistic services, alongside several micro-firms and numerous freelance translators ^{Footnote 5}. There is limited data available on the evolution of the private sector translation industry in Canada, but industry studies conducted by national and international industry organizations point to the private sector's increasing capacity to produce quality translations. For example, the number of firms with revenues exceeding $500K per annum has increased from 34 in 1997 to 74 in 2006.

30. Interviews with program stakeholders indicated that there is a large and growing capacity in the private sector, with companies offering a full range of translation services. Additionally, private sector representatives interviewed for this evaluation highlighted the capacity of medium-sized firms to sub-contract work to freelance translators in order to manage work flow. As such, the Evaluation found that the continued need for the Program has diminished somewhat since there is evidence of a stronger capacity in the private sector to satisfy federal government demand for translation services than at the time the Program was first established within the Translation Bureau. However, as noted below, the private sector capacity is not exactly equal in all respects to the public sector. It should be noted that there is currently a legislative requirement for the Translation Bureau to offer this Program; however, there is not a mandatory requirement to use these services. Despite the optional use, federal clients rely heavily on the Program to provide them with translation services, as evidenced by the fact that the Program provided an estimated 76% of translation services used by the Federal government in fiscal year 2011–12.

31. Program stakeholders interviewed for this Evaluation confirmed that the private sector plays an important role in the delivery of the Program, but indicated that the industry could not presently meet all their needs. Interviewees noted that the majority of translation work required, at minimum, a quality assurance review before delivery to the client. Both the Translation Bureau and Canada's medium-sized translation firms can provide such a service. However, smaller private sector translation firms (micro-firms) and freelance translators do not offer this same level of service. As well, for some federal organizations, security concerns are paramount. The survey of private sector translation providers conducted for this Evaluation found that micro-firms and freelance translators did not offer a broad range of translation services: 81% of respondents did not have a facility security clearance of any kind, 70% had only basic or no personal security clearance, and 32% had an independent quality assurance process.

Alignment with government priorities

32. Program alignment is determined by assessing program linkages with federal government priorities and with departmental strategic outcomes. The Program's alignment with federal priorities and with the departmental strategic outcome was assessed through a review of the following: federal priorities related to Program outcomes and the Program design; and, Departmental strategic outcomes and priorities related to Program outcomes and outputs. The Evaluation found that there is an alignment between the Translation and Other Linguistic Services Program and federal and Departmental priorities.

33. The Translation and Other Linguistic Services Program supports Government of Canada priorities related to official languages. As the most important single provider of translation services to federal government organizations, the Bureau plays a supporting role in assisting these organizations in meeting the requirements of the Official Languages Act.

34. As well, the Program aligns with the Federal government's priority of centralizing administrative services. The Administrative Services Review which was launched as part of the 2010 Budget emphasizes the adoption of a whole-of-government approach to administrative functions. As a centralized service provider, coordinating private sector outsources with translation firms and freelance translators as well as servicing over 150 federal organizations, the Program model corresponds closely to this approach. Interviews with Program stakeholders emphasized the benefits of this centralized model, particularly regarding potential avoidance of duplication and cost savings typically associated with a centralized approach to service delivery and contracting.

35. The Translation and Other Linguistic Services Program aligns with departmental priorities in two ways. Firstly, it provides a necessary service to federal departments and agencies. This aligns with the Department's strategic outcome of meeting the program needs of federal institutions. Secondly, by facilitating contracts with the private sector, the Program aligns with PWGSC's commitment to remove barriers to federal government work for this sector. The Program also plays a positive role in contracting with small- and medium-sized private sector enterprises, which was emphasized by private sector stakeholders interviewed for this evaluation.

Alignment with federal roles and responsibilities

36. To determine if the Program is an appropriate role and responsibility for the federal government, three options were reviewed: the potential for decentralization to other federal departments and agencies; the potential for devolution of program responsibility to another level of government; and, the potential for devolution to the private sector. Based on the examination of these options, the Evaluation found that the Translation and Other Services Program is currently an appropriate role and responsibility for the federal government. The decentralization of the Translation Bureau's management and coordination role to other government departments is possible; however, decentralization of the provision of translation service to departments is not possible. A transfer to the private sector is possible but would involve transition costs. The devolution of roles and responsibilities of the Program to other levels of government would not be appropriate.

37. When considering the roles and responsibilities of translation among federal government departments, two aspects are considered: firstly, the role and responsibility of ensuring departments have access to these services; secondly, the role and responsibility for providing the service. The Evaluation has found that the responsibility for ensuring access to translation services is already decentralized but that a further decentralization towards self-provision of these services by federal organizations would require a change in legislation and would encounter transition costs.

38. The Official Languages Act outlines requirements for federal organizations to communicate in both official languages. The responsibility for meeting the requirements of the Act rests with the federal organization to which it applies. It is the responsibility of the head of each organization to ensure the access of departmental programs to the necessary translation services so as to ensure that the official language requirements are met. In order to meet their departmental needs to access translation services, individual departments or agencies either procure services from the Translation Bureau, directly from the private sector, or through an in-house translation co-ordination unit (which outsources services from either the Translation Bureau or the private sector). As well, a few departments translate some documents internally, using staff in the Information Services (IS) or Administrative Services (AS) occupational categories.

39. Decentralization to individual departments and agencies of the responsibility for management and coordination of translation services provided by the private sector through contracts is possible and several departments have recently opted to take on this responsibility. However, it is not possible for individual departments and agencies to take on the responsibility for actually providing themselves with in-house translation services, given the Program's sole employer status for translators (the TR occupational group) within the federal government.

40. It would not be appropriate to devolve these responsibilities to other levels of government. The Official Languages Act is federal legislation. As such, other levels of government are not mandated to undertake federal roles and responsibilities as they are outlined in the Official Languages Act. The Evaluation did not assess the capacity of other levels of government to provide services similar to those of the Program.

41. The Translation Bureau Act and the Translation Bureau Regulations mandate the Translation Bureau to provide translation services to other government departments on request. As discussed above, the private sector already plays an important role in the delivery of the Program'sservices. While the devolution of the Program as an entity responsible to ensure federal organizations have access to translation services is not possible, devolution of responsibility for the provision of the services by the private sector through contracts is possible. However, the transfer would involve transition costs. A further discussion of the role of the private sector in the delivery of the Program and related opportunities and challenges is found in this Report under Alternative Delivery.

Conclusions: Relevance

42. The Evaluation found that there is a demonstrable need for translation services by federal departments and agencies which stems from The Official Languages Act. There is also a legislative requirement for the Translation and Other Linguistic Services Program to provide services to departments and agencies that request them. Currently 150 federal organizations use the Program for some or all of their translation requirements.

43. The Evaluation was unable to document the rationale for the Program's establishment in 1934; however, objectives were set for the Program at the time the Translation Bureau became a Special Operating Agency providing an optional service in 1995. The extent to which these objectives have been achieved was not assessed by the Evaluation but they were found to continue to provide a valid rationale for the Program.

44. The Evaluation found that the Canadian private sector for translation services is becoming increasingly more capable of providing the same range of services directly to federal government clients than it was in 1995 however its capacity is not exactly equal in all respects to that of the public sector.

45. The Evaluation found that the Program is aligned with federal priorities related to official languages and the centralization of administrative services. The Program is also aligned with PWGSC's priorities, as it provides a necessary service and supports the removal of barriers for small- and medium-sized enterprises within the Translation Sector.

46. The Evaluation found that the responsibilities for ensuring access to translation services is already decentralized (i.e. the responsibility of individual departments). Further decentralization of roles and responsibilities to individual departments, such as those of service management and coordination of translation services provided by the private sector through contracts, is possible and several departments have taken on this responsibility. Decentralization of the responsibility for the provision of in-house translation services to individual departments and agencies is not possible, given the Program's sole employer status for translators (the TR occupational group) within the federal government. It is not appropriate to devolve the role and responsibility of this Program to other levels of government. Devolution of responsibility for the provision of the services to the private sector is possible but would involve transition costs.

Performance

47. Performance is the extent to which a program is successful in achieving its objectives and the degree to which it is able to do so in a manner that demonstrates efficiency and economy. Alternative delivery options represent other potential means to achieve program outcomes and for reducing program costs.

Outcome achievement

48. The Evaluation examined the degree to which the Program achieved its expected immediate, intermediate, and ultimate outcomes. The expected outcomes of the Program are identified in italics below, followed by an assessment of the extent to which they have been achieved.

49. The Program's achievement of its immediate outcome was assessed through a review the following: client satisfaction with the quality and timeliness of the services received; and achievement of planned targets. Client surveys, both conducted by the Program and by the evaluation, were used to measure these indicators. Based on these indicators, the evaluation has found that the Program is achieving this outcome.

50. Clients are satisfied with the quality of translations procured through the Translation Bureau. The fiscal year 2011–12 client survey conducted by the Translation Bureau found that 67% of respondents were satisfied or very satisfied with the quality of translation work and 13% were neutral. The survey conducted as part of the Evaluation found somewhat more positive results, with 77% of respondents to this survey reporting being satisfied or very satisfied with the quality of translation and 11% being neutral. Furthermore, 37% of respondents indicated that there 'never' or 'rarely' was a need to revise Translation Bureau products. The survey did not capture the reasons why or the extent to which other clients (63% of respondents) were revising translated materials, nor did it capture specific reasons for satisfaction or dissatisfaction with Translation Bureau transactions.

51. Clients are relatively satisfied with the timeliness of translations procured through the Translation Bureau. Program performance data for fiscal year 2011–12 shows that the Program is meeting the deadlines negotiated with clients 95% of the time. As well, the fiscal year 2011–12 Translation Bureau client survey reports that 85% of clients were satisfied or very satisfied with the timeliness of their translations. The client survey conducted for this evaluation confirmed client satisfaction with 92% of respondents reporting they were satisfied or very satisfied with the ability of the Program to meet its deadlines.

52. When considering overall client satisfaction with translation services, the Program is meeting planned targets. The Translation Bureau has a target of achieving 85% client satisfaction with overall services. The survey conducted by the Translation Bureau reported that 81% of respondents were satisfied or very satisfied with the Translation Bureau's translation services overall. Though the survey conducted by the Office of Audit and Evaluation for this evaluation did not measure overall client satisfaction, other factors which may contribute to overall satisfaction (such as ease of accessing the service) were rated highly.

53. The Program's achievement of its intermediate outcomes was assessed through a review of the following: volume of translation transactions managed by the Translation Bureau and an analysis of the achievement of immediate outcomes. Multiple lines of evidence were used to measure these indicators. Based on these indicators, the Evaluation has found that the Program is likely making a contribution in the area of its intermediate outcomes, but that the contribution cannot be quantified.

54. The Translation Bureau is the largest single supplier of translation services to federal government organizations: since 2006–07, the Program has managed, on average, 74% of total federal government spending on translation ^{Footnote 6}.

55. To the extent that the Program provides quality and timely translation services, as evidenced in the section on immediate outcomes, on such a large scale (as evidenced by it being the single largest supplier to the Government of Canada), it is likely contributing to the capacity of federal departments and agencies and the judiciary to operate in both official languages (and in other languages as needed) as well as the ability of other government operations to fulfill their obligations under the Official Languages Act. The extent of this contribution, however, has not been quantified by the Evaluation.

56. As the ultimate outcome is also shared with other Translation Bureau programs, and given that achievement of the ultimate outcome includes the contributions of multiple government departments and agencies involved in the promotion of Canada's official languages, it was not feasible for the Evaluation to assess the extent to which it has been achieved.

Conclusions: Outcome achievement

57. The Translation and Other Linguistic Services Program provides quality and timely translation services to federal departments and agencies and the judiciary.

58. The Translation and Other Linguistic Services Program is currently an important translation service provider and, to the extent that clients report satisfaction with the quality and timeliness of its products, it is likely supporting federal departments and agencies and the judiciary in operating effectively in both official languages and in fulfilling their obligations under the Official Languages Act. However, the Evaluation was unable to quantify the contribution made by the Program to the achievement of this outcome.

59. Given the Program's indirect contribution to the ultimate outcome and that its achievement is dependent on multiple Translation Bureau programs as well as other government departments and agencies involved in the promotion of Canada's two official languages, it was not feasible for the Evaluation to assess achievement of the ultimate outcome.

Design and delivery

60. The Program design and delivery was reviewed as part of the Evaluation. In addressing design and delivery, the Evaluation considered the degree to which the Program is gathering the necessary performance data to inform Program improvement and support accountability. The Evaluation also considered the Program model and the degree to which it facilitates or hinders the provision of quality services at a reasonable cost. The Evaluation found that performance data is supporting Program reporting and accountability, and Program improvement; however, unique features of the Program's design limit its flexibility to meet its mandate and provide value-for-money.

61. The Translation Bureau (which administers the program) collects information to monitor the service performance as well as to allocate work among offices and translators, among internal translators, and between internal translators and contracted translators. Data collected includes the percentage of translation deadlines met, utilization rates of internal translators (measured in billable hours), and productivity rates for internal staff. The Translation Bureau also collects data on the volume and profitability of work billed at different rates (e.g., general, specialized, ultra-specialized, urgent), profitability by client, and profitability of internal versus outsourced translation work. Furthermore, the Translation Bureau conducts additional studies, such as client satisfaction surveys, to gauge its achievement of outcomes. Data on volume, revenue and costs are used to monitor the Translation Bureau's financial situation on an ongoing basis to update budget forecasts, make adjustments to expenditures, and support strategic planning. Translation Bureau'smulti-year business plans outline robust performance measurement and targets and responsive planning that addresses the gaps that these measures highlight. Finally, performance measures are reported regularly through various media including a Translation Bureau dashboard, as well as annual reports and plans. As such, the Evaluation found evidence that adjustments have been made continually to aspects of Program delivery based on these measures and the Program has reported on these measures as part of its reporting and accountability requirements.

62. The Translation and Other Linguistic Services Program provides translation services based on a model that includes: a request management and billing system, production of translations (internally or contracted), and a quality assurance function. The private sector firms interviewed as part of the Evaluation, as well as a review of industry data, has shown the model employed is a traditional one for translation service delivery. A traditional model employs internal translation staff and quality assurance staff and leverages additional resources (outsourced translations by private sector) to meet increases in the Program's workload and/or requirements for specialized work. Some level of quality assurance is provided by the Program before the product is returned to the client regardless of whether it is completed by internal or contracted translators.

63. The Evaluation has identified four unique elements to the Program's design and delivery. The first unique element is that the Program is required, by the Translation Bureau Regulations, to meet all requests made of it by federal organizations. As a result of this requirement, the Program has staffed to ensure that this requirement can be met. This is a contributing factor in the Program becoming the largest single employer of translators in the country. The second unique element is the program's size. Program stakeholders have identified the Program's size as having an impact on delivery, stating that it has enabled investment, development and integration of language technologies to assist in the faster production of quality translations. While the Evaluation found evidence of these investments, it did not assess the extent to which efficiencies may have been achieved from their introduction.

64. The third unique element of the Program is its rigorous contractor selection process that complies with the federal government ethics of sound stewardship and fair, open and competitive procurement practices. While the Program can justify the repeated use of well performing contractors, it must follow procurement practices that ensure the fairness of the tendering process and that qualified contractors have equitable access to government contracts. The Translation Bureau has to balance the need to respond quickly to requests for services against the requirement to ensure fairness, transparency and to demonstrate accountability in contracting processes. This impacts on the ability of the Program to quickly outsource time sensitive work, and to economically outsource small translation work.

65. The fourth unique element of the Program is that it charges clients by the hour whereas the private sector standard is to charge by the word. At the intake stage, clients are quoted a cost for the translation of a document based on an assumed productivity rate of 209 words per hour in 2012–13. However, data from the time reporting system used by internal translators indicates a real productivity rate of 249 words per hour in 2012–13. The nominal rate of 209 words per hour likely underestimates the productivity of external contractors the Program employs as well, as the productivity rates for the private sector range from 250 to 300 or more words per hour.

66. The evaluation has found three implications stemming from the fourth design element: Firstly, the result is likely that the data produced by the Translation Bureau on the total volume of words translated annually and on the average cost per word contains a margin of error. Secondly, the use of a billing system based on an hourly fee and on a nominal productivity rate, makes it difficult to compare the costs of the Translation Bureau with the costs of the private sector where the standard practice is to charge by the word. Finally, the use of this hourly billing system also reduces transparency of billing to clients, as noted in a recent study of pricing and costing commissioned by the Translation Bureau.

67. From these design elements emerge several key challenges for the Translation Bureau. Firstly, the mandatory requirement to respond to all requests, combined with the large base of indeterminate employees, limits the organization's flexibility in responding to fluctuations in demand. Where a private sector firm can simply turn down business when it lacks resources, the Translation Bureau must, effectively, ensure it always has sufficient resources. Employment of a large base of indeterminate employees, supplemented with private sector contractors, ensures it can meet its mandatory service requirements; however, this approach limits its ability to quickly reduce resources and costs in response to declines in demand for its services.

68. Secondly, higher salary costs for public sector employees and lower levels of productivity compared to the private sector, result in higher costs for translation services provided by the Translation Bureau compared to the private sector.

69. Thirdly, the need for the Translation Bureau to adhere to federal government requirements for fairness, transparency and accountability when outsource translation work, increases both the time required and the costs of contracting compared to private sector firms that sub-contract work to other firms or freelance translators in order to meet client requirements.

70. Finally, the Translation Bureau, as a federal government organization, must comply with a wide range of policy, accountability and reporting requirements in such areas as human resources, employment equity, financial management, business planning and management, departmental performance reporting, official languages, sustainable development; and other areas. Meeting these requirements also results in higher costs for services provided by the Translation Bureau compared to those provided by the private sector.

Conclusions: Design and delivery

71. The performance data produced and compiled in support of the Translation and Other Linguistic Services Program are very comprehensive. Ongoing performance data are sufficiently comprehensive to support internal monitoring of operations and operational planning; while the volumetric and financial data are sufficient to support ongoing expenditure management and longer-term strategic planning and business transformation.

72. The Program is operating within a traditional model for the provision of translation services. Because the Program is very large, and operating within a government context, there are some limitations on the ability for the Program to maximize efficiencies. The Program differs from to the private sector in its per-hour versus per-word billing, making cost comparison between the two difficult.

Efficiency and economy

73. Demonstration of efficiency and economy is defined as an assessment of resource utilization in relation to the production of outputs and outcomes. A program has high demonstrable efficiency and economy when resources maximize outputs at least cost and when there is a high correlation between minimum resources and outcomes achieved.

74. Efficiency refers to the extent to which resources are used such that a greater level of output is produced with the same level of input or a lower level of input is used to produce the same level of output. The Evaluation assessed the operational efficiency of the Program, which reviewed how inputs are being used and converted into outputs that support the achievement of expected outcomes. Operational efficiency was assessed through a review of the following: indirect costs as a percentage of total costs; the proportion of both management and administrative support staff to professional staff and analysis of utilization and productivity rates. Based on these indicators, the evaluation found that, while utilization rates are close to optimal, there is potential to improve the operational efficiency of the Translation and Other Linguistic Services Program by improving overall productivity and utilization.

75. Based on financial information provided by the Program, indirect costs amounted to 14.6% of the total costs of the Translation and Other Linguistic Services Program in fiscal year 2011–12 and to 14.7% in fiscal year 2012–13. A previous Evaluation of a Program with a similar service provision model within government showed similar amounts of indirect costs. The Program's indirect costs include its share of the Translation Bureau overhead costs (e.g., the Chief Executive's Office, resource management, strategic planning) as well as costs charged by PWGSC for services, such as accommodation, human resources, information management/information technology (IM/IT) services ^{Footnote 7}, and finance provided to the Translation Bureau by PWGSC. ^{Footnote 8}

76. Management staff made up 6% of all staff. ^{Footnote 9} The Evaluation found that this is approximately equal to the percentage of management staff across the federal government, which is 6.9%.

77. In fiscal year 2012–13, administrative support accounted for 27% of all staff. ^{Footnote 10}This is higher than the percentage of administrative support staff for comparable organizations across the government, which average 22%. Based on discussions with Translation Bureau managers, the 5% difference reflects, in part, inefficient and labour-intensive business processes that are currently under review. Recognizing, however, that costs of the Program were higher than in comparable federal organizations, the Program has recently tried to address this issue by consolidating some administrative activities and, hence, reduce its overall administrative costs. At this early time, the Evaluation is unable to assess the degree to which this consolidation effort will result in cost savings.

78. According to the classification standard for government translators, an average of 70% to 80% ^{Footnote 11} of a translator's time should be spent on translation work or related, billable, activities. This allotment does not include provisions for statutory holidays or leave. Utilization rates are higher in the private sector (typically 80-85%). Based on Translation Bureau data, the actual average billable utilization rate of internal translators in the Translation and Other Linguistic Services Program was 69% (1255/1820 hours) in fiscal year 2012–13, below what would be expected based on the collective agreement ^{Footnote 12}.

79. The operational efficiency of the Program can also be measured in terms of the productivity of translators, as measured by the average number of words per hour translated. For fiscal year 2012–13, internal translators at the Translation Bureau translated at the average rate of 249 words per hour. This rate is below private sector productivity rates, which range from 250 to 300 or higher in some cases. The Translation Bureau reports that it is planning to address this issue through the use of translation technologies.

80. Economy refers to minimizing the use of resources. Economy is said to have been achieved when the cost of resources used approximates the minimum amount of resources needed to achieve expected outcomes. The Evaluation assessed the economy of the Program based on the current delivery model (for the consideration of the ability of other models to deliver the same outcomes as the Program, please refer to the section on Alternative Delivery). Economy was assessed through a review of the following: the cost-recovery status of the Program and a sensitivity analysis of internal/external work allocation options for the current delivery model. Based on these indicators, the Evaluation found that while the Program can improve its economy, there are limitations as to what can realistically be achieved by the Translation and Other Linguistic Services Program in reducing its costs using the existing delivery model.

81. Currently, the Translation and Other Linguistic Services Program is not charging its clients sufficient fees to allow them to recover the full costs of the Program. In recent years, the costs incurred by the Translation and Other Linguistic Services Program have been considerably higher than the fees it charges to clients. Consequently, it has been unable to recover all of its costs. For example, the Evaluation calculated that in fiscal year 2012–13, the Program charged its clients, on average, $85.15 per hour for its services, while its combined average cost for work done internally (by salaried translators) and externally (by private sector contractors) was $88.44 per hour. As a result, the Program operated at a loss of $5.0M ^{Footnote 13} in this fiscal year. As a program employing a full cost recovery model, operating at a loss is not sustainable.

82. There are a variety of mechanisms, detailed below, within the current delivery model that can be improved which would have a positive impact on the economy of the Program and allow the Program to reduce its cost per word. This would enable the Program to recover its full costs, assuming fees charged to clients were to remain constant.

83. Currently, work that is outsourced by the Program to the private sector is profitable, while work that is done internally is loss generating. The primary explanation for this is the low fee per word charged by contractors, who are, most typically, freelance translators working directly with the Translation Bureau and who have lower fixed or overhead costs. Unfortunately, reducing costs by increasing the proportion of work translated by contractors is not possible in the short term, given that internal resource levels are relatively fixed and the Program has experienced a reduction in the volume of its translation work over the last few years, as noted earlier in this report. Consequently, it has been necessary to transfer a higher percentage of work to its internal translators in order to maintain their utilization rates.

84. It may be possible to increase utilization rates (billable hours) of internal translators beyond the current maximum of 70% to 80%; however this would require a change to the classification standard for translators within government. It may also be possible to improve productivity (i.e., words translated per hour) rates beyond 300, which is a private sector benchmark; however, this would be a substantial increase from current productivity levels and would likely take some time to achieve. Keeping in mind these challenges, the Evaluation conducted a prospective sensitivity analysis of the potential impacts on the Translation Bureau's cost per word of three factors: the allocation of work to internal and external translators, respectively; improvements to utilization rates; and improvements to productivity rates. The results are summarized in Exhibit 2 below.

Exhibit 2: Potential impacts on the cost per word of efficiency improvements

85. As the bolded text in Exhibit 2 indicate, by achieving the likely maximum possible improvements to utilization (75%) and productivity (300 words per hour) rates for internally translated work, the Program could reduce its combined (internally and externally translated) cost per word to $0.35 per word. This cost per word is composed of two components that are not shown in Exhibit 2: the cost per word for outsourced work, which the Evaluation assumed remains stable at $.36 per word under all these scenarios; and the cost per word for internally translated work which, under these "best-case" scenarios (75% utilization and 300 words per hour), declines to $.34 per word. Thus, the achievement of these "best-case" scenarios would be highly dependent on the improvements to the utilization and productivity rates for internal translators. As noted in the previous paragraph, however, achieving these improvements in utilization and productivity rates for internal translations involves significant challenges for the Translation Bureau.

86. The Bureau could achieve very close to the same economy by transferring a much greater portion of its work to the private sector, as work done by the private sector currently costs the Program $0.36 per word. This amount is comprised of $.19 per word in fees charged by the private sector and $.17 per word in Translation Bureau administrative and indirect costs. The Evaluation was unable to assess whether further reductions below this cost per word would be possible.

Conclusions: Efficiency and economy

87. The efficiency of operations of the Translation and Other Linguistic Services Program could be improved. The proportion of administrative support staff to professional staff is higher than for comparable federal organizations and productivity rates of translators are low compared to private sector translators. Utilization rates are lower than defined by the classification standard for translators within government.

88. The Program is not maximizing its use of resources to the point where it can recover its full costs. There is potential to address this by improving productivity, utilization, and/or increased leveraging of the private sector.

Alternative delivery

89. Alternative delivery refers to the potential for other delivery models to deliver on the expected outcomes of the Program. In examining alternative delivery models, three options were reviewed: increased use of the private sector by the Translation Bureau; private sector delivery of translation services with departments managing their own translation functions; and a change to the program funding model from a full cost-recovery model to a vote-netting model. In assessing options for alternative delivery, the potential benefits or disadvantages of each model and/or transition to it were reviewed. The Evaluation has found that, while there are challenges associated with each alternative, there are potential benefits to considering alternative delivery models for the Program.

90. One option is for the Program to make more use of the private sector to deliver translation services whereby it would likely be able to recover 100% of its costs at current fee levels.

91. This option would necessitate a significant shift of translation work from internal translators to contracted translators from the private sector. However, there are challenges associated with this; principally the costs associated with a dramatic reduction in the internal workforce and the number of years it would take to achieve this change. Further, such a change is not possible immediately, given recent declines in business volume and the negative impact on the utilization rates of internal translators of immediately transferring more work to contracted translators. In addition, certain departments, for reasons of security, require the use of public servants - often on-site - to meet their translation needs. Nevertheless, this option likely can be implemented in the medium term (4-6 years), using attrition and other workforce adjustment tools.

92. This option would also likely necessitate more cost-effective contracting processes in order to maximize potential savings. The Evaluation assessed costs associated with contracting for translation services for a sample of departments who used alternatives to the Program to meet their translation needs. One of the departments interviewed indicated that the direct cost of managing their outsourced translation services was only 5%, or $0.01 per word. This low cost is the result of a single contract with one firm and of highly automated, decentralized job transmittal, invoicing, and payment processes. By contrast, the Translation and Linguistic Services Program incurs direct management costs of $.10 per word when contracting. This higher cost is due to contracting processes that are substantially more complex and labour intensive than the departmental example cited above, due to the need to ensure a fair, open and competitive processes for individual translations distribution of the large volume of work being outsourced; to the extensive quality assurance carried out on this work; and to the management of a large database of contractors.

93. Another option is for each federal department or agency to outsource their translation services with the private sector. The Evaluation found that the private sector's fees are considerably less than fees charged by the Program. A sample of departments interviewed provided us with data on their costs to outsource translation services from the private sector. These departments paid, on average, fees between $0.22 and $0.26 per word for translation work. This is considerably less than the average $0.40 per word charged by the Program^{Footnote 14}.

94. Program stakeholders interviewed for this Evaluation expressed concern that these comparisons did not take into account the value-added benefits of obtaining translation services from the Translation Bureau compared to contracting with the private sector. Primarily, program services are intended to be a full-service package that includes ease of access to Translation Services; urgent, specialized, and ultra-specialized services; treatment of secret documents; and quality assurance processes to ensure product quality and consistency. Stakeholders also mentioned that federal clients obtaining services from the Program did not require additional translation management functions within the department as the Program offered all these services as part of their fee.

95. While this is true for some departments, the Evaluation found that others have established in-house translation co-ordination units within their departments even when requesting at least a part of their translation service needs to be met by the Translation Bureau.

96. Direct costs incurred by these departments for managing the translation work ranged from $0.01 to $0.06 ^{Footnote 15} per word. Other departments, while not able to quantify the amount, acknowledged that the cost of managing translations within the department increased the cost per word considerably over what was charged by private sector translators. One department examined as part of this Evaluation, which outsources solely with the private sector, calculated the full costs of establishing their own in-house translation co-ordination unit ^{Footnote 16} and found that, when considering all the costs associated with translation work, their costs were on par with fees charged by the Program. This Department also noted, however, that they still preferred an in-house translation coordination unit as it allowed for better control over the translation services it received.

97. Furthermore, the Evaluation also found that there was variability in terms of the level of service that the in-house translation coordination units provided. The department with the lowest direct cost provided minimal management services which amounted to payment and administration of contracts whereas the department with higher direct costs provided a range of services, including a quality assurance function. It is important to note that representatives interviewed from departments who use both the Translation Bureau and the private sector to meet their translation needs were as satisfied with the Program's services and products as they were with private sector services and products, as were clients who were surveyed as part of the Evaluation.

98. The Evaluation was able to assess the perceived quality of translation services coordinated through in-house translation coordination units through its survey. The Evaluation found that 89% of clients of in-house translation co-ordination units were satisfied or very satisfied with the deliverables they received. Furthermore, 20% of clients of in-house translation co-ordination services said that they had to review translated documents 'often' or 'always'. Satisfaction was also high for timeliness, with 90% of survey respondents indicating their satisfaction for with timeliness from this provider. The evaluation did not assess the comparability, in terms of volume or nature of material, of texts submitted to in-house translation coordination services with those submitted to the Translation Bureau.

99. These results are marginally more positive when compared to survey results for Translation Bureau clients. However, differences are not statistically significant, indicating that differences in satisfaction are likely due to chance rather than an actual difference. This indicates that in-house translation co-ordination units are likely able to provide products and services that are at least equal to arrangements where the coordination of translation services is organized by the Program on behalf of a client department.

100. Despite apparent lower costs of a decentralized, private sector service provider model and the apparent equivalence in terms of client satisfaction, there are some challenges in adopting this model:

• For security reasons, some departments prefer internal translators who are public servants.
• Some departments may lack the critical mass to establish and manage contracts with the private sector and would prefer a centrally managed service within the federal government.
• The private sector's capacity to provide a comprehensive range of translation services has increased; however, at this time the industry is still somewhat fragmented, with a large number of freelance translators with limited capacity to provide an integrated range of services, especially quality assurance. In the long term, the private sector could adapt to increases in demand.

101. For these reasons, this option is likely not viable in the short-to-medium term and likely is not viable for departments lacking very large volumes of translation work, due to a lack of economies of scale. Further, this would run counter to the government's current direction with respect to centralization of services. It is possible that, in the longer-term, this option may be more viable for some departments, particularly as the industry responds to the change in demand by increasing its capacity to provide resources and facilities with higher security clearances and if procurement instruments (such as government-wide standing offers) were available to enable procurement from the private sector at a lower administrative cost for smaller organizations.

102. Another option for the Program is to move from funding under a revolving fund, which requires full recovery of government wide costs, to funding under a net vote arrangement. Under this option, costs normally paid on behalf of government departments by central agencies (i.e. salary increments, health care costs, and accommodation) would be funded through appropriation (net vote) and would not need to be recovered in the fees charged, as they would be under a revolving fund. Program costs would continue to be recovered from clients through fees. This would reduce the cost per word and would enable the Translation Bureau's fees to be based on cost factors comparable to those departments incur when they deal directly with the private sector (i.e., translation fees and direct costs only). This option was considered viable by several Translation Bureau stakeholders interviewed for this Evaluation.

103. This option has the advantage that the cost incurred by the Program would likely be more consistent with the out-of-pocket costs departments incur when they deal directly with the private sector. Program stakeholders believe that this would likely increase the volume of work sent to the Program and thereby allow for improved efficiencies and economies. Economies of scale, if achieved, would enable lowering of fees as savings are passed on to the client. However, it is not clear that the economies projected by this model would allow the Program to charge fees equivalent to the private sector. Furthermore, although the cost to the Translation Bureau would be diminished, overall costs to the government would not be changed as costs would still be paid by the government - just under a different mechanism.

104. The Evaluation has not fully explored the advantages and disadvantages associated with this option in relation to the cash-flow and financial management flexibility of various funding models. Consideration of this model would necessitate balancing the advantages mentioned above with these and other factors, such as increased use by the Translation Bureau of private sector translators.

Conclusions: Alternative delivery

105. There are viable options to achieve Program outcomes that merit further exploration. The first option, increased use of the private sector, lowers costs for federal departments and agencies. The immediate adoption of this option involves significant challenges but these are likely manageable in the short-to-medium term. The second option, complete private sector delivery, also appears to reduce costs for federal departments and agencies but not to the same degree as option one; however, this model also faces challenges and may be viable only in the medium or long-term, as it requires the industry to continue expanding its capacity. The third option of moving to a net vote arrangement may make comparisons between Program and private sector delivery fairer, but the advantages and disadvantages were not explored in detail.

Conclusions: Performance

106. The Evaluation found that the Translation and Other Linguistic Services Program provides quality and timely translation services to federal organizations. Overall, clients are satisfied with the Program's products and services. Limitations in the evaluation's methodology prevented the isolation and quantification of the Translation and Linguistic Services Program's contribution to the achievement of the shared intermediate outcomes. However, to the extent that the Program has met its immediate outcome (the provision of quality and timely translation services to federal departments and agencies and the judiciary), the Program is likely helping departments to operate effectively in both official languages and to meet their official language obligations. However, the Evaluation was unable to quantify or isolate the contribution of the Program to these intermediate outcomes. It was not feasible for the Evaluation to assess achievement of the ultimate outcome.

107. The Translation and Other Linguistic Services Program has in place a performance measurement system that is adequate to support numerous aspects of internal operational and financial performance. Operating with a traditional service delivery model, being a large service provider, and being within the Federal Government limits the ability of the Program to gain efficiencies. The Program differs from the private sector in its per-hour versus per-word billing, making a cost comparison between the two difficult.

108. The efficiency of the operations of the Translation and Other Linguistic Services Program could be improved. The ratio of administrative support staff to professional staff is high and productivity rates of translators as well as translator utilization rates are below private sector norms. The Program is also not recovering its full costs. Improvements can be made to address these issues.

109. It is not clear whether potential improvements to the efficiency and economy of operations will be sufficient by themselves to enable the Program to reduce its costs to the point where they are competitive with costs incurred by departments that deal directly with the private sector.

110. In terms of alternative delivery, increased use of the private sector by the Translation Bureau is the most viable option in the medium term; while, in the longer-term, other options could be considered.

Management response

111. Overall, the Translation Bureau Management concurs with the Evaluation's findings as well as its recommendations, and is pleased to note that the evaluation team concluded that there is a need for translation services and that it is aligned with Departmental and Government-wide policies and priorities.

Recommendations and management action plan

Recommendation 1: The Chief Executive Officer for the Translation Bureau should, in the short-term, improve the productivity of internal translation staff to levels closer to private sector norms.

• Management action plan 1: The Translation Bureau will establish a more transparent and result-oriented quantitative performance management system for its translators. More specifically:
  • 1.1 It will adopt a new productivity system focusing on concrete quantitative production; and
  • 1.2 It will increase the production rate by 10% by September 2014 to bring it closer to private sector norms.

Recommendation 2: The Chief Executive Officer for the Translation Bureau should, in the long term, continue to seek ways to reduce program delivery costs with consideration given to alternative delivery models.

• Management action plan 2: The Translation Bureau will take concrete action in order to reduce long-term program delivery costs. More specifically:
  • 2.1 It will reform its procurement processes in order to adopt a rationalized organization-wide strategy; and
  • 2.2 It will take every measure to maximize the use of its internal resources, reduce its fixed cost and increase the use of the private sector whenever possible:
    • 2.2.1 In the short term, the Translation Bureau's staffing freeze will continue, and only critical positions will be replaced;
    • 2.2.2 It will establish a human resources steering committee and will provide strict staffing guiding principles;
    • 2.2.3 It will centralize main functions such as procurement and other common tasks to achieve efficiencies; and
    • 2.2.4 It will expand telework and consolidate office space.

Recommendation 3: The Chief Executive Officer for the Translation Bureau should take steps to ensure that its billing practices are based on actual productivity rates (actual words per hour) rather than standardized rates.

• Management action plan 3: The Translation Bureau will modify its billing practices in order to adopt a more transparent approach based on actual volume by moving towards a word-based billing. More specifically:
  • 3.1 In the short-term:
    • 3.1.1 It will establish an hourly rate per client in order to recognize each client's unique needs in terms of demand/volume profile, instead of two rates for general and specialized text;
    • 3.1.2 It will have one "rush premium" rate of 35% instead of two (30% and 50%);
    • 3.1.3 It will establish a single rate for the on-site translator service; and
    • 3.1.4 It will introduce a fair and transparent discount policy to establish client loyalty, as a means of increasing business volume.
  • 3.2 In the long-term, it will move towards a cost per word to align with industry norms and client demand.

About the evaluation

Authority

The Deputy Minister for Public Works and Government Services Canada approved the conduct of this evaluation, on recommendation by the Audit and Evaluation Committee, as part of the 2012–17 Risk-Based Multi Year Audit and Evaluation Plan.

Evaluation objectives

The Evaluation examined the Translation and Other Linguistic Services Program, delivered by the Translation Bureau within PWGSC. This evaluation had two objectives:

• To determine the relevance of the program: the continued need for the program, its alignment with governmental priorities and departmental strategic outcomes and its alignment with federal roles and responsibilities.
• To determine the performance of the program: the achievement of its expected outcomes and a demonstration of the efficiency and economy of the program.

Approach

The Evaluation was conducted in accordance with the Standard on Evaluation for the Government of Canada. The Evaluation of this Program was done in conjunction with the evaluation of related Translation Bureau Programs: the Terminology Standardization Program and the Conference Interpretation Program. Planning and research took place between August 2012 and March 2013. To assess the Evaluation issues and questions, the following lines of evidence were used.

Document review: An initial document review provided an understanding of the program and its context to assist in both the planning phase and in providing background data for the Program. Documents reviewed included documents provided by the Program (including process reviews completed by consultants: an efficiency review (2011), a benchmarking analysis (2012) and a study of the Program's pricing approach and methodology (2012)), as well as documents written about the Program. They included: legislative and policy documents; business cases; annual and multi-year plans; and departmental documents such as annual Reports on Plans and Priorities and Departmental Performance Reports. Additionally, secondary documents were reviewed which provided data on the Program context.

Financial analysis: The Evaluation reviewed Program financial data and related comparative data as well as additional information provided by the Program on the cost of salaries and benefits for professional, administrative support and management staff. A thorough analysis of direct and indirect costs was conducted as well as a review of productivity and utilization data. This information informed the analysis of program efficiency and economy. For the purposes of trend analysis, the Evaluation extended its analysis of some financial data to fiscal year 2006–07.

Interviews: Interviews with Program Stakeholders: Interviews were conducted with Managers and Senior Managers within the Translation Bureau (14), officials of federal organization translation co-ordination units (6), private sector translation and linguistic services providers (4). The interviewees provided information about the Program's activities, outputs, expected outcomes, stakeholders, environment/context, relevance, and performance. The qualitative analysis of the interviews provided information about the Program's activities, outputs, expected outcomes, stakeholders, relevance, and performance from the perspective of program managers, client departments, and other related stakeholders. Interview guides were used.

Survey: Two online surveys were conducted in this evaluation.

• Survey of Translation Bureau Clients: Authors of texts that were sent to the Translation Bureau for translation were asked to provide their feedback on their experience with the Bureau and its products to compare this experience with that of alternatives (such as internal translation coordination units and directly with the private sector). A total of 1,000 people were invited to complete the survey. 291 people responded, representing a response rate of 29.1%. The total population is estimated at 6,390.
• Survey of Private Sector Translation Providers: Private sector translators who are on the Translation Bureau's contractor database were invited to participate in a survey. The survey focused on providing a profile of the translation and linguistic services industry as well as assessing the Translation Bureau's quality, timeliness and cost of the services. A total of 500 people were invited to complete the survey. 187 people responded, representing a response rate of 37.2%. The total population is estimated at 2,219.

Review of program Performance Data: The Translation Bureau generates performance data which was reviewed for this evaluation. This included data on program activities, outputs and outcomes; examples include data from Termium Plus^®, client satisfaction surveys and from reports and studies conducted by external parties.

Environmental scan: Jurisdictions reviewed in the environmental scan were selected based on the availability of data and it's comparability with the federal government and/or Canadian context. A total of eight jurisdictions were reviewed. The team reviewed information available online to assess the model and obtain baseline/comparable data, where available. Subsequently, the evaluation team contacted officials of these jurisdictions to request interviews. Five either participated in an interview or submitted responses to interview questions electronically, while three provided information but did not complete the interview questionnaire.

Limitations of the evaluation of the Translation and Other Linguistic Services Program

The conduct of this evaluation was part of a larger engagement which piloted a 'cluster' approach to evaluating Translation Bureau programs that had not yet been subject to an evaluation: the Terminology Standardization Program, the Translation and Other Linguistic Services Program, and the Conference Interpretation Program. A Logic model that depicted the outcomes that were common to all three programs was prepared by the Translation Bureau. The data collection methodologies employed for the cluster engagement did not support the isolation and quantification of a program's specific contribution to the achievement of the Translation Bureau's intermediate and ultimate results. Reference to this limitation is made within the Report.

The 'cluster' engagement employed a risk-based approach to determining scope and level of effort, placing a greater emphasis on the assessment of the Translation and Other Linguistic Services Program, given its greater materiality (in terms of full time employees (FTEs) and $), sensitivity and the current transformation/modernization initiatives underway in the Program. As such, the overall risk of this Program is the highest among the three that were simultaneously evaluated. In addition, the Translation and Other Linguistic Services Program accounts for 85.9% of Translation Bureau expenditures for fiscal year 2011–12 and 4.8% of direct program spending for PWGSC. As a result of these factors, the evaluation team chose to dedicate a higher level of effort to evaluating this Program and employed a traditional approach to reporting.

A number of challenges were experienced in the design, data collection and analysis of the Evaluation of the Translation and Other Linguistic Services Program. These challenges, as well as related limitations and strategies used to mitigate their impact, are outlined below.

It was difficult to compare the Program internationally, as the nature of Canada's language profile is unique. The Translation and Other Linguistic Services Program is the largest translation services provider in Canada and as such this made comparisons to other service providers difficult. The Evaluation conducted comparisons with Canadian firms which are the other major suppliers of translation services to federal clients. There is limited data available on the translation services industry in Canada. The Evaluation obtained industry data that was slightly dated but which nonetheless provided information about trends in the industry. This information was corroborated through interviews with major private sector service providers as well as a survey of private sector providers who were primarily freelancers.

The Program provides an optional service to over 150 client departments and agencies. There are also some federal organizations that do not use the Program's services at all. The Evaluation obtained client feedback about the Program from a selection of these clients. To mitigate the effects of a small sample size, the Evaluation used Public Accounts of Canada data to isolate federal organizations into different groups: 1 - Organizations that used the Program's services exclusively (both larger and smaller organizations); 2 - Organizations that used a combination of the Program's services and the private sector; and 3 - Organizations that used the private sector almost exclusively. PWGSC was able to obtain interviews with representatives from these different groups. Despite this, it is important to note that the sample was small and, as such, this feedback reflects the experiences of these departments and cannot necessarily be generalized.

The evaluation of Translation and Other Linguistic Services Program focused on the use of existing data to inform reporting. Efforts were made to draw information on the specific contribution by the Program to the achievement of shared Translation Bureau outcomes from existing data however it was not possible to do so. Additionally, one particular line of evidence intended to support an assessment of the achievement of the intermediate outcome did not yield information that was useful to the evaluation. The evaluation team reviewed the Office of the Commissioner of Official Languages report card ratings of a select number of Translation Bureau clients to assess any linkages between use of the Program's services and positive rating outcomes. Unfortunately, the Evaluation was unable to attribute report card ratings to the Program's activities.

Finally, efforts were made to draw linkages between the Program's outputs and outcomes but limitations in existing data had an impact on our ability to assess the overall efficiency of the program. To mitigate this issue, PWGSC focussed this assessment of efficiency on operational rather than allocative efficiency. This allowed the Evaluation to draw some conclusions about efficiency, even if only a more narrow scope was applied.

Reporting

Findings were documented in a Director's Draft Report, which was reviewed by the Office of Audit and Evaluation's Quality Assessment function. The Translation Bureau's Chief Executive Officer was provided with the Director's Draft Report and a request to validate facts and comment on the Report. A Chief Audit and Evaluation Executive's Draft Report was prepared and provided to the Chief Executive Officer, Translation Bureau for acceptance as the Office of Primary Interest. The Office of Primary Interest was requested to respond with a Management action plan. The Draft Final report, including the Management action plan, was presented to PWGSC's Audit and Evaluation Committee for the Deputy Minister's approval in November 2013. The Final report will be submitted to the Treasury Board Secretariat and posted on the PWGSC website.

Project team

The Evaluation was conducted by employees of the Office of Audit and Evaluation, overseen by the Director of Evaluation and under the overall direction of the Chief Audit and Evaluation Executive. The Evaluation was reviewed by the Quality Assessment function of the Office of Audit and Evaluation.