2010-602 Evaluation of the Shared Travel Services Initiative
(Final Report)

March 24, 2011

Table of Contents

• Main Points
• Introduction
• Profile
• Focus of the Evaluation
• Findings and Conclusions
  • Relevance
    • Continuing need
    • Alignment with federal and departmental priorities
    • Alignment with federal roles and responsibilities
  • Conclusions: Relevance
  • Performance
    • Outcome Achievement
      • Awareness of Shared Travel Services Initiative (STSI)
      • Uptake of STSI tools
      • Uptake - Policy requirements
      • Uptake - External and internal barriers
      • Enhancement of the travel experience
      • Administrative and direct travel costs
      • Accountability, transparency, and oversight for air travel
    • Economy
      • The departmental surcharge and STSI cost recovery
      • Discount agreements and STSI
      • Service delivery model
    • Efficiency
      • Efficiency ratio (outputs over costs)
      • Potential savings of the Online Booking Tool
  • Conclusions: Performance
• General Conclusions
• Management Response
• Recommendations and Management Action Plan
• About the Evaluation
• Appendix A: Process Chart of STSI
• Appendix B: STSI and the Cost of Travel
• Appendix C: Evaluation Matrix

Main Points

What was evaluated

i. The Shared Travel Services Initiative (STSI) provides travel services for federal departments and agencies. It serves two interrelated purposes: (a) to provide an automated mechanism for travel and (b) to provide business intelligence and customized reports to help manage travel more effectively and negotiate more cost-efficient contracts with travel suppliers. These services are included under sub-activity 1.7.11 of the Program Activity Architecture, and represent approximately $10.3M in annual revenue. This report presents the results of the evaluation of the Shared Travel Services Initiative program. The evaluation covered the period of 2003/04 to 2009/10.

Why it is important

ii. Travel expenses for the Government of Canada totalled $1.59 billion in 2009/10, with a daily average of 2,000 air and rail bookings and over 6,500 federal employees on travel status on any given day. The Government of Canada spends more on travel than any other enterprise in Canada in order to deliver its programs and services to Canadians. For example, travel takes place for such things as weights and measures inspections, food inspections, international trade negotiations, enforcement of environmental regulations, tax and other audits, and mapping the regions and resources of Canada. As the second largest country in the world, in terms of landmass, with a population density of 3.4 persons per square kilometre, travel plays an integral role in helping federal organizations deliver on their mandates across regions. By travelling, Government of Canada employees can deliver on region-specific objectives; provide services and seminars to geographically remote locations; and benefit from face-to-face communications with other public servants and citizens.

iii. STSI is intended to have oversight responsibility over the federal travel commodity. The concept of STSI was developed by the Treasury Board of Canada Secretariat (TBS) in response to issues raised in a 1995 Auditor General Report. The Auditor General commented on the lack of travel information and the inefficiencies with administering travel.

iv. In 2000, the travel modernization team, comprised of TBS and Public Works and Government Services Canada (PWGSC) staff, received approval to introduce a solution to the federal government's travel needs. In 2004, STSI was transferred to PWGSC and a seven-year contract (with a possible two-year extension) was awarded for $416.8 million, including all options. In accordance with the contract, departments are charged for using the components of STSI, which results in revenue to the vendor. STSI's staff, who manage the contract, operate on a cost recovery basis, where the revenues captured by the program equal the expenditures the program incurs. The revenue sources are from airline and railway cash rebates, commission reimbursements, travel card rebates, and a departmental surcharge administered by the program to recoup its expenditures.

What was found

v. The STSI's rationale is sound. Federal employees need to travel in order to deliver departmental programs and mandates. As such, departments require an efficient mechanism to plan and arrange travel. There is also a need for the federal government to increase its accountability and oversight of travel in order to efficiently assess the cost of travel. PWGSC is required to deliver STSI under the Treasury Board Common Services Policy, and the use of STSI is mandatory for departments. Despite its mandatory status, some departments have not implemented all of the components, particularly the Expense Management Tool.

vi. Overall, the performance of the program was mixed. Uptake of STSI tools varied among departments and has not reached the expected uptake presented in the 2003 STSI Business Case, especially for the Expense Management Tool and the Designated Travel Card. STSI operates in a policy environment that is not conducive to achieving expected uptake levels. The lack of uptake is not in spirit with the Common Services Policy and is not conducive towards the benefits of economies of scale. As it stands, the uptake levels are achievable, but it requires a strong commitment from deputy heads, as seen with Public Works and Government Services Canada among others.

vii. For those departments who do not expense claims through the Expense Management Tool or an equivalent system, the practice of manual claim submissions continues, which is not in line with best practices. Nevertheless, over the scope of the evaluation virtually all air and rail travel was booked through the program. With respect to the policy barriers, and the barriers internal and external to the program, much has been accomplished since the implementation of the program.

viii. The potential benefits of the program, and the business intelligence it captures, will only be fully understood with increasing departmental uptake. There are areas where the government lacks the business intelligence to effectively account and manage its travel, especially for accommodation and car travel expenditures. Capturing this information will further enable STSI to more effectively provide oversight and manage the travel commodity. The low uptake has had financial consequences, such as the surcharge administered to fund the program. Originally intended to be a short-term solution, the current funding model of the program has caused some concern with clients. Moving forward, further savings are attainable as the uptake of services increase. Until such an increase transpires, the program will not be able to capitalize on the potential administrative savings that were initially expected. The program's delivery model, leveraging the private sector to deliver the components while retaining the procurement function, was in line with best practices.

Management Response

The Shared Travel Services Initiative within the Acquisitions Branch accepts the evaluation findings and intends to act on the recommendations of the evaluation by implementing their Management Action Plan detailed as follows.

Recommendations and Management Action Plan

• Recommendation 1: The Assistant Deputy Minister, Acquisitions Branch, should coordinate with the Treasury Board Secretariat to increase the use of the Expense Management Tool (or equivalent) within the federal government.

  • Management Action Plan 1.1: The Assistant Deputy Minister, Acquisitions Branch, will propose a meeting with Treasury Board Secretariat representatives to address the challenges of implementing an electronic travel expense management tool in the Government of Canada. Items to be addressed will include:

    1. The issue of the current Common Services Policy

    2. The requirement for departments to implement some type of expense management tool

    3. The requirement for departments who elect not to use the Shared Travel Services expense management tool to provide detailed travel expense data to Public Works and Government Services Canada for commodity management purposes

    4. The Treasury Board Secretariat commitment to achieve the agreed-upon goals

    5. An implementation plan including milestones and outcome targets

  • Management Action Plan 1.2: The Assistant Deputy Minister, Acquisitions Branch, will report back to the Deputy Minister, Public Works and Government Services Canada on the outcomes of the above meeting.

• Recommendation 2: The Assistant Deputy Minister, Acquisitions Branch, should engage the Treasury Board Secretariat to resolve the oversight gap of accommodation and car travel expenditures.

  • Management Action Plan 2.1: The Assistant Deputy Minister, Acquisitions Branch, will propose a meeting with Treasury Board Secretariat representatives to address the challenges of bookings made outside of the Shared Travel Services. Items to be addressed will include:

    1. Methods to ensure accommodation and car rental expenditures are properly captured

    2. Issues with the Common Services Policy

    3. The Treasury Board Secretariat commitment to achieve the agreed-upon goals

    4. An implementation plan including milestones and outcome targets

  • Management Action Plan 2.2: The Assistant Deputy Minister, Acquisitions Branch, will report back to the Deputy Minister, Public Works and Government Services Canada on the outcomes of the above meeting.

• Recommendation 3: The Assistant Deputy Minister, Acquisitions Branch, should work with the Treasury Board Secretariat to address the policy barriers to STSI uptake in order to achieve maximum value.

  • Management Action Plan 3.1: The Assistant Deputy Minister, Acquisitions Branch, will propose a meeting with Treasury Board Secretariat representatives to address the challenges of policy issues that negatively impact Shared Travel Services uptake. Items to be addressed will include:

    1. The lack of adherence to the Common Services Policy

    2. The incongruity between the National Joint Council Travel Directive and the Policy on Travel Cards

    3. The direct costs faced by the program as a result of low uptake e.g. payments to contractors

    4. The Treasury Board Secretariat commitment to achieve the agreed-upon goals

    5. An implementation plan including milestones and outcome targets

  • Management Action Plan 3.2: The Assistant Deputy Minister, Acquisitions Branch, will report back to the Deputy Minister, Public Works and Government Services Canada on the outcomes of the above meeting.

• Recommendation 4: The Assistant Deputy Minister, Acquisitions Branch, should pursue a more sustainable, permanent funding model for STSI.

  • Management Action Plan 4.1: The Assistant Deputy Minister, Acquisitions Branch, will put forward a funding plan for Shared Travel Services that removes the requirement for surcharges and the dependency on air/rail rebates.

Introduction

1. The Public Work and Government Services Canada Shared Travel Services Initiative (STSI) program provides travel services for federal departments and agencies. This report presents the results of the evaluation of the program. The examination covered the period of 2003/04 to 2009/10. The Audit and Evaluation Committee of PWGSC approved this evaluation as part of the 2009/10 to 2013/14 Risk-Based Multi-Year Audit and Evaluation Plan.

Profile

Background

2. The concept of STSI was developed by the Treasury Board of Canada Secretariat (TBS) in response to issues raised in a 1995 Report of the Auditor General. These include a lack of transparency, accountability, and business intelligence information with respect to the way the government managed travel. The Auditor General also observed that data was not organized or consolidated for senior management to efficiently assess the need for and costs of travel.

3. The Auditor General indicated that central agencies and departments acknowledged that the cost of travel administration needed to be reduced. Although no studies had been conducted to determine the costs, the Auditor General estimated that the average cost of administering travel was between 25% and 35% of travel expenditures. Reducing the time spent on the administration of travel, according to the Auditor General, would free employees to better meet operational requirements. The Auditor General claimed that new technologies and re-engineered processes could improve control and reduce the cost of travel and travel administration.

4. In 2000, the Government of Canada travel modernization team, comprised of TBS and PWGSC staff, received approval to introduce a proposed solution to the Government's travel needs. In 2004, a seven-year contract (with a possible two-year extension) was awarded to a private sector company to develop and deliver a travel solution. The contract at the time was valued at $417 million. The projected value of the contract was based on the inclusion of the two year extension and on expected levels of uptake for the services. The $417 million was intended to manage $17.5 billion worth of travel spend over nine years. However the expected levels of uptake did not materialize, and as consequence, neither did the expected value of the contract. Service uptake and program costs are further discussed in the performance section of the report.

5. Once the contract was awarded, the travel modernization team was transferred to PWGSC and with the contract, became the Shared Travel Services Initiative. In 2006, PWGSC's Office of Audit and Evaluation completed a formative evaluation of STSI in which progress towards meeting its goals was assessed. The focus was the identification of problems encountered during the development and early adoption of STSI tools and services.

6. In 2008, PWGSC and TBS conducted a joint mid-term review of the management of government travel. The mid-term review concluded that the program was best in class. It recommended that an end-to-end enterprise approach be revisited at the end of the current contract in 2013 and that in the interim, an optional expense claims modules be permitted as long as spend data was fed back to PWGSC. In 2009, the government approved a contract extension with the service provider until 2013 and amended the contract so that an optional approach could be taken with the Expense Management Tool. At the time of this evaluation, the Request for Information for the 2013 contract had been released.

7. In 2009/10, Government of Canada travel expenses totalled $1.59 billion, with a daily average of 2,000 air and rail bookings and over 6,500 federal employees on travel status.

Program Description

8. STSI is intended to provide a travel management solution for the Government of Canada (GC). It serves two interrelated purposes. First, STSI provides an automated mechanism for: trip planning and authorization; travel booking; travel expense recording; and submitting and approving expense claims. Second, STSI provides business intelligence and customized reports to help: (a) departmental senior managers more effectively manage and explain their travel expenditures; and (b) PWGSC to negotiate more cost-efficient contracts with travel suppliers, e.g., airlines, hotels and car rental agencies. STSI services are available to all federal departments and agencies, and the program is delivered by the private sector on a fee-for-service basis.

9. The STSI program includes the following components:

• Online Portal: Web-based gateway for travel services and information. It provides federal travellers secure access to online travel tools and services, including electronic travel authorization, the Accommodation and Car Rental Directory, and current travel information and advisories.

• Online Booking Tool: Allows federal employees to book air and rail travel, hotels, and car rentals—within Canada and the United States—securely from their own desktop.

• Travel Call Centre: Designed to complement the Online Booking Tool, the call centre operates seven days a week, 24 hours a day, to assist travellers who have special needs, complex travel arrangements, or are unable to access the Online Booking Tool.

• Designated Travel Card: A unique credit card with travel benefits issued to federal business travellers for all expenses incurred while travelling, including hotels, car rentals, meals, local transportation, and travel advances. It is designed to provide an automatic transaction feed to the Expense Management Tool.

• Designated Responsibility Centre Travel Card: A credit account for each responsibility centre in a department for all air and rail fares booked by federal travellers within the responsibility centre. This card also provides an automatic transaction feed to the Expense Management Tool.

• Expense Management Tool: An expense claim system that streamlines the process by automating the submission, approval and tracking of travel requests and expense reports. The Expense Management Tool ensures consistent application of government travel and expense policies through built-in policy rules; it also automatically calculates travel allowances and exchange rate conversions.

• Amex@Work reporting tool: Provides a consolidated view of departmental and government-wide travel spending, allowing departmental senior managers to monitor and analyse travel costs and policy compliance at various levels of detail.

A graphical process chart of the components of STSI is provided in Appendix A.

Roles and Responsibilities

10. The Shared Travel Service Initiative is compromised of the following directorates. In addition, a voluntary group of departmental representatives oversee its implementation.

• Shared Travel Services Directorate: As the business owner and technical authority, the senior director is accountable to provide the professional leadership for the management, operations and delivery of the program.

• Shared Travel Services Renewal Project Directorate: Manages the renewal and enhancement of the new Travel Management Services contract. As well, the Directorate works with the senior director to ensure smooth platform migration and to equip the organization with the resources to effectively manage the program.

• Policies, Processes, and Systems Division: Manages the Travel Card Program and provides subject matter expertise for the Travel Authorization Number Registry, the Online Booking Tool, and the Travel Agency. In addition, they manage corporate and administrative functions.

• Client Services and Training Division: Manages the Training Team, and oversees the provision of consultative support, development of a training strategy, and training tools and aids. Client Services and Training oversees client support and assists the client departments in achieving and optimizing travel management practices.

• Design, Configuration, and Testing Division: Oversees the Design and Configuration Team, which is responsible for the Government's view on design issues, research, and recommendations for all components of the STSI. In addition, they manage acceptance testing, incremental threat risk assessments, incremental privacy impact assessments, and the customer questionnaire.

• Account Management and Business Analytics Division: Manages and promotes the implementation and adoption of the solution in departments and agencies. They also manage the Business Analysis team, which supports management decision-making and provides data and analytical support to STSI teams.

• Travel Advisory Board: Voluntary group of departmental representatives that oversee the service and implementation of the STSI by departments and agencies against its key objectives and milestones. The Board assists STSI in adopting best practices and determines whether adjustments should be made to STSI to improve its implementation and delivery.

Resources

11. The gross expenditure authority for STSI in 2009/10 was $9.77M and it operated with 38 resources. The program operates on a cost recovery basis. It is funded from net-voted revenues and revenues from the Optional Services Revolving Fund. Revolving fund revenues are generated from airline and railway cash rebates and service provider commission reimbursements^{Footnote 1}, while net-vote revenues are generated from travel card rebates and departmental surcharges administered twice annually by the program.

12. In 2009/10, STSI's revenue of $10.3 million consisted of $4.5 million in departmental surcharges, $3.5 million in air/rail cash rebates, $1.8 million in travel card rebates, and $0.5 million in hotel and car rental commissions.

13. In addition to the surcharge, departments are charged for using the components of STSI. These fees are paid to the contractors who deliver the various tools and services. In 2009/10, the Online Booking Tool charged $20.79 per booking, the Travel Call Center charged $46.26 per booking, and the Expense Management Tool charged on average $13 per claim submitted.^{Footnote 2} The program's association with travel fees is displayed in Appendix B.

Program Logic Model

14. A logic model is a visual representation that links a program's activities, outputs, and outcomes; provides a systematic and visual method of illustrating the program theory and shows the logic of how a program, policy, or initiative is expected to achieve its objectives. It also provides the basis for developing the performance measurement and evaluation strategies, including the evaluation matrix.

15. The program's logic model was based on the logic model developed for the 2006 formative evaluation, a detailed document review, and meetings with program managers. It was subsequently validated with program managers. The logic model is presented in Exhibit 1.

Exhibit 1: Logic Model

A larger image of Exhibit 1: Logic Model with a long description is available on a separate page. Due to the size of the image, it may not display properly.

Focus of the Evaluation

16. The objective of this evaluation was to determine the program's relevance and performance in achieving its planned outcomes in accordance with the Treasury Board Policy on Evaluation.

17. An evaluation matrix—including evaluation issues, questions, indicators and data sources—was developed during the planning phase and is presented in Appendix C. Multiple lines of evidence were used to assess the program. These include:

• Document Review: A literature review was conducted to contextualize the program both nationally and internationally; provide theoretical background for the program model; provide baseline data against which the program could be assessed; and identify alternative delivery models through an analysis of other jurisdictions.

• Interviews: The evaluation team conducted interviews with key program and PWGSC managers. In addition, the evaluation team conducted interviews with representatives from client departments and agencies, Treasury Board of Canada Secretariat staff, and with the contractor and subcontractors of the travel services.

• Performance Data Analysis: Performance data was analysed in order to measure the program's outcomes. The evaluation team leveraged both internal performance reports and raw data provided by the program when appropriate.

• Client Survey: The program, through the primary contractor, annually surveys its users. The population surveyed includes travellers, travel arrangers, account verifiers and travel approvers. The results of the surveys were incorporated where relevant.

• Financial Analysis: Financial data related to the program's budgets, revenues and expenditures were reviewed and analyzed to understand the effectiveness of the cost recovery model of the program. All financial information was provided by the program, with the exception of the total federal government travel expenditures, which was provided by another group within Acquisitions Branch not related to the program.

18. More information on the approach and methodologies used to conduct this evaluation can be found in the About the Evaluation section at the end of this report.

Findings and Conclusions

19. The findings and conclusions below are based the multiple lines of evidence used during the evaluation and are presented by evaluation issue (relevance and performance).

Relevance

20. Relevance is measured by the extent to which the program addresses a demonstrable and continuing need; is aligned with federal and departmental priorities and is an appropriate role and responsibility for the federal government.

Continuing need

21. Continuing need assesses the extent to which the program continues to address a demonstrable need and is responsive to its clients. Lines of evidence reviewed to evaluate continuing need include: the ongoing validity of its original rationale; the legislation surrounding its delivery; and the use of the program's services by clients. Based on these criteria, the evaluation found that there is a continuing need to deliver STSI to federal organizations.

22. The rationale for the program remains sound. Federal employees need to travel in order to deliver departmental programs and mandates. There is a need for the federal government to increase its accountability and oversight of travel in order to efficiently assess the cost of travel. Additionally, the need to reduce the cost of planning, arranging and administering travel through the use of new technologies continues since departments require an efficient mechanism to plan and arrange travel, and to submit expense claims to ensure compliance with appropriate legislation and policies.

23. PWGSC is required to deliver STSI under Appendix E, section 6.12 of the Treasury Board Common Services Policy. Appendix E also makes the use of STSI mandatory. As a mandatory program, PWGSC must offer all the components of the program while Departments listed under Schedules I, I.1, or II of the Financial Administration Act are required to use the program, unless exempted by Treasury Board or granted the necessary delegation of authority from the common services organization. According to TBS, not using the STSI services for government travel would make the applicable departments non-compliant with the policy.

24. The policy states that certain services are designated as mandatory when a government-wide interest or consideration prevails over, or coincides with, the interests of individual departments and agencies. Such considerations can include: economies of scale; access to centres of expertise or specialization; the need for a high level of consistency; or attaining the government's social, economic, or environmental objectives.

25. With regards to the use of the program's services by clients, clients interviewed were unanimous in their opinion that there is a demonstrable need for STSI and supported the concept of a centralized, standardized, common travel service. However, and despite the program's mandatory status, departmental uptake of the components have not been unanimously implemented across the GC due to various barriers. These barriers are discussed in the performance section of the report. Of the 106 federal institutions identified by STSI as clients, 96 institutions use the Travel Call Centre, 80 use the Online Booking Tool, and 95 use the Designated Travel Card.

26. While the Department of National Defence in the past has resisted implementing the Online Booking Tool, it is currently embarking on a pilot project to test the feasibility of full adoption. The Department wishes to further automate their current travel practices as well as implement more built-in controls and are regarding STSI as a potential solution. Most other non-users of the three tools are small organizations or are not part of Schedules I, I.1, or II of the Financial Administration Act.

27. At the time of the evaluation, 25 of 106 federal institutions had adopted the Expense Management Tool. Although the adoption rate is low for this tool, the need for travel-related business intelligence continues to exist. At the time of this evaluation, some federal organizations who have not implemented the Expense Management Tool have implemented an equivalent system. In a number of cases, these systems predate the program. For instance, the Department of National Defence, Environment Canada, and the Public Service Commission have implemented their own expense claims system, which runs counter to the Common Services Policy, but is aligned with the new modular approach being pursued by STSI since 2009/10. For the federal organizations that do not use an expense claims system, the practice of manual claims submissions is not aligned with best practices in government or private sector. The information captured by an expense claims system would help government organizations to more effectively manage their travel and to ensure that the government remains accountable to the Canadian public.

28. In all, a continuing need for the program persists. The relevance of the program's rationale continues and PWGSC has an obligation to provide the program in accordance with the Common Services Policy. Although not all components have been universally implemented, virtually all federal organizations use at least some of the components.

Alignment with federal and departmental priorities

29. Alignment with federal and departmental priorities is determined by assessing the program linkages with federal government priorities and with departmental strategic outcomes. The evaluation found that STSI is aligned with federal and departmental priorities.

30. The March 2010 Speech from the Throne emphasized restoring fiscal balance. The Speech outlined steps towards restraining federal program spending, including:

• Freezing departmental operating budgets (i.e., total amount spent on salaries, administration and overhead); and
• Launching a review of administrative services to improve efficiency.

31. The 2010 Budget proposed significant actions to reduce the growth in operating expenditures. This included freezing departmental spending on travel, conferences, and hospitality at 2008/09 levels. As a common service, STSI reduces duplication by eliminating the need for departments to produce their own travel solution. One of the objectives of the program is to reduce travel expenditures through cost control and improved travel management. By capturing travel intelligence, STSI enables federal organizations to travel more effectively through better travel management, thereby reducing costs.

32. The November 2008 Speech from the Throne committed the federal government to pursuing innovative reforms to the administration of programs and services in order to build partnerships with third parties and the private sector to deliver better services at a lower cost. The service delivery method aligns to this priority. The tools and services of STSI are outsourced, while PWGSC provides oversight and stewardship. The procurement of discount rates (such as airline discount agreements) is conducted by Acquisitions Branch, in order to leverage the federal government's market influence.

33. The program also supports the objectives of the Federal Accountability Act by strengthening auditing and accountability within departments. As a centralized expense tracker, STSI's Expense Management Tool supports transparency and accountability through the centralization of expenditure information. The service also provides enhanced auditing and reporting capabilities, data warehousing, and built-in policy compliance.

34. In the 2009/10 and 2010/11 Report on Plans and Priorities, PWGSC's strategic outcome was to provide "high quality, central programs and services that ensure sound stewardship on behalf of Canadians and meet the program needs of federal institutions." While not specifically mentioned, STSI aligns to this outcome by providing a government-wide centralized common service.

35. The program is also within the scope of the Administrative Services Review conducted by the Privy Council Office. The goal of this review is to assess administrative services and overhead costs across the government to identify opportunities to improve access to government services, streamline the delivery of services, and reduce the costs of program delivery.

36. Approximately half of the federal managers the evaluation interviewed identified STSI as a priority for their department. Some clients have shown major commitment to the program, enforcing policy compliance within their respective departments, including the Expense Management Tool.

37. The program is aligned with government priorities. The 2010 Budget proposed significant actions to reduce the growth in operating expenditures, and the 2008 Speech from the Throne committed the government to pursuing innovative reforms to the administration of programs and services.

Alignment with federal roles and responsibilities

38. To determine whether or not the program is aligned with the roles and responsibilities of the federal government, three elements were examined: a) whether the responsibility for the program could be transferred to another level of government (i.e.: to the provinces); b) whether the responsibility for the program could be transferred to the private sector; and c) whether the responsibility for the program could be decentralized to individual departments and agencies.

39. The program is for federal government travel. It addresses needs that are internal to the federal government. As such, devolution of responsibility to a different level of government would not be viable.

40. The program offers services that are available to the private sector. Since there are private sector solutions available to deliver the components, PWGSC outsourced the service components but retained overall responsibility by providing stewardship and oversight of the contract. If the federal government were to build similar components internally, it would not be appropriate since the federal government would be creating duplicate services that are readily available within the private sector. As is, it is appropriate for PWGSC to outsource the components. Further information on service delivery is discussed in the performance section of this report.

41. Travel is an activity that supports program delivery across the federal government. As a common service provider it is appropriate that the responsibility for the program reside within PWGSC. Achieving critical mass for negotiating discounts with airlines, railways, hotels, and car rental agencies requires the collaboration of federal organizations. Decentralizing the responsibility of travel services to individual departments and agencies would require them to potentially duplicate these services, contrary to the principles of the Treasury Board Common Services Policy. The policy established travel services as mandatory for the purpose of economies of scale.

42. Both the private sector and other federal government jurisdictions have integrated, or are integrating, a centralized end-to-end solution for travel management similar to STSI. This trend can be seen in the national governments of Australia, United Kingdom, and the United States as they seek to increase savings through consolidation of travel information and better travel management.

43. On the whole, the program is aligned with roles and responsibilities of the federal government. Travel is a common service that supports program delivery, making PWGSC the logical provider of STSI. Moreover, decentralization of federal travel services would run contrary to economies of scale and current international trends.

Conclusions: Relevance

44. Based on the continuing need; alignment with federal and departmental priorities; and alignment with federal roles and responsibilities, the evaluation concluded that the program remains relevant.

45. PWGSC is obligated to provide the program in accordance with Common Services Policy. Although not all departments have bought into every component of the system, most of all federal organizations have implemented the Travel Call Center, Online Booking Tool and the Travel Card. With regard to expense claims, for departments who have not implemented either the Expense Management Tool or an equivalent system, the current practice of manual claims submission is not aligned with best practices. Additionally, the objectives of the program align with federal priorities as well as PWGSC's strategic outcome.

46. As travel is a common service that supports program delivery, it is appropriate that PWGSC provides STSI. The private sector has the capacity to deliver the components. PWGSC is well suited to provide oversight of its delivery and carry out the appropriate procurement requirements.

Performance

47. Performance is the extent to which a program or initiative is successful in achieving its objectives (outcomes) and the degree to which it is able to do so in a cost-effective manner that demonstrates efficiency and economy.

Outcome Achievement

48. Performance can be measured as the extent to which a program or initiative is achieving the targets of their outcomes. STSI's outcomes, as presented in its logic model in Exhibit 1, are designed so that their achievement will aid PWGSC in accomplishing its strategic outcome. This evaluation examined the performance of the STSI in providing federal organizations high-quality travel services.

Awareness of STSI

49. Program awareness is the extent to which the stakeholders of the program are aware of the travel services and tools available to them. Awareness is measured through an annual survey of travellers, travel arrangers, account verifiers, and travel approvers, administered by the primary contractor of STSI. According to the survey, awareness of the end-to-end STSI solution had increased from 41% in 2005/06 to 95% in 2009/10. The survey also measured awareness by service. The Expense Management Tool received the lowest rating, achieving only 44% awareness in 2009/10, which was an increase from 33% in 2007/08. The low awareness of the Expense Management Tool was unsurprising since the component's awareness depends on the buy-in of departments, which was relatively low.

Uptake of STSI tools

50. Uptake is measured by the use of travel services among the federal government client base (i.e. actual travellers). Uptake was measured in this matter since departmental implementation does not guarantee actual usage from travellers. There is a direct relationship between uptake and achieving the program's outcomes. When the program was first being developed, the financial and qualitative benefits to be achieved by implementing the STSI solution were established. These benefits assumed certain levels of government-wide uptake of the program's components. Required uptake levels included:

• 80% usage of the electronic booking service,
• 90% adoption of the government travel card, and
• 100% usage of the electronic expense claim processing system.

51. According to the initial business case, achieving these usage rates among travellers would ensure that STSI could operate on a cost recovery basis and generate the desired benefits. The targets were based on certain assumptions, such as an enforced mandatory status as per the Common Services Policy. At the time of the evaluation, the services were effectively optional (as discussed later in the report). Although PWGSC has assumed the responsibility for uptake, authority to enforce its usage does not reside within the department, but with deputy heads. The expected uptake and actual uptake are presented in Exhibit 2. For further information concerning how uptake was calculated and the limitations of the data, the methodology is discussed in the About the Evaluation section of the report.

Table Summary

This table represents the STSI Uptake by Travelers for the Online Booking Tool, the Designated Travel Card and the Expense Management Tool.

Exhibit 2: Summary of STSI Uptake by Travellers

STSI Components	2003 Expected Uptake	Uptake as of 2009/10
On-line Booking Tool	80%	43%
Designated Travel Card	90%	35%
Expense Management Tool	100%	4%

Source: Expected Uptake taken from the 2003 STSI Business Case

52. Online Booking Tool - When booking air or rail travel, a traveller can use either the Online Booking Tool or the Travel Call Center. Both provide the same service—the difference being the cost to departments. In 2009/10, the cost of using the Online Booking Tool was $20.79, whereas the cost of using the Travel Call Centre was $46.26.

53. The Online Booking Tool is the preferred tool for domestic and trans-border bookings because of its lower cost.^{Footnote 3} From 2005/06 to 2009/10 uptake increased from 19% to 43% for all domestic and trans-border bookings. A 2009 Online Booking Tool Benchmark study reported an industry average of 73% uptake, while the 2003 Business Case stated an 80% uptake was achievable. These latter figures were based on the notion that, wherever feasible, the Online Booking Tool would be mandated over the Travel Call Centre. The target has been achieved by some departments. For instance, PWGSC operates with an adoption rate above the 80% threshold at 82%.

54. Collectively, the Online Booking Tool and the Travel Call Centre achieved virtually 100% uptake for air and rail bookings. However, the program was unable to achieve the same rate of uptake for hotel bookings and car rentals. Many travellers preferred to book hotels and rent cars directly with the suppliers, especially for trips that do not involve air or rail, such as day trips. These travellers did not perceive a reason why they should incur the fee associated with the booking tools (please refer to Appendix B for applicable fees). Even when booking air or rail travel through the STSI tools, some travellers still preferred to personally call hotels and car rental companies directly.

55. The uptake of STSI booking tools for hotel and car rentals could not be assessed since the number of overall bookings made directly by the traveller or the arranger was not known. Without the Expense Management Tool or another automated mechanism to collect hotel and car rental data, there was no practical means to compile the booking data of individual travellers on a department or government-wide basis.

56. Designated Travel Card^{Footnote 4} - From 2005/06 to 2009/10, travel expenses paid for through the Designated Travel Card increased from $200 million to $236 million. According to the program's quarterly reports, this amount is "sub-optimized." In 2006, it was predicted that $680 million worth of expenditures was achievable. At the time of the evaluation, Designated Travel Card uptake was 35%. According to the program, the 90% uptake target was established based on an expected amendment to the National Joint Council Travel Directive. However, the required amendment did not occur (please refer to the Policy requirements section of the report for further details).

57. Expense Management Tool - The Expense Management Tool has been under utilized and has experienced slower than expected uptake. The Expense Management Tool was to be the primary source of consolidated travel expense information for the government, providing oversight and business intelligence. In 2009/10, $64.5 million worth of expenditures was recorded by the Expense Management Tool. Of the $1.59 billion travel expenditures of the federal government, the Expense Management Tool provided 4% worth of coverage. In comparison, the 2003 Business Case assumed that 100% uptake was achievable. When the target was established, it was expected that the mandatory status of the service within the Common Services Policy would make the uptake level attainable.

58. In response to the low uptake, starting 2009/10 the government approved a "modular." approach with the Expense Management Tool as prescribed in the 2008 mid-term review (please refer to the background section of the report). This approach has effectively made the service optional despite the Common Services Policy and has made the likelihood of attaining 100% adoption low.

59. The uptake of STSI tools was mixed. The notable success of the program was that virtually all air and rail travel was booked through the program, either through the Online Booking Tool or the Travel Call Centre. Overall, uptake of STSI tools has not reached the expected levels presented in the 2003 STSI Business Case and as a result, has limited the program from fully achieving the program's other outcomes as defined in its logic model.

Uptake - Policy requirements

60. There are two aspects to STSI uptake: 1) whether the department or agency has implemented the tool, and 2) the actual uptake of the STSI tool by employees within the department or agency.

61. The ExpenseManagement Tool has experienced low uptake, with 25 federal institutions having implemented the tool. This despite the fact that the Common Services Policy identifies STSI as mandatory, including an "expense claims system".

62. According to the program, there is a definite lack of compliance by a number of departments, and enforcement actions have not been employed. TBS states that adherence to the policy resides with deputy heads. For instance, in 2007/08 PWGSC was one of the first departments to enforce compliance with the Common Services Policy and as of 2009/10, they were the highest ranked department in terms of usage. However most other departments have not followed suit, and in response to the lack of compliance in 2009/10 the government approved an optional approach with the Expense Management Tool. Studies demonstrate that private sector organizations that make their travel solutions mandatory and enforce compliance experience much higher adoption rates.

63. Section 3.3 of the Treasury Board Directive on Travel Cards and Travellers Cheques states that the Designated Travel Card is the recommended method of payment to be used by employees for authorized government travel expenses. However, section 1.6.1 of the 2008 National Joint Council Travel Directive states that the use of the Designated Travel Card is at the discretion of the travelling employee. Since the directive was negotiated between the federal government and public service bargaining agents, it is adhered to more closely than the Common Services Policy. As it stands, the Designated Travel Card is only mandatory for Executive Groups, Executive Equivalents, and Excluded Groups as per section 6.1.6 of the Directive on Travel Cards and Travellers Cheques.

64. As for STSI's booking components (the Online Booking Tool and the Travel Call Center), senior managers in most departments mandate the use of either STSI booking tool in accordance with the Common Services Policy. However, senior managers often do not promote the use of the Online Booking Tool over the Travel Call Centre where practical. The authority to mandate travellers to use one booking tool over another does not rest with STSI, although the program does promote the Online Booking Tool over the Travel Call Center.

65. Over the course of the evaluation, PWGSC was an example of a department that mandated the Online Booking Tool (as well as the Expense Management Tool). As such, PWGSC was the top ranked department with respect to Online Booking Tool adoption, recording 82% in the third quarter of 2009/10 and achieving the expected uptake for the service. The second and third ranked departments were the Canadian Food Inspection Agency with 74% uptake and Citizenship and Immigration Canada with 71% uptake.

66. Another issue is the policy environment within the federal government. Adherence to security and privacy requirements has made it difficult for STSI to innovate at the same rate as its private sector counterparts. For instance, over the scope of the evaluation the Online Booking Tool could not be accessed by the internet (it could only be accessed through the governments secured intranet network) and could not provide all available air and rail routes.^{Footnote 5} The lack of internet availability made it difficult for federal employees to use the service when on travel status. For the Expense Management Tool, the National Joint Council Travel Directive requires physical receipts, preventing the service from being fully electronic. The program has made some progress addressing these barriers, such as obtaining an exemption from original written signatures for the Expense Management Tool, which has further streamlined the expense claims.

67. By design, the Expense Management Tool has built-in policy compliance, which is one of the reasons clients regard the tool as non-user friendly as it must navigate through the policies. However, by the same token, some clients have indicated that the built-in compliance makes expensing claims much simpler compared to the manual approach. For instance, Department of Fisheries and Oceans Canada submitted the Expense Management Tool as a case study to the government's Web of Rules initiative, as an example of a solution that encourages efficiency and reduces red-tape.

68. The policy environment that STSI operates within is not conducive towards achieving the desired uptake levels. Without a valid mandatory impetus backing the services, it is questionable whether the program can meet its expected levels of uptake.

Uptake - External and internal barriers

69. Aside from policy conflicts, the evaluation has identified other barriers that have contributed to the current uptake of the program. These barriers can be categorized as either departmental or end-user barriers. Departmental barriers are reasons why senior departmental managers do not implement STSI for their department, while end-user barriers are reasons why the travellers do not use the program.

70. For the Expense Management Tool, some departments expressed concern over the transaction fee of the claim submissions. This was of particular concern to departments with high volumes of low-dollar travel expenses such as day trips. In some departments, day travellers' expense claims are paid out of petty cash or submitted using manual or electronic forms. This is a relatively simple, inexpensive process for reimbursing travellers. The Canada Revenue Agency is one such organization that requires a large number of day trips to fulfill their mandate. As a consequence, at the time of the evaluation the Agency was in the process of implementing its own Software Application Program (SAP) travel expense claims module so as not to incur a transaction fee. As for travellers, some were resistant to using the Expense Management Tool since they were comfortable with manually claiming expenses and did not view the tool as being user-friendly.

71. Another contributor to uptake may have been the launch time of the Expense Management Tool. When STSI was first offered in 2004, the Expense Management Tool was not yet available, making the program not truly "end-to-end" as it was intended. Clients waited until 2006 for the component to be available. The program believes delivering the STSI service in this fragmented manner may have contributed to the lack of uptake for the Expense Management Tool.

72. In light of the Expense Management Tool's low adoption across the government, and despite its mandatory status as per the Common Services Policy, for the 2009/10 fiscal STSI was pursuing an optional approach with the tool as recommended by the 2008 mid term report. Without the mandatory imperative to adopt the tool departments were unwilling to commit to the Expense Management Tool and were instead awaiting the new expense claims system that will emerge from the 2013 contract. Meanwhile, other departments utilizing the SAP departmental financial management system indicated they would delay adoption until the results of the Canada Revenue Agency tool were known. At the time of this evaluation, 36 government organizations have adopted SAP as their financial management system, representing approximately 85% of government spending, although none have yet implemented the travel module. Interviewees indicated that it was unknown if departments who do not use SAP could implement an SAP travel expenses claims module to record their expenditures or if the module could have the connectivity that the STSI components share. These two draw backs make the SAP solution an impracticable option for the whole of government. Additionally, an SAP module would need to be purchased, adapted and maintained by participating departments. Implementing an SAP module would result in duplication work and would add an additional parallel system among the current portfolio of Expense Management Tool equivalents.

73. For the Designated Travel Card, the primary reason that represented travellers did not use the card was that they preferred to use their own credit card and collect personal benefits through business travel. Additionally, some travelers perceived that they did not travel enough to warrant acquiring a travel card. A client organization interviewed pointed out that departments may be hesitant to promote the card since it involves active management of the travel cards within the organization, as well as increasing departmental risk through outstanding balances.

74. Barriers preventing further uptake of the STSI travel tools are well known to STSI and have been extensively documented. The annual survey, the 2008 Mid-term Review and the Travel Advisory Board meetings have captured the on-going barriers to departmental uptake. The array of departmental and end-user barriers are summarized in Exhibit 3.

Table Summary

This table represents the annual survey, the 2008 Mid-term Review and the Travel Advisory Board meetings have captured the on-going barriers to departmental uptake. The array of departmental and end-user barriers have been summarized in this table.

Exhibit 3: Summary of STSI Performance Barriers

STSI Components	Departmental Barriers	End-User Barriers
Online Booking Tool	Not all booking content was available on the Online Booking Tool Lack of accessibility (not available via internet) Does not cater to unique departmental needs	Preference to speak with a Travel Counsellor - resistance to change
Designated Travel Card	Requires active management of travel cards Increases departmental risk through outstanding balances	Preference for their personal credit card
Expense Management Tool	Some departments would prefer to use an SAP travel module Requires a user fee per claim - perceived fiscal concern by some departments Requires department-wide implementation - resistance to change	Not user friendly Preference for manual claims Lack of departmental support

Enhancement of the travel experience

75. The travel experience refers to the end-to-end process of travel, from planning a trip to the final phase of claiming expenses. The program's original business case states that prior to STSI "the travel program, policy, and processes no longer provide employees with services that are up to date, efficient and that represent best business practices."

76. The contractor's survey results found that user satisfaction was generally high. Exhibit 4 illustrates the percentage of respondents who were satisfied, very satisfied, or extremely satisfied with the components of the program.

Table Summary

A table illustrates the percentage of respondents who were satisfied, very satisfied, or extremely satisfied with the components of the program.

Exhibit 4: Overall satisfaction regarding the components of STSI
(among travellers, travel arrangers, account verifiers, and travel approvers)

(Percentage)	Jan. '05	Aug. '05	Aug. '06	Oct. '07	Nov. '08	Nov. '09
Designated Travel Card	89	93	94	93	94	95
Travel Call Centre	80	86	93	92	93	94
Portal	-	86	80	74	79	77
Online Booking Tool	-	83	81	77	83	82
Expense Management Tool	-	-	-	41	39	54
Overall Solution	-	-	88	79	84	82

Source: Annual service provider survey, 2005/06 to 2009/10

77. Of clients who use the service, only the Expense Management Tool had a satisfactory rating of less than 60%. The main driver of dissatisfaction was that the service was not perceived as user-friendly.

78. Most client departments interviewed for the evaluation were of the opinion that STSI has enhanced the travel experience for employees. Departments noted the value of a standardized, streamlined e-service for end-users across the government. In departments that have adopted the Expense Management Tool, travellers valued the quicker reimbursement times. Some noted that, for infrequent travellers, the learning curve was high, especially for the Expense Management Tool.

79. The tools and technology of the program have enhanced the travel experience of employees for the better. The major barrier towards greater satisfaction is the learning curve.

Administrative and direct travel costs

80. By streamlining and automating the administration of travel, the implementation of STSI was to have generated cost savings by reducing the time required to plan, book, and report on travel. When the program was first implemented, STSI projected an increase in total travel expenditures of 5% per year, resulting in an estimated total travel spend of $18.3 billion over the nine-year period of the STSI contract if STSI was not implemented. The program estimated that over the nine-year period, the end-to-end solution would reduce overall travel spend by $811 million net of program costs. Of the total savings, the largest portion was an estimated $362.6 million through soft benefits (i.e. time savings). The remaining savings were expected from reductions in direct travel costs ($236.3 million), supplier discounts ($112 million), and administrative resources ($45 million) and other various savings. It is important to note that these savings exclude from consideration the role played by STSI as the point of access to PWGSC's negotiated travel rates. As the provider of travel bookings services, STSI is the sole mechanism that travellers have in order to access the discount agreements negotiated by PWGSC. The discount percentages and STSI's role in enabling negotiations are discussed later in the report.

81. At the time of this evaluation, there was insufficient evidence to quantify any administrative savings. The proposed savings were thought to emerge from the Expense Management Tool, given that automating expense claims would be more time efficient than the manual approach. According to the program's survey, 40% of respondents who have used the component believed that the Expense Management Tool was quicker for inputting expense claims than the manual process.

82. The client departments interviewed who use and support the Expense Management Tool agreed that automating the travel process had decreased the administrative burden. These clients stated that once the learning curve was overcome, the end-to-end solution was more efficient than the legacy process. However, departments were unable to determine whether the soft savings of automation offset the Expense Management Tool fee.

83. A number of departments and agencies that have adopted the Expense Management Tool indicated that they are experiencing administrative savings. Natural Resources Canada made the Expense Management Tool mandatory in February 2008, and integrated it into the department's travel processes. Natural Resources Canada reported to STSI that, prior to implementation, travel expense management was paper-based, inconsistent, and had a 50% error rate. Once the expense tool was operational, their error rate fell to 13%. Natural Resources Canada also reported that the Expense Management Tool has: improved the employee claims experience; minimized planning, approving and processing workload; and reduced its end-to-end travel management expenses.

84. Similarly, the Canadian Air Transport Security Authority (CATSA) identified some benefits of implementing STSI. With the Expense Management Tool, reimbursement time decreased to approximately 15 days, down from 30. The STSI solution also allowed CATSA to restructure its travel management process and streamline the processing of travel requests and expense claims. CATSA also claimed that different types of errors are flagged immediately with users correcting their own mistakes. CATSA is not one of the federal organizations identified as having to comply with the Common Services Policy. The department willingly implemented STSI based on the assessed value of the program's services.

85. The Canada School of Public Service also reported benefits. At the School, the Expense Management Tool reduced the paper burden which resulted in less chance of lost claims, less time spent processing, and faster reimbursement to travellers.

86. For direct travel costs, STSI was to have generated cost savings by allowing the government to better control and manage travel expenditures. The STSI 2003 Business Case estimated direct travel savings of $236.3 million over nine years. However, the estimate was based on key assumptions that have not been realised. Due to a lack of sufficient data, the evaluation was not able to quantify direct savings.

87. STSI places an emphasis on behavioural change by providing departments with intelligence on their air and rail travel practices. For instance, the program tracks Online Booking Tool adoption (since the fee is lower than the Travel Call Center), lowest airfare selected, airfare purchased in advance of ten days, and business class usage. If a department chose to improve practices based on this intelligence, it would result in savings for the department. It is also important to note that STSI is the only mechanism that travellers have to access the discounted travel rates negotiated by PWGSC. The discount percentages are further discussed later in the report.

88. Over the scope of the evaluation, the increased adoption of the Online Booking Tool has resulted in direct savings for departments. From 2005/06 to 2009/10, the increase of the tool's adoption from 19% to 43% has realized quantifiable savings of $5 million. As for the other indicators, they have more or less remained stable over this timeframe. If a department managed its travel using the intelligence generated by the program, this would have resulted in cost savings directly relating to the program. However, these savings are not readily quantified or tracked.

89. The client departments interviewed indicated that there was no evidence to determine if the program had successfully lowered direct travel costs. Clients mentioned other factors that have influenced government spending, such as the cap on travel expenditures introduced in the 2010 Budget. Client departments also commented that they no longer receive card or travel rebates from airline agreements. Prior to the program, these rebates would have been given to the department but are now transferred directly to STSI, as part of the program's cost recovery funding model. In conjunction with the STSI surcharge, clients perceived that they had lost previous benefits.

90. Taken as whole, departments that have implemented all the automated STSI components, including the Expense Management Tool, have claimed cost reductions, while other departments were unsure of the program's financial benefits. STSI enables departments, especially for air travel, to manage their travel expenditures more effectively. However, effective travel management resulting in cost savings are the responsibility of departments and cost avoidances are difficult to quantify.

Accountability, transparency, and oversight for air travel

91. Before STSI, ensuring travel policy compliance was a manual process reliant on an individual verifier's and approver's interpretation and enforcement. Without a centralized system to capture travel intelligence, departmental senior managers lacked timely and reliable information to hold travellers and managers accountable for travel management to the government and the Canadian public. With built-in electronic travel policy compliance and expense information fed centrally to STSI, the Expense Management Tool was intended to be the primary mechanism to address these concerns. Additionally, this information would enable STSI to more effectively manage the travel commodity and fulfill its intended oversight responsibilities.

92. In 2009/10, the Expense Management Tool only claimed $64.5 million worth of expenditures. However, the Online Booking Tool and the Travel Call Center also provide oversight, since virtually all air and rail booking information is captured centrally by the program. In 2009/10, air and rail expenditures totalled $387 million across the federal government. By adding the two data sources together (and removing the overlap of same expenditures captured by both services), STSI provided oversight of 27% of all federal government travel expenditures.^{Footnote 6} For expenses excluding airfare, such as accommodations and car travel expenditures (such as car rentals and personal vehicles), STSI has not captured a significant amount of expenses. Without having a clear understanding of these expenditures, the federal government cannot manage its travel practices in the most cost effective manner. For instance, without specific data, it is not clear whether using a personal vehicle, renting a car, or implementing a departmental car fleet is the most cost-effective option for managing car travel expenditures.

93. The client departments interviewed generally believed that the program has increased accountability, transparency, and oversight of travel within their departments. Clients, whether users of the Expense Management Tool or not, value the travel intelligence provided by the program.

94. Through the program, the federal government can now effectively account for almost one third of its travel practices (and almost 100% of its air/rail practices). If the federal government could expense more claims through the Expense Management Tool (or an equivalent), then federal organizations would be better positioned to account to the Canadian public the cost of government travel and STSI would be better equipped to manage the travel commodity.

Economy

95. Economy refers to the extent to which there is correlation between resources used and outcomes achieved. A program has high demonstrable economy when there is high correlation between minimum resources used and outcomes achieved.

The departmental surcharge and STSI cost recovery

96. The program operates on a cost recovery basis with revenue from travel card rebates, departmental surcharges, airline and railway cash rebates, and booking commission reimbursements.

97. The departmental surcharge was conceived as a short-term funding mechanism for STSI. When approved in 2003, the program was granted authority to administer departmental surcharges to clients per air/rail booking to help finance program start-up costs. The authority was scheduled to expire at the end of 2006/07, after which program costs were expected to be covered by the other revenue sources. However, with a lower than expected uptake, the revenue necessary to cover the costs of the program was not recovered.^{Footnote 7}

98. In response, PWGSC's authority to administer surcharges was extended in 2007 and again in 2009 to cover any revenue shortfalls. As of 2010/11, STSI retained authority to continue to use the client surcharge as a source of funding through to 2012/13. At the time of the evaluation, the surcharge was $10 per air or rail transaction.

99. From 2007/08 to 2009/10, departmental surcharges have accounted for 41% ($15 million) of STSI operating revenues. The surcharge has not been used as a short term solution, but as a principal source of ongoing program revenue. As an additional cost to clients, the surcharge runs counter to the savings that the program was intended to achieve. In interviews, client departments noted their frustration over continuing to pay the surcharge, since they were of the understanding the surcharge would discontinue.

100. As for the other sources of revenue, the amounts recouped by the program fluctuate dependent on travel volume. These revenues are received on a monthly basis while the travel surcharge is administered twice annually. The surcharge billed to clients varies according to the shortfall of the other revenue sources. The varying surcharge makes it unpredictable for clients. For instance, in interviews clients indicated that the varying semi-annual surcharge was arduous to plan against.

101. If the program had achieved its critical uptake as assumed in the 2003 Business Case, other sources of revenue would be sufficient for cost recovery and the surcharge would no longer be required. In 2009/10, travel card rebates generated $1.8 million worth of revenue for the program^{Footnote 8}. If travel card usage across the government was higher, it could potentially eliminate the need for the surcharge. It was noted by the service provider of the travel cards that if $1 billion worth of expenditures was charged to the cards, this volume would generate approximately $4.1 million in revenue.

102. Another issue concerning revenue involves the negotiated rebates. The negotiated airline and rail agreements rebates have decreased from a 4% rebate to 1.75%. This was done in pursuit of more competitive discount rates. Although the air and rail spend increased over the time period of the evaluation, total rebates have decreased. For instance, in 2005/06 rebates totalled $5.2 million, while in 2009/10, rebates fell to $3.5 million. The revenue sources of the program, from 2005/06 to 2009/10, are detailed below in Exhibit 5.

Exhibit 5: Revenue sources of program from 2005/06 to 2009/10

Source: Figures provided to the evaluation team by the program

Text description of Exhibit 5: Revenue sources of program is available on a separate page.

103. Lastly, low uptake has under some circumstances increased costs. The terms and conditions of the contract with the service provider stipulate that STSI must compensate the provider if the uptake of the Expense Management Tool fails to achieve a pre-defined threshold. As part of pursuing an optional approach with the Expense Management Tool, the service contract with the supplier was amended for 2009/10. To compensate the provider for the removal of exclusivity, PWGSC agreed to pay a $4 million one-time payment and an additional $1 million per year until the end of the initial 7-year contract. For the contract extension period (2011/12 to 2012/13), the contract amendment stipulates a $1.56 million yearly payment to the primary contractor if the monthly expense claim rate does not reach 10,000 per month.

104. As such, in 2009/10, STSI paid $1.56 million. This fee is a component of the program's operating expenditures and must be recovered through the revenue sources of the program, including the surcharge.

105. The viability of the program's funding model was based on the prediction that the program could achieve an expected level of uptake. As discussed previously in the report, the environment in which STSI operates in makes it unlikely that the levels can be achieved. As a consequence, the program is unable to generate the sufficient revenues necessary to recover costs without the surcharge. However, the surcharge was originally intended to be a short-term solution and as is there are some limitations to how it is administered, which has caused some concern with clients.

Discount agreements and STSI

106. The program promoted that a centralized travel solution would allow for the collection of the travel intelligence necessary for improved discount agreements. The Expense Management Tool facilitates commodity management activities by capturing all travel expenses in granular detail. According to the 2003 Business Case, this information would be fundamental for trend analysis, supporting fact-based decision-making, and negotiating optimal rates.

107. According to the program, although information captured was being used to an extent, additional information from the Expense Management Tool would be advantageous towards negotiating discount agreements with travel suppliers. However, information on air and rail travel was also being collected through the Online Booking Tool and the Travel Call Centre. With this information, STSI was able to collect the average air ticket price, the most popular domestic routes, business class tickets, and the most popular airlines.

108. For the scope of the evaluation, the competitive bidding process conducted by the Traffic Management Directorate determined the negotiated agreements. Negotiations with airlines are a cooperative process where either party may walk away within 60 days, depending on market conditions. In that sense, it is the supply market that determines the negotiated rates. STSI assisted negotiations through its emphasis on lowest air fare selection. STSI's operating method, where travellers are enticed to choose the lowest airfare, pressures airlines to offer the lowest rates when bidding on discount agreements. The expenditure information captured by the program provided PWGSC with understanding of its expenditures, but did not directly lower rates through leverage of information.

109. As of 2009/10, the 15 airline discount agreements covered 80% of flying volume, with discounts ranging from 10% to 50%, for a weighted average of 12.5%. According to the 2008 mid-term review commissioned by PWGSC and TBS, discount rates were within industry benchmarks.

110. Guaranteeing business volume could produce steeper discounts. In such an agreement, the federal government would guarantee a certain volume of purchases from qualified suppliers. Business intelligence would play an important role in such contract negotiations, since the government would require a complete understanding of travel expenditures in order to commit to guarantees. If the necessary volume is not achieved, the supplier could have the right to cancel the discount or administer a penalty. As such, because of the risk with such an approach, knowledge of travel volumes would be essential. Within the federal government, the guaranteed business volume strategy could be applied to airline agreements, but not to hotel bookings or car travel expenditures as the program lacks sufficient data given the low uptake of the Expense Management Tool.

111. Further uptake of the Expense Management Tool could provide the federal government with the necessary data for better strategic negotiations. However, the evaluation team observed that it was the competitive market that was the primary driver of discount agreements, and not necessarily the granular information captured by the program.

Service delivery model

112. The service delivery model of a program is strongly correlated with the economy of the program. When alternative program delivery methods exist, it is necessary to use the delivery method that maximizes the use of resources. For instance, a delivery model that is aligned with best practices would demonstrate an optimal use of resources. Upon review, the service delivery model of STSI was aligned with best practices.

113. As previously mentioned, the service components of STSI have been outsourced to the private sector while PWGSC provides the stewardship of the contract and the procurement function. When the contract was launched in 2004, it resulted in an amalgamation of multiple companies uniquely formed to develop and deliver the full range of required program tools and services. Meanwhile, PWGSC retained the procurement aspect of negotiating rates within PWGSC's Traffic Management Directorate.

114. According to a 2007 survey conducted by a research group who specializes in corporate and government best practices, outsourcing travel tools while retaining procurement responsibility was in line with best practices. The survey indicated that 92% of companies surveyed outsourced their online booking, and 87% negotiated their own annual rate negotiations. The survey concluded that companies were more inclined to outsource transactional activities while maintaining close control over procurement. This was substantiated with a peer group study, which concluded that organizations prefer to deal directly with the airline operators when negotiating agreements. Six out of eight respondents reported better discounts on airfares when negotiating with airlines directly instead of dealing indirectly through agencies.

115. With the federal government having the largest travel account in Canada, it is appropriate for it to procure its own rates. Outsourcing travel services to the private sector is also appropriate and is in line with the best practices in other jurisdiction (both in industry and other governments). Overall, during the scope of the evaluation the program was using an optimal service delivery model.

Efficiency

116. Efficiency refers to the extent to which the resources are used in relation to outputs. A program has high demonstrable efficiency when resources maximize outputs at least cost.

Efficiency ratio (outputs over costs)

117. STSI is intended to have oversight responsibility over the federal travel commodity. To this end, the program was granted funding authority in the amount of $9.7 million for 2009/10 (and $9.5 million on-going until the end of the current contract in 2013). However, the expenditures of the program have not always been constant, especially in the earlier years when STSI had high implementation costs.

118. Considering the program's intended oversight responsibility over federal travel, an efficiency ratio can be established between the total commodity that passes through the program's outsourced components over the cost required to manage the commodity. This ratio demonstrates how efficient the program is at managing government travel expenditures over time. Ideally, for every dollar spent by the program, the amount of commodity that passes through the program would increase on a yearly basis, demonstrating an increase in efficiency.

119. In 2005/06, STSI oversaw $576 million in travel dollars with a budget of $25.3 million, an efficiency ratio of 23:1. In 2008/09, $751 million in travel expenditures passed through the components, with a budget of $9.5 million, an efficiency ratio of 79:1^{Footnote 9}. To place the figures into perspective, if the program captured all of the federal government's travel expenditures ($1.6 billion), the efficiency ratio would be 173:1. Exhibit 6 demonstrates the program's efficiency ratio from 2005/06 to 2009/10.

Exhibit 6: Efficiency ratio between travel expenditures managed by the program and the operating expenditures of STSI

Source: Ratio calculations performed by the Office of Audit and Evaluation

Text description of Exhibit 6 is available on a separate page.

120. If uptake increases and more travel expenditures pass through STSI, the efficiency ratio will rise. The program can also increase the ratio by decreasing program costs. According to the program, half of the program's costs are allocated to various fees unrelated to the day-to-day managing of the contract. For instance, as previously mentioned, the compensatory fees for the Expense Management Tool when uptake is below the agreed upon threshold. STSI is also heavily reliant on contractors for daily operations. In 2009/10, 82% of the program's operating expenditures were from the per diem expenses of professional services.

121. According to the analysis, the program has demonstrated increasing efficiency as more travel passes through the program. Greater efficiencies are possible by further increasing managed travel and decreasing program costs.

Potential savings of the Online Booking Tool

122. When STSI was first implemented, direct savings to departments were expected from the Online Booking Tool, whose fees would be considerably less than booking through the Travel Call Center. In 2009/10, the cost for booking online was $20.79 compared to $46.26 for the call center. However, more than half (57%) of domestic and trans-border flights were booked through the call center in 2009/10.

123. Higher uptake for the Online Booking Tool from 2005/06 to 2009/10 would have resulted in significant savings for the government and individual departments. If all eligible trips were booked online, $23 million in savings could have been realized from booking fees alone. However, 100% uptake is unrealistic and is not aligned with any available benchmark studies. The 2003 STSI Business Case indicated that an 80% uptake of the Online Booking Tool could be achieved, which would have resulted in $16 million in savings. As previously mentioned, a corporate benchmarking study showed an average online adoption rate of 73%. This type of adoption would have resulted in $14 million worth of savings. The variance analysis is summarized in Exhibit 7.

Exhibit 7: Variance analysis of booking fees with regards to different uptake scenarios for the Online Booking Tool from 2005/06 to 2009/10 (in millions)

Source: Variance calculations performed by the Office of Audit and Evaluation

Text description of Exhibit 7 is available on a separate page.

124. The variance analysis demonstrates that there are significant savings yet to be realized. As previously mentioned, PWGSC operates with an adoption rate above the 80% threshold at 82%, demonstrating the viability of the service and the achievability of the savings. Moving forward, this continuing upward adoption trend will result in increased direct savings for federal organizations.

Conclusions: Performance

125. For outcome achievement, one of the primary indicators for a service provider is uptake. For STSI, the outcome results for uptake were mixed. Over the scope of the evaluation, virtually all air and rail travel was booked through the program, yet uptake across all of the services has not met the initial expectations set out by the program. There are a variety of factors, both internal and external to the program, that have contributed to the current uptake. These include: lack of strong policy, resistance to change, perceived fiscal concerns and lack of user-friendliness of the components. The initial uptake targets were established based on certain assumptions, such as an enforced mandatory status. With the Designated Travel Card and Expense Management Tool effectively being optional despite the Common Services Policy, the likelihood of achieving their uptake targets is low. As it stands, the uptake levels are achievable, but it requires a strong commitment from deputy heads, as seen with Public Works and Government Services Canada among others.

126. The program has enhanced the travel experience of employees. Satisfaction levels for the components were high. The major barrier to greater satisfaction was the learning curve.

127. The program has increased the transparency of travel practices within the federal government and can now effectively account for approximately a third of its travel expenditures. To increase this figure, further efforts are required to capture accommodation expenses and car travel expenditures. As it stands, with approximately two thirds of government travel not captured by the program, there are areas where the government lacks the business intelligence necessary to effectively account for and manage its travel.

128. Overall, there is still opportunity for greater outcome achievement. Although much has been accomplished since the implementation of the program, achievement of outcomes is heavily dependent on program uptake.

129. Over the review period of this evaluation, the program's efficiency was increasing. The potential savings of the Online Booking Tool are yet to be fully realised and the uptake accomplished by PWGSC is proof that the savings are achievable.

130. The viability of STSI's funding model was based on uptake levels that have not come to pass. As a consequence, the program has had to administer a surcharge. As this surcharge was originally intended to be a short-term solution, there are some limitations to how it is administered, which has caused some concern with clients.

131. The delivery model of outsourcing travel services to the private sector and retaining the procurement function was aligned with the best practices of other jurisdiction both in industry and other governments.

General Conclusions

132. Relevance: STSI addresses a demonstrable need, is aligned with government priorities, and is consistent with federal roles and responsibilities. While the program was not legislated, PWGSC is mandated to provide the service and applicable departments are mandated to use the service in accordance with the Treasury Board Common Services Policy. Despite its mandatory status, some departments have not implemented all of the components, especially the Expense Management Tool. For departments that have not implemented the Expense Management Tool or an equivalent system, the continuing practice of manual claims submissions is not aligned with best practices.

133. Performance: Uptake of STSI tools varied and have not reached the expected levels presented in the 2003 STSI Business Case. However, STSI operates in a policy environment that is not conducive to achieving expected uptake levels. The lack of uptake is not in spirit with the Treasury Board Common Services Policy and is not conducive towards the benefits of economies of scale. Without the necessary uptake levels, it will be difficult for the program to achieve the rest of its outcomes. With that said, virtually all air and rail travel is booked through the program.

134. The potential benefits of the program, and the business intelligence it captures, will only be fully understood with increasing departmental uptake. With approximately two thirds of government travel not captured by the program, there are areas where the government lacks the business intelligence necessary to effectively account and manage its travel, especially for accommodation and car travel expenditures. Capturing this information will further enable STSI to more effectively provide oversight and manage the travel commodity.

135. The low uptake has had financial consequences, such as the surcharge administered to fund the program. Originally intended to be a short-term solution, the current funding model of the program has caused some concern with clients. Moving forward, further savings are attainable. For instance, the increasing uptake of the Online Booking Tool will result in further cost savings for the federal government. However, until uptake increases, the program will not be able to capitalize on the potential administrative savings that were initially expected. The service delivery model of the program was appropriate, as leveraging the private sector while retaining the procurement function was in line with best practices.

Management Response

The Shared Travel Services Initiative within the Acquisitions Branch accepts the evaluation findings and intends to act on the recommendations of the evaluation by implementing their Management Action Plan detailed as follows.

Recommendations and Management Action Plan - Acquisitions Branch

Recommendation 1: The Assistant Deputy Minister, Acquisitions Branch, should coordinate with the Treasury Board Secretariat to increase the use of the Expense Management Tool (or equivalent) within the federal government.

• Management Action Plan 1.1: The Assistant Deputy Minister, Acquisitions Branch, will propose a meeting with Treasury Board Secretariat representatives to address the challenges of implementing an electronic travel expense management tool in the Government of Canada. Items to be addressed will include:

  1. The issue of the current Common Services Policy

  2. The requirement for departments to implement some type of expense management tool

  3. The requirement for departments who elect not to use the Shared Travel Services expense management tool to provide detailed travel expense data to Public Works and Government Services Canada for commodity management purposes

  4. The Treasury Board Secretariat commitment to achieve the agreed-upon goals

  5. An implementation plan including milestones and outcome targets

• Management Action Plan 1.2: The Assistant Deputy Minister, Acquisitions Branch, will report back to the Deputy Minister, Public Works and Government Services Canada on the outcomes of the above meeting.

Recommendation 2: The Assistant Deputy Minister, Acquisitions Branch, should engage the Treasury Board Secretariat to resolve the oversight gap of accommodation and car travel expenditures.

• Management Action Plan 2.1: The Assistant Deputy Minister, Acquisitions Branch, will propose a meeting with Treasury Board Secretariat representatives to address the challenges of bookings made outside of the Shared Travel Services. Items to be addressed will include:

  1. Methods to ensure accommodation and car rental expenditures are properly captured

  2. Issues with the Common Services Policy

  3. The Treasury Board Secretariat commitment to achieve the agreed-upon goals

  4. An implementation plan including milestones and outcome targets

• Management Action Plan 2.2: The Assistant Deputy Minister, Acquisitions Branch, will report back to the Deputy Minister, Public Works and Government Services Canada on the outcomes of the above meeting.

Recommendation 3: The Assistant Deputy Minister, Acquisitions Branch, should work with the Treasury Board Secretariat to address the policy barriers to STSI uptake in order to achieve maximum value.

• Management Action Plan 3.1: The Assistant Deputy Minister, Acquisitions Branch, will propose a meeting with Treasury Board Secretariat representatives to address the challenges of policy issues that negatively impact Shared Travel Services uptake. Items to be addressed will include:

  1. The lack of adherence to the Common Services Policy

  2. The incongruity between the National Joint Council Travel Directive and the Policy on Travel Cards

  3. The direct costs faced by the program as a result of low uptake e.g. payments to contractors

  4. The Treasury Board Secretariat commitment to achieve the agreed-upon goals

  5. An implementation plan including milestones and outcome targets

• Management Action Plan 3.2: The Assistant Deputy Minister, Acquisitions Branch, will report back to the Deputy Minister, Public Works and Government Services Canada on the outcomes of the above meeting.

Recommendation 4: The Assistant Deputy Minister, Acquisitions Branch, should pursue a more sustainable, permanent funding model for STSI.

Management Action Plan 4.1: The Assistant Deputy Minister, Acquisitions Branch, will put forward a funding plan for Shared Travel Services that removes the requirement for surcharges and the dependency on air/rail rebates.

About the Evaluation

Authority

The Audit and Evaluation Committee of Public Works and Government Services Canada (PWGSC) approved this evaluation as part of the 2010 to 2015 Risk-Based Multi Year Audit and Evaluation Plan.

Evaluation Objectives

The evaluation examined the Shared Travel Services Initiative (STSI), within Acquisitions Branch. This evaluation had two objectives:

• To determine the relevance of the program: the continued need for the program; its alignment with governmental priorities and strategic outcomes; and its consistency with federal roles and responsibilities.
• To determine the performance of the program: the achievement of its expected outcomes; and a demonstration of the efficiency and economy of the program.

Approach

The evaluation was conducted in accordance with the Evaluation Standards of the Government of Canada and the Office of Audit and Evaluation of PWGSC. The evaluation took place between February and October 2010 and was conducted in three phases: planning, examination, and reporting. To assess the evaluation issues and questions, the following lines of evidence were used:

Document Review: An initial document review provided an understanding of the program and its context to assist in the planning phase. A number of documents were reviewed to assess the continued relevance of STSI with respect to departmental and government-wide priorities. They included policy, Speeches from the Throne, Auditor General Reports, PWGSC Reports on Plans and Priorities, and Departmental Performance Reports.

The evaluation team also reviewed a wide range of documents to gain insight into intended and actual results. These documents included quarterly operational reports, Departmental Performance Reports, financial reports and plans, budgetary allocations, business plans, and client satisfaction surveys.

Literature Review: A literature review was conducted to contextualize the program both nationally and internationally; provide theoretical background for the program model; provide baseline data against which the program could be assessed; and identify alternative delivery models through an analysis of other jurisdictions.

Interviews: The evaluation team conducted interviews with key program and PWGSC managers (n=4). In addition, the evaluation team conducted interviews with client departments and agencies (n=11), with Treasury Board of Canada Secretariat staff (n=5), and with the primary contractor and a subcontractor (n=2). The qualitative analysis of the interviews provided information about the program's activities, outputs, expected outcomes, stakeholders, relevance, and performance from the perspective of program managers, client departments, and other related stakeholders. Interview guides were used.

Survey: The program, through the primary contractor, annually surveys its users. The population of those surveyed include travellers, travel arrangers, account verifiers and travel approvers. The results of the surveys were incorporated where relevant.

Financial Analysis: Financial data related to the program's budgets, revenues and expenditures were reviewed and analyzed to understand the effectiveness of the cost recovery model of the program. Based on the document review, the review of financial files, and the literature review, the evaluation team analysed the efficiency of the program. All financial information was provided by the program with the exception of the total federal government travel expenditures, which was provided by another group within Acquisitions Branch not related to the program.

Performance Data Analysis: Performance data from the program was analysed in order to measure the performance of the program's outcomes. The evaluation team leveraged both internal performance reports and raw data provided by the program when appropriate.

Expert Opinion: The evaluation team relied on the key expert opinion of a representative from the Association of Corporate Travel Excutives. The organization is a not-for-profit association established to provide executive-level global education and networking. The association spans all of business travel, from corporate buyers to agencies to suppliers and serves more than 6,000 executives in over 80 countries.

Limitations of the Methodology

Document Review: Efforts were made to ensure that all data provided by the program were reviewed and documented. Due to the large volume of data and literature review, and the complex nature of the travel environment, the evaluation team prioritized documents based on the relevance, usefulness and comprehensiveness of the document. As a result some documents were relied on more heavily than others. However, every effort was made to systematically identify and categorize data from all documents. Data from documents was also verified with supporting evidence from other documents or research methodologies.

Literature Review: A wide range of literature was available on travel solutions dealing primarily with private sector organizations. Efforts were made to include relevant material from reliable sources to enable an unbiased and neutral assessment.

Interviews: The evaluation team interviewed senior full-time travel officers and Travel Advisory Board members in departments and agencies with respect to the relevance and performance of the program.Representatives from STSI contractors were interviewed. The evaluation team did not interview representatives from other suppliers of end-to-end travel services so as to not interfere with the upcoming Request for Proposal process.

Survey: The survey used by the evaluation team was conducted by the primary contractor of the components. In such a situation, there is a potential conflict of interest. To mitigate this risk, the evaluation team utilized the raw data from the survey. The survey was administered to travellers, travel arrangers, account verifiers, and travel approvers. As such, the survey did not reach the demographic of those who do not travel.

Data Analysis: To calculate uptake of the program's services, the evaluation team relied on the performance measurements provided by the program as well as a methodologies derived by the evaluation team.

For the uptake of the Online Booking Tool, uptake is calculated by all domestic and trans-border travel booked through the tool divided by all domestic and trans-border bookings booked through the Online Booking Tool and Travel Call Center. The calculation excludes bookings made by the Department of National Defence, since they currently only use the Travel Call Center. It also excludes international flights. This methodology was derived by the program. It was unclear whether the expected uptake percentage of 80% included or excluded the bookings made by departments who do not use the service; and whether it excluded international flights. Regardless, the evaluation team compared the data under the assumption that it did exclude departments who do not use the service and excluded international flights.

For uptake of the Designated Travel Card, the methodology in the report was derived by the evaluation team. The program calculated usage of the Designated Travel Card based on Expense Management Tool claims submitted by travelers. Due to the limited information available data from the Expense Management Tool, it lacked representation across the federal government. The team instead calculated uptake by the amount of expenditures charged through the travel card divided by the "optimal amount" as recommended by the Travel AcXess Voyage Quarterly reports. These are reports submitted by the travel card subcontractor to the program. This methodology was utilized since it provides a complete scope of the federal government. It also ties into the program's funding model, where a higher uptake is directly related to the program's viability as a cost recoverable organization. One limitation with the methodology is the comparison with the expected uptake of 90%. As the uptake is referred to as "adoption", an uptake methodology were the number of travelers using the card divided by the total number of travelers would be more appropriate. However, at the time of the evaluation, such data was not actively tracked.

For uptake of the Expense Management Tool, the methodology in the report was derived by the evaluation team. The program calculated Expense Management Tool usage by the number of expense claims divided by number of air/rail trips for the equivalent time period. Due to the limitations of the denominator, there are fewer air/rail trips than expenses claimed. As such, the methodology did not provide an overview of uptake across the federal government and the evaluation team chose not use the data. The team instead calculated uptake by the amount of expenses recorded within the system divided by the total travel commodity within the federal government. This methodology was utilized since it provides a complete scope of the federal government. One limitation with the methodology is the comparison with the expected uptake of 100%. As the uptake is referred to as "usage", an uptake methodology were the number of Expense Management Tool claims divided by the total number of traveler claims would be more appropriate. However, at the time of the evaluation, the total number of traveler claims was not actively tracked.

Reporting

The evaluation team documented findings in a Director's Draft Report, which was internally reviewed by the Office of Audit and Evaluation's quality assessment function. The evaluation team provided the OPI Director General with the Director's Draft Report and a request to validate facts and comment on the report. A Chief Audit Executive's Draft Report was prepared and provided to the Assistant Deputy Minister, Acquisitions Branch, for acceptance as the Office of Primary Interest. The Office of Primary Interest was requested to respond with a Management Action Plan.

The Draft Final Report, including the Management Action Plan, will be presented to PWGSC's Audit and Evaluation Committee for the Deputy Minister's approval in March 2011. The Final Report will be submitted to the Treasury Board of Canada Secretariat and posted on the PWGSC website.

Project Team

The evaluation was conducted by employees of the Office of Audit and Evaluation, overseen by the Director of Evaluation and under the overall direction of the Deputy Chief Oversight Officer. The evaluation was reviewed by the quality assessment function of the Office of Audit and Evaluation.

Appendix A: Process Chart of STSI

Text description of Appendix A: Process Chart of STSI is available on a separate page.

Legend

OBT

Online Booking Tool

TAN

Travel Authorization Number

EMT

Expense Management Tool

DFMMS

Departmental Financial and Materiel Management System

Appendix B

Table Summary

This table represents the fees and revenues of STSI and the clients and departments that expense and receive them.

STSI and the Cost of Travel

Fees/ Charges paid to	Fees/Charges paid by
	STSI	Contractor and Subcontractors
Client Department	$10/booking (surcharge)	$20.79/On-line Booking $46.26/Travel Call Centre	An average $13/Expense claim submittedFootnote 10
Travel Card Provider	1.75% rebate from travel cards
Airlines/Railways	1.75% rebate from travel cards
Hotels and Car Rental Companies	Listing fee for Accommodations and Car Rental Directory Commissions paid when booked through booking tools

Appendix C

Table Summary

The table in Appendix C, the evaluation matrix outlines and summarizes the evaluation issues, questions, indicators and identifies the data sources that address the questions. The evaluation matrix is a tool that helps the evaluation team consider the most appropriate and feasible data collection method for each of the questions identified in the evaluation plan.

Evaluation Matrix

Evaluation Questions and Issues	Indicators	Methodology and Data sources
Relevance
1. To what extent does STSI continue to address a demonstrable need and is responsive to the needs of federal public servants, departments and Canadians?	Evidence on Demonstrable Needs Number of government travel trips & dollar expenditures on travel Rationale for introducing centralized travel management programs (government, other organizations), e.g., the need for greater accountability & transparency in travel Canadians' need for transparency and accountability in government spending Rationale for travel management as part of federal government's e-commerce approach Extent to which STSI has adapted in response to the changing needs of government and the changing environment of the travel industry	1. Literature Review Review literature to identify why other jurisdictions & large non-GC organizations implement modern travel management programs Library Internet Industry publications Pertinent reports commissioned for the program 2. Document Review & Data Analysis Review & content analysis of key GC & PWGSC documents to assess the rationale for STSI & its responsiveness TB mandates & policies TBS Submissions/Memorandum of Cabinet (MC) Speeches From the Throne (SFT) Note: 2010 is relevant Dept Performance Reviews (DPR) & Reports on Plans & Priorities (RPP) for PWGSC 3. Interviews Interviews with key stakeholders to obtain perspectives & evidence on demonstrable needs & responsiveness to need TBS, STSI, OGDs Director General (DG) Travel Advisory Board, Senior Project Advisory Committee (SPAC)
2. To what extent is STSI aligned with GC and PWGSC priorities and strategic outcomes?	Extent of alignment with the relevant: Throne Speeches (SFT) Points from 09-10 & 10-11 Min of Finance Budgets Acts / legislation TB Policies Government priorities on: generation of savings; reduction & avoidance of costs accountability transparency promotion of e-solutions shared services model PWGSC Strategic Outcomes	1. Document Review Review of key GC documents to assess the extent of STSI's consistency with Objectives, Outcomes & GC, PWGSC policies & priorities Program Activity Architecture (PAA) Strategic Outcomes Key policy documents: SFTs, Min Fin Budgets, DPR, RPPs, policy statements, Management Accountability Framework (MAF) TB Common Services Policy STSI Business Plan & related documents 2. Interviews Interviews with key stakeholders to assess alignment with priorities & strategic outcomes STSI, TBS DG Travel Advisory Board
3. To what extent are the roles and responsibilities of the STSI appropriate to the federal government?	High-level policy statements on shared services, travel modernization, ecommerce, etc. Evidence: of clearly defined, documented and communicated roles for PWGSC and other federal organizations, i.e., managing the travel service contract on behalf of government that lessons learned have been compiled and utilized (best practices) on what other jurisdictions do on private sector comparisons in terms of delivering travel services	1. Document Review Review of key GC documents to assess the extent that STSI's roles, responsibilities & structure are appropriate SFTs; DPR & RPPs STSI RACI model STSI Business Plans & TB Submissions Interis' 2008 reports on STSI's mid-term role; STSI's response & any submissions to TB 2. Literature Review Info on alternative delivery models and other jurisdictions 3. Interviews Interviews with key stakeholders on relation of STSI to federal roles & responsibilities STSI, TBS, PWGSC, OGDs DG Travel Advisory Board
PERFORMANCE -- Expected Outcomes (Effectiveness): To what extent is the STSI achieving its expected outcomes?
Immediate Outcome 1: Increased awareness of travel services and tools	Awareness: Level of general STSI awareness of end users over time Level of awareness of the individual components of STSI over time Measures of training activity by STSI & departments	1. Survey Review of results of sponsored survey 2. Interviews Interviews with key stakeholders on awareness and methods to improve awareness DG Travel Advisory Board, TBS, OGDs
Immediate Outcome 2: Increased uptake of travel tools by government departments and travellers	Number & rates of the use (uptake) of STSI's various tools and services (displayed, where possible, by department, tool, over time, type of employee & level of seniority, region, etc.) Adoption rate of Central payment vehicle for Air spend (5 yrs) Adoption rate of bookings going through Online Booking Tool (5 yrs) (by type of booking) Adoption rate of Expense Management Tool (EMT) (5 yrs) Government and STSI travel Spend (5yrd) (STSI Travel Estimate/GC Total Expenditures) Identifying barriers to uptake, and the resolution of such barriers Measure of employees' use of alternative travel tools Trends in use of travel tools in private sector & other jurisdictions (benchmarks)	1. Document Review Review of key GC/STSI documents to assess the extent of uptake by employees, travellers & departments; reasons for levels of uptake and barriers to uptake 2. Data Analysis Analysis of uptake data, including training STSI PMS data Data on complaints & STSI's responses STSI PMS data on training (if available, data by travellers, arrangers, managers & others) Financial Reports and Data 3. Literature Review Analysis of uptake data from private sector & other jurisdictions, including reasons for uptake Relevant industry or other government benchmarks to measure effectiveness of travel management tools 4. Interviews Interviews with key stakeholders on uptake & barriers to uptake DG Travel Advisory Board TBS OGD 5. Survey Review of results of sponsored survey
	Extent of buy-in to STSI by senior departmental managers Senior management statements or directives on use of STSI tools, services & reports by department & employees Departmental participation in SPAC & DG Travel Advisory Board	1. Document Review Review of key GC documents to assess the extent of acceptance & commitment by SFTOs & Senior Management Departmental travel policies 2. Interviews Interviews with key stakeholders to assess the extent their participation in the program and perceived barriers to uptake STSI, TBS DG Travel Advisory Board, OGDs, SFTOs
Intermediate Outcome 3: Enhanced travel experience for employees	Satisfaction & Complaints: Levels/rates of satisfaction with the suite of STSI tools for employees & OGDs Departments' & employees' error rates with STSI tools Number and type of complaints related to STSI & its tools Reimbursement time for expenses Type and volume of complaints about travel management systems in other jurisdictions & large organizations Evidence on Responsiveness to Needs OGDs' satisfaction with STSI services & tools OGDs' satisfaction with issue & complaint resolution Time to resolve issues STSI adaptable to changing client needs & industry standards	1. Literature Review Review literature to identify relevant models, benchmarks, experiences & trends in other jurisdictions & large organizations with modern travel programs: media reports, Internet, Library, Industry publications 2. Data Analysis Analysis of data on employee travel experience, satisfaction levels, complaints with STSI services & tools STSI PMS data 3. Document Review Review of STSI documents assessing the employee & OGD satisfaction with STSI services & tools 4. Interviews With key stakeholders & employees to gather data to assess travel experiences with STSI services & tools (see Immediate Outcome 4) DG Travel Advisory Board 5. Survey Review of results of  sponsored survey 6. Focus Group End users of STSI
Intermediate Outcome 4: Decreased administrative cost/burden and decreased travel costs	Admin costs/burden Administrative activities changed (reduced, increased, added) due to the use of STSI OGDs' estimated changes in admin costs (dollar savings) from using STSI & its various tools & services (positive or negative); variations over time (links to introduction of new tools) Best practices that have lowered and/or avoided admin costs Measures of the time & effort for OGDs & travellers to participate in the STSI system and in comparison with the alternatives available to them Travel Costs (hard savings achieved by STSI) Evidence of overall reductions in travel costs attributable to STSI (& net of travel volume changes)	1. Document Review Review of key GC documents to assess the extent of potential & actual savings 2. Data Analysis Average Discounts and rebates negotiated across Airline Agreements Accommodations Procurement Car Rental Procurement Financial Reports 3. Interviews Interviews with key stakeholders to gather perception of administrative cost/burden decreases & direct travel cost decreases due to the program STSI, OGDs DG Travel Advisory Board SPAC large organizations using centralized travel management services suppliers of end-to-end centralized travel management services Procurement Group 4. Survey Review of results of sponsored survey
Intermediate Outcome 5: Increased accountability, transparency and oversight of travel in the GC	Uptake of EMT adoption rate (5yrs) Measures of Departments that receive & use regular business intelligence reports from STSI Satisfaction levels with STSI reports Extent of use of STSI business intelligence Changes in OGD processes/procedures & their impact Feedback from OGDs' senior managers on overall satisfaction with STSI's tools & services, business intelligence reports & need for changes in the program Measures of use by senior managers of available travel cost reports (increases in use of reports over time & by department to show the link between Outputs & Outcomes) Sharing of best practices of use of business intelligence with OGDs	1. Literature Review Review literature to identify nature of relevant success stories, benchmarks, experiences & trends in other jurisdictions & large organizations with modern travel programs 2. Document Review Review of documents on improved management and stewardship of travel within OGDs Comparative analysis of benchmarks 3. Data Analysis EMT adoption rate (5 yrs) 4. Interviews Perception of EMT's contribution to the outcome Interviews with key stakeholders on improved management and oversight of travel within OGDs & on overall perspectives on STSI as a modern shared travel program DG Travel Advisory Board 5. Survey Review of results of sponsored survey
PERFORMANCE -- Efficiency and Economy: To what extent does the STSI demonstrate efficiency and economy?
Demonstrated Economy: Has the program operated in the most economical manner?	Ability to offer best travel rates [adaptation to price & other changes offered by STSI's competitors] Comparison with potential savings from alternative travel mechanisms used by other federal departments Comparison with savings from similar travel regimes in other jurisdictions & large organizations Comparison of STSI fees & charges with potential admin savings from similar travel regimes in other jurisdictions & large organizations	1. Data Analysis Analysis of data on administrative & travel costs compared to other methods of supplying travel services STSI PMS data Financial Reports 2. Literature Review Review of literature to identify relevant models, benchmarks, experiences & trends in other jurisdictions & large organizations with modern travel programs that can provide markers on admin & direct savings info on ideal systems & best practices; data from other jurisdictions &/or large organizations comparisons of STSI's tools & services with 'ideal' modern, integrated electronic travel program
Demonstrated Efficiency: Has the program operated in the most efficient manner?	Effectiveness of STSI's cost-recovery model (including fees & charges paid relative to STSI's costs & distribution of rebates to departments) Ratio of travel expenditures managed by the program to the budget of the program Client Perception of Value for Money for rates charged Benchmarks for travel rates from other jurisdictions Travel Cost Savings (hard savings achieved by STSI) Evidence of overall savings in travel costs attributable to STSI (& net of travel volume changes) Quantification of savings from negotiated travel arrangements, including rebates	1. Data Analysis STSI PMS Data Financial Reports Program Revenues Program Expenditures 2. Literature Review Review of literature to identify benchmarks, experiences & trends in other jurisdictions, large organizations with modern travel programs and the Canadian market that can provide comparative markers. 3. Interviews Interview with Stakeholders to discern if the program surcharges achieve value for money If there are methods to reduce cost of delivery for the program OGDs, STSI, DG Travel Advisory Board