2009-602 Evaluation of the IT Shared Services Telecommunications Line of Business (Final Report)

March 24, 2011

Table of Contents

• Main Points
• Introduction
• Program Profile
• Focus of the evaluation
• Findings and Conclusions
  • Relevance
    • Continuing Need
    • Alignment with federal and departmental priorities
    • Alignment with federal and departmental roles and responsibilities
  • Conclusions: Relevance
  • Performance
    • Outcome Achievement
      • Uptake and Defragmentation of Telecommunications Services
      • Quality of Services
        • Client Satisfaction
        • Service Reporting
      • Cost savings and reductions
      • Information Technology (IT) Security Risk Reduction
    • Efficiency and Economy
      • Benchmarking
      • Costing
      • Revolving Fund
      • Mandatory Service
      • Alternative Delivery
  • Conclusions: Performance
• General Conclusions
• Mamagement Response
• About the Evaluation
• Appendix A: Evaluation Matrix

Main Points

What was examined

1. The Public Works and Government Services Canada (PWGSC) Information Technology and Shared Services (ITSS) organization provides IT services to Government of Canada departments and agencies across four lines of business: Telecommunications Services, Data Center Services, Distributed Computing services, and IT Security Services. This report presents the results of the evaluation of the ITSS organization's Telecommunications Services line of business (the program) which offers a suite of voice and data telecommunications services, divided among data network infrastructure services and voice network services.

Why it is important

2. Telecommunications services provide core business telecommunications tools – voice and data services - to federal departments and agencies, in support of their program mandates. They achieve cost savings through aggregation of service requirements and government process improvements while providing a Government-wide perspective to address the rapidly evolving nature of the telecommunications industry, associated technologies, and their application in government.

3. PWGSC is legislated as a common services organization under the Department of Public Works and Government Services Act. The Treasury Board Common Services Policy holds PWGSC responsible for providing telecommunications services and infrastructure for federal departments and agencies, which includes planning and constructing informatics infrastructure to support central and other common services. The ITSS organization was created as a result of the 2005 restructuring of the Information Technology Services Branch to build service capacity through the establishment of an IT Centre of Excellence to deliver IT services government-wide.

What was found

4. Overall, the program remains relevant to the Department and the federal government. PWGSC is legislated to be a common services provider and is required under the Treasury Board Common Services Policy to provide optional IT services on request to other government departments. The evaluation found that there is a continuing need for the program, as evidenced by the continued demand and use by client departments. The program was found to be aligned with the priorities of the federal government and PWGSC. As the telecommunications service provider for federal organizations, the program was found to be fulfilling an appropriate role and responsibility for the government.

5. In terms of performance, the program has made considerable progress towards achieving its intended outcomes. The evaluation found that the uptake of the program's optional services was increasing, thereby reducing the fragmentation of telecommunications infrastructure in the Government of Canada. As well, clients were generally satisfied with the quality of services, although some question the value for money of data services. The program was able to demonstrate cost savings and reductions for some services. In cases where cost savings were not quantifiable, the program's activities were believed to have increased procurement efficiencies, which would result in a net cost benefit overall. With regard to the program's ability to reduce IT risks to program delivery, client perceptions could not be validated due to a lack of performance data. Overall, while the program had a strong foundation for performance measurement, gaps existed between the performance measures and outcome achievement indicators.

6. The program was unable to effectively monitor the efficiency of its activities since the costing methodology during the evaluation period did not allocate indirect costs to the appropriate cost drivers. In addition, the use of the competitive tender process for procuring telecommunications services from the private sector were at times not sufficiently robust to use for benchmarking activities.

7. The revolving fund within which the program operates was consistently cost recoverable over the five-year evaluation period. Since there is one mandatory service within the program's service offering, the requirements for demonstrating economy were somewhat different from optional services. The program was unable to demonstrate the mandatory service's economy. No viable alternative methods for program delivery existed.

Management Response

8. The Information Technology Branch agrees with the recommendations and conclusions of the report and intends to act on the recommendations of the evaluation by implementing their Management Action Plan detailed as follows.

Recommendations and Management Action Plan

Recommendation 1: he Chief Executive Officer, Information Technology Services Branch should develop and implement a costing methodology that is consistent with industry best practices, in consultation with the Chief Financial Officer.

• Management Action Plan 1.1: An Activity Based Costing (ABC) framework has been developed and will be implemented in the Information Technology Services Branch at the start of Fiscal Year 2011/2012. This initiative will implement a costing methodology that is continuously consistent with recommended practices. The ABC project is aligned with Treasury Board Secretariat (TBS) costing policy and practices.
• Management Action Plan 1.2: The costing methodology used for each telecommunication service is described and maintained along with its updates forthcoming in a consolidated document. This document will detail the enhanced expected outcomes of applying the costing methodology for each service. This enhanced document will be reviewed annually to ensure that the costing methodology for each service has been reviewed as part of the life cycle management framework.

Recommendation 2: The Chief Executive Officer, Information Technology Services Branch should develop and implement a benchmarking strategy to ensure competitive pricing for its clients and demonstrate value-for-money, in consultation with the Chief Financial Officer.

• Management Action Plan 2.1: The Information Technology Services Branch is developing and implementing a benchmarking strategy to ensure competitive pricing for its clients and demonstrate value-for-money. The strategy will be updated with additional benchmarking framework elements consistent with industry standards and best practices.
• Management Action Plan 2.2: All telecommunications services will have their benchmarks completed using the updated methods outlined in the Information Technology Services Branch benchmarking strategy. This process is performed with the intent of ensuring that departments and agencies are getting value-for-money.
• Management Action Plan 2.3: The existing process to ensure that periodic third-party benchmarks are conducted for key services will also be augmented as per the life cycle needs and demands of the products and services catalogue.

Recommendation 3: The Chief Executive Officer, Information Technology Services Branch should conduct needs analyses for each of the data services and develop an action plan to address the issues pertaining to value-for-money, client relations, and uptake for the data services portfolio.

• Management Action Plan 3.1: A telecommunications strategy and requirements document is being updated and will incorporate a needs analysis for each of the data services.
• Management Action Plan 3.2: An accompanying action plan will be updated/developed to address the issues pertaining to value-for-money, client relations, and uptake for the entire telecommunications portfolio including each of the data services portfolios. The first updated iteration of the strategy and requirements document will be completed this year, and a process will be implemented to ensure that it is reviewed on a semi-annual basis.

Recommendation 4: The Chief Executive Officer, Information Technology Services Branch should consolidate and formalize the performance measurement strategy to better report on outcomes.

• Management Action Plan 4.1: The Information Technology Services Branch will update the iBOC (ITSB operational Branch monthly performance reporting process and committee) performance measurement strategy to better report on outcomes. The strategy will continue to be aligned with TBS-Chief Information Officer Branch (CIOB) policies and consistent with industry standards and best practices.
• Management Action Plan 4.2: The telecommunications performance measurement report and dashboard will be enhanced and used at iBOC based on the Information Technology Services Branch performance measurement strategy. This performance management dashboard will be comprised of key objectives and corresponding performance indicators. The report and dashboard will be reported on and reviewed regularly at the Telecommunications Product Portfolio Review Committee (PPRC) and the Product and Services Executive Committee (PSEC).

Introduction

1. The PWGSC ITSS organization provides IT services to Government of Canada departments and agencies across four lines of business: Telecommunications Services, Data Center Services, Distributed Computing Services, and IT Security Services. This report presents the results of the evaluation of the ITSS organization's Telecommunications Services line of business (the program). The Audit and Evaluation Committee of PWGSC approved this evaluation as part of the 2009/10-2013/14 Risk-Based Multi-Year Audit and Evaluation Plan.

Program Profile

Background

2. Telecommunications services include the transmission of data and voice within and across an organization. Data network services include the provision and ongoing support of electronic data and communications networks, which includes all software as well as wiring, switches, hubs, routers and all other hardware required to support data communications between computing devices. The voice communication services include the provision of local and long-distance services globally, as well as fax services, voice mail, video-conferencing, secure voice and other related services, which include all carrier software and hardware environments.

3. Telecommunications expenditures for the federal government, not including salaries, totalled approximately $1.18 billion in Fiscal Year (FY) 2008/09. Since FY 2006/07, the growth of federal government telecommunications expenditures has been more than double (9.5%) the growth of total IT expenditures for the federal government (4.35%).

4. In 2005, Treasury Board approved an initiative to transform the planning and development of IT products, services, systems and infrastructure in support of a comprehensive IT Infrastructure across the Government of Canada. The aim was to build servicing capacity through the establishment of an IT Centre of Excellence that would deliver IT Services government-wide. In October 2005, PWGSC's Information Technology Services Branch (ITSB) was restructured in support of these changes and to better position the ITSS organization for large-scale growth.

5. In 2008, ITSB developed five-year business plans for both ITSS and the Telecommunications and Informatics Common Services Revolving Fund, which the program operates within (please see the resources section). One of the goals of the ITSS organization was to have a government-wide set of common telecommunications infrastructure and services so as to eliminate duplication and provide a more efficient delivery model for telecommunications services.

Resources

6. The ITSS organization operates on a cost recovery basis. At the time of the evaluation, the Telecommunications line of business was the only line of business operating fully within the Telecommunications and Informatics Common Services Revolving Fund, with the other three lines of business operating primarily with net voted revenues. For instance in FY 2008/09, Telecommunications represented 94% ($221 million) of the revenues that passed through the fund, with the other 6% ($14 million) being partial revenues from the other lines of business. In all, the other lines of business in FY 2008/09 accumulated to $210 million of net voted revenues. At the time of the evaluation, the IT Security line of business was expected to transfer into the revolving fund in FY 2009/10.

Authority

7. The program operates under the authority of the Department of Public Works and Government Services Act which states that "the Department shall operate as a common service agency for the Government of Canada, and its activities as a common service agency shall be directed mainly toward providing the departments, boards, and agencies of the Government of Canada with services in support of their programs."

8. The program also operates under the authority of the Treasury Board Common Services Policy where PWGSC is designated as a common services organization. Under section 5.2 of the policy, PWGSC is responsible for providing telecommunications services and infrastructure to the federal government, which includes planning and constructing informatics infrastructure to support central and other common services. Under the Treasury Board Policy on Management of Information Technology, PWGSC is responsible for the management and operation of common and shared IT services in consultation with departments and the Treasury Board of Canada Secretariat. All of the program's services, with the exception of the Secure Channel Network, are provided on an optional basis.

9. The program's services are funded by the Telecommunications and Informatics Common Services Revolving Fund (the Revolving Fund), granting additional authorities to the Minister of PWGSC. Section 5.2(1) of the Revolving Funds Act provides the Minister of PWGSC with the authority to make expenditures out of the Consolidated Revenue Fund for the purposes of paragraphs 6(g) and 15(b) in the Department of Public Works and Government Services Act. The two relevant sections of the Act define the powers, duties, and functions of the Minister as:

   • Section 6(g): provides the Minister of PWGSC with the mandate to provide planning and co-ordination of telecommunications services for departments, boards, and agencies of the Government of Canada.
   • Section 15(b): provides the Minister of PWGSC with the mandate, on the request of a department, board or agency of the Government of Canada, to provide information management and information technology systems and services.

Program Governance

10. At the time of the evaluation, the ITSS organization was managed across six functions. Each of these functions supported all four lines of business within ITSS, including the Telecommunications line of business. After the evaluation examination period, the ITSS organization was restructured, though the organization still employs a matrix governance structure.

11. The ITSS organization has several working groups and committees in place to inform the various aspects of operations. The organization has one external committee, chaired by the Chief Executive Officer (CEO) of ITSB, called the IT Shared Services Council. Members typically include Assistant Deputy Ministers from IT and Corporate services in other federal departments and agencies, one representative from the Small Agency Administrative Network, the Chief Information Officer and the Deputy Chief Information Officer of the Government of Canada, and representatives from academia and IT research organizations. The council is a forum whereby members can provide strategic advice and guidance to the CEO of ITSB on business plans and strategies to ensure they adequately address the needs of clients and the Government of Canada as an enterprise. It acts as a senior oversight body for the IT Shared Services Transformation Program by reviewing and identifying the principal risks of the IT Shared Service's business growth strategy and implementation plan.

Delivery Models

12. The services provided by the program are delivered via three models, described below.

    • Brokered service with mark-up: A contract is established by ITSS with a service provider from the private sector for direct delivery of the services to the client organizations. The program invoices the client for the cost of the services delivered (for the service provider) plus a mark-up to cover ITSS' administration costs to broker the service, and then pays the service provider for services rendered. Brokered services are predominantly variable costs, meaning costs that vary directly with the volume of production or activity.

      This model applied to Long Distance Telephony Services, Calling Card Services, the Government Teleconferencing Service, Toll-Free Voice Services, Converged Network Services (Enterprise and Transport), Managed Network Services, Government Managed Cabling Services, the Secure Channel Network^{Footnote 1}, Fixed Satellite Services, and Mobile Satellite Services.

    • Brokered service without mark-up: Client organizations use procurement vehicles established by the program and pay the service providers directly. This model does not generate revenue for the program; some overhead costs are recovered through service level agreements.

      Both Local Access Service and the Wireless Communications Service are bundled into the Basic Suite, which covers direct costs for the program and some associated indirect costs. Clients of Networking Equipment Services sign Service Level Agreements that cover direct costs for the program as well as indirect costs.

    • Internally managed service: The program delivers the services and hardware to client organizations through civil servants and consultants. Internally managed services are predominantly fixed costs for the Department, costs that remain relatively unchanged in total regardless of the volume of production or activity within a fairly wide range of volume.

      This model applies to Managed Telecommunications Services, the PWGSC local area network, the Shared Metropolitan Area Network Service, and the Application Response Monitoring Service. The PWGSC local area network is unique among these services in that the department owns the assets and equipment. For the other services, the program leases the assets and equipment from the private sector.

Logic Model

13. A logic model is a visual representation that links a program's activities, outputs and outcomes; provides a systematic and visual method of illustrating the program theory; and shows the logic of how a program, policy or initiative is expected to achieve its objectives. It also provides the basis for developing the performance measurement and evaluation strategies, including the evaluation matrix.

14. For the Evaluation Framework for IT Shared Services (2007-607), the evaluation team developed a logic model for the ITSS organization on the basis of information provided by ITSB and other stakeholders. ITSS staff reviewed this model and validated its logic. The logic model was included in the evaluation framework report that was presented to the Audit and Evaluation Committee in July 2008.

15. Given that the original logic model included desired results for all four of the lines of business in the ITSS organization (i.e., Telecommunications Services, Data Centre Services, Desktop Services, and IT Security), it was necessary to tailor the pre-existing logic model to reflect the particular characteristics of the Telecommunications Services line of business (the program). The revised logic model was reviewed and approved by the program at the onset of the evaluation. The logic model is presented in Exhibit 1.

Exhibit 1: Program Logic Model

A larger image and text description of Exhibit 1: Program Logic Model is available on a separate page. Due to the size of the image, it May not display properly.

Focus of the Evaluation

16. The objective of this evaluation was to determine the program's relevance and performance in achieving its planned outcomes in accordance with the Treasury Board Policy on Evaluation.

17. An evaluation matrix—including evaluation issues, questions, indicators and data sources—was developed during the planning phase. Multiple lines of evidence were used to assess the program. These include:

    1. Document Review: The document review provided an understanding of the program and its context. Documents reviewed included documents provided by the program, as well as documents written about the program. Other relevant documents include: legislation and policy; Speeches from the Throne; PWGSC Reports on Plans and Priorities; and Departmental Performance Reports.
    2. Literature Review: A literature review was conducted to contextualize the program both nationally and internationally; provide theoretical background for the program model; provide baseline data against which the program could be assessed; and identify alternative delivery models through an analysis of other jurisdictions.
    3. Interviews: The evaluation team conducted interviews with PWGSC managers and staff, both internal and external to the program (n=23). In addition, the evaluation team conducted interviews with client departments and agencies (n=16) and with Treasury Board Secretariat staff (n=3). The interviewees provided information about the program's activities, outputs, expected outcomes, stakeholders, relevance and performance from the perspective of program managers, client departments, and other related stakeholders.
    4. Survey: A survey was developed by the evaluation team to capture the clients' perspective on the performance of the program. The survey provided additional depth and evidence to the findings. In total, 84 departments and agencies were invited to participate. This represented 98% of the program's revenues in 2008/09. There were 58 respondents representing 84% of the program's revenues in 2008/09. The survey response rate was 69% overall.
    5. Financial Analysis: Financial data related to the program's budgets, revenues and expenditures were reviewed and analyzed in order to better understand which departments and agencies had adopted telecommunications services; which services had been adopted; and the relative profitability of each service to the program.
    6. Data Analysis: Data from the Acquisitions Spend Cube were analysed in order to obtain spend information for voice and data networks across the Government of Canada. This information was used calculate the relative market share of the program for telecommunications. In addition, program performance data was reviewed to determine to what degree the program was achieving its intended outcomes in the area of quality of service. The scope of the data analysis covered five fiscal years (FY 2004/05 to FY 2008/09). Where appropriate, the scope changed accordingly:
       • Uptake measurements (FY 2004/05 to FY 2008/09)
       • Savings/reductions measurement (FY 2006/07 to FY 2008/09)
       • When available and pertinent, FY 2009/10 information was utilized
    7. Expert Opinion: The evaluation team leveraged expert opinion to evaluate the alignment of the program's activities with industry best practices.

18. More information on the approach and methodologies used to conduct this evaluation can be found in the About the Evaluation section at the end of this report. The evaluation matrix is provided in Appendix A.

Findings and Conclusions

19. The findings and conclusions below are based on the multiple lines of evidence used during the evaluation. They are presented by evaluation issue (relevance and performance).

Relevance

20. Relevance is measured by the extent to which the program: addresses a demonstrable and continuing need; is aligned with federal government priorities and the departmental strategic outcome; and, is an appropriate role for the federal government.

Continuing Need

21. Continuing need is assessed by three primary elements: relevance of original program rationale, legislative or policy requirements and use of the program's services. Based on this, the evaluation found that there is a continuing need for the program.

22. The rationale for the program stems from the overarching issue of the fragmentation of IT across the federal government. Since there is no common telecommunications infrastructure across the Government of Canada, federal organization established their telecommunications infrastructures independently, resulting in over 125 separate voice and data networks across the federal government as of 2008. In situations where multiple departments share one office building, each department has typically been responsible for managing its own infrastructure. This led to a duplication of systems and an increase in costs. The intention of the program is to address this issue by streamlining telecommunications infrastructure across federal organizations so as to reduce costs for the government as a whole. There remains duplication of telecommunications infrastructure, therefore the original rationale for the program remains relevant.

23. With regard to legislative and policy requirements, PWGSC is legislated under section 5 of the Department of Public Works and Government Services Act to operate as a common service organization. Sections 6 (g) and 15 (b) legislate PWGSC as the provider of information technology and information systems services on request to all Government of Canada departments and agencies. The Department is also mandated as a common services organization under the Treasury Board Common Services Policy. According to the policy, PWGSC is obligated to provide telecommunications services on request to federal government organizations on a cost-recovery basis. While the program must provide the services, use of the services by federal organizations is optional, with the exception of the Secure Channel Network, which was made mandatory by Treasury Board in 2006. According to the Treasury Board Common Services Policy, the Department's responsibilities with respect to telecommunications include:

    • developing the government-wide telecommunications architecture and identifying requirements for interfaces with departmental architectures;
    • planning and managing the common government telecommunications infrastructure and co-ordinating the use of shared telecommunications facilities and services; and
    • identifying opportunities for strategic, operational, and economic benefits for the federal government.

24. The Treasury Board Policy on Management of Information Technology outlines an expectation for "increased use of common or shared IT assets and services by departments and agencies to ensure efficiency gains." Under this policy directive, "deputy heads are responsible for ensuring that common or shared IT assets and services are used in departments to avoid duplication, when such assets and services are available and appropriate."

25. Despite there being no mandatory policy requirements for the usage of the optional services provided by the program, the uptake of the program's services was relatively high and increased over the course of the evaluation period. The evaluation was able to estimate that the program had an approximate uptake between 32% and 44% in FY 2008/09, which increased from an uptake between 24% and 29% in FY 2004/05 (uptake is discussed in more detail in the performance section of this report).

26. The need for the program remains strong. The original rationale for the program remains relevant. There is a demonstrated demand for the program's services despite the mixed policy support for the program. While PWGSC is obligated as a common services organization to provide optional services to other federal organizations, there is minimal policy support requiring other departments to use the program's services.

Alignment with federal and departmental priorities

27. Alignment with federal and departmental priorities is determined by assessing the degree to which the issue the program is intended to address is discussed in both federal and departmental priority-setting reports and documents. The evaluation found that the rationale for the program is aligned with numerous federal and departmental priorities. With increased attention across the federal government towards reducing costs, the program is intended to help deliver on cost saving priorities.

28. The March 2010 Speech from the Throne emphasized a commitment to restoring fiscal balance. As part of that commitment, the speech outlined steps towards restraining federal program spending overall. The program is also within the scope of the Administrative Services Review conducted by the Privy Council Office, the results of which have yet to be made known. The goal of this review is to assess administrative services and overhead costs across the government to identify opportunities to improve access to government services, streamline the delivery of services, and reduce the costs of program delivery. The savings that can be achieved by effective implementation of shared services is in line with these priorities of the federal government. The program fits within this priority because one of its goals is that "Government of Canada departments will share a common set of telecommunications infrastructure and services, eliminating duplication, and providing a more efficient delivery model."

29. Previous Speeches from the Throne also outlined commitments that align with the program's activities. The November 2008 Speech from the Throne committed the government to pursuing "innovative reforms to the administration of programs and services… It will build partnerships with third parties and the private sector to deliver better services at a lower cost." The ITSS organization is committed to growing shared services by "working with public sector and industry partners to offer products and services that address the Government's IT business needs via ongoing consultations."

30. The Treasury Board Secretariat's Management Accountability Framework includes an element for the "organizational use of common or shared IT assets and services to avoid duplication, when such assets and services are available and appropriate." The program provides the means for organizations to meet this element of the framework, though this element examines the usage of all IT services, not just telecommunications. According to an August 2009 presentation on the results of a five-year evaluation on the framework, IT management was placed on a rotational basis for large departments and agencies whereby these organizations will be assessed on IT management every three years, with the exception of those who were rated as below acceptable.

31. In terms of Departmental priorities, the 2009/10 Report on Plans and Priorities included a commitment by PWGSC to deliver a "consistent set of government-wide information technology infrastructure products and services to enable government departments to deliver on priorities." In addition, the PWGSC strategic outcome is to provide "high quality, central programs and services that ensure sound stewardship on behalf of Canadians and meet the program needs of federal institutions." The program's governance model is designed to help meet this outcome because it aims to drive IT Shared Services' products and services as well as ensure it continues to meet the needs of federal departments and agencies.

32. The various federal and departmental priorities outlined above provide a relevant context within which the program continues to operate. With a view towards contributing to the reduction of infrastructure and administrative costs, the program aligns with federal and departmental priorities.

Alignment with federal and departmental roles and responsibilities

33. To determine whether or not the program is aligned with the roles and responsibilities of the federal government, three elements were examined: a) whether the responsibility for the program could be transferred to another level of government (i.e.: to the provinces); b) whether the responsibility for the program could be transferred to the private sector; and c) whether the responsibility for the program could be decentralized to individual departments and agencies.

34. The program addresses needs that are internal to the federal government; as such, transferring responsibility to a different level of government is not appropriate. While the telecommunications services provided by the program are available within the private sector, the federal government needs to retain overall responsibility for the program. The government leverages the private sector appropriately to deliver telecommunications services in consolidated contracts to other federal organizations. An analysis on alternative program delivery models is presented in the performance section of this report.

35. Since the Treasury Board Common Services Policy established IT as a core competency for PWGSC, responsibility for the coordination of telecommunications services for the federal government is most appropriate for PWGSC. While some federal organizations May have the capability to assume responsibility of their own telecommunications services, there is no other federal organization with the ability to have complete responsibility for this program. For these reasons, the program is strongly aligned with federal and departmental roles and responsibilities.

Conclusions: Relevance

36. There is a continuing need for the program. PWGSC is legislated as a common services organization to provide services, though the use of the services provided by the program is optional (with the exception of the Secure Channel Network, which is mandatory). While there is no formal requirement, by way of policy or legislation, for other federal organizations to use the services of the program, there is continued demand for the program's services. The program is strongly linked to federal and departmental priorities and is aligned with federal and departmental roles and responsibilities.

Performance

37. Performance is the extent to which a program or initiative is successful in achieving its objectives and the degree to which it is able to do so in a cost-effective manner that demonstrates efficiency and economy.

Outcome Achievement

38. The outcomes presented in a logic model are designed so that their achievement will aid the Department in achieving its strategic outcomes. As such, the evaluation examined the degree to which the program's activities are sufficient for achieving its intended outcomes. While the program did have some performance measures in place, they were not sufficient to measure the achievement of outcomes. The evaluation team developed and implemented alternative methodologies in order to measure outcome achievement. The evaluation was able to determine that the program was making progress towards achieving its outcomes, as discussed below.

Uptake and Defragmentation of Telecommunications Services

39. Uptake is the measurement of a service's use among the federal government.^{Footnote 2} Since the program's services are predominantly optional, when adopters make the commitment to purchase and use such services, this is a key indicator demonstrating the program's value. High uptake is also integral to the success of the program; the more adopters buy into the services, the greater extent the program can accomplish its objective of reducing IT fragmentation.

40. At the time of the evaluation, there were no performance measures in place for the federal government to adequately measure the uptake of ITSS' four lines of business. While the program measures business volume on a monthly basis, federal organizations are not required to report all their IT expenditures to one central agency.

41. To estimate the overall uptake of the program, we measured the program's revenues as a percent of the federal government's total spend on telecommunications goods and services as reported by the Acquisitions Branch Spend Cube Database. Based on this calculation, it was estimated that the program had an approximate uptake between 32% and 44% in FY 2008/09, which increased from an uptake between 24% and 29% in FY 2004/05.^{Footnote 3} The program believes that the uptake calculation May be distorted due to the sheer size of some of the departments that have opted out of the services. For instance, the Department of National Defence procures some of its own services and represents a significant portion of government wide telecommunications purchases. From an organizational standpoint, nearly all of the organizations listed under the Financial Administration Act used at least one of the program's optional services during the review period of the evaluation. For instance, in FY 2008/09, the service Local Access Services had 141 clients.

42. Client departments who did not subscribe to some of the program services identified the barriers to their uptake of these services. In many cases, there was a perception from clients that the costs of the data services were too high. In these cases, the clients May not have a total understanding of the complete cost for a service. When a client is quoted a price from a service provider that is lower than the cost of the same service provided by PWGSC, the client May not be taking into account the full costs if they were to manage the service internally (i.e.: administration costs, technical support costs, etc) which is essentially the value-add of the program. As well, clients May not see the value-add of the consolidation of the telecommunications infrastructure across the Government of Canada, which ultimately leads to increased efficiencies, decreased costs related to duplication, and a modernization of networks and systems. Other barriers were client service related, such as complaints not being addressed in a timely manner, a delay in receiving pricing quotes, and a lack responsiveness to client needs.

43. Through alternative methodologies, the evaluation found that uptake of the program's services was increasing over time. Communication related issues were identified by other federal organizations as primary barriers of the uptake of the program's services.

Quality of Services

44. For optional services, client satisfaction is an important measure of the program's ability to meet its client's needs because clients have the ability to opt out of the services with which they are not satisfied. A large contributor to client satisfaction is high quality service. The program must remain competitive in terms of quality of service and value for money if they expect to retain clients and progress on the achievement of intended outcomes related to uptake. Overall, the evaluation found that client satisfaction with the program's services was high, particularly with the voice services. Perceptions of value for money for data services were found to be somewhat weaker than for voice services. Inadequate communication was found to be the predominant client satisfaction issue for the program.

Client Satisfaction

45. In order to measure client satisfaction, the evaluation team developed and administered a client satisfaction survey. Results of the survey were generally positive. While stakeholders were highly satisfied with the value for money of voice services, clients were less satisfied with the value for money provided by the data services; although these results May have been skewed by the price of the Secure Channel Network. Regardless, stakeholders believed that the program was well managed and credible. Key results from the client satisfaction survey are presented in Exhibit 2.

Table Summary

This table represents the key results from the client satisfaction survey.

Exhibit 2: Client Satisfaction Survey Results

	Voice	Data
Satisfaction rate with the quality of services	93%	80%
Would recommend services to other potential clients	93%	75%
Services were well managed and credible	94%	67%
Services provided value for money	91%	51%

46. Although satisfaction rates were high, some users provided their views on areas in which the program could improve its quality of service overall. Such areas for improvement included: integrated solutions; the use of smaller procurement vehicles; and streamlining the processes with fewer Service Level Agreements (SLA).

47. Also, the evaluation found that in certain cases, the program was not able to meet the needs of some organizations. Some interviewees believed they had special high security needs that the program could not meet with its current approach to shared services; some had sought exemption from the mandatory data service offered by the program.^{Footnote 4} These were cases where the situational requirements would lead to service solutions that would far exceed the needs of most organizations and therefore a shared services model would not be appropriate.

48. The program implemented a portfolio review process in March 2009 as a way of obtaining feedback from clients and assessing customer satisfaction. The portfolio reviews occurred on a periodic basis (approximately once a month) and only one client was surveyed at a time. Eleven clients had undergone a portfolio review at the time of the evaluation. The program is able to establish baseline measurements for each individual organization through this process. However, due to the portfolio review's data collection process at the time of the evaluation, not enough information could be collected at the onset of the portfolio review process to create a program-wide baseline, nor could the results be extrapolated to reflect the general population.

49. Overall, client satisfaction was generally high. Clients indicated areas that could be improved upon to further increase client satisfaction. The program had implemented a process to monitor client satisfaction of individual departments; however the results of the process could not be generalized to reflect the program's client base as a whole.

Service Reporting

50. The SLAs signed between the program and its clients were monitored by the Service Management and Delivery (SM&D) Sector within ITSB. The program reports to senior management monthly on the negotiated standards within the SLAs. Beyond the monthly operational reports, the program posted quarterly reports on the SM&D metrics website.

51. Upon review of the program's monthly operational reports, the evaluation found that the program met the service level agreement targets from October 2008 through January 2010. The evaluation found that 64% of survey respondents believed that the reports met the information needs of their organization. However, some client interviewees perceived that the operational reports from the program were insufficient.

52. According to industry experts, the metrics collected by SM&D were "a great starting point at providing the indicators with which to measure service performance." However, the metrics data were highly technical and largely operational in nature. The data collected were largely related to the availability of services, which measured the program's performance on an output level, rather than an outcome level. The industry experts indicated, "operational measures are not, on their own, performance indicators." In addition, industry experts indicated that many of the indicators used by the program did not state a benchmark value. The industry experts indicated that the availability of services without benchmark requirements, or understanding the impact of service disruptions, makes service improvement decisions difficult from a business point of view.

53. In addition, the evaluation found that communication was an issue in terms of quality of service reporting. For example, interviewees identified issues regarding service outages. In these instances, the client organizations had to implement short-term solutions to resolve the service outage due to slow responses from the program. Interviewees indicated that the delay in response from the program was attributed to differences in perceptions of network availability. In other words, the program's systems indicated that there was connectivity while the client was experiencing an outage. Survey respondents further corroborated this finding wherein only half of respondents believed that the reports for overall network availability reflected actual availability.

54. Overall, service reporting was found to be operational in nature which did not provide clients with sufficient information to meet their expectations. According to industry experts, the current service reporting metrics are a strong foundation for building performance measurement indicators.

Cost savings and reductions

55. Cost savings and reductions^{Footnote 5} are among the most prominent outcomes for the program's clients. The shared services model is expected to be able to negotiate the best prices with suppliers through the ability to leverage economies of scale. As such, the program aims to be able to demonstrate cost savings and reductions for its services. The evaluation found that the program was able to deliver demonstrable cost savings and reductions for the Government of Canada through its shared services procurement model. Where cost savings and reductions could not be documented, anecdotal evidence alluded to the fact that the program's activities were contributing to a net cost benefit for its clients.

56. Given that cost savings and reductions can come in many forms, such as contractual amendments, cost avoidance, and cost consolidation, it is not possible to fully calculate a total percentage or dollar value of cost savings realized from the program's activities. For example, of the program's 17 services available to the federal government, only two had adequate documentation to demonstrate quantifiable price/cost reductions to all clients over the last three fiscal years (FY2006/07 to FY2008/09). However, two other services were able to demonstrate cost savings for some of their clients by comparing the price the department paid for a similar service prior to transitioning to the PWGSC service. In addition, due to the nature of contracts in the telecommunications industry, some contractual amendments were made outside the evaluation period that reduced the cost of services. The cost savings realized through these contractual amendments/service transitions are outlined in Exhibit 3.

Table Summary

This table represents the cost savings realized through these contractual amendments/service transitions.

Exhibit 3: Demonstrated Cost Savings

Timeframe of Cost Saving/Reduction	Service	Percentage Cost Savings/Reductions Realized
FY 2008/09 to FY 2009/10	Fixed Satellite Service	17% to 44%
FY 2009/10	Government Teleconferencing Service	38% to 43%
FY 2005/06 to FY 2006/07	Mobile Satellite ServicesFootnote 8	12% to 44%Footnote 6
FY 2005/06 to FY 2009/10	Converged Network ServiceFootnote 9	47%
FY 2009/10Footnote 10	Network Equipment Support Services	33% to 77%Footnote 7
FY 2005/06	Long Distance Telephony Services	63%
FY 2004/05	Calling Card Services	41% to 56%
FY 2010/11	Toll Free Services	48% to 60%
FY 2010/11	Wireless Communications Services	25% to 43%
FY 2010/11	Secure Channel Network	56%

57. In addition to contractual amendments, some procurement activities May have led to significant savings for the Government of Canada. A primary example would be the cost savings associated with having the program deliver the services as opposed to having each organization procure the services individually, and the costs associated with the duplication of effort for each contract. The evaluation found anecdotal evidence which indicated some cost savings and reductions May have been achieved for other services. Most notably, client interviewees believed that the contract for the Wireless Communications Services provided some of the best prices they had ever seen.

58. Similarly, when Network Equipment Support Services, a service that provides standing offers on common networking equipment to the whole of GC, replaced the legacy service that provided standing offers to only 23 government departments, cost decreases were realised. While it is difficult to directly associate these savings to the buying power of the service, as other factors such as the age of the components and their widespread availability could have had an effect, the consolidation and modernization of services are intrinsically understood as being beneficial.

59. It should also be noted that there are instances where achieving cost savings or reductions are prioritised lower than one May assume. For certain mature services, the program did not reduce prices so as to leverage the higher margins and allow price reductions on newer services which naturally have higher costs in the earlier stages of development. Two mature services had among the highest contribution margins.^{Footnote 11} Neither of these services had undergone any price reductions over the timeframe in question. However, this practice aligned with the ITSS Strategic Business Plan 2009-2014, which states that it is an acceptable practice to "leverage higher margins on mature services, to offset the reduced financial contribution of new or declining services." While cost reductions are not realized directly for these services, leveraging a higher margin creates a net benefit for the program's clients as a whole.

60. For most services, the program had demonstrated cost savings or reductions. In some cases, cost savings could not be quantified; however savings were implied through an increased modernization of services. The program strategically leveraged the success of mature services to deliver cost savings on other services.

IT Security Risk Reduction

61. One of the intended outcomes of the program was to reduce the IT security risks to program delivery through a centralized common service. The rationale for this outcome was that PWGSC would reduce the number of entry points into IT infrastructure and integrate safeguarding measures into its services.

62. At the time of the evaluation, the program did not have performance measures in place to assess the achievement of this outcome. Without performance data, the evaluation team relied on client's perspectives on the efficacy with which the program has reduced IT security risks to their program delivery. As such, one third of survey respondents perceived that the data network services had reduced the IT risks to their organization's program delivery. Some interviewees believed that had the Secure Channel Network included a common firewall, the program would be more effective at achieving this outcome. However, a common firewall was never a component of the service offering for the Secure Channel Network, nor was it the intention of the service. Program interviewees noted that the service was intended to reduce entry points into the Government of Canada IT infrastructure. The program was unable to quantify the reduction of entry points, but was confident that the reduction had been realized.

63. Since this outcome is largely attributed to the Secure Channel service which spans two lines of business, it is difficult to ascertain whether or not client's perspectives were directed towards the network component which is under the program's service offering, or the rest of the service which was under the IT Security line of business. Without performance data to validate the client's perceptions, it cannot be concluded whether or not the program effectively achieved this outcome.

Efficiency and Economy

64. Demonstration of efficiency and economy is defined as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. Efficiency refers to the extent to which resources are used such that a greater level of output is produced with the same level of input or, a lower level of input is used to produce the same level of output. Economy refers to minimizing the use of resources. A program has high demonstrable economy and efficiency when resources maximize outputs at least cost and when there is high correlation between minimum resources and outcomes achieved. Overall, the evaluation was unable to effectively conclude on efficiency and economy due to insufficient benchmarking and costing activities. Despite this, the evaluation found that the revolving fund within which the program operates posted surpluses and that there were no viable alternatives to program delivery.

Benchmarking

65. Benchmarking is an important exercise for determining if the price of services is comparable or better to private sector alternatives. When providing optional services in a revolving fund, the Treasury Board Guide on Revolving Funds states that "prices should be based on market value for the sale…this will help ensure the efficient utilization of scarce resources coupled with a fair economic return to the public." The evaluation found that in the absence of formal, all inclusive benchmarking activities, the program cannot demonstrate efficiency and economic return for its services.

66. The ITSS Strategic Business Plan 2009-2014 states, "benchmarking will form the most important basis for price list development." However, at the time of the evaluation, the program relied on the Request for Proposals process in lieu of formal benchmarking. The program relied on this process under the assumption that the competitive tender process would result in the best possible prices for services. What is not taken into account with the Request for Proposals process is the total cost of services. These would include the internal personnel costs associated with PWGSC tendering and managing the contracts, and in some cases, managing part of the services. Without benchmarking exercises, it is difficult for the program to determine if their clients are receiving a fair economic return.

67. A review of other jurisdictions indicated that other governments have successfully implemented benchmarking activities in this respect. For example, the United States Government's General Services Administration (the procurement arm of the U.S. federal government) has implemented a benchmarking tool. It has developed a Cost per Person Model to help clients understand telecommunications costs and helps the government manage its inputs.

Costing

68. The Treasury Board Guide to Costing states that, "quality, timely costing information supports decision making and performance monitoring… All financial proposals and decisions are strengthened when there is a clear understanding of their complete resource implications." Accurate costing information is critical to establishing fair and appropriate pricing, managing the efficiency of operations and generating cost reductions. The evaluation found that the costing system for the program's services was not conducive to effectively optimizing operational efficiency. Indirect costs were not allocated on an individual cost driver basis, so the program could not accurately determine the full cost of delivering its services.

69. According to program interviewees, the costing system for the program's services was overly complex such that determining accurate costs for each service was a difficult process. The program did not have a streamlined costing system; different sectors gathered cost information independently from one another. ITSS' Business Planning and Management Services Division was required to manually consolidate this information.

70. At the time of the evaluation, the SIGMA financial management system was in its first year of implementation. As such, there were difficulties in accurately matching the costs recorded by SIGMA with those recorded internally within the Telecommunications. Consequently, the program used SIGMA reports more for challenging numbers rather than for determining the costs of services because the financial coding used in SIGMA was not always correct.

71. According to the program, indirect costs represent a variety of cost areas, primarily including human resources associated with finance, contract management, procurement support (from Acquisitions Branch); technology architecture and design; marketing and sales; and business analysis. As of FY 2008/09, the approach to indirect costs was to allocate them to the individual lines of business based on the percentage of the total ITSS revenues per line of business. This is despite the recommendation of the Treasury Board Guide on Revolving Funds which states that programs are to use activity based costing for services in revolving funds. By allocating indirect costs on a non-cost driver basis, the costs allocated to the program's services could be under or overstated. As well, this methodology limits the program's ability to determine whether or not they are achieving greater outputs with less or the same amount of inputs.

72. The program was aware of the need to establish a more reliable costing system. As such, the program had begun to pilot a new costing tool, Decimal Suite. The program was confident that this system would rectify the issues with the manual processes as well as bring consistency in costing across the four lines of business. However, at the time of this evaluation, the piloting of the system was in early stages and could not be assessed.

Revolving Fund

73. Revolving funds provide a business oriented funding mechanism for delivering services on a "commercial or quasi-commercial basis." In such an operating environment, revolving funds are intended to promote good business practices since they must be self-sufficient by recovering all the incurred costs. Revolving funds also require a program to have an effective costing system, since recovering the costs with properly priced services is one of its basic principals. According to the ITSS Strategic Business Plan 2009-2014, the revolving fund is the preferred mechanism for "all in-catalogue service delivery revenues at steady state." The Telecommunications line of business used the Telecommunications and Informatics Common Services Revolving Fund to fund its operations. The program represents the majority, 94%, of the revenues that pass through the fund (FY 2008/09). The evaluation found that the revolving fund for the program consistently posted surpluses over the evaluation period.

74. From FY 2004/05 to FY 2008/09, the revolving fund consistently posted surpluses. According to the Treasury Board Guide on Revolving Funds, revolving fund surpluses May be used to offset future deficits or future capitalization. The 2009 Mandate and Viability Report for the fund forecasted that surpluses from the Telecommunications line of business would subsidize the IT Security line of business. The fund's report acknowledged this indicating that "the 2009/10 Annual Reference Level Update Business Plan requests Treasury Board approval for access to Telecommunications accumulated surplus to be used to offset anticipated deficits from the IT Security business line." The 2009/10 Annual Reference Level Update was approved; this approval is important because this is contrary to the Treasury Board Guide on Revolving Funds, which states that there must be an "absence of cross subsidization between business lines."

Mandatory Service

75. The Secure Channel Network is the only mandatory service within the suite of services delivered by the program. As such, the guidelines for its demonstrable economy are somewhat different from the rest. The Treasury Board Common Services Policy indicates that, "as a guiding principle, mandatory services provided by common service organizations are funded mainly through appropriation, and optional services are funded mainly by full cost-recovery through a revolving fund or net-voting authority." In the case of the Secure Channel Network, Treasury Board approved the service being funded through the revolving fund with other optional services. The evaluation found that the costing methodology of the program over the course of the evaluation period was not optimal for demonstrating the economy of the mandatory service. It is not clear if the service was respecting the special policy requirements of mandatory services.

76. The Common Services Policy indicates, "where optional services are funded by a revolving fund, rates charged to departments must recover the full costs of the common service organization in order to break even at the level of the overall operation over a reasonable period of time." Typically, a reasonable period of time has been five-years as per the five-year business plans.

77. Although the revolving fund posted surpluses, the weaknesses in the costing methodology, and more specifically the allocation of indirect costs, resulted in an inability to determine whether individual services within the program were fully cost recoverable. As such, it cannot be concluded whether or not the Secure Channel Network was demonstrating economy, nor if it was respecting the policy requirements related to mandatory services.

Alternative Delivery

78. The delivery method of a program is strongly correlated with efficiency and economy. While alternative program delivery methods exist, it is necessary to use the delivery method that maximizes the use of inputs. The evaluation identified two potential alternatives for program delivery: private sector delivery and decentralization to other federal organizations. The telecommunications industry in Canada is not optimally structured for full delivery of the program's services. While some federal government organizations May have the capacity to organize their own telecommunications services, none have the capacity, or are well-positioned, to provide full delivery of telecommunications services for the federal government.

79. The telecommunications landscape in Canada contains a mix of national and regional service providers. In 2008, the market share by value for fixed line telecommunications^{Footnote 12} industry was largely dominated by Bell Canada with 66% of the total market share; Telus was a distant second with 26% of the market share; Manitoba Telecom Services with 8% of the market and other service providers (including regional providers) with 1% of the market. For wireless services, the competitive market was significantly different wherein Rogers Wireless represented 37% of the market share; Bell Canada represented 30% of the market share; Telus represented 29%; and other service providers (including regional providers) represented 4% of the market share. With a diverse and competitive landscape, there is a need for the federal government to have an internal organization engaged in a competitive tender process for the federal government. This not only ensures optimal pricing and contracting for services, but also helps maintain a competitive environment open to all players within the market.

80. Considering that there may be an opportunity for large and medium federal organizations to assume responsibility for their own competitive tender processes for telecommunications services given that they already have some economies of scale, the need for small federal organizations to collaborate to achieve the critical mass remains. The Department is able to leverage economies of scale by coordinating the procurement of telecommunications services for other federal departments and agencies. This reduces administrative costs related to the procurement process. As noted earlier, some federal organizations have special needs that are above and beyond those of most clients. As such, tailoring common services to meet the special needs of some federal organizations May not be cost effective.

81. Overall, as a centralized, federal government-based agent for procuring and delivering telecommunications services, the program provides greater value-add to client departments than the other available alternatives.

Conclusions: Performance

82. With regard to the achievement of outcomes, the program had a strong foundation for performance measurement; however there were gaps in the linkage between performance data and the measurement of outcome achievement. Through alternative methodologies, the evaluation found that the program was largely on track towards achieving its outcomes. Uptake was increasing, thereby reducing the fragmentation of telecommunications infrastructure across the Government of Canada. Clients were satisfied with the quality of services provided by the program.

83. The program was able to provide cost savings and cost reductions to its clients. While not all cost savings or reductions could be directly quantified, there was evidence that increased efficiencies stemming from mass procurement would result in a net cost benefit to clients and the Government of Canada as a whole.

84. At the time of the evaluation, performance measures were not in place to adequately measure the program's ability to reduce IT risks to program delivery. While clients were able to provide their opinion on the program's efficacy at reducing IT risks to their organization's program delivery, the lack of performance data resulted in an inability to validate the client's perceptions.

85. With respect to the efficiency of the program, the program's costing methodology during the evaluation period was not conducive to ensuring optimal efficiency and economy. While the program uses a competitive tendering process to ensure optimal prices for services procured from the private sector, its usage as a benchmarking tool is incomplete as it does not include the total cost for the client (where the clients are charged a mark-up on services). Overall, the costing and benchmarking practices of the program during the evaluation period were not sufficient for determining efficiency.

86. In regard to the economy of the program, the revolving fund was consistently cost recoverable over the five-year period. Surpluses generated over the five-year were earmarked to support the transition of the IT Security line of business into the revolving fund. With the mandatory service having somewhat different requirements relating to the demonstration of economy, the costing methodology led to an inability to demonstrate economy for the mandatory service. As well, it was unclear if the program was aligned with the special policy requirements pertaining to mandatory services.

87. As for the overall performance (outcome achievement, efficiency and economy) of the program, outcome achievement was strongly demonstrated for most outcomes. The costing methodology of the program was not conducive to efficiently managing operations and demonstrating greater economy for its services. In addition, as a centralized, federal government-based service provider, there are no viable alternatives for delivery of the program's services.

General Conclusions

88. There is a continuing need for the program and its services, as evidenced by the continued demand for its optional services. The program is aligned with the priorities of the federal government and PWGSC. Its role as the telecommunications service provider for federal organizations is the most appropriate in the interests of the government and the program therefore assumes an appropriate role and responsibility.

89. The program was performing well in terms of achieving its outcomes. In achieving its outcomes, the program was generating a net benefit to its clients and the federal government as a whole. The economy and efficiency of the program could be optimized through improvements to costing and benchmarking activities. The revolving fund within which the program operates consistently posted surpluses, thereby demonstrating cost recovery across the program as a whole.

Management Response

90. The Information Technology Branch agrees with the recommendations and conclusions of the report and intends to act on the recommendations of the evaluation by implementing their Management Action Plan detailed as follows.

Recommendations and Management Action Plan

Recommendation 1: The Chief Executive Officer, Information Technology Services Branch should develop and implement a costing methodology that is consistent with industry best practices, in consultation with the Chief Financial Officer.

• Management Action Plan 1.1: An Activity Based Costing (ABC) framework has been developed and will be implemented in the Information Technology Services Branch at the start of Fiscal Year 2011/2012. This initiative will implement a costing methodology that is continuously consistent with recommended practices. The ABC project is aligned with TBS costing policy and practices.
• Management Action Plan 1.2: The costing methodology used for each telecommunication service is described and maintained along with its updates forthcoming in a consolidated document. This document will detail the enhanced expected outcomes of applying the costing methodology for each service. This enhanced document will be reviewed annually to ensure that the costing methodology for each service has been reviewed as part of the life cycle management framework.

Recommendation 2: The Chief Executive Officer, Information Technology Services Branch should develop and implement a benchmarking strategy to ensure competitive pricing for its clients and demonstrate value for money, in consultation with the Chief Financial Officer.

• Management Action Plan 2.1: The Information Technology Services Branch is developing and implementing a benchmarking strategy to ensure competitive pricing for its clients and demonstrate value-for-money. The strategy will be updated with additional benchmarking framework elements consistent with industry standards and best practices.
• Management Action Plan 2.2: All telecommunications services will have their benchmarks completed using the updated methods outlined in the Information Technology Services Branch benchmarking strategy. This process is performed with the intent of ensuring that departments and agencies are getting value-for-money.
• Management Action Plan 2.3: The existing process to ensure that periodic third-party benchmarks are conducted for key services will also be augmented as per the life cycle needs and demands of the products and services catalogue.

Recommendation 3: The Chief Executive Officer, Information Technology Services Branch should conduct needs analyses for each of the data services and develop an action plan to address the issues pertaining to value-for-money, client relations, and uptake for the data services portfolio.

• Management Action Plan 3.1: A telecommunications strategy and requirements document is being updated and will incorporate a needs analysis for each of the data services.
• Management Action Plan 3.2: An accompanying action plan will be updated/developed to address the issues pertaining to value-for-money, client relations, and uptake for the entire telecommunications portfolio including each of the data services portfolios. The first updated iteration of the strategy and requirements document will be completed this year, and a process will be implemented to ensure that it is reviewed on a semi-annual basis.

Recommendation 4: The Chief Executive Officer, Information Technology Services Branch should consolidate and formalize the performance measurement strategy to better report on outcomes.

• Management Action Plan 4.1: The Information Technology Services Branch will update the iBOC (ITSB operational Branch monthly performance reporting process and committee) performance measurement strategy to better report on outcomes. The strategy will continue to be aligned with TBS-CIOB policies and consistent with industry standards and best practices.
• Management Action Plan 4.2: The telecommunications performance measurement report and dashboard will be enhanced and used at iBOC based on the Information Technology Services Branch performance measurement strategy. This performance management dashboard will be comprised of key objectives and corresponding performance indicators. The report and dashboard will be reported on and reviewed regularly at the Telecommunications Product Portfolio Review Committee (PPRC) and the Product and Services Executive Committee (PSEC).

About the Evaluation

Authority

The Audit and Evaluation Committee of Public Works and Government Services Canada (PWGSC) approved this evaluation as part of the 2009-2010 to 2013-2014 Risk-Based Multi Year Audit and Evaluation Plan.

Objective

The evaluation was conducted in accordance with the Evaluation Standards of the Government of Canada and the Office of Audit and Evaluation of PWGSC. The evaluation took place between August 2009 and March 2010 and was conducted in three phases: planning, examination and reporting. The evaluation had two objectives:

• To determine the relevance of the program: the continued need for the program, its alignment with governmental priorities and its consistency with federal roles and responsibilities.
• To determine the performance of the program: the achievement of its expected outcomes and a demonstration of the efficiency and economy of the program.

Approach

To assess the evaluation issues and questions, the following lines of evidence were used.

Document Review: An initial document review provided an understanding of the program and its context to assist in the planning phase. Documents reviewed included documents provided by the program, as well as documents written about the program.

A number of documents were reviewed to assess the continued relevance of ITSS with respect to departmental and government-wide priorities. They included legislation and policy, Speeches from the Throne, PWGSC Reports on Plans and Priorities, and Departmental Performance Reports.

The evaluation team also reviewed a wide range of documents to gain insight into intended and actual results. These documents included monthly operational reports, financial reports and plans, budgetary allocations, Information Technology Services Branch annual reports and business plans, and client satisfaction surveys.

Literature Review: A literature review was conducted to: contextualize the program both nationally and internationally; provide theoretical background for the program model; provide baseline data against which the program could be assessed; and identify alternative delivery models through an analysis of other jurisdictions.

Interviews: The evaluation team conducted interviews with PWGSC managers and staff, both internal and external to the program (n=23). In addition, the evaluation team conducted interviews with client departments and agencies (n=16) and with Treasury Board Secretariat staff (n=3). The qualitative analysis of the interviews provided information about the program's activities, outputs, expected outcomes, stakeholders, relevance and performance from the perspective of program managers, client departments, and other related stakeholders. Interview guides were used.

Survey: A survey was developed by the evaluation team to capture the clients' perspective on the performance of the program. It was launched on January 20, 2010 and closed on February 10, 2010. The initial contact list was based on membership on the Telecommunications Management Board and the Telecommunications Advisory Panel. The list was supplemented by names from the Chief Information Officers Council (CIO) and the Heads of IT distribution list. The survey provided additional depth and evidence to the findings. In total, 84 departments and agencies were invited to participate. This represented 98% of the program's revenues in 2008/09. There were 58 respondents representing 84% of the program's revenues in 2008/09. The survey response rate was 69% overall.

Financial Analysis: Financial data related to the program's budgets, revenues and expenditures were reviewed and analyzed in order to better understand which departments and agencies had adopted telecommunications services; which services had been adopted; and the relative profitability of each service to the program. Based on the document review, the review of financial files and the literature review, a costing analysis considered the cost-effectiveness of the program.

Data Analysis: Data from the Acquisitions Spend Cube were analysed in order to obtain spend information for voice and data networks across the Government of Canada. This information was used calculate the relative market share of the program for telecommunications. In addition, program performance data was reviewed to determine the degree to which the program was achieving its intended outcomes in the area of quality of service. The scope of the data analysis covered five fiscal years (FY2004/05 to FY2008/09). Where appropriate, the scope changed accordingly:

• Uptake measurements (FY 2004/05 to FY 2008/09)
• Savings/reductions measurement (FY 2006/07 to FY 2008/09)
• When available and pertinent, FY 2009/10 information was utilized

Expert Opinion: The evaluation team levereged expert opinion in three areas. First, the Information Technology Infrastructure Library (ITIL) is a set of recommended concepts and practices for managing IT services. Performance measures and data were reviewed and analyzed by a certified ITIL professional to assess the extent to which the Telecommunications Services program has incorporated industry best practices. Second, the team spoke with Gartner, an information technology research firm, to verify that the IT infrastructure was beening refreshed in accordance with industry practices. Finally, the team reviewed analyses of costs for data network services which were conducted by external expert consultants.

Limitations of the Methodology

Document Review: Efforts were made to ensure that all data provided were reviewed and documented. Due to the large volume of data provided by the program, the large volume of data available for literature review and the complex nature of the telecommunications environment, the evaluation team prioritized documents based on the relevance, usefulness and comprehensiveness of the document. As a result, some documents were relied on more heavily than others. However, every effort was made to systematically identify and categorize data from all documents. Data from documents was also verified with supporting evidence from other documents or research methodologies.

Literature Review: An extensive and wide range of literature was available on IT shared services in public sector organizations and the outsourcing of local area network services. Efforts were made to include relevant material from reliable sources to enable an unbiased and neutral assessment.

Interviews and Survey: When there is a lack of performance data available, two common tools used by Evaluators to measure the performance of programs are interviews and surveys. It is important to recognize that both interviewees and survey respondents provide their opinions based on their individual perspective.

For interviews, product vendors and service suppliers to PWGSC were not interviewed.

For surveys,every department and agency has a slightly different organizational structure for voice and data network services. As such, there was not always a single point of contact for both services. In several cases, the management of voice services was located in the accommodations group. In some instances, voice and data services were organized on a regional basis. For the survey, respondents were encouraged to consult with their colleagues (where appropriate) to ensure a comprehensive response on behalf of their department or organization.

Data Analysis: To calculate market share and federal government expenditures on IT infrastructure, the evaluation team used Spend Cube data furnished by Acquisitions Branch. Acquisitions Branch assembled federal spend information submitted by departments and agencies. The branch indicated that the submitted data represented approximately 85% of total federal government spend. Acquisitions Branch extrapolated the remaining 15% of spend from the Public Accounts of Canada. All transactions were classified by subcategories that best describe the expenditures based on the item description, vendor name (and industry), department line object coding, and economic object coding. Acquisitions Branch categorizes the expenditures. Validation was conducted directly with departments and agencies to ensure transactions were classified accurately.

However, there were three caveats when using Spend Cube data in reporting:

• First, the sheer volume of data prevented department representatives from reviewing all transactions, so the goal was to validate 80% of the federal government spend submitted to the Spend Cube. Acquisitions Branch did not validate the information derived from the Public Accounts of Canada.

• Second, there could be discrepancies between economic object coding and sourcing tree coding because of differences in coding structures.

• Third, the fact that transactions were manually coded to the spend management's sourcing tree meant that there was always the possibility for human error.

For the uptake calculations, uptake was defined as revenues of the program divided by the total telecommunications spend for goods and services for the federal government. However, the program at times did not track revenues for some of its services. This was the case for services that were directly brokered between the vendor and the clients through a contract established by the program. Under these circumstances, the evaluation team, in consultation with the program, removed the appropriate sub-categories from the Spend Cube data. This was deemed appropriate since these services must be procured through the contract established by the program.

Secondly, when appropriate, sub-categories were removed from the Spend Cube for services that the program did not compete in so as to have a better representation of the attainable market. For instance, specialized services such as watercraft telecommunications and Department of National Defence satellites address needs that are not common to other government organizations and as such, would not be services provided by the program. These sub-categories were removed based on consultation with the program, who vetted the sub-categories. Furthermore, the sub-categories and their limited definitions at times represented a challenge for the uptake calculation as it was unclear if the sub-categories should have been kept or removed from the uptake denominator. To mitigate the uncertainty, a range was provided to best represent the uptake of the program. Even so, the uptake number in the report May under or over state actual uptake.

The use of multiple lines of enquiry served to mitigate these limitations.

Reporting

The evaluation team documented findings in a Director's Draft Report, which was internally reviewed by the Office of Audit and Evaluation's quality assessment function. The evaluation team provided the Office of Primary Interest (OPI) Director General with the Director's Draft Report and a request to validate facts and comment on the report. A Chief Audit Executive's Draft Report was prepared and provided to the Chief Executive Officer, Information Technology Services Branch, for acceptance as the OPI. The OPI was requested to respond with a Management Action Plan. The Draft Final Report, including the Management Action Plan, was presented to PWGSC's Audit and Evaluation Committee for the Deputy Minister's approval in January 2011. Based on the Committee's comments, the Draft Final Report and the Management Action Plan was approved in March 2011. The Final Report will be submitted to the Treasury Board Secretariat and posted on the PWGSC website.

Project Team

The evaluation was conducted by employees of the Office of Audit and Evaluation, overseen by the Director of Evaluation and under the overall direction of the Deputy Chief Oversight Officer.

Appendix A

Table Summary

An evaluation matrix is a table that outlines the indicators and data sources used to address the evaluation questions. It is derived from the logic model of the program.

Evaluation Matrix

Evaluation Questions	Indicators	Existing Source(s) of Data
Relevance
1. To what extent does the Telecommunications Services program continue to address a demonstrable need (i.e. increasing standardization, increasing consolidation, and decreasing fragmentation in IT infrastructure across the Government of Canada with respect to telecommunications services?)	1.1 – Number of GC users using common solutions (voice and data network services) in the Telecommunications Services program compared to the total number of users 1.2 – Number of GC users using shared solutions (voice and data network services) in the Telecommunications Services program compared to the total number of users 1.3 – Establishment of GC enterprise-wide procurement instruments for voice and data network services 1.4 – Estimated relative market share for ITSS with respect to total procurement in the GC for voice and data network services (based on data from Acquisitions Branch) 1.5 – Comparisons with the success of other jurisdictions in standardizing and consolidating their IT infrastructure	Document Review Monthly operational reports Information Technology Services Branch business plans Data and Statistics Financial reports and plans Client expenditures statistics Procurement data from Acquisition Branch for GC IT infrastructure expenditures Interviews and Surveys of Key Stakeholders Senior ITSS managers Chief Information Officers and senior managers at client organizations Central Agency representatives Acquisitions Branch staff Acquisitions Branch staff
2. Does the Telecommunications Services program continue to be consistent with PWGSC and GC priorities?	2.1 – Extent of alignment between ITSS activities, outputs, and intended outcomes with departmental and government policies and priorities	Document Review DPWGSC Act 1996 Common Services Policy and other relevant GC policies PWGSC and ITSB business plans PWGSC and ITSB websites and other communications materials PWGSC Program Activity Architecture (PAA) and Management Resources and Results Structure (MRRS) TBS mandates to departments Interviews Senior ITSS managers Central Agency representatives CIO's and senior managers at client organizations
3. To what extent is the Telecommunications Services program aligned with the role and responsibilities for the federal government?	3.1 – Extent to which the Telecommunications Services leverages private sector expertise in the delivery of the program (i.e., acts as managed broker)	Document Review GC-wide procurement vehicles for voice and data network services
Performance: Achievement of Expected Outcomes
4.a To what extent is the Telecommunications Services program successful in capturing or creating uptake of IT shared services?	4.a.1 – Total value of voice and data network services provided annually (i.e. total revenues) by the Telecommunications Services program 4.a.2 – Number of departments and agencies participating as a Wave Partner or Strategic Partner for the Telecommunications Services program. 4.a.3 – The total value of ITSS Telecommunications Services for each department and agency 4.a.4 – Estimated market share with respect to total procurement in GC for voice and data network services, (based on data from Acquisitions Branch)	Document Review Information Technology Services Branch business plans Service Level Agreements Data and Statistics Financial reports and plans Client adoption statistics Procurement data from Acquisition Branch for GC IT infrastructure expenditures Interviews and Surveys of Key Stakeholders Senior ITSS managers CIOs and senior managers at client organizations Central Agency representatives Acquisitions Branch staff Finance Branch staff
4.b What are the barriers to uptake?	4.b.1 – Client perceptions of barriers to uptake 4.b.2 – Client perceptions of their level of awareness, credibility and trust in the Telecommunications Services program
5.a To what extent is ITSS successful in meeting negotiated departmental and agency service requirements?	5.a.1 – Number of times that ITSS has met or exceeded negotiated standards in Service Level Agreements (e.g., Availability Management, Change Management, or Incident Management) 5.a.2 – Average time to implement service requests by type 5.a.3 – Client satisfaction ratings for service quality 5.a.4 – Number and type of complaints 5.a.5 – Number and type of incident reports and problem reports	Document Review Service management performance measures Service Levels Agreements Incident reports and problem reports Complaints Data and Statistics Quality of service statistics Client satisfaction ratings Interviews and Surveys of Key Stakeholders Senior ITSS managers Chief Information Officers and senior managers at client organizations Expert Opinion Industry experts with respect to the use of industry best practices for performance measurement for IT service management
5.b To what extent has ITSS delivered improved service to departments and agencies over time?	5.b.1 – Creation of Service Level Agreements and performance baselines for the first time (for some departments and agencies) 5.b.2 – Increase in the percentage of ITSS clients who have Service Level Agreements that include service levels 5.b.3 – Improvements over time in the scope of services provided to Wave Partners and Strategic Partners. For example, this May include the introduction of robust business continuity and disaster recovery measures 5.b.4 – Improvements over time in the performance standards reflected in Service Level Agreements. For example, this May include the expansion of support from regular business hours to 7/24. 5.b.5 – Client perceptions of service quality 5.b.6 – Extent to which the Telecommunications Services incorporates industry best practices into performance measures for service management
6. To what extent is the Telecommunications Services Program contributing to reduced IT risks to program delivery (e.g. aging IT infrastructure, security, privacy and disaster recovery)?	6.1 – Client perceptions of the Telecommunications Services' contribution towards reducing IT risks to program delivery Aging IT Infrastructure 6.2 – Estimates of the average age of IT assets (once part of ITSS) versus average age of IT assets in new rigorous continual evergreening environment with respect to voice and data networks 6.3 – Increase in the number of departments and agencies with robust business continuity and disaster recovery measures (once part of ITSS) with respect to voice and data network services 6.4 – Comparisons with industry benchmarks for age of IT assets and warranty with respect to voice and data network services IT Security and Disaster Recovery 6.5 – Time required to apply critical security-related patches by software or hardware vendors with respect to voice and data network services 6.6 – Increases over time in the percentage of ITSS products and services that include a security profile with respect to voice and data network services 6.7 – Increases over time in the percentage of ITSS products and services that include a Threat and Risk Assessment 6.8 – Compliance with other applicable parts of the Management Information Technology Security (MITS) policy and other GC security policy instruments for security 6.9 – Increases in the number of departments and agencies with robust business continuity and disaster recovery measures (once part of ITSS) 6.10 – Comparisons with industry best practices with respect to IT security	Document Review Reports to the Deputy Minister and Central Agencies Information Technology Services Branch annual reports IT audit reports on policy compliance Data and Statistics Policy compliance statistics Interviews and Surveys of Key Stakeholders Senior ITSS managers Chief Information Officers and senior managers at client organizations Central Agency representatives
Performance: Demonstration of Efficiency & Economy
7. To what extent is the Telecommunications Services program the most appropriate and efficient means for achieving GC IT infrastructure cost objectives and outcomes?	Alternative Means 7.1 – Use among other jurisdictions of outsourced shared services for IT infrastructure 7.2 – Client perceptions of value-for-money and the impact of the use of Telecommunications Services on their total costs 7.3 – Experiences of large departments and agencies who chose not use Telecommunications Services for optional services Cost Savings 7.4 – Reductions in price of IT infrastructure services (i.e., amount paid by clients) 7.5 – Reductions in cost to deliver IT infrastructure services over time. 7.6 – Cost to deliver IT infrastructure services compared to industry benchmarks for total cost of ownership in government	Literature Review Experiences of other jurisdictions in using outsourced shared services for IT infrastructure Document Review Reports to the Deputy Minister and Central Agencies Product Service Executive Committee (PSEC) decisions Contracts and procurement vehicles Data and Statistics Financial reports and plans Pricing data Costing data Interviews and Surveys of Key Stakeholders Senior ITSS managers Chief Information Officers and senior managers at client organizations Central Agency representatives Acquisitions Branch staff Finance Branch staff Expert Opinion Industry experts with respect to industry benchmarks and best practices for cost management