2009-715 Audit of Written Agreements between ITSB and Selected Branches (Final Report)

March 24, 2011

Table of Contents

• Main Points
• Introduction
• Focus of the Audit
• Statement of Assurance
• Observations
  • Content and Specificity of Written Agreement
    • Insufficient scope and depth on services provided to client branches
  • Identificaton of Critical Systems Within Written Agreements
    • No inclusion of all critical systems within written agreements
  • Service Standards and Associated Reporting
    • Service levels identified in written agreements and performance against these service levels is not always measured and reported
• Conclusion
• Management Response
• Recommendations and Management Action Plan
• About the Audit

Main Points

What we examined

• i. Written agreements between an information technology (IT) service provider and its customer are intended to define the roles, responsibilities and accountabilities of both parties; outline the types of services to be provided, as well as the associated costs of those services and service levels at which the services will be provided; and define the process for reporting and monitoring of services, costs, and performance against service levels. Public Works and Government Services Canada (PWGSC) Information Technology Services Branch (ITSB) provides IT services to other PWGSC branches. The nature and scope of these services are captured in written agreements.

Why it is important

• ii. Branches depend on IT services provided by ITSB to deliver their programs and services. Significant branch investment is managed, in part, through written agreements for the provision of IT services. Written agreements are important, as they are the formal binding arrangement that help service providers to manage client expectations and assist clients to manage and measure the quality of the IT services received.

What we found

• iii. We found that written agreements for IT services were in place, but that improvements could be made to help client branches to better understand their services and associated costs, and better manage and measure the quality of IT services received.

• iv. We noted that branches were responsible for the development of the business continuity plan. We found that the Department's critical systems in support of critical services were identified through the PWGSC-wide Business Continuity Plan. However, there is no agreed-upon consolidated list of all critical systems maintained by the Department. ITSB maintains a list of critical systems that are funded for disaster recovery. However, branch critical systems that were not funded for disaster recovery were not included within ITSB's list. Without specific references to branch critical systems within the written agreements it is possible that confusion exists on whether critical systems emergency requirements are covered, or not, by ITSB's services.

• v. Finally, we noted that written agreements contained few service levels. Performance against some service levels was monitored by ITSB, but results were not reported to the client.

Management Response

Information Technology Services Branch reviewed the report and accepts the audit recommendations found therein and will address these recommendations by completing the implementation of the attached management action plan.

Recommendations and Management Action Plan

Recommendation 1: The Chief Executive Officer, Information Technology Services Branch, in consultation with client branches, should examine written agreements to ensure they are sufficiently complete, and include adequate detail on services, costs, and service levels so they are monitored.

• Management Action Plan 1.1: Create a SLA (Service Level Agreement) working group with branch clients to obtain feedback on areas of improvement in the Application SLA.

• Management Action Plan 1.2: Develop the standard Application SLA template for 2011-2012 that will be a standard for all PWGSC branches. Template will improve the branch client's understanding of the service levels; costs; improve quality and quantity of service levels; and performance against service levels.

• Management Action Plan 1.3: Implement the standard Application SLA template for 2011-2012 with all PWGSC branches.

• Management Action Plan 1.4: Create a URA (User Recovery Agreement) working group with branch clients to obtain feedback on areas of improvement in the User Recovery Agreements.

• Management Action Plan 1.5: Develop the standard Application Change URA (User Recovery Agreement) template for 2012-2013 with branch client's that will be a standard for all PWGSC branches. Template will improve the branch client's understanding of the impact of changes on service levels, costs, and improve quality and quantity of service levels, and performance against service levels.

• Management Action Plan 1.6: Implement the standard Application Change URA template for 2012-2013 with all PWGSC branches.

Recommendation 2: The Chief Executive Officer, Information Technology Services Branch, in consultation with client branches, should establish an agreed-upon definition of critical systems, maintain a list of these critical systems, and ensure that all identified critical systems are covered by a written agreement.

• Management Action Plan 2.1: The Application SLA template will be modified to include a definition of critical systems and the 'level of criticality' of applications.

• Management Action Plan 2.2: The 2011-2012 Application SLA will indicate the branch client's critical systems and include the related IT service (i.e. Tier 1, 2, or 3) and the related downtime.

• Management Action Plan 2.3: ITSB currently maintains a list of ITSB-supported branch applications in a database. ITSB to update the application database (OMIS) based on the Business Continuity of Critical IT (BCCIT) report from 2010 and upon receiving branch client approval.

Recommendation 3: The Chief Executive Officer, Information Technology Services Branch, in consultation with client branches, should improve the quality and quantity of service levels established within written agreements, as well as ensuring that performance against service levels are monitored and reported to client branches.

• Management Action Plan 3.1: Develop a client-focused service management framework, in consultation with client branches, to improve service levels and determine reporting frequency on the quality and quantity of key service levels based on existing ITSB information.

• Management Action Plan 3.2: Implement the client-focused service management framework.

• Management Action Plan 3.3: Agreed upon service levels with reporting and monitoring frequency will be embedded in the new standard templates for both Application SLAs and Change URAs.

• Management Action Plan 3.4: Create a matrix working group including branch clients to improve the quantity and quality of service levels with a plan to incorporate with the 2012-2013 SLA improvements.

• Management Action Plan 3.5: Develop a service level performance framework that includes monitoring and reporting with client branches.

Introduction

• 1. Public Works and Government Services Canada's (PWGSC) mandate is to be a common service agency for the Government of Canada's various departments and agencies. With a strong focus on quality services and sound financial stewardship, PWGSC provides optimum value by enabling other government departments and agencies to deliver their programs and services to Canadians.

• 2. As the federal government's largest common service organization, it is important for PWGSC to clearly understand and meet the needs of its clients. As such, PWGSC launched its Client Service Strategy in 2010-2011 with a view to further improving the Department's relationship with government clients. Part of this initiative involves clarifying roles and expectations between the Department and its clients by developing an improved framework for agreements with client organizations. The framework is also intended to enable strategic discussions between clients and PWGSC. The implementation of the improved framework is done progressively starting in 2010-11 and continuing in 2011-12 with the Department's largest clients.

• 3. In addition to providing services to Government of Canada departments and agencies, most PWGSC branches also provide services to other branches within the Department. The Information Technology Services Branch (ITSB) provides information technology (IT) services to clients that are external as well as internal to PWGSC. IT services provided by ITSB to PWGSC branches include: ongoing support for business applications, significant changes to existing applications or implementation of new applications, and maintenance and support of IT infrastructure services. The nature and scope of the services provided to other PWGSC branches are contained in written agreements. The written agreements serve to assist branches in managing their IT investment.

• 4. Written agreements between ITSB and its clients define the roles, responsibilities, and accountabilities of both parties, outline the types of services to be provided, as well as the costs associated with those services, and levels at which the services will be provided, and define the process for reporting and monitoring of services, costs, and performance against service levels. Service levels are the target for agreed-upon services between the IT service provider and its clients. Examples of service standards include availability or downtime of a particular business application over a specified period, speed with which ITSB will respond, or working hours during which the services will be provided.

• 5. The PWGSC Service Management Agreement Framework was established by ITSB to provide branches with a high-level framework for existing written agreements. Sustainability Agreements cover management, maintenance, minor patching, and minor enhancements of existing business applications. Change Agreements cover significant changes to existing applications, as well as development of new applications and services. The costs associated with the services contained in these agreements are paid by client branches to ITSB.

• 6. In addition, there is an Enterprise Service Level Agreement between ITSB and all client branches to cover the ongoing provision of maintenance, support and security upgrades for shared IT infrastructure services. Annually, each branch transfers funds to ITSB for the shared service costs of data centre services, distributed computing services, network services, and IT security services. For the purpose of this audit, written agreements are comprised of Sustainability Agreements; Change Agreements; and the Enterprise Service Level Agreement.

• 7. Clear written agreements are particularly important for services provided to support critical systems, as the implications associated with the integrity and operations of these systems could have a significant impact on the operations of the Department. A critical system is defined as a system that supports a service that is considered critical. Services are considered critical if a compromise in availability or integrity could result in a high degree of injury to the health, safety, security, or economic well-being of Canadians or to the effective functioning of the Government of Canada.

• 8. The Accounting, Banking, and Compensation Branch (ABCB) is the PWGSC branch with the largest IT investment, managed in part through written agreements with ITSB. ABCB also has the greatest number of critical systems supported by ITSB. The Real Property Branch (RPB) has the second-largest branch IT investment. For 2009-2010, infrastructure sustain services costs managed through the Sustainability Agreements amounted to $31.1 million for ABCB and $21.1 million for RPB. Costs managed through Change Agreements for 2009-2010 for ABCB were $11.2 million, and $7.3 million for RPB. The total shared IT infrastructure services costs managed through the Enterprise Service Level Agreement was $99.1 million for all client branches in 2009-2010.

Focus of the Audit

• 9. The objectives of the audit were to determine whether appropriate written agreements were in place between selected branches and ITSB, and whether ITSB met selected branch requirements as stated in the written agreements.

• 10. The audit focused on Sustainability and Change Agreements between ITSB and two selected PWGSC branches: ABCB and RPB. The audit also reviewed the Enterprise Service Level Agreement between ITSB and all other PWGSC branches. Even though the audit focused exclusively on written agreements between ITSB and two selected PWGSC branches, it is expected that the results of this audit could also serve to improve agreements with other client branches as well as clients from outside PWGSC. More information on the audit objective, scope, approach, and criteria can be found in the "About the Audit" section at the end of this report.

• 11. The audit focused on the content of written agreements. The audit was not intended to assess the overall management of departmental IT investments. As such, the audit did not examine how written agreements serve in managing the IT investments nor did it examine other complementary processes and documents that ITSB and client branches may have put in place to manage their IT investments.

Statement of Assurance

• 12. This audit was conducted in accordance with the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing.

• 13. Sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the findings and conclusions in this report and to provide an audit level of assurance. The findings and conclusions are based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were agreed-upon with management. The findings and conclusions are only applicable to the entity examined for the scope and time period covered.

Observations

Content and Specificity of Written Agreement

Insufficient scope and depth on services provided to client branches

• 14. Clear and complete written agreements are required to avoid misunderstandings between the parties involved as they allow service providers to manage client expectations and clients to manage and measure the quality of the IT services received. Without an appropriate level of detail within these agreements and in the absence of complementary processes and documentation, there is a risk that ITSB and client branches may have a different understanding of the services to be provided, their cost, or whether the services are being provided at the expected level.

• 15. We expected that service agreements would provide sufficient detail to assist clients to manage and measure the services they receive.

• 16. We found that while written agreements are in place, they often lacked details such as: a complete description of the services to be provided, what they include, or when the services are expected to be rendered. More specifically, in the Change and Sustainability Agreements between ITSB and ABCB, descriptions of services were incomplete or not included within the Agreement. For example, the Change Agreements pertaining to the Compensation Report and the Central and Public Accounts Reporting Directorate and Financial Reporting Products Directorate did not contain any service descriptions. For the above-mentioned examples we noted that clients had to rely on descriptions of services contained in other internal documents apart from the written agreements.

• 17. We found that, for many IT services, only a single cost figure was contained in the written agreements, representing the sum of costs for all services covered by the agreements. There was no high-level breakdown of this figure that would allow client branches to understand the cost distribution among the services received. Enterprise Service Level Agreement costs which are documented within the Infrastructure Sustain Services Reports list single figure costs for Enterprise Service Level Agreement costs such as telecommunication services and disaster recovery services. The Infrastructure Sustain Services Report is submitted by ITSB to each branch annually. For example, the Infrastructure Sustain Services Report for the Receiver General (ABCB) for 2008-2009 and 2009-2010 allocated a single figure cost of $2.7M for ABCB disaster recovery. In itself, this sole amount contained in the Infrastructure Sustain Services Report did not allow ABCB to undertake any cost analysis of the invoiced amount. The client branch was not provided with adequate information to substantiate the amounts charged each year and had to rely on complementary processes and documents to determine if the amounts were reasonable. By having to refer to multiple documents, it can be less efficient for client branches to perform their review of amounts charged than having the required information in one single document, such as in a written agreement.

• 18. We also noted that IT security requirements were only addressed at a high level in the Enterprise Service Level Agreement. There was no detailed documentation in these Agreements of the specific system security and privacy requirements for confidentiality, integrity, and availability of the information managed through these written agreements. Although IT security requirements are captured in other separate documents, it is important that written agreements specify security requirements since the enterprise level security practices may be too broad to adequately cover all the requirements for each application. Specific security and privacy provisions should be contained in the written agreements (critical systems as a minimum) between ITSB and client branches since the requirements may be different for each application.

• 19. Without all-encompassing written agreements, client branches need to rely on complementary processes and documents to manage their IT investment. This could result in a misunderstanding about services to be provided by ITSB to client branches. Currently, without clear descriptions in the written agreements and in the absence of complementary processes and documents, this can create challenges for client branch managers to link the amounts invoiced by ITSB to the specific services provided. Finally, without a clear description of the security requirements and associated roles and responsibilities within the written agreements, there is a risk that client branches may not have all the information in this particular document to ensure that their systems are adequately protected. ITSB is working with some of its client branches to develop a new standard Service Level Agreement template and to improve service descriptions.

Identificaton of Critical Systems Within Written Agreements

No inclusion of all critical systems within written agreements

• 20. A business impact analysis review was conducted across all branches at PWGSC in 2009 and 2010 to develop business continuity plans for each branch. These were designed to put into place strategies for continuous delivery and timely resumption of critical services in the event of a crisis. Part of the process involved identifying each branch's critical services and the critical systems supporting these services. The branches were responsible for the development of the business continuity plan, including the identification of critical services and critical systems supporting these services and the strategy to maintain these services operational in the event of a crisis. ITSB was responsible for determining with client branches the technology necessary to continue critical services, and coordinate this capability within agreed-upon resources.

• 21. Maintaining a complete, updated, and agreed-upon list of critical systems that support these critical services is important, as it allows ITSB and client branches to mutually agree on the systems that need to be operational in the event of a crisis. Without an agreed-upon list of critical systems and clarity in written agreements on the level of services provided in support of these critical systems, there is a risk of confusion on whether these systems are covered for disaster recovery.

• 22. We expected that a common list of critical systems for all critical services within PWGSC would exist. We also expected that the written agreements between ITSB and client branches would indicate whether these critical systems are covered by the Agreement and if covered, their associated requirements in the event of a crisis would be formally documented in the agreement.

• 23. We found that a number of branch-specific critical services are dependent on an array of critical systems supported by ITSB. While a list of critical systems is maintained by ITSB, it does not include all critical systems identified as part of the business impact analysis review. The list of critical systems maintained by ITSB is limited to those systems that have an identified and funded disaster recovery solution.

• 24. We also found that some systems critical requirements may not be covered by a written agreement as management may have identified alternative approaches or accepted the risks. This was found to be the case for the critical requirements of the Real Property Management System, a national system that supports a number of asset and facilities management functions related to federally owned and operated buildings in Canada. The Real Property Management System critical requirements were not covered by a written agreement.

• 25. Without specific references to branch critical systems within the written agreements it is possible that confusion exists on whether critical systems emergency requirements are covered, or not, by ITSB's services. As such, related critical services may not have the level of service required to properly support them during a crisis. As a result, in the event of system failure, recovery may not be addressed in a timely manner or the system could be unavailable for undefined periods of time, which could result in an inability to deliver the required critical services.

Service Standards and Associated Reporting

Service levels identified in written agreements and performance against these service levels is not always measured and reported

• 26. Service levels help to define the expected performance of the IT service provider. They are used to measure and report on the achievement of performance against the commitments made in the agreements with client organizations.

• 27. Proper monitoring and reporting of performance against service levels is crucial, as it allows ITSB and its client branches to identify service delivery issues and to measure the progress of service improvements.

• 28. We expected that service levels would be specified for the services described within the written agreements, and that reporting requirements contained within the written agreements would be used to assist with monitoring the achievement of service levels.

• 29. We found that few service levels were included within the written agreements and specified reporting requirements were not always performed. As a result, client branches had to rely on complementary processes and documents to monitor the performance of services. Within every Change Agreement, three service levels are to be monitored and reported upon. These are: has the service met user requirements; has the service been delivered on time; and has the service been delivered within budget. These service levels were to be reported upon quarterly, but we found that they were not. Other service levels were monitored by ITSB, but the information was not reported to client branches as required. For instance, the ESLA contained service levels related to the availability of business applications. These were monitored by ITSB but results were not communicated to the client branches.

• 30. Without proper monitoring and reporting of service levels and in the absence of complementary processes and documents, ITSB and client branches may not be able to fully assess the services and determine whether these achieved the expectations set out in the written agreements. This is particularly important for client branch managers to support certification that services have been rendered in accordance with the agreement. Without proper monitoring and reporting of service levels within the written agreements, clients have to rely on complementary processes and documents to assess performance. It would be beneficial for ITSB and its client branches to have a common understanding of the performance achieved using agreed-upon service levels.

Conclusion

• 31. Written agreements exist between ITSB and its client branches for the provision of IT services. However, the agreements themselves did not contain the appropriate level of information with respect to the nature of the services, their costs, roles and responsibilities of the parties involved, and security requirements. As a result, client branches need to rely on complementary processes and documents to manage their IT investment. In addition, not all client branch critical systems are specifically covered within the written agreements. This could create challenges in the event of a crisis, as some client branches may assume that their critical systems' emergency requirements are covered by the agreement.

• 32. Finally, it is difficult to conclude whether selected branch requirements identified in the written agreements were met by ITSB. Few branch requirements service standards and reporting requirements are specified within the written agreements. When reporting requirements were described within an agreement, reports were, in most cases, not delivered to client branches.

Management Response

Information Technology Services Branch reviewed the report and accepts the audit recommendations found therein and will address these recommendations by completing the implementation of the attached management action plan.

Recommendations and Management Action Plan

Recommendation 1: The Chief Executive Officer, Information Technology Services Branch, in consultation with client branches, should examine written agreements to ensure they are sufficiently complete, and include adequate detail on services, costs, and service levels so they are monitored.

• Management Action Plan 1.1: Create a SLA (Service Level Agreement) working group with branch clients to obtain feedback on areas of improvement in the Application SLA.

• Management Action Plan 1.2: Develop the standard Application SLA template for 2011-2012 that will be a standard for all PWGSC branches. Template will improve the branch client's understanding of the service levels; costs; improve quality and quantity of service levels; and performance against service levels.

• Management Action Plan 1.3: Implement the standard Application SLA template for 2011-2012 with all PWGSC branches.

• Management Action Plan 1.4: Create a URA (User Recovery Agreement) working group with branch clients to obtain feedback on areas of improvement in the User Recovery Agreements.

• Management Action Plan 1.5: Develop the standard Application Change URA (User Recovery Agreement) template for 2012-2013 with branch client's that will be a standard for all PWGSC branches. Template will improve the branch client's understanding of the impact of changes on service levels, costs, and improve quality and quantity of service levels, and performance against service levels.

• Management Action Plan 1.6: Implement the standard Application Change URA template for 2012-2013 with all PWGSC branches.

Recommendation 2: The Chief Executive Officer, Information Technology Services Branch, in consultation with client branches, should establish an agreed-upon definition of critical systems, maintain a list of these critical systems, and ensure that all identified critical systems are covered by a written agreement.

• Management Action Plan 2.1: The Application SLA template will be modified to include a definition of critical systems and the 'level of criticality' of applications.

• Management Action Plan 2.2: The 2011-2012 Application SLA will indicate the branch client's critical systems and include the related IT service (i.e. Tier 1, 2, or 3) and the related downtime.

• Management Action Plan 2.3: ITSB currently maintains a list of ITSB-supported branch applications in a database. ITSB to update the application database (OMIS) based on the Business Continuity of Critical IT (BCCIT) report from 2010 and upon receiving branch client approval.

Recommendation 3: The Chief Executive Officer, Information Technology Services Branch, in consultation with client branches, should improve the quality and quantity of service levels established within written agreements, as well as ensuring that performance against service levels are monitored and reported to client branches.

• Management Action Plan 3.1: Develop a client-focused service management framework, in consultation with client branches, to improve service levels and determine reporting frequency on the quality and quantity of key service levels based on existing ITSB information.

• Management Action Plan 3.2: Implement the client-focused service management framework.

• Management Action Plan 3.3: Agreed upon service levels with reporting and monitoring frequency will be embedded in the new standard templates for both Application SLAs and Change URAs.

• Management Action Plan 3.4: Create a matrix working group including branch clients to improve the quantity and quality of service levels with a plan to incorporate with the 2012-2013 SLA improvements.

• Management Action Plan 3.5: Develop a service level performance framework that includes monitoring and reporting with client branches.

About the Audit

Authority

The audit was approved by the Audit and Evaluation Committee of Public Works and Government Services Canada as part of the 2009-2014 Risk-Based Audit and Evaluation Plan.

Objective

The objectives of the internal audit were to determine whether:

• Appropriate written agreements are in place between selected branches and the Information Technology Services Branch (ITSB); and
• ITSB has met selected branch requirements as stated in the written agreements.

Scope and Approach

The audit was conducted from May 2009 to August 2010, covering from April 2008 to March 2010. It focused on the examination of written agreements between ITSB and two selected branches: Real Property Branch (RPB) and Accounting, Banking and Compensation Branch (ABCB). For the latter, the audit focused on written agreements related to the Receiver General and Compensation Services of the Branch. Written agreements between ITSB and ABCB for Major Crown Projects were not specifically selected for examination. The purpose of the audit was to assess and evaluate the effectiveness of the written agreement process in meeting selected branch requirements. Even though the audit focused exclusively on written agreements between ITSB and two selected PWGSC branches, it is expected that the results of this audit could also serve to improve agreements with other clients branches as well as clients from outside PWGSC.

The audit focused on the content of written agreements. The audit was not intended to assess the overall management of departmental IT investments. As such, the audit did not examine how written agreements serve in managing the branches overall IT investments nor did it examine other complementary processes and documents that ITSB and client branches may have put in place to manage their IT investments.

The survey phase included a review of the relevant processes, procedures, and guidelines for written agreements. Preliminary interviews were held with key staff within ITSB, RPB and ABCB and a detailed risk assessment was undertaken to determine which areas of the service-level management process should be included in the audit.

During the examination phase, in-depth interviews were conducted with key departmental personnel. Relevant processes, controls, and documentation were reviewed and tested. Based on analysis of the information and evidence collected, the audit team formulated audit observations, which were validated with the appropriate managers.

The audit was conducted in accordance with the Institute of Internal Auditors' International Standards for the Professional Practice of Internal Auditing.

Criteria

The criteria used to conclude against the audit objectives were derived from Control Objectives for Information and Related Technology (COBIT® 4.1), the Information Technology Infrastructure Library (ITIL) Framework, and the 2009 Policy on Government Security.

The criteria were that:

• Written agreements contain defined roles, responsibilities, and accountabilities of both the client branches and ITSB, as well as a description of the services, service levels, and associated costs;

• Branches have written agreements that cover their critical services that have IT dependencies; and

• Service levels are defined in written agreements, and reported on by ITSB. Reports on achievement of service levels are provided to the branches.

Audit Work Completed

Audit fieldwork for this audit was substantially completed on August 15, 2010.

Audit Team

The audit was conducted by members of the Office of Audit and Evaluation and an audit consultant. It was overseen by the Director Internal Audit, and under the overall direction of the Deputy Chief Oversight Officer.

The audit was reviewed by the quality assessment function of the Office of Audit and Evaluation.