2008-608 Evaluation of Heritage Conservation Services (Final Report)

January 27, 2011

Table of Contents

• MAIN POINTS
• INTRODUCTION
• PROFILE
  • Background
• EXHIBIT 1: LOGIC MODEL
• FOCUS OF THE EVALUATION
• FINDINGS AND CONCLUSIONS
  • RELEVANCE
  • Continuing Need
  • Alignment with government and departmental priorities
  • Appropriate role and responsibility for federal government
  • Conclusions with respect to relevance
  • PERFORMANCE
  • Outcome Achievement
  • Protecting heritage character of federal assets
  • Knowledge transfer of heritage conservation practices
  • Client satisfaction with services
  • Program performance measurement
  • Cost-effectiveness of services
  • Service delivery model
  • Conclusions with respect to performance
• GENERAL CONCLUSIONS
• MANAGEMENT RESPONSE
• RECOMMENDATIONS AND MANAGEMENT ACTION PLAN
• ABOUT THE EVALUATION
• APPENDIX A: MAIN STAKEHOLDERS OF THE HERITAGE CONSERVATION NETWORK
• APPENDIX B: EVALUATION MATRIX

MAIN POINTS

What we evaluated

i. This evaluation examined the ongoing relevance and the performance of Public Works and Government Services Canada (PWGSC) heritage conservation services. In PWGSC, responsibility for delivering heritage conservation services is divided amongst the Real Property Branch and PWGSC regional offices and is collectively known as the Heritage Conservation Network (the HC Network). These services are included under sub-activity 1.2.3 (Professional and Technical Services) of the Program Activity Architecture, and represent approximately $12M in annual revenue for the Real Property Services Revolving Fund and approximately $1.48M in appropriation funding.

Why it is important

ii. Buildings are tangible evidence of our history and "when important parts of Canada's built heritage are lost, future generations of Canadians are deprived of access to key moments of their shared history.^{Footnote 1}" Heritage conservation activities not only assist PWGSC in meeting its obligations as the third largest custodian of federally owned heritage buildings, but also provide support to other government departments as a common service provider for professional and technical advice on heritage conservation activities.

iii. PWGSC is legislated as a common service organization under the Department of Public Works and Government Services Act. The Treasury Board Common Service Policy grants PWGSC responsibility to provide heritage conservation services to federal departments and agencies. All federally owned buildings over 40 years of age are reviewed to determine potential heritage value. If a building receives a heritage designation subsequent maintenance and repairs (interventions) need to respect the heritage defining elements of the building.

What we found

iv. There is a continuing need under Treasury Board Policy requirements for PWGSC to provide heritage conservation services to other government departments and to protect the heritage character of its own buildings. While there is no legislative requirement for use, demand from other government departments for PWGSC heritage conservation services is increasing.

v. Through the HC Network PWGSC is providing sound stewardship advice on the heritage assets in the federal government's custodianship. The HC Network is largely achieving its stated outcomes and received high client satisfaction ratings in terms of the level and quality of service provided. However it has not fully met functional lead obligations such as knowledge transfer and capacity building for heritage conservation within the HC Network, the federal government, and non-government sectors. The HC Network was not reaching its full potential to meet its mandate and service its clients. This is attributed in large part to its informal governance structure and absence of clear reporting relationships.

Management Response

vi. The Real Property Branch accepts the evaluation findings and intends to act on Recommendations 1 to 4 of the evaluation by implementing their Management Action Plan detailed below. The actions outlined in this Management Action Plan reflect the specific policy context in which heritage conservation services are provided - as established by other federal agencies - and external factors over which PWGSC has limited influence. These include the certainty of client demand and development of specialized private sector capacity.

vii. The Finance Branch accepts the evaluation findings and intends to act on Recommendation 5 of the evaluation by implementing their Management Action Plan detailed below.

Recommendations and Management Action Plan

Recommendation 1: The Assistant Deputy Minister, Real Property Branch, should strengthen the governance and organizational structure of the Heritage Conservation Network in line with current and future client expectations.

• Management Action Plan 1: Produce and implement a Strategic Plan for the Heritage Conservation Network that will clarify and formalize the structure, roles, reporting relationships and business model of the Network.

• Management Action Plan 1.1: Quantify current and expected client expectations through consultation with client groups internal and external to PWGSC.

• Management Action Plan 1.2: Analyze existing and future private sector apacity, in consultation with the regional Directors of Professional and Technical Services, and the private sector.

• Management Action Plan 1.3: Draft a Strategic Plan for the Heritage Conservation Network - including the clear definition of the Heritage Conservation Directorate's Service Management functions (see Recommendation 2 below), and present to Real Property Branch Management Committee for approval.

Recommendation 2: The Assistant Deputy Minister, Real Property Branch, should review the core service management functions of the Heritage Conservation Directorate to align available resources and priorities.

• Management Action Plan 2: Review the Heritage Conservation Directorate's core service management functions as they relate to its mandate to provide strategic stewardship advice to the Program Management and National Accommodation and Portfolio Management - Real Estate Services Sectors, and to provide functional direction to the Network.

• Management Action Plan 2.1: Consult with regional offices and other Real Property Branch Sectors to review Branch and Sector priorities and resource allocations.

• Management Action Plan 2.2: Prepare recommendations for changes to the current service management functions of the Heritage Conservation Directorate based on priorities and revised resource allocations.

• Management Action Plan 2.3: Draft a Mandate Statement for the Heritage Conservation Directorate that lists its core service management functions.

• Management Action Plan 2.4: Integrate this statement into the Heritage Conservation Network Strategic Plan for the approval of the Real Property Branch Management Committee (see Recommendation 1 above).

Recommendation 3:The Assistant Deputy Minister, Real Property Branch, should develop and implement an ongoing performance measurement strategy to assess the effectiveness of the Heritage Conservation Network and to track overall performance, including financial results.

• Management Action Plan 3: Develop and implement a Performance Measurement Strategy for the Heritage Conservation Network, in collaboration with Client Consultancy and Real Property Solutions, and based on the approved Strategic Plan (see Recommendation 1 above).

• Management Action Plan 3.1: Determine, in consultation with regional offices and other Real Property Branch Sectors, the appropriate metrics of performance measurement.

• Management Action Plan 3.2: Establish process by which performance targets will be set and tracked on an on-going basis by regional offices and the Heritage Conservation Directorate.

• Management Action Plan 3.3: Establish format and schedule of annual year-end report of Network performance to be submitted to the Real Property Branch Management Committee.

Recommendation 4: The Assistant Deputy Minister, Real Property Branch, should develop and implement a stakeholder communications strategy for the migration to service management.

• Management Action Plan 4: Based on the approved Strategic Plan (see Recommendation 1 above), develop and implement a stakeholder communication strategy in collaboration with regional offices and the Client Consultancy and Real Property Solutions Sector.

• Management Action Plan 4.1: Develop and implement a communication strategy to inform clients and the private sector of the Network's strategic implementation of service management.

• Management Action Plan 4.2: Develop and implement communication protocols and establish any needed structures in order to engage clients and the private sector in the systematic identification and resolution of issues.

Recommendation 5: The Chief Financial Officer, Public Works and Government Services Canada, should develop and implement an ongoing strategy to assist the business line to improve financial reporting capabilities and to address data comparability issues.

• Management Action Plan 5: Finance Branch will take the following actions to ensure the situation is being addressed:

• Management Action Plan 5.1: The regions and organizations will be consulted to fully understand the issues and to assess their needs in order to improve the financial reporting requirements for the PWGSC's activities and to find solutions to support financial business requirements.

• Management Action Plan 5.2: The centers of expertise in Finance such as Chart of Accounts, Financial Management Advisors as well as SIGMA experts will provide advice and guidance to the regional people in order to assist them with the recording of financial information associated with the PWGSC's activities in the financial system and guidelines will be developed in order to ensure standardization across Canada.

INTRODUCTION

1. This report presents the results of the evaluation of Public Works and Government Services Canada's (PWGSC) heritage conservation services. These services are provided to federal custodians of built heritage by the Real Property Branch's Heritage Conservation Directorate and its regional counterparts, collectively known as the Heritage Conservation Network (the HC Network). The Audit and Evaluation Committee of PWGSC approved this evaluation as part of the 2008-2011 Risk-Based Multi-Year Audit and Evaluation Plan.

PROFILE

Background

2. The Government of Canada owns over 1,300 federally designated heritage buildings, landscapes and engineering works and maintains over 220 National Historic Sites and commemorative monuments. Conservation planning and management are essential to preserve these built heritage assets for future generations.

"Built heritage raises our awareness about how Canadian society has developed, helps us better understand the present and prepare for the future. It fosters a sense of belonging and helps our communities to flourish" ^{Footnote 2}

3. The HC Network provides multi-disciplinary heritage conservation services to federal departments and agencies that own and manage federal built heritage assets, including PWGSC.

4. The HC Network has a team of experienced conservation architects and technologists to provide a range of professional heritage planning, design, and advisory services for the conservation of federal heritage buildings, national historic sites and commemorative monuments across Canada and abroad. These conservation professionals have extensive experience in collaborating with multi-disciplinary experts and in applying heritage conservation policies and standards to a broad range of projects and building types. Services include: strategic planning for conservation projects; historic building envelope screening; environmental monitoring; condition assessment of historic buildings; historic structures reports; conservation guidelines for historic buildings; maintenance guidelines for historic buildings; preliminary treatment options and analysis; schematic design, design development, working drawings and specifications; and site review for building conservation projects. Recent projects include the Vimy Memorial Restoration Project, Vimy France and the conservation guidelines for the Public Archives of the National Library.

5. The HC Network also has a team of professional engineers and technologists who have extensive experience in working on assets while ensuring adherence to heritage policies, standards and guidelines. Service by this team places emphasis on the strategic up front activities such as pre-planning, monitoring, testing, structural analysis, condition assessment, design review, and design management. Structural safety and due diligence are of paramount importance. HC Network engineers and technologists work with clients to define and achieve the most appropriate structural solution. The main objective is to execute minimal but reliable interventions to ensure long-term structural integrity while protecting heritage values. Recent projects include Peace Tower, West Block and East Block of the Parliament Buildings and the Postern Tunnel of the Fortress of Louisbourg.

6. The HC Network provides heritage recording services to conservation specialists and decision-makers by providing up-to-date comprehensive information, documenting the collective heritage for historic buildings, cultural landscapes, monuments, engineering works and a wide range of artifacts. The heritage recording experts within the HC Network have been continually advancing their professional practices and information management, and have been exploring emerging technologies and new media in this field.

7. The architects and engineers providing the professional conservation expertise were originally part of a Parks Canada operational unit but were transferred to PWGSC in 1988 as a result of the PWGSC Ministerial Task Force on Program Review. In order to improve the management of real property, all federal real property contracting services and resources were consolidated in the then Department of Public Works and all federal and architectural staff were assigned to Public Works except for those required by departments to fulfill their role as a knowledgeable client.

8. The HC Network is an informal operating body. The PWGSC team members with heritage conservation expertise are located in Halifax, Quebec City, Gatineau, Ottawa, Cornwall, Winnipeg, and Calgary. The accountability for attaining self-sufficiency and other performance targets is shared between the Assistant Deputy Minister, Real Property Branch and Regional Directors General.

Program Activities

9. The HC Network carries out four main service delivery activities: provision of technical heritage conservation advice and expertise to federal custodians (including PWGSC); knowledge transfer for heritage conservation practices; assistance to Parks Canada to carry out its regulatory function related to federally-owned built heritage and assistance to PWGSC in protecting Federal Heritage Buildings in its custodianship.

Technical Advice and Expertise to Federal Custodians of Built Heritage

10. The HC Network provides technical advice and expertise for federal heritage buildings, landscapes and engineering works (including federal National Historic Sites) to federal custodians on a fee-for-service basis. These activities include: providing engineering, architectural, and landscape architectural advice; managing contracts on behalf of client departments; providing heritage recording services; and documenting and assessing the condition of heritage assets.

Knowledge Transfer for Heritage Conservation Practices

11. The HC Network is involved in knowledge transfer activities related to federal heritage conservation. These activities include initiatives to support nationally consistent services as well as joint initiatives with other jurisdictions, academic institutions, and non-governmental organizations to access industry best practices, influence heritage conservation practice, and support the development of private sector capacity. It also develops best practices and standards for PWGSC and provides technical input for policy and legislation. The appropriation base covers the costs of the above activities.

Assist ParksCanada to Carry out its Regulatory Function Related to Built Heritage

12. The HC Network provides professional and technical conservation advice to Parks Canada on a fee-for-service basis. As outlined in the Treasury Board Policy on the Management of Real Property federal custodians are responsible for working with the Federal Heritage Building Review Office (FHBRO) to ensure that all the buildings they administer that are 40 years of age or older are assessed to determine their level of heritage designation. Parks Canada's FHBRO uses an interdepartmental multi-disciplinary advisory committee to assess a building's heritage value based on historical and environmental significance. Recommendations are then made to the Minister of the Environment, who is responsible for approving federal built heritage designations.

13. HC Network staff are included in the membership of Parks Canada's FHBRO Interdepartmental Review Committee. The HC Network staff provide professional advice to the Interdepartmental Review Committee to determine the heritage character of federal assets. Heritage character is a combination of a building's heritage value and character-defining building features that must be protected to preserve its value. This professional advice is provided by the HC Network on a fee-for-service basis to Parks Canada.

14. In addition HC Network staff assist Parks Canada's FHBRO in reviewing proposed interventions to federally-owned heritage buildings. An intervention is defined as any action that affects heritage character. This professional advice is provided by the HC Network on a fee for service basis to Parks Canada.

15. As the custodian of 148 federal heritage buildings PWGSC must also submit proposed interventions to Parks Canada's FHBRO for review. The HC Network prepares the intervention reports for PWGSC heritage assets for submission to Parks Canada's FHBRO. In order to ensure professional objectivity, HC Network staff members who participate in the FHBRO review process are not involved in the preparation of the intervention report under review.

Assist PWGSC to Protect Federal Heritage Buildings in its custodianship

16. As the third largest custodian of designated federal heritage buildings PWGSC is responsible for 148 heritage buildings across the country. The Accommodation Portfolio Management and Real Estate Services Sector of the Real Property Branch (RPB) is responsible for managing 118 of these heritage buildings and the Parliamentary Precinct Branch is responsible for the remaining 30.

17. To assist PWGSC to meet its custodial obligations the HC Network provides, through appropriation base funding: input into the development of RPB policies and procedures to preserve heritage character; input into the PWGSC investment planning, asset and project management process; and produces annual reports on the stewardship activities undertaken for RPB federal heritage buildings. As an example, the HC Network and the RPB Accommodation, Portfolio Management and Real Estate Services Sector created the policy and procedures for the Stewardship of Federal Heritage Buildings to align with the Treasury Board Policy on the Management of Real Property to ensure that PWGSC respects and conserves heritage character of its designated heritage buildings throughout their life cycle.

Stakeholders

18. HC Network stakeholders within PWGSC include RPB Accommodation, Portfolio Management and Real Estate Services Sector; RPB Program Management Sector; and the Parliamentary Precinct Branch.

19. In addition to PWGSC there are 20 other federal custodian departments of heritage assets. The HC Network provides services to 10 of these federal custodian departments. Parks Canada is the largest custodian of federally owned built heritage and the largest client of the HC Network.

20. On the world stage the Heritage Conservation Network is involved with UNESCO—the United Nations Educational, Scientific and Cultural Organization—on the International Council on Monuments and Sites (ICOMOS). This international non-governmental organization of professionals is dedicated to the conservation of the world's historic monuments and sites.

21. Ultimately the general public is a stakeholder in these services. According to the Guide to Working with the Federal Heritage Building Review Office "federal built heritage includes buildings and places that recall the lives and history of the men and women who built this country. They are significant to all Canadians, be they young or old, recent arrivals to Canada or long-time residents. Built heritage raises our awareness about how Canadian society has developed and helps us better understand the present and prepare for the future."^{Footnote 3}

22. The roles and responsibilities of the primary stakeholders internal and external to PWGSC involved in federal heritage conservation activities are outlined in Appendix A.

Resources

23. The HC Network is made up of 80 full-time equivalent positions across the country and in 2007-08 generated $9.7M in fee revenue and $1.8M in consulting fees for private sector firms. The fees for the program's services were billed through the Real Property Services Revolving Fund. The Heritage Conservation Directorate received an appropriation base of $1.48M in 2008-09.

Logic Model

24. A logic model is a visual representation that links a program's activities, outputs and outcomes; provides a systematic and visual method of illustrating program theory; and shows the logic of how a program, policy or initiative is expected to achieve its objectives. It also provides the basis for developing performance measurement and evaluation strategies, including the evaluation matrix.

25. A logic model of the HC Network was developed based on a detailed document review, meetings with program managers and interviews with key stakeholders. It was subsequently validated with program staff and is presented in Exhibit 1.

EXHIBIT 1: LOGIC MODEL

A larger image and text description of Exhibit 1 : Logic Model is available on a separate page. Due to the size of the image, it may not display properly.

FOCUS OF THE EVALUATION

26. The objective of this evaluation was to determine relevance and performance of PWGSC heritage conservation activities delivered by the HC Network, including cost-effectiveness, in achieving planned outcomes. The evaluation also explored alternative ways of achieving the expected results.

27. An evaluation matrix—including evaluation issues, questions, indicators and data sources—was developed during the planning phase. Multiple lines of evidence were used to assess the program. These include:

1. Document Review: Over 90 documents were reviewed to gain an understanding of the Heritage Conservation Network and its context. In addition financial data, performance measurement and other types of data were reviewed and the analysis of this data contributed to assessing the success of the Network.
2. Interviews: The evaluation team conducted interviews with 20 program stakeholders internal and external to PWGSC. The qualitative analysis of the interviews provided information about the program's activities, outputs, expected outcomes, stakeholders, relevance and performance from the perspective of program managers and clients.
3. Client Survey: A client survey was designed and distributed to 65 clients, internal and external, to PWGSC. Thirty-six responses were received. The individuals completing the surveys represent their own opinions and experiences with the Heritage Conservation Network.
4. Information obtained from the Federal Heritage Buildings Review Office website and the National Historic Sites websites was used to create templates to capture the number of heritage assets worked on by the Heritage Conservation Network from April 1, 2004 to March 31, 2009. The Heritage Conservation Network supplied the data regarding heritage interventions.

28. More information on the approach and methodologies used to conduct this evaluation can be found in the About the Evaluation section at the end of this report. The evaluation matrix is provided in Appendix B.

FINDINGS AND CONCLUSIONS

29. The findings and conclusions below are based the multiple lines of evidence used during the evaluation and are presented by evaluation issue (relevance and performance).

RELEVANCE

30. Relevance was assessed based on the extent to which the HC Network: addressed a demonstrable and continuing need; was aligned with federal government priorities and the departmental strategic outcome; and, was an appropriate role for the federal government.

Continuing Need

31. Continuing need assesses the extent to which the program continues to address a demonstrable need and is responsive to its clients. Lines of evidence reviewed to evaluate continuing need include: the use of the program's services by clients; the ongoing validity of its original rationale; and, the legislation surrounding its delivery. Based on these criteria, the evaluation found that there is a continuing need to deliver Heritage Conservation Services to custodians of federal built heritage.

32. There has been an increasing demand for HC Network Services by federal heritage building custodians. Heritage conservation services are defined as optional services under the Treasury Board Common Services Policy. While PWGSC is required under the Policy to offer these services, custodians are not required to use them and may obtain these services though other sources. Despite the optional status of these services, during the period 2005-06 to 2008-09, the number of federally-owned heritage buildings worked on per year by the HC Network has increased from 177 to 333 which represents an increase of 87%. During the same period, the number of federal national historic sites worked on per year increased from 49 to 76 which represents an increase of 55%.

33. The original rationale for the program remains valid. Originally, the architects and engineers of the HC Network were part of a Parks Canada operational unit but were transferred to PWGSC in 1988 as a result of the PWGSC Ministerial Task Force on Program Review. The aim of the transfer was to improve the management of real property by consolidating all federal real property contracting services and resources in the then Department of Public Works. Staff involved with the transfer noticed two benefits from emerging from the transfer: more rigour brought to the processes and, by being part of a Common Service Organization, the HC Network could provide services to other departments, thereby increasing its sphere of influence for the conservation of federal built heritage.

34. Despite the fact that Canada ratified the World Heritage Convention in 1976 it is the only G8 country that does not have legislation protecting its federally-owned historic places. The Treasury Board Policy on the Management of Real Property obligates custodians of federal heritage to determine the level of heritage designation of all buildings that are forty years of age or older to respect and conserve the heritage character of buildings throughout their life cycles.

35. The Department of Public Works and Government Services Act provides the authority for PWGSC as a common service agency to provide heritage conservation services to custodians of federally owned built heritage through the HC Network. The Act states that "the powers, duties and functions of the Minister extend to and include all matters over which Parliament has jurisdiction, not by law assigned to any other department, board or agency of the Government of Canada, relating to… the provision to departments of advice on or services related to architectural or engineering matters affecting any public work, federal real property or federal immovable."

36. The HC Network also operates under the authority of Treasury Board policies. The Treasury Board Common Services Policy designates PWGSC as an optional Common Service Organization for built heritage conservation activities under Annex F sections 5.12.1 and 5.13.1 where "optional services are based on the dual principles of user choice and user pay… to meet their requirements, departments may select from optional services provided by Common Service Organizations (CSO) or arrange for other sources of supply, including internal supply, shared services with other departments, or suppliers outside of government." ^{Footnote 4}. The HC Network also operates under the Treasury Board Policy on the Management of Real Property.

37. Although there is no legislation specifically mandating the use of HC Network Services, the evaluation determined that there is a continuing need for the program as evidenced by the increasing demand for services, the ongoing validity of the original rationale for the program, and the Treasury Board policy requirements surrounding its delivery.

Alignment with government and departmental priorities

38. Alignment with government and departmental priorities is determined by assessing the program linkages with federal government priorities and with departmental strategic outcomes. The evaluation found that the HC Network contributes to the achievement of PWGSC objectives.

39. Although recent Speeches from the Throne and federal budgets have not specifically identified built heritage as a priority, in 2009, under Canada's Economic Action Plan $323M was allocated to repair and restore federally-owned buildings (which may or may not require interventions to heritage assets). An additional $75M was allocated to bolster tourism by investing in upgrades for National Historic Sites. This increased spending has intensified the demand for heritage conservation services. Further as noted previously, Canada ratified the World Heritage Convention, recognizing the importance of conserving heritage.

40. PWGSC's strategic outcome is to provide high-quality, central programs and services that ensure sound stewardship on behalf of Canadians and meet the program needs of federal institutions. The HC Network Services align with this outcome by providing engineering and architectural expertise to federal custodians. Further they support federal custodians to discharge their stewardship responsibilities by assisting them to meet their policy obligations to protect the heritage character of federal assets. Seventy-five percent of survey respondents rated PWGSC Heritage Conservation Services as very important in protecting the heritage character of federal assets while the remaining 25% indicated the services were somewhat important.

41. Treasury Board Policy on the Management of Real Property requires that the heritage character of federal buildings be respected and conserved throughout their life cycle. Federal custodians of built heritage are responsible for working with the Parks Canada Federal Heritage Building Review Office (FHBRO) to ensure that the buildings they administer, that are 40 years of age or older, are evaluated to determine their level of heritage designation. If Parks Canada's FHBRO determines that the asset possesses heritage character, it will receive either a recognized or classified designation. Classified is the highest level of designation awarded. While the Parks Canada mandate includes providing policy leadership for federal heritage assets, the technical and professional heritage conservation expertise resides with PWGSC.

42. Under the Policy on the Management of Real Property, custodians are required to consult with Parks Canada's FHBRO before demolishing, dismantling or selling a recognized heritage building and to consult Parks Canada's FHBRO before taking any action that could affect the heritage character of a classified building. While the assessment process for federal buildings is mandatory under policy, it is not mandatory for custodians to follow Parks Canada's FHBRO's advice or recommendations on preserving heritage character.

43. In the matter of federal heritage assets in the custodianship of PWGSC, investment decisions balance all functional requirements for the building including the heritage conservation advice related to the property. According to the Real Property Branch Procedure for the Stewardship of Federal Heritage Buildings, "if the project will not follow the conservation advice received, compliance with the Treasury Board (TB) policy cannot be achieved. PWGSC's position should be documented and endorsed by the appropriate delegated authority, usually the Regional Investment Management Board. This justification should be provided to the Regional Heritage Coordinator for all designated buildings and to FHBRO in the case of classified buildings."^{Footnote 5}

44. The evaluation found that the services are not closely aligned with current priorities articulated in Speeches from the Throne. However, the services are very closely aligned with the strategic outcome of PWGSC by providing federal custodians with engineering and architectural advice and services to meet their obligations under the Treasury Board Policy on the Management of Real Property.

Appropriate role and responsibility for federal government

45. To determine if the program is an appropriate role for the federal government, three elements are reviewed: potential program delivery by another level of government, private sector delivery and decentralization to other departments and agencies. Based on these criteria, the evaluation found that HC Network Services are an appropriate role for the federal government.

46. Given the assets are owned by the federal government and the program addresses needs and requirements that are specific to the federal government, devolution of federal responsibility to other levels of government or the private sector is not an option.

47. The federal government has a role to play in the preservation of its federal heritage assets as these sites and buildings are an integral part of Canadian history. "The value of built heritage comes from what it can teach us about the lives and history of those who built this country. It comprises learning sites for all Canadians, be they young or old, recent immigrants to Canada, or long-time residents. It is also a source of tourist revenue for communities and helps to preserve the environment by capitalizing on existing structures."^{Footnote 6}

48. The Government of Canada owns more than 40,000 buildings, more than 20,000 of which have had their level of heritage designation evaluated and over 1,300, or 3%, have been designated^{Footnote 7}. The HC Network plays an important role not only in conserving heritage buildings that have already been designated, but as the federal building inventory ages, it also will play a role in providing services to historic assets identified in the future.

49. Other levels of government do not have jurisdiction over federal assets. In addition, neither other levels of government nor the private sector can contextualize heritage interventions within the overall management framework for federal real property. Heritage considerations need to be balanced throughout the building life cycle with other considerations such as available funding, sustainability and greening initiatives, and usefulness of space.

50. While engineering and architectural services are available in the private sector, as noted above, devolution of federal responsibility to the private sector is not viable. In addition, since private sector architectural fee structures are based on the value of work to be undertaken, a conflict of interest could arise. At the time of this evaluation, the program was adjusting its service model to maximize its use of private sector services while retaining its overall responsibilities related to federal heritage interventions. Further information on this service evolution is discussed in the Performance section of this report.

51. Total decentralization to client departments would not likely result in greater efficiencies for the Government of Canada. In response to the 1988 PWGSC Ministerial Task Force on Program Review, to improve the management of real property, all federal real property contracting services and resources were consolidated in the then Department of Public Works and all federal and architectural staff were assigned to Public Works except for those required by departments to fulfill their role as a knowledgeable client. This centralized or centre of expertise approach is reinforced by the Common Services Policy that states departments must not recreate internal architectural and engineering expertise in lieu of using PWGSC optional services.

52. In addition, 64% of survey respondents indicated that they were unaware of alternatives to the heritage conservation services offered through PWGSC that would be of greater benefit to them, 11% indicated they were aware of alternatives and 25% indicated they didn't know. Only 17% of survey respondents indicated that they felt that there is no ongoing need for centralized heritage conservation services.

53. The evaluation found that heritage conservation services are an appropriate role for the federal government. It also found that a federal common service organization is the most appropriate model to ensure the ongoing protection of the heritage character of federal assets.

Conclusions with respect to relevance

54. While the use of the HC Network's services is optional, the demand for its heritage conservation services is increasing. The number of federally-owned heritage buildings and the number of historic sites worked on per year by the HC Network increased from April 2005 to March 2009. In addition over one-third of survey respondents indicated that they expect their need for the HC Network services to increase over the next five years. By increasing its sphere of influence for federal heritage buildings, the HC Network is ensuring federal custodians have consistent and reliable information to enable them to make informed decisions regarding their heritage assets.

55. Although there is no legislative requirement for the protection of federal heritage assets there are policy requirements for PWGSC to deliver heritage conservation services. In accordance with the Treasury Board Policy on the Management of Real Property all federal government custodians must respect and conserve the heritage character of federal buildings throughout their life cycle. Furthermore, as a common services organization PWGSC must provide heritage conservation expertise and professional advice to other custodian departments as an optional service under the Common Services Policy. Finally, the current model of federal accountability for the provision of heritage conservation services appears to be the most appropriate.

PERFORMANCE

56. Performance is the extent to which a program or initiative is successful in achieving its objectives and the degree to which it is able to do so in a cost-effective manner that demonstrates efficiency and economy.

Outcome Achievement

57. Performance can also be measured as the extent to which a program or initiative is achieving its targets or standards. The outcomes presented in a logic model are designed so that their achievement will aid the Department in achieving its strategic outcomes. This evaluation examined the performance of the HC Network in providing PWGSC and other federal custodian departments with advice and expertise to protect the heritage character of federal assets.

Protecting heritage character of federal assets

58. Heritage character is a combination of a building's heritage value and character-defining building features that must be protected to preserve its value. Success in achieving this outcome would be evidenced by appropriate interventions being carried out by the HC Network and federal custodians, as well as federal custodian departments having reliable information provided by the HC Network to make informed decisions.

59. HC Network staff are included in the membership of Parks Canada's FHBRO Interdepartmental Review Committee. The HC Network provides professional advice to the Interdepartmental Review Committee to determine the heritage character of federal assets. In addition the HC Network assists Parks Canada's FHBRO in reviewing proposed interventions to federally-owned heritage buildings. An intervention is defined as any action that affects heritage character.

60. Over the past five years, the sphere of influence for the services of the HC Network has been significant. The HC Network has provided technical and advisory services for 501 of Parks Canada's 1,314 FHBRO-designated federal heritage buildings. Over the same period of time it also worked on 98 of 220 federal national historic sites. Specifically with respect to PWGSC, over the past five years the RPB Accommodation, Portfolio Management and Real Estate Sector utilized the services of the HC Network to conduct heritage interventions on 51 of its 118 federal heritage buildings (43% of the RPB federal heritage inventory).

61. Since 2004, the HC Network has provided services on an ongoing basis to the four largest custodians of federal heritage buildings and national historic sites: Parks Canada, the Department of National Defence, PWGSC and the Department of Fisheries and Oceans. These four custodians are responsible for 1,107 federal heritage buildings (84% of the total federal inventory) and 207 national historic sites (94% of total federal national historic sites).

62. To monitor and protect the heritage character of PWGSC assets, the HC Network has implemented a formal reporting process to monitor activities related to the 388 buildings in the custodianship of RPB. As of March 2009, of the 388 buildings, 118 were heritage buildings; 144 were under 40 years of age; 89 had been evaluated but not designated; and 37 had not been submitted to FHBRO to determine heritage character. In 2008-09 PWGSC submitted 105 projects for FHBRO review on a selection of its 118 heritage buildings.

63. Through its work with FHBRO to determine heritage character and review interventions for federal built assets, and by providing professional conservation and advice services to federal custodians, we can conclude the HC Network contributes to appropriate heritage interventions and provides reliable information leading to the protection of federal heritage assets.

Knowledge transfer of heritage conservation practices

64. Heritage conservation is a multi-discipline expertise that requires ongoing interaction within communities of practice to transfer knowledge and streamline approaches. However, in 2007-08 and 2008-09 activities with other jurisdictions and communities of practice were curtailed to allocate resources to the staffing actions required to meet the current and anticipated demand for heritage conservation services. In addition work planned in 2008-09 to reconcile sustainability and heritage considerations was delayed, as were initiatives related tos tandardizing/optimizing project delivery practices.

65. The informal governance structure of the HC Network^{Footnote 8} also hinders the ability of the HC Directorate to provide appropriate leadership to increase knowledge transfer activities internal and external to government. The HC Network exists to: share best practices; rationalize resources across the country for national projects; and develop and implement policies and procedures related to heritage conservation activities. However, an informal structure has limitations and results in different approaches and levels of service across the country. Such limitations include inconsistent financial reporting structures and inconsistent use of performance measurement indicators and baselines.

66. Some changes to the structure have already emerged. A change in formal reporting relationship was made to respond to client concerns with the level of service available in the Ontario Region. In addition to managing the functional leadership role for heritage conservation and service delivery to clients of the National Capital Area, as of October 2009 the Director, Heritage Conservation Directorate became responsible for managing the Ontario Region Heritage Canals and Engineering Works staff. The Western Region office, which also serves clients in the Pacific Region, has indicated that it would be beneficial if the reporting relationships for heritage conservation were reviewed.

67. Increased client demand and the current informal governance structure of the HC Network have impacted on the development of knowledge transfer for heritage conservation practices internal and external to the federal government community.

Client satisfaction with services

68. Since optional services under the Treasury Board Common Service Policy are based on the dual principles of user choice and user pay, client satisfaction is a key indicator in determining success of the program.

69. The HC Network has received high client satisfaction ratings from federal custodians. The Office of Audit and Evaluation sent a survey to 65 federal clients internal and external to PWGSC and received 36 responses. Survey respondents indicated that advisory services provided by HC Network staff met their needs. Eighty-six percent indicated they were overall satisfied and none indicated dissatisfaction with the advisory services. Ninety-four percent of survey respondents indicated they had access to reliable conservation advice and expertise through the HC Network. In addition 97% of clients surveyed indicated that PWGSC Heritage Conservation staff provided sound conservation advice

70. When clients were asked about the ability to effectively manage contracts, 73% of survey respondents indicated that contracts managed by PWGSC were on budget. Six percent indicated that contracts were not on budget while 21% were unsure.

71. The high client satisfaction ratings coupled with the increasing demand for services indicate that the HC network meets or exceeds the expectations of its clients.

Key survey results indicate high client satisfaction

• 97% indicated they received support to their program from the Network's activities
• 94% indicated they had access to reliable heritage conservation advice and expertise through the Network
• 83% indicated the Network's services provided value-added advice/guidance
• 86% indicated overall satisfaction with the service - none expressed dissatisfaction

Program performance measurement

72. The HC Network does not use consistent performance measurement systems or targets to monitor performance. Two of the six regions providing heritage conservation services utilize the (International Organization for Standardization) ISO 9001:2000 standard to track and measure performance. Processes are in place and monitored to ensure continual improvement in all aspects of a project (definition, implementation, and close-out) in the Western Region and the HC Directorate. While no overall service standards exist for the HC Network, RPB has established that employee positions funded through the revolving fund should maintain a level of 90% billable hours^{Footnote 9} to ensure that the billing rate is sufficient to cover revolving fund expenditures. For the two regions that submitted this information only the HC Directorate serving the National Capital Area has achieved this target within the past three years.

73. No targets had been established with respect to the potential reach of the HC Network as services were provided based on the available funding of its clients. However, overall the HC Network appears to have contributed to the protection of heritage character by providing advice and expertise on projects affecting 38% of federal heritage buildings and 44% of national heritage sites over the past five years.

74. The HC Directorate and the Western Region were the only HC Network offices that had administered client satisfaction surveys upon completion of a project but response rates were low (only 7 of 31 surveys were returned in 2008-09 in the Directorate, and 21 of 110 were returned in the December 2008 survey for the Western Region).

75. For the HC Directorate, trend analysis from 1997 to 2009 based on 120 responses demonstrated an increase in client satisfaction. During this period results indicate that: meeting client expectations has increased from 85% to 97%; understanding client needs rose from 73% to 94%; keeping clients informed increased from 74% to 85%; meeting deadlines increased from 73% to 91%; and overall satisfaction with the quality of work rose from 83% to 97%. This trend is supported by the data from the client survey conducted as part of this evaluation.

76. While in the Western Region client satisfaction surveys were administered annually so as to maintain the ISO 9001:2000 standard, the region was only able to provide survey results for the years 2000 to 2004 and 2008. The surveying methods for the region changed from year to year (for the years where data was provided), invalidating any possible trending analyses. However for 2008 surveys were administered twice within the year, April and December. The results from these two surveys indicate that customer satisfaction decreased from April to December for many of the criteria. In April, 100% of clients indicated that they were kept informed and that work delivered was completed. In December this dropped to 86% indicating that they were kept informed and 82% indicating that work delivered was completed. Overall client satisfaction remained high at 92% in December (100% in April).

Cost-effectiveness of services

77. The HC Network services were billed as part of the Real Property Services Revolving Fund, which in addition to architectural and engineering activities, includes property management, holdings and divesture, and support services. To assess cost-effectiveness, percentage of billable time for revolving fund staff and revenue breakdowns by client department, the evaluation team would have required financial information for fiscal years 2006-07, 2007-08, and 2008-09 for the HC Network. Since not all regions separately tracked work for heritage and contemporary building interventions, the evaluation was not able to review data solely attributable to heritage interventions.

78. Two of the regions were able to provide specific heritage conservation financial data on the performance of projects against budget. An analysis of this data could not be performed due to data comparability and integrity issues. This is due in large part to the fact that Parks Canada is the largest client for those two regions, and as an agency it has the flexibility to carry funds forward and thereby adjusting project completion dates. Also, due to Parks Canada's mandate that includes a Regulatory function that is responsive in nature, funds allocated to their projects were continuously shifted between projects and programs. The HC Network also indicated that there were inconsistencies in the reporting and tracking of projects such that the final work plans did not always provide the explanation for over or under spending. As such, we are unable to conclude on the cost-effectiveness of the program.

Service delivery model

79. To reflect industry best practices RPB is implementing a Corporate Real Estate Model to modernize its service approach by using the most appropriate delivery mechanisms and refocusing from service delivery to service management. Service delivery is the direct provision of services to clients whereas service management is the management of contract mechanisms with the private sector to provide services to clients. To support this model the HC Directorate was specifically tasked with providing leadership for the regional members of the HC Network for the following areas:

• subject matter expertise for heritage buildings, including preservation /conservation requirements and technology approaches/techniques
• the development/promulgation of national standards, policies, best practices, risk management and lessons learned related to heritage
• quality management and monitoring relating to heritage conservation
• performance monitoring and reporting; regulatory compliance and risk management in relation to heritage conservation
• service improvement and procurement strategies for providing heritage services

80. The HC Directorate has made progress in some areas of its service management implementation. For the HC Network to align with the RPB corporate real estate model of service management, capacity building for private sector delivery of heritage conservation work was identified as a requirement. The creation of a standing offer to increase use of the private sector for heritage conservation services was originally planned for 2005-06 but due to time and resource constraints was not put in place until August 15, 2008. The standing offer was awarded to four firms for an estimated value of $4.5 million.

81. However, there are some concerns regarding the migration from service delivery to service management. Parks Canada indicated that the current pricing structure is a concern and that it was unsure of the added value that PWGSC would bring to managing contracts as opposed to directly providing the services. The efficiency of migration from service delivery to service/contract management would need to be fully explained and clarified. Furthermore, only 56% of survey respondents indicated that they believed PWGSC heritage conservation services currently saves their organization time and resources through contract management.

82. As a Schedule 2 department Parks Canada is not bound to adhere to the Common Services Policy, allowing it to create an internal unit of architects and engineers to meet its heritage conservation needs. Parks Canada has indicated that it does not wish to recreate an internal unit for professional and technical advice and is hesitant to use the private sector for professional advice but must consider all options to keep costs down. As Parks Canada is the largest custodian of federally-owned built heritage it is also the HC Network's largest client. While there are specific service agreements between PWGSC and Parks Canada the overarching Memorandum of Understanding for the long-term provision of heritage conservation services has not been updated since 2001.

83. In addition the Canada Research Chair, School of Architecture, University of Montreal, expressed concern that over time HC Network resources could lose their professional skill sets if they do not actively work on projects and only review the work of others. Such negative impacts would not appear immediately but the quality of the advice could deteriorate in the long-term. The Canada Research Chair also identified staff retention as an issue of concern, as heritage conservation is a creative field and solely commenting on projects would not be fulfilling for staff.

Conclusions with respect to performance

84. The HC Network was progressing towards achieving its stated outcomes. Departments and agencies had access to professional and technical advice to protect federal heritage assets. However, the program was having difficulty meeting its functional lead obligations such as knowledge transfer and capacity building for heritage conservation.

85. The HC Network was not reaching its full potential when operating as an informal body. The formalization of the HC Network could rationalize resources across the country to assist regions with less capacity or specialized expertise to meet client demand. In addition formalization could address service standardization and performance reporting issues.

86. Since not all regions separately tracked work for heritage and contemporary building interventions, the evaluation was not able to review data solely attributable to heritage interventions and therefore the evaluation could not conclude on the overall cost-effectiveness of heritage conservation activities. However, over 70% of survey respondents indicated that contracts managed by PWGSC were on budget.

87. Some clients are concerned about the future added value of the services if there is a full devolution from service delivery to service management.

GENERAL CONCLUSIONS

88. Relevance: There were no legislative requirements to protect federal heritage buildings. However there was a continuing need for heritage conservation activities internal and external to PWGSC based on obligations from Treasury Board policy requirements. Demand for the services continues to increase demonstrating that the HC Network is meeting a need of the federal community.

89. Performance: The HC Network was providing sound stewardship advice on federal heritage buildings to clients internal and external to PWGSC. The HC Network was largely achieving its stated outcomes and its clients were satisfied with the level and quality of service provided. The contribution of the HC Network to the revolving fund could not be fully measured.Heritage conservation is a multi-discipline expertise that requires ongoing interaction within communities of practice to transfer knowledge and streamline approaches and the HC Directorate had difficulty maintaining, initiating and leading these activities for the regional members of the HC Network.

MANAGEMENT RESPONSE

90. The Real Property Branch accepts the evaluation findings and intends to act on Recommendations 1 to 4 of the evaluation by implementing their Management Action Plan detailed below. The actions outlined in this Management Action Plan reflect the specific policy context in which heritage conservation services are provided - as established by other federal agencies - and external factors over which PWGSC has limited influence. These include the certainty of client demand and development of specialized private sector capacity.

91. The Finance Branch accepts the evaluation findings and intends to act on Recommendation 5 of the evaluation by implementing their Management Action Plan detailed below.

RECOMMENDATIONS AND MANAGEMENT ACTION PLAN

Recommendation 1: The Assistant Deputy Minister, Real Property Branch, should strengthen the governance and organizational structure of the Heritage Conservation Network in line with current and future client expectations.

• Management Action Plan 1: Produce and implement a Strategic Plan for the Heritage Conservation Network that will clarify and formalize the structure, roles, reporting relationships and business model of the Network.

• Management Action Plan 1.1: Quantify current and expected client expectations through consultation with client groups internal and external to PWGSC.

• Management Action Plan 1.2: Analyze existing and future private sector capacity, in consultation with the regional Directors of Professional and Technical Services, and the private sector.

• Management Action Plan 1.3: Draft a Strategic Plan for the Heritage Conservation Network - including the clear definition of the Heritage Conservation Directorate's Service Management functions (see Recommendation 2 below), and present to Real Property Branch Management Committee for approval.

Recommendation 2: The Assistant Deputy Minister, Real Property Branch, should review the core service management functions of the Heritage Conservation Directorate to align available resources and priorities.

• Management Action Plan 2: Review the Heritage Conservation Directorate's core service management functions as they relate to its mandate to provide strategic stewardship advice to the Program Management and National Accommodation and Portfolio Management - Real Estate Services Sectors, and to provide functional direction to the Network.

• Management Action Plan 2.1: Consult with regional offices and other Real Property Branch Sectors to review Branch and Sector priorities and resource allocations.

• Management Action Plan 2.2: Prepare recommendations for changes to the current service management functions of the Heritage Conservation Directorate based on priorities and revised resource allocations.

• Management Action Plan 2.3: Draft a Mandate Statement for the Heritage Conservation Directorate that lists its core service management functions.

• Management Action Plan 2.4: Integrate this statement into the Heritage Conservation Network Strategic Plan for the approval of the Real Property Branch Management Committee (see Recommendation 1 above).

Recommendation 3: The Assistant Deputy Minister, Real Property Branch, should develop and implement an ongoing performance measurement strategy to assess the effectiveness of the Heritage Conservation Network and to track overall performance, including financial results.

• Management Action Plan 3: Develop and implement a Performance Measurement Strategy for the Heritage Conservation Network, in collaboration with Client Consultancy and Real Property Solutions, and based on the approved Strategic Plan (see Recommendation 1 above).

• Management Action Plan 3.1: Determine, in consultation with regional offices and other Real Property Branch Sectors, the appropriate metrics of performance measurement.

• Management Action Plan 3.2: Establish process by which performance targets will be set and tracked on an on-going basis by regional offices and the Heritage Conservation Directorate.

• Management Action Plan 3.3: Establish format and schedule of annual year-end report of Network performance to be submitted to the Real Property Branch Management Committee.

Recommendation 4: The Assistant Deputy Minister, Real Property Branch, should develop and implement a stakeholder communications strategy for the migration to service management.

• Management Action Plan 4: Based on the approved Strategic Plan (see Recommendation 1 above), develop and implement a stakeholder communication strategy in collaboration with regional offices and the Client Consultancy and Real Property Solutions Sector.

• Management Action Plan 4.1: Develop and implement a communication strategy to inform clients and the private sector of the Network's strategic implementation of service management.

• Management Action Plan 4.2: Develop and implement communication protocols and establish any needed structures in order to engage clients and the private sector in the systematic identification and resolution of issues.

Recommendation 5: The Chief Financial Officer, Public Works and Government Services Canada, should develop and implement an ongoing strategy to assist the business line to improve financial reporting capabilities and to address data comparability issues.

• Management Action Plan 5: Finance Branch will take the following actions to ensure the situation is being addressed:

• Management Action Plan 5.1: The regions and organizations will be consulted to fully understand the issues and to assess their needs in order to improve the financial reporting requirements for the PWGSC's activities and to find solutions to support financial business requirements.

• Management Action Plan 5.2: The centers of expertise in Finance such as Chart of Accounts, Financial Management Advisors as well as SIGMA experts will provide advice and guidance to the regional people in order to assist them with the recording of financial information associated with PWGSC's activities in the financial system and guidelines will be developed in order to ensure standardization across Canada.

ABOUT THE EVALUATION

The evaluation examined PWGSC's heritage conservation activities delivered by the Real Property Branch Heritage Conservation Network. This evaluation had two objectives:

• To determine the relevance of PWGSC's heritage conservation activities delivered by the Heritage Conservation Network: The continued need for the program, its alignment with governmental priorities and its consistency with federal roles and responsibilities.
• To determine the performance of PWGSC's heritage conservation activities delivered by the Heritage Conservation Network: Achieving its expected outcomes and demonstrating efficiency and economy.

Approach

The evaluation was conducted in accordance with the Evaluation Standards of the Government of Canada and PWGSC's Office of Audit and Evaluation. It took place between March and November 2009 and was conducted in three phases: planning, examination, and reporting. To assess the evaluation issues and questions the evaluation team used the following lines of evidence:

Document Review: An initial document review was carried out to gain an understanding of the Heritage Conservation Network and its context to assist in the planning phase. Over 90 documents were reviewed in this process. Data from these documents were collected in a matrix and analyzed. A second phase of the document review was conducted to collect and assess program/Heritage Conservation Network data (such as financial, performance measurement, and other types of data already collected by the Network). The analysis of this data contributed to assessing the success of the Network.

Literature Review: As Canada is the only G8 country that does not have legal protection for federally-owned historic places it was difficult to identify reasonable standards of comparison. Efforts were made to include relevant material to enable an unbiased and neutral assessment.

Interviews: The evaluation team conducted interviews with 20 program stakeholders internal and external to PWGSC. The qualitative analysis of the interviews provided information about the program's activities, outputs, expected outcomes, stakeholders, relevance and performance from the perspective of program managers and client departments.

Client Survey: A client survey was designed, reviewed, and approved by PWGSC Communications and Public Opinion Research and distributed to 65 clients, internal and external, to PWGSC. Thirty six responses were received. The individuals completing the surveys represent their own opinions and experiences with the Heritage Conservation Network.

Other Methodologies: Information obtained from the Federal Heritage Buildings Review Office website and the National Historic Sites websites was used to create templates to capture the number of heritage assets worked on by the Heritage Conservation Network from April 1, 2004 to March 31, 2009. The Heritage Conservation Network supplied the data regarding heritage interventions.

Limitations of the Methodology

Document Review: Discrepancies were noted between the various databases (PWGSC and Parks Canada) used to track the tombstone data for designated heritage assets.

Financial Information: Financial data was not available for all regions. The Atlantic Region was unable to provide data and the Quebec and Ontario Regions were only able to provide data for fiscal year 2008-09. However, when compared to the number of heritage buildings located in each region, the financial information supplied for 2008 09 represented coverage of approximately 87%.

Performance Data: In 2008 09 the Heritage Conservation Directorate produced the Real Property Annual Report on the Stewardship of Federal Heritage Buildings. However the evaluation found data integrity issues with the report caused by discrepancies between various databases used to track these assets such as the Parks Canada FHBRO and PWGSC system. The Parks Canada FHBRO on-line database indicated 170 buildings in the custodianship of PWGSC. At the time of this report, the on-line database had not been updated since 2004 to reflect the disposal of 42 buildings, the addition of another 21 buildings, and the removal of a building that was disposed of by PWGSC in 2002.

Reporting

Findings were documented in a Director's Draft Report which was internally cleared through the Office of Audit and Evaluation's quality assessment function. The Director General, Professional and Technical Programs, was provided with the Director's Draft Report and requested validation of facts and comments on the report. A Chief Audit Executive's Draft Report was prepared and provided to the Associate Deputy Minister of Real Property for acceptance as the Office of Primary Interest. The Office of Primary Interest prepared and submitted a Management Action Plan. The Draft Final Report, including the Management Action Plan, was originally presented to PWGSC's Audit and Evaluation Committee for the Deputy Minister's approval in May 2010. The Final Report will be submitted to the Treasury Board Secretariat and posted on the PWGSC website.

Project Team

The evaluation was conducted by employees of the Office of Audit and Evaluation, overseen by the Director of Evaluation and under the overall direction of the Deputy Chief Oversight Officer.

The evaluation was reviewed by the quality assessment function of the Office of Audit and Evaluation.

APPENDIX A

Table Summary

Main Stakeholders of the Heritage Conservation Network List of stakeholders interested in federal heritage conservation and their role as part of the Heritage Conservation Network.

MAIN STAKEHOLDERS OF THE HERITAGE CONSERVATION NETWORK

Party	Role
Accommodation, Portfolio Management and Real Estate Services, Real Property Branch	Provides stewardship for the PWGSC real property portfolio of 118 buildings (excluding the 30 parliamentary precinct buildings managed by PWGSC's Parliamentary Precinct Branch).
Heritage Conservation Directorate, Real Property Branch	Provides engineering, architectural, landscape architectural and technical conservation advice and expertise to federal custodian departments across Canada. The Directorate is also an advisor to the RPB, Accommodation, Portfolio Management and Real Estate Services Sector. Provides professional and technical advice to Parks Canada Federal Heritage Buildings Review Office to assist it to carry out its regulatory function for federally owned built heritage.
Heritage Conservation Network	Collective term used to refer to the Heritage Conservation Directorate and PWGSC Regional Offices that provide engineering, architectural and technical conservation advice and expertise to federal custodian departments.
Parks Canada	Parks Canada's Mandate: "On behalf of the people of Canada, we protect and present nationally significant examples of Canada's natural and cultural heritage and foster public understanding, appreciation and enjoyment in ways that ensure their ecological and commemorative integrity for present and future generations."
Parks Canada, Federal Heritage Buildings Review Office	The primary objective of the Federal Heritage Buildings Review Office (FHBRO) is to assist federal custodians to protect their heritage buildings, in accordance with the Treasury Board Policy on Management of Real Property.
Parliamentary Precinct Branch	Responsible for the stewardship of the 30 designated federal heritage buildings in the Parliamentary Precinct.
Professional and Technical Services Management, Real Property Branch	Leads a national program of professional and technical services that supports service management and service delivery in the areas of operations and maintenance, planning, design, renovation and construction of federal facilities.
Program Management Sector, Real Property Branch	Responsible for providing Real Property Branch leadership in strategic planning, resource management and policy activities.

APPENDIX B

Table Summary

This table represents the Evaluation Matrix, including evaluation issues, questions, indicators and data collection methods and data sources, developed during the planning phase.

EVALUATION MATRIX

Evaluation Questions	Indicators	Data Sources
Relevance: Continued need for the program
To what extent does the program address a demonstrable need?	1. Degree to which program contributes to the conservation of federal built heritage Percentage of classified or designated federal buildings (as per FHBRO Register) for which Heritage Conservation Directorate (HCD)/Heritage Conservation Network (HCN) provided conservation services during the past five years (preferable if this info was available for a period of five years to compare trends) Percentage of historic sites and monuments (as per total defined by Parks Canada) for which HCD/HCN provided conservation services (preferable if this info was available for a period of five years to compare trends) Percentage of FHBRO evaluation processes in which HCD/HCN has participated (preferable if this info was available for a period of five years to compare trends) 2. Current vs. predicted future demand for the service Number of buildings that will need to be evaluated for heritage character (i.e. over 40 years old) over the next five years Current number of projects per year vs. total number of federal heritage buildings in FHBRO federal database 3. Percent change in revenues over five years Note, move from service delivery to service management may invalidate this performance measure 4. Extent or need for centralized services - who would perform function of each of business lines if program decentralized? Interview with FHBRO representative	1. Document Review Real Property Business Plan Implementation of Corporate Real Estate Model for Heritage Conservation Directorate PWGSC Financial data Speech from the Throne Department of Public Works and Government Services Act Federal Real Property and Federal Immovables Act Historic Sites and Monuments Act Canadian Environmental Assessment Act Policy on the Management of Real Property Policy Framework for the Management of Assets and Acquired Services Common Services Policy 2007 Report of the Auditor General - Status Report Federal Heritage Building Review Office Register 2. Stakeholder interviews and/or surveys Client departments PWGSC Real Property Branch Accommodation, Portfolio Management and Real Estate Services PWGSC Parliamentary Precinct Branch Parks Canada Federal Heritage Building Review Office (FHBRO)
Relevance: Alignment with Government of Canada priorities
Does the program align with Government priorities?	5. Degree to which the program aligns with federal government policies and priorities Review of Acts and Policy instruments Stakeholderinterview with FHBRO 6. Degree to which the program aligns with departmental strategic outcomes (High quality, central programs and services that ensure sound stewardship on behalf of Canadians and meet the program needs of federal institutions). Review of Real Property Business Plan Investment Management Board Terms of Reference Common Services Policy 7. Degree of alignment with Real Property Branch Corporate Real Estate Model Implementation of Corporate Real Estate Model for Heritage Conservation Directorate % Increase in services contracted out to private sector firms	1. Document Review Real Property Business Plan Terms of Reference for the Investment Management Board Implementation of Corporate Real Estate Model for Heritage Conservation Directorate Speech from the Throne Budget 2009 Department of Public Works and Government Services Act Federal Real Property and Federal Immovables Act Historic Sites and Monuments Act Canadian Environmental Assessment Act Policy on the Management of Real Property Policy Framework for the Management of Assets and Acquired Services Common Services Policy 2007 Report of the Auditor General - Status Report 2. Stakeholder interviews and/or surveys Managers of the Heritage Conservation Directorate Regional Offices involved in the delivery of conservation services Parks Canada Federal Heritage Buildings Review Office
Relevance: Alignment with federal roles and responsibilities
Is the program consistent with federal roles and responsibilities?	8. Degree to which the program is consistent with federal roles and responsibilities 2007 Report of Auditor General (why it is important) Review of legislation 9. Evidence that the program is responsive to federal government policies and priorities Departmental policies/procedures to support legislation and TB Policies	1. Document Review Business plans/strategic plans 2. Literature review Speech from the Throne Department of Public Works and Government Services Act Historic Sites and Monuments Act Canadian Environmental Assessment Act Policy on the Management of Real Property Policy Framework for the Management of Assets and Acquired Services Common Services Policy Standards and Guidelines on Conservation of Historic Places in Canada
Efficiency and Economy: Effectiveness
Is the program achieving its intended immediate, intermediate and ultimate outcomes	10. Degree to which the program is meeting performance targets Historical performance for 90% target for billable rate Historical performance for Quality Management Audits 11. Change in knowledge transfer activities for heritage conservation, internal and external to Government of Canada (GC) Number and scope of knowledge transfer activities Historical spending on knowledge transfer activities (three to five years) Number of employee participating in knowledge transfer activities per year (past three to five years) 12. Change in number of private sector firms qualified to perform heritage interventions 13. Percentage change of salaries versus operating expenditures 14. Change in nature of services provided over time (i.e. service management activities vs. service delivery activities) 15. Recommendations from HCD are included in PWGSC asset management plans/presented to the Investment Management Board 16. Recommendations of HCD are carried out by Other Government Departments (OGD) clients (excluding Parks Canada) 17. Recommendations of HCD are carried out by Parks Canada 18. Recommendations for Parliamentary Precinct projects carried out by Parliamentary Precinct Branch (PPB). 19. Rate of compliance with PWGSC Heritage Policy (as per compliance report) 20. Level of client satisfaction with services: Percentage of projects delivered on time Percentage of projects delivered on budget Percentage of clients satisfied overall with services Percentage of clients indicating they would use services again Percentage of clients incorporating HCD advice in their investment plans Benchmark results against other RPB services (as per Real Property Optional Services (RPOS) client satisfaction survey)	1. Document Review Parks Canada Year End Report Parliamentary Precinct Year End Report Reports/Recommendations to PWGSC Investment Management Board Program performance targets and results (if available) Historical data on knowledge transfer initiatives Procurement data on the number of qualified bidders per heritage contract awarded Historical Financial data Real Property Optional Services Client Satisfaction Survey 2. Stakeholder interviews and/or survey Client departments PWGSC Real Property Branch Accommodation, Portfolio Management and Real Estate Services PWGSC Parliamentary Precinct Branch
Demonstration of Efficiency and Economy
Can the program achieve the same results at a lower cost?	21. Assessment of resource utilization and the production of outputs and progress towards expected outcomes 22. Evidence that the program uses performance measurement to inform decision making 23. Breakdown of costs and revenues by regions/business lines 24. Comparison of PWGSC spending vs Parks Canada and Department of National Defence (DND) (overall spending per building) 25. Percent of budgetary spending of other countries	1. Document Review Business plans/strategic plans Financial plans and reports 2. Literature Review Comparison to other jurisdictions 3. Stakeholder interviews and/or survey Senior program management Parks Canada Federal Heritage Buildings Review Office
Are there alternative arrangements for reaching the same results more cost-effectively?	26. Is the program structured appropriately to deliver the services in the most effective way? 27. How do other countries define heritage? 28. Does PWGSC follow internationally recognized practices? 29. How do other countries prioritize heritage interventions?	1. Document Review Business plans/strategic plans Financial plans and reports 2. Literature Review Findings in other jurisdictions with respect to alternative arrangements 3. Stakeholder interviews and/or survey Senior program management Parks Canada Federal Heritage Buildings Review Office Interviews with other jurisdictions