2009-603 Final Report Evaluation of Selected PWGSC Large Space Office Accommodation Projects (Final Report)

March 24, 2011

Table of Contents

• Main points
• Recommendations and Management Action Plan
• Introduction
• Profile
  • Background
  • Governance
  • Activities
  • Authority
  • Resources
  • Stakeholders
  • Logic Model
• Focus of the Evaluation
• Findings and Conclusions
  • Relevance
    • Continuing Need
    • Alignment with Federal and Departmental Priorities
    • Appropriate Role and Responsibility for the Federal Government
    • Conclusions: Relevance
  • Performance
    • Outcome Achievement
    • Project Delivery
    • Client Satisfaction with Real Property Branch (RPB) Services and Facilities
    • Management Policies and Practices & Project Management Skills and Capacity
    • Real Property Investment
    • Real Property Portfolio Optimization
    • Strategic Portfolio Management
    • Efficiency and Economy
    • Efficiency
    • Economy
    • Lessons Learned From Other Jurisdictions
    • Conclusions: Performance
• General Conclusions
• Management Response
• About the Evaluation
• Appendix A: Evaluation Matrix

Main Points

What was evaluated

i. Public Works and Government Services Canada (PWGSC) provides office accommodation space to federal departments and organisations. The evaluation examined the ongoing relevance and performance of Real Property Branch's (RPB) provision of Large Space Office Accommodation projects.

ii. Due to the size of the real property portfolio, case studies of six accommodation projects were undertaken. Three of the projects examined were located in the National Capital Area (Skyline, 3000 Merivale, and 1600 Star Top) and three in the Atlantic and Western Regions (Shediac, Charlottetown, and Yellowknife). RPB undertook the Skyline project using the full-funding model while the remaining five projects utilized the split-funding model.

iii. In the PWGSC Program Activity Architecture, large space accommodations is situated under Federal Accommodation, Federal Holdings and Professional and Technical Services.

Why it is important

iv. Based on the 1996 Department of Public Works and Government Services Act and Treasury Board's Common Services Policy, RPB is the mandated provider of office accommodation services to federal government departments and agencies. This mandate sees RPB manage one of Canada's largest real estate portfolios, coordinating $8.6 billion in federal real property for 110 federal organizations and 265,000 public servants in 1,800 locations across Canada. RPB is the sole provider of safe, healthy, and affordable accommodations to federal organizations. These accommodations support the effective delivery of federal programs and services.

v. As real property management is a core function of PWGSC, it is essential that RPB demonstrate value for money and rigorous stewardship in the use of public funds for the planning and delivery of large scale office accommodation projects. Together, the six projects examined in the case studies cover close to 190,000m² of usable space, $692 million in project costs, and accommodations for over 8,000 federal public servants.

What we found

vi. The evaluation found that there is a continuing need for RPB's role in large space office accommodation projects; that this role is aligned with the policies and priorities of the Government of Canada; and that both legislation and central agency policy mandate RPB's role in the provision of safe, healthy, and affordable accommodation of federal public servants. RPB seeks to maximize value for money and ensure rigorous stewardship in the provision of office accommodations through its systematic consideration of a range of delivery options; a large capital program; plans for further projects in the next five years; its focus on 'on time, on budget, and within scope' delivery; and its implementation of accommodation standards.

vii. RPB achieved considerable success with the Star Top, Shediac, and Yellowknife projects. However, there was room for improvement in project implementation with the Skyline, Charlottetown, and Merivale projects. Case studies revealed a number of challenges in front-end planning, client management, and risk management. However, the National Project Management System is increasing rigour and discipline in the management of RPB's large space office projects.

viii. The literature review revealed that no single delivery model is appropriate for all circumstances and that a number of delivery strategies, options, and practices could potentially be used to strengthen the efficiency and economy of RPB project outcomes. These options include public private partnerships, gateway reviews, management contracting, construction management, and design-build.

ix. The funding model chosen for the Skyline project played a minor role in the achievement of project outcomes by enabling comprehensive and consistent project reporting. While RPB experienced difficulties with accountability for decisions on fit-up expenditures, this unique funding model was not a major contributor to project outcomes. Although the Skyline project's use of the full-funding model served to increase complexity, this was a result of it being the first accommodation project to use this model. Relative complexity is expected to decline through increased familiarity from future use. It was found that the funding model used for a given project had little impact on the achievement of immediate, intermediate, and long-term outcomes.

Management Response

x. The Real Property Branch accepts the evaluation findings and intends to act on the recommendations of the evaluation by implementing their Management Action Plan detailed as follows.

RECOMMENDATIONS AND MANAGEMENT ACTION PLAN

Recommendation 1: The Assistant Deputy Minister, Real Property Branch should enhance lessons learned practices to further improve project management practices on large space accommodation projects.

• Management Action Plan 1.1: The National Project Management System Quality Management Area is being updated to enhance lessons learned practices.

  The improved lessons learned practices will be linked to the National Project Management System Continual Improvement process already in-place to ensure that improvement of project management practices will be incorporated into the National Project Management System for the benefit of other project teams.

Recommendation 2: In using the full-funding model, the Assistant Deputy Minister, Real Property Branch, should clarify accountability structures.

• Management Action Plan 2.1: The National Project Management System framework requires projects to complete a project charter during the initial stages of a project. The purpose of the project charter is to create a high level agreement between the client department and PWGSC that is intended to establish a framework for the implementation. The intent of the charter is to obtain client agreement on all of the key parameters of the project (scope, time, and cost) prior to Lease Project Approval (LPA) or Preliminary Project Approval (PPA) and to confirm client commitment to expend client funds and resources in developing and/or delivering the defined project. For PWGSC space projects, it is to be utilized in conjunction with the Statement of Requirements to articulate and confirm understanding and agreement related to project goals and objectives, guiding principles, roles and responsibilities, and issue resolution processes. In complex cases, projects will complete a Memorandum of Understanding in addition to a project charter. The Memorandum of Understanding will define additional objectives complete with related roles and responsibilities.

  It is critical that projects identify proper accountabilities, priorities, governance and roles and responsibilities in this document prior to commencement of the project. The full-funding model would require a more comprehensive project charter in order to clarify accountability with regards to processes and procedures for dealing with project issues and reporting.

Recommendation 3: In using the full-funding model, the Assistant Deputy Minister, Real Property Branch, should develop guidelines for the application of this model to address project management issues: project capacity for cost tracking and reporting; the most appropriate size and complexity for the projects involved; client management issues, such as client readiness to accept RPB's enhanced role and to utilize RPB processes and systems, and the need for clear accountabilities for both RPB and client departments.

• Management Action Plan 3.1: As stated in Recommendation 2, the National Project Management System framework requires projects to complete a project charter during the initial stages of a project. The purpose of the project charter is to create a high level agreement between the client department and PWGSC that is intended to establish a framework for the implementation. This document should be utilized by project to address any and all roles and responsibilities including clear accountability, priorities and governance for all project parameters.

  When properly used, the project charter would address all issues that occurred during the implementation of the Skyline project with regards to cost reporting, accountabilities, roles and responsibilities and processes.

  PWGSC will leverage existing reporting systems including SIGMA (SAP) to meet specific project reporting requirements. This will ensure that existing financial controls will ensure accurate and reliable reporting.

Introduction

1. This report presents the results of the Evaluation of Selected Public Works and Government Services Canada (PWGSC) Large Space Office Accommodation Projects. The evaluation is based on six case studies of Real Property Branch (RPB) large space office accommodation projects since 2000. The Audit and Evaluation Committee of PWGSC approved this evaluation as part of the 2009/10 to 2013/14 Risk-Based Multi-Year Audit and Evaluation Plan. The evaluation was conducted in accordance with the evaluation standards of the Government of Canada and the Office of Audit and Evaluation at PWGSC.

Profile

Background

2. PWGSC manages one of Canada's largest real estate portfolios, coordinating $8.6 billion in federal real property for 110 federal organizations and 265,000 public servants in 1,800 locations across Canada.

3. In the PWGSC Program Activity Architecture, planning and delivery of large space accommodation projects is an activity situated in Federal Accommodation, Federal Holdings, and Professional and Technical Services, under Accommodation and Real Property Assets Management.

4. PWGSC does not formally define what constitutes 'large space'. For the purposes of the evaluation, case studies of three types of office accommodation projects were selected as they provided a broad coverage of RPB projects. These include large projects in which RPB had a lead role but which did not require Treasury Board approval, as they fell below the Treasury Board definition of large projects (i.e., $30 million); projects over $30 million for which Treasury Board approval was required; and projects under $20 million where relevant lessons learned could be gained.

5. The evaluation of large space office accommodation projects focuses on six large space office accommodation projects. These six projects represent a wide variety of sizes, timeframes, delivery options, and funding models. The projects ranged from $1 million to $242 million in up-front costs^{Footnote 1} (e.g., land acquisition, construction, and preparation of accommodation for initial occupancy) and accommodated between 185 and 3,380 full-time equivalents (FTEs). All but one of the projects (3000 Merivale) has been completed. They are summarized in Exhibit 1.

Table Summary

This table is a summary of the status of the project, the delivery option and funding model used for the six large office accommodation projects that the evaluation focused on.

Exhibit 1: Summary of Six Large Scale Office Accommodation Projects: Delivery Option and Funding Model

Project	Status	Delivery Option		Funding Model
		Crown-owned	Capital lease	Full-funding	Split-funding
Skyline (National Capital Region (NCR))	Completed	X*Footnote 2		X
3000 Merivale (NCR)	Ongoing		X		X
1600 Star Top (NCR)	Completed		X		X
Jean Canfield Building (Charlottetown, Prince Edward Island (PEI))	Completed	X			X
Michel C. Leger Building (Shediac, New Brunswick (NB))	Completed		X		X
Greenstone Building (Yellowknife, Northwest Territories (NWT))	Completed	X			X

6. PWGSC provides federal departments and organizations with affordable work environments, a full range of real property services, and strategic and expert advice t hat supports the Government of Canada in the delivery of programs to Canadians. Based on the needs of client departments, RPB provides real property solutions while respecting Treasury Board Secretariat (TBS) and PWGSC policies and priorities. These six projects also illustrate a full range of client needs, including accommodating increased staffing and operations, consolidating federal space, and increasing facility functionality.

7. Of the six case studies, three projects were Crown-owned ^{Footnote 3} (Skyline, Charlottetown, and Yellowknife) and three were capital lease^{Footnote 4} (3000 Merivale, Star Top, and Shediac).

8. Five projects (3000 Merivale, Star Top, Shediac, Charlottetown, and Yellowknife) were implemented using the split-funding model and one (Skyline) used the full-funding model. The split-funding model refers to the situation in which RPB is responsible for, and funds, the base building and the tenants focus on tenant fit-up. Under this model, tenant fit-up costs^{Footnote 5} such as Information Management and Information Technology, equipment, and furniture are funded by client departments. The split-funding model is used in the vast majority of office accommodation projects that RPB undertakes.

9. RPB developed a new full-funding model for fit-up in 2005. The initial implementation of this model was the Skyline project. Under this model, TBS transferred to PWGSC the funding required to cover the fit-up and moving costs for Agriculture and Agri-food Canada and the Canadian Food Inspection Agency, including security, furniture, equipment, and the relocation and implementation of information management and information technology infrastructures.

Governance

10. The responsibility for large space office accommodations rests with the Director General (DG), Accommodation, Portfolio Management and Real Estate Services Sector, who reports directly to the Assistant Deputy Minister, RPB. Within the sector, the DG oversees four directorates: Accommodation Management, National Portfolio Management, Valuation and Payment in Lieu of Taxes Programs, and National Real Estate Services. Each is headed by a director who reports to the DG.

Activities

11. RPB's large space office accommodation activities are divided across six Program Activity Architecture sub-activities:

• Federal Accommodation: Provision of general-purpose office and common-use facilities and related real property services to departments and agencies; setting fit-up and accommodation standards for federal office accommodation.

• Federal Holdings: Acting as a manager and custodian for a diverse portfolio of federal facilities, such as housing, special properties, marine structures, land transportation, and surplus properties.

• Professional and Technical Services: These services fall into three broad categories (advisory, project delivery, and property and facility management) and target six major asset-based market segments (office, transportation, secure facilities, laboratories, land, and culture).

• Disposition Incentives: Assisting routine disposals of federal real property.

• Payments-in-lieu of Taxes: Involves the payment of funds to local taxing authorities across Canada (where custodian departments have a real property presence) in lieu of municipal property taxes.

• Parliamentary Precinct: Custodial activities related to the ongoing maintenance and renovation of the Parliamentary Precinct buildings on Parliament Hill and the North side of Sparks Street.

Authorities

12. Large space office accommodations are provided by PWGSC's Real Property Branch under the authority of the Department of Public Works and Government Services Act, which conferred upon the Minister of Public Works and Government Services the mandate for "the provision of accommodation and other facilities for departments…[and] the construction, maintenance, and repair of public works, federal real property, and federal immovables." In addition, the 2005 Treasury Board Common Services Policy states that "PWGSC is the designated custodian of general-purpose office facilities…under its residual responsibility for federal real property."

Resources

13. According to the PWGSC 2010/11 Report on Plans and Priorities, the total Accommodation and Real Property Asset Management program activity is budgeted at $4.5 billion in gross expenditures in 2010/11.

Stakeholders

14. The list of stakeholders is extensive, encompassing nearly the entire federal government. PWGSC's Real Property Branch currently provides accommodation services to 265,000 employees across 110 federal departments and agencies.

Logic Model

15. A logic model is a visual representation that links a program's activities, outputs, and outcomes; provides a systematic and visual method of illustrating program theory; and shows the logic of how a program, policy, or initiative is expected to achieve its objectives. It also provides the basis for developing performance measurement and evaluation strategies, including the evaluation matrix.

16. A logic model for the Large Space Office Accommodation Program, presented in Exhibit 2, was developed as part of the evaluation framework conducted prior to this evaluation. The logic model was based on a document review and interviews with program managers and was subsequently validated with senior management.

Exhibit 2: Logic Model

Text description of the Exhibit 2: Logic Model is available on a separate page.

Focus of the Evaluation

17. The objective of this evaluation was to determine the ongoing relevance and performance of RPB's Large Space Office Accommodation activities, including economy and efficiency, in achieving expected outcomes. The evaluation also explored alternative ways of achieving the expected results.

18. Due to the size of the real property portfolio, a case study approach was used. Projects were selected to provide an accurate cross-section of variables, including location, price, customer requirements, timeframes, and funding models.

19. The following is a list of the projects that were examined in this evaluation: Skyline, 3000 Merivale, 1600 Star Top (National Capital Region); Shediac, Charlottetown (Atlantic); and Yellowknife (Western). For purposes of this evaluation report, 'program' refers to PWGSC activities undertaken in relation to the six aforementioned large space office accommodation projects.

20. An evaluation matrix—including evaluation issues, questions, indicators, and data sources—was developed during the planning phase. Multiple lines of evidence were used to assess the program. These include:

1. Document Review: More than 200 documents were collected and reviewed in this evaluation. Most contained information and data on the six selected projects. Some RPB documents addressed broader approaches to managing the RPB real property portfolio. The documents included approval documents, project close-out documents, project charters, and client satisfaction surveys. The review also included external client satisfaction surveys conducted by RPB regarding project management services for Skyline, 3000 Merivale, and Charlottetown.

2. Literature Review: Over 50 documents from international jurisdictions were reviewed in order to gain a better understanding of delivery options and project delivery best practices. In addition, Canadian documents were reviewed that provided information on the use of a variety of delivery options for large public infrastructure projects in Canada.

3. Stakeholder Interviews: Thirty-one stakeholders were interviewed for the evaluation. These consisted of stakeholders from TBS (2), as well as the six projects: Skyline (14), Yellowknife (5), Charlottetown (3), Shediac (2), 3000 Merivale (3), 1600 Star Top (2). Interviews were conducted with RPB as well as client departments for all projects except for 1600 Star Top. For this project stakeholder interviews were limited to RPB personnel due to non-availability of Department of National Defence (DND) personnel, many of whom are in rotational positions. The stakeholders were asked a number of questions relating to the relevance, success, and cost-effectiveness of the large space office accommodation projects.

21. More information on the approach and methodologies used to conduct this evaluation can be found in the About the Evaluation section at the end of this report. The evaluation matrix is provided in Appendix A: Evaluation Matrix.

Findings and Conclusions

22. The findings and conclusions below are based on the multiple lines of evidence used during the evaluation. They are presented by evaluation issue (relevance and performance).

Relevance

23. Relevance is measured by the extent to which the program: addresses a demonstrable and continuing need; is aligned with federal and departmental priorities; and is an appropriate role for the federal government.

Continuing Need

24. Continuing need is assessed by three primary elements: continued relevance of original program rationale, legislative or policy requirements, and use of the program's services. Based on these criteria, the evaluation team found a continuing need for the program.

25. PWGSC was formed in 1993 through the amalgamation of Public Works Canada, the Translation Bureau, Supply and Services Canada, and the Government Telecommunications Agency. The mandate of PWGSC is to be a common service agency for the Government of Canada's various departments, agencies, and boards. The Department of Public Works and Government Services Act conferred upon the Minister of Public Works and Government Services the mandate for "the provision of accommodation and other facilities for departments…[and] the construction, maintenance, and repair of public works, federal real property, and federal immovables." At the present time, real property and accommodations services continue to be the primary activity of the Department, with the Federal Accommodation sub-activity accounting for 40% of direct program spending.

26. Additional legislative and policy requirements emphasize the continuing need for PWGSC to provide federal accommodation services. The 2005 Treasury Board Common Services Policy states that "PWGSC is the designated custodian of general-purpose office facilities…under its residual responsibility for federal real property." As per its 2010/11 Report on Plans and Priorities, PWGSC offers a mandatory service for the provision of office accommodation to government departments and agencies, with the overall goal of providing "safe, healthy, and affordable facilities." As such, PWGSC has reaffirmed the original rationale of providing accommodation to federal clients.

27. Demand for federal accommodation services continues to exist. PWGSC will be engaged in large space office accommodation projects for the foreseeable future. According to the draft PWGSC Investment Plan 2010/11 to 2014/15 RPB has 11 large space office accommodation projects planned, totalling $50 million, while Parliamentary Precinct Branch has another five projects planned. Key client stakeholders interviewed for this evaluation highlighted the importance of RPB's role in the six projects examined and that RPB's services continue to be required. Key stakeholders from Treasury Board Secretariat recognized the importance of RPB's role in identifying and analyzing delivery options and in selecting a preferred option.

28. Overall, the evaluation found that there is a continued need for PWGSC to provide office accommodations, including large space office accommodations, to federal clients.

Alignment with Federal and Departmental Priorities

29. Alignment with federal and departmental priorities is determined by assessing the degree to which the issue the program is intended to address is discussed in both federal and departmental priority-setting reports and documents.

30. The provision of efficient, cost-effective accommodation for public servants directly supports the federal government's objectives in Budget 2010 to modernize federal infrastructure while reducing costs and leveraging economies of scale by consolidating office accommodations.

31. PWGSC's 2008/09 Departmental Performance Report indicates that the Federal Accommodation and Holdings activity "manages real property assets under the custody of PWGSC on behalf of federal organizations [to provide] work environments and facilities that enable federal organizations to deliver programs and services." The PWGSC 2010/11 Report on Plans and Priorities indicates that the Accommodation and Real Property Assets Management Program is "how PWGSC provides departments and agencies with office and common use accommodations."

32. Overall, the evaluation found that RPB's activities in the provision of large space office accommodation align with federal and departmental priorities.

Appropriate Role and Responsibility for the Federal Government

33. To determine whether or not the program is aligned with the roles and responsibilities of the federal government, three elements were examined: whether the responsibility for the program could be transferred to another level of government (i.e., to the provinces); whether the responsibility for the program could be transferred to the private sector; and whether responsibility for the program could be decentralized to individual departments and agencies.

34. Provincial Devolution: Other levels of government do not have jurisdiction over federal assets, nor do they have a responsibility to ensure that federal government departments have adequate accommodations for their employees. This responsibility lies solely with the federal government.

35. Private Sector: The private sector has an important role to play in the provision of large space accommodation. The Government of Canada is continually seeking ways to improve service and engage with the private sector to ensure a strong public-private relationship, especially in the provision of federal accommodations. While the private sector is leveraged by PWGSC in fulfilling its mandate, the private sector role is limited to service delivery. The ultimate responsibility for providing office accommodation to its departments and organizations rests with the federal government. As the overall objective of the private sector is to maximize profit for shareholders, it is not appropriate to privatize the responsibility for ensuring that federal accommodations meet the needs of government clients; respect federal policies, guidelines, and directives regarding federal accommodations; and provide value for money. The federal government has a unique obligation to provide sound stewardship of federal public funds. A federal government department such as PWGSC is in the best position to consistently apply federal policies, technical codes, and standards, as well as assume responsibility for such applications.

36. Federal Decentralization: Decentralization to client departments would not likely result in greater efficiencies for the Government of Canada. Though the six projects represent different funding relationships between PWGSC and client departments, all projects are centrally managed within PWGSC. RPB is a locus of government expertise in real property and real property management. In addition, PWGSC is responsible for ensuring that Treasury Board accommodation directives are respected. As there is increased complexity and risk with large space accommodation projects, it is especially important that individuals involved with the project have the expertise and knowledge to maximize value for money and minimize project overruns. This centralized or centre of expertise approach is reinforced by the Common Services Policy that states, "PWGSC is the designated custodian of general-purpose office facilities provided on an obligatory basis to departments and agencies and of other special federal facilities assigned to PWGSC under its residual responsibility for federal real property. This function is considered a program of government, not a common service."

37. Overall, the evaluation found that the provision of large space accommodations is an appropriate role and responsibility for the federal government. It also found that a centralized federal organization, such as PWGSC, is the most appropriate model to ensure the ongoing maximization of value for money while providing appropriate accommodations to federal institutions.

Conclusions: Relevance

38. In conclusion, there is a continuing need for PWGSC to address the requirements of federal organizations for large space office accommodation. RPB's activities related to large space office accommodation investments continue to be aligned with PWGSC's mandate and government policies and priorities. RPB's roles and responsibilities in providing client departments with large space office accommodation are appropriate.

Performance

39. Performance is the extent to which a program or initiative is successful in achieving its objectives and the degree to which it is able to do so in a cost-effective manner that demonstrates efficiency and economy. This evaluation examined the Large Space Office Accommodation Program's achievement of its immediate, intermediate, and long-term outcomes, as shown in the Logic Model contained in the Profile section.

Outcome Achievement

40. The outcomes presented in the logic model are designed so that their achievement will aid the Department in achieving its strategic outcome. As such, the evaluation examined the degree to which the program's activities are sufficient for achieving its intended outcomes.

Project Delivery

41. A project was considered to be on time if the timeframes stated in the approved Effective Project Approval (EPA) or Lease Project Approval (LPA) documentation were met. Based on the above criteria, three of the five completed projects met this requirement and two (Skyline and Charlottetown) did not. At the time of the evaluation, the ongoing project at 3000 Merivale was experiencing scheduling issues.

42. A project was considered to be on budget if actual costs were less than or equal to projected costs in the approved EPA/LPA. While all five completed projects met this criteria, three (Yellowknife, Charlottetown, and Skyline) did so only by virtue of updated budget approvals to match increasing costs. Every budget revision required Treasury Board Approval, and was accompanied by an explanation of why assumptions had changed. With Skyline, it is important to note that cost estimates were generic as there was no precedent for a project of similar scale. Significant time was needed by the project team to develop costs for items such as base systems upgrades, furniture and equipment, security, moves, and critical information technology work, all of which occurred after the initial estimates. By the end of the project, the list of costs had become much longer than originally anticipated. A summary of the budget and schedule revisions for the completed projects is presented in Exhibit 3.

Table Summary

This table identifies the complexity, delivery option, budget and schedule revisions for each project. A project was considered to be on time if the timeframes stated in the approved Effective Project Approval (EPA) or Lease Project Approval (LPS) were met. A project was considered to be on budget if actual costs were less than or equal to projected cost in the approved EPA/LPA.

Exhibit 3: Summary of Budget and Schedule Revisions

Project	Complexity	Delivery Option	Initial Estimate		Number of Budget Revisions (EPAs/ LPAs)	Final
			Budget (in million $)	In-Service Date		Budget	In-Service Date
Skyline (NCR)	High	Originally Crown-Owned**Footnote 6	$117.4	April 2006	5	$121.5	March 2008
1600 Star Top (NCR)	Moderate	Capital Lease	$117.1	July 2004	1	$117.1	July 2004
Jean Canfield Building (Charlottetown, PEI)	Moderate	Crown-owned	$48.0	March 2006	3	$56.4	April 2008
Michel C. Leger Building (Shediac, NB)	Low	Capital Lease	$10.6	December 2007	1	$10.6	December 2007
Greenstone Building (Yellowknife, NWT)	Low	Crown-owned	$28.0	Early 2005	2	$28.0 plus 2.5%	October 2005

43. A project was considered to be within scope if it met approved EPA/LPA objectives. Project scope includes FTE occupancy capacity, as well as space requirements for offices, storage, parking, and special purpose space. Based on this criteria, four (Skyline, Shediac, Star Top, and Yellowknife) of the five completed projects were finished within EPA scope. For the remaining completed project (Charlottetown) the scope was reduced (removal of 60 planned parking spaces) to meet the budget. The ongoing project at 3000 Merivale is experiencing scope issues. At the time of the evaluation, it appeared that the approved EPA occupancy requirement of 3,800 FTEs would not be met. The building will accommodate only 3,200 FTEs—a 15.8% reduction.

44. Project complexity was based on a broad spectrum of criteria: overall budget, number of budget revisions, extent of client requirements, building size (usable area), and number of FTE accommodations, as well as other unique circumstances.

• The Skyline project was rated High complexity due to the large number of budget revisions and extensive client involvement. The delivery method was switched from Crown-owned to capital lease, further adding to the project's complexity. The projected usable area and FTE accommodations for Skyline were the highest of the six projects examined. Skyline was also a unique project: it utilized the full-funding model (whereby PWGSC was responsible for client fit-up in addition to normal accommodation activities while decisions were handled by a joint committee between PWGSC and the client departments), which increased complexity. The added necessity of co-locating the headquarters of both Agriculture and Agri-food Canada and the Canadian Food Inspection Agency also increased the workload and complexity of this project through increased client requirements. While the full-funding model contributed to complexity, it is expected that familiarity with the full-funding model will increase through future use, reducing complexity.

• The Charlottetown project was rated Moderate. While the budget was less than half that of the larger projects in the National Capital Area, Charlottetown experienced a two year delay as a result of contaminated soil on site. In addition, Charlottetown was subject to three budget revisions. Ultimately, a scope reduction resulting in the elimination of the planned underground parking was necessary in order to remain within budget.

• 1600 Star Top received a Moderate complexity rating. While its budget was only slightly smaller than that of Skyline, Star Top was a straightforward project with only one budget revision. Although RPB experienced some difficulty with client requirements and needs identification, issues were resolved in the early stages and did not result in significant budget, scope, or schedule changes. In addition, Star Top's usable area was only 28% the size of Skyline's, and was designed to accommodate only 900 FTEs, while Skyline was built to accommodate 3,380. Although the budgets were comparable, Star Top used the traditional split-funding model, which served to reduce complexity.

• Shediac and Yellowknife both received a Low complexity rating. Both projects had small budgets ($10 million for Shediac, $28 million for Yellowknife) and received comparatively few budget revisions. As well, both these projects were for much smaller scale accommodations (3,912 square meters in Shediac and 7,058 square meters in Yellowknife).

45. Overall, the evaluation found that while the five projects were completed on budget—albeit in three projects only by virtue of updating the budget estimates as the costs increased—RPB experienced difficulties in achieving on-time objectives for three projects and scope objectives for one project. While some projects demonstrated higher levels of risk and complexity, the evaluation was unable to demonstrate a direct correlation between complexity and lower project success ratings.

Client Satisfaction with RPB Services and Facilities

46. Stakeholders provided no negative feedback regarding the moves to new accommodations. Client departments noted the usual disruptions of moving (e.g., packing, labelling) but indicated that the moves were handled with minimal disruption.

47. Despite cost and scheduling issues, stakeholder interviews confirm clients were satisfied with accommodations in all five completed projects in terms of the provision of productive workspaces and facilitation of program delivery for client organizations. The only completed client occupant survey was for the Yellowknife project and this indicated—with some caveats—a significant degree of overall satisfaction with accommodation.

48. However, during interviews clients expressed dissatisfaction over the implementation of three of the six projects (3000 Merivale, Charlottetown, and Skyline). Areas of client dissatisfaction included planning, responsiveness, and quality assurance.

49. Stakeholders for 3000 Merivale indicated that key objectives for this project will not be met. There will be a lower than planned occupancy capacity and the Royal Canadian Mounted Police (RCMP) National Operations Centre will not be integrated into the new complex as planned. The final outcomes of this project are not yet apparent, as the project is not scheduled to be completed until June 2011.

50. Client satisfaction surveys conducted by RPB for the Charlottetown and Skyline projects reflected lower levels of satisfaction with the project implementation process, particularly with respect to project scheduling and risk management.

51. However, satisfaction improved over time at Skyline. In the end, project management services were considered successful by both RPB and its clients. Regarding the 3000 Merivale project (in the implementation phase at the time of the evaluation) the client indicated that the situation had improved.

52. Overall, client satisfaction was mixed. While all clients expressed satisfaction with moving and accommodations, clients expressed dissatisfaction over the implementation of the Merivale, Charlottetown, and Skyline projects.

Management Policies and Practices & Project Management Skills and Capacity

53. Clients had mixed views on RPB's skills and capacities in project management. Based on stakeholder interviews, views on the quality of RPB's project management fell into two categories: clients recognizing quality project management and clients expressing room for improvement in project management.

54. Shediac, Yellowknife, and 1600 Star Top were projects where clients recognized that RPB provided good project management. Interviews for these projects indicated that RPB was perceived by clients as a good project manager. This is consistent with projects that were delivered on time, on budget, and within scope.

55. RPB demonstrated some best practices in all six projects:

• Yellowknife was the first building in the Canadian north, and the first building in the federal office accommodation portfolio, to receive a gold rating in the Leadership in Energy and Environmental Design building certification system;

• Shediac used an arms-length process for assessing tenders and for awarding a long-term capital lease contract to the most appropriate real property developer;

• Skyline utilized innovations in project management. This included a master schedule and database developed in-house by the RPB team to keep the project on track, and the co-location on site of the core project management team. These new measures facilitated communication and understanding between all parties involved (PWGSC, Canadian Food Inspection Agency, and Agriculture and Agri-food Canada);

• 3000 Merivale utilized the PWGSC National Project Management System, methodological requirements, and lessons learned for each phase of the project;

• 1600 Star Top demonstrated the value of a good working relationship between RPB, the contractor, and the client. This enabled the project to overcome initial deficiencies in needs identification and helped deliver the project on time; and

• Charlottetown was designed to Leadership in Energy and Environmental Design gold certification standards and used an innovative integrated facility development approach.

56. Alternatively, regarding Skyline (completed) and 3000 Merivale (ongoing), clients indicated that RPB's project management could be improved. Interviews showed a lower level of client satisfaction with RPB's project management services. These projects experienced challenges as a result of insufficient front-end planning and risk management, changes from original scope, and evolutions in cost from EPA through to completion.

57. Weaknesses in client management, project management, and risk management were identified for all six projects.

• Client Management - Limited use of client satisfaction surveys: While client satisfaction surveys are considered standard practice, only two projects (Charlottetown and Skyline) had completed these surveys and only Yellowknife had completed an occupancy survey. As the Skyline project was completed in December 2009, it was too early to conduct an occupancy survey.

• Client Management - Challenge of managing high client expectations for both involvement and outcomes: Skyline and 3000 Merivale were characterized by high client expectations related to service delivery and facility requirements, which RPB had difficulty meeting. These projects were also marked by a high level of client involvement in the management of the project, which at times strained working relationships.

• Client Management - Challenge of changing client requirements during the life of the project: With extended project delivery times—seven years for the Skyline project—there is an increased likelihood of change as a result of evolving client programs and needs. These changes caused disruption to the planning, funding, and scheduling of the project. In some particularly large projects RPB had difficulty managing these fluid requirements.

• Client Management - Insufficient front-end planning: In three projects (Skyline, 3000 Merivale, and Charlottetown) difficulties in front-end planning contributed to problems later in the projects. These included delays in project delivery, revisions to scope, and changes to approved project budgets. In other cases, stakeholders suggested that RPB did not take client views fully into account during the front-end planning process.

• Project Management - Insufficient use of standard project documents: Some TBS management practices were not followed. In the six projects standard project documents were absent. Four (Yellowknife, Charlottetown, Shediac, and 1600 Star Top) lacked 'evergreen' risk management plans, close-out reports, and reports on lessons learned, while three projects (Yellowknife, Star Top, and Shediac) were unable to produce approved project charters.

• Project Management - Challenges with project management personnel/expertise: Project success can be adversely affected by a shortage of project management personnel, insufficient expertise, or excessive staff turnover. Some clients noticed a lack of expertise to manage all functions and a loss of project knowledge as a result of staff turnover. In a number of RPB interviews, it was also noted that RPB's internal capacity to manage large projects has been reduced. Some RPB staff commented that "retirements and transfers from PWGSC real property have led to diminishing real property expertise in high value functions such as asset management and project management." High staff turnover will likely continue to be a challenge during the coming years.

• Risk Management - Insufficient risk management: While RPB does have some risk management practices as part of its project management approach, these practices are not always followed. Projects failed to demonstrate advanced capability in risk management, such as the use of 'evergreen' risk registers, which allow for the evolving identification, assessment, and mitigation of risks throughout the project. Even where practices had been followed, they have not always been effective in managing risk. Weaknesses were evident in the original risk and project management plan for Skyline.

58. In 2006, RPB established the National Project Management System to address management issues. The National Project Management System has become the standard methodology for project management. Although five projects pre-date the implementation of the National Project Management System, 3000 Merivale applied the system from the beginning and several other projects adopted the methodology during implementation.

59. The National Project Management System is intended to improve project management through the establishment of set control points with attendant deliverables and approvals during project planning and implementation. This allows for effective progress monitoring and timely decisions concerning corrective action. The requirement to obtain sign-off on key deliverables ensures projects receive greater scrutiny at all stages of delivery. Furthermore, the National Project Management System supports improved project management of client requirements and expectations through the Statement of Requirements and Preliminary Project Plan, as well as increased control over project risks through the Investment Analysis Report.

60. It should be noted that TBS recently awarded PWGSC a rating of three out of four on its Organizational Project Management Capacity Assessment based on the strength of the National Project Management System in promoting good project management practices. As well, in the Spring 2010 Report on the rehabilitation of the Parliament Buildings, the Auditor General of Canada noted that, "PWGSC has been implementing the National Project Management System to manage its projects. Overall…project management practices…were generally sound." The Auditor General also found that while PWGSC has improved its costing methodology and estimates based on lessons from other projects, it would benefit from a more comprehensive approach for the capture and transfer of lessons learned.

61. Overall, the evaluation found that while there are areas for improvement, RPB demonstrated best practices in several key areas, including building design and client relations. The National Project Management System is intended to improve project management, planning, and implementation, as well as provide increased control over project risks.

Real Property Investment

62. RPB demonstrated prudence in the use of federal resources in all six projects. This was reflected in RPB's investment analysis reports. Investment analysis reports assessed the extent to which preferred options were strategically aligned with RPB's plans, strategies, and public policy requirements (e.g., accessibility, heritage, environment). Investment analysis reports for both preliminary and effective project approvals identified key project risks and the strategies and initiatives to measure, contain, mitigate, or eliminate them. Throughout the project, RPB monitored contract performance and methodically tracked allocated resources.

63. Overall, the evaluation found that RPB made good use of federal resources in real property investment, reflected in the Branch's investment analysis reports.

Real Property Portfolio Optimization

64. RPB's real property portfolio is optimized and aligned with the Branch's National Investment Strategy and the National Portfolio Plan. These documents are the most recent and strategic statements on the Government's real property portfolio. The National Investment Strategy identifies eight broad objectives for the office accommodation portfolio. There is evidence that these six projects contributed to the federal portfolio at the national, regional, and community levels, consistent with the PWGSC 2010/11 Report on Plans and Priorities and the National Investment Strategy. All six projects involved a consolidation of accommodation from two or more sites into one site; brought client departments into modern facilities; and reduced average office space at the community, regional, and national portfolio levels. Three projects (Yellowknife, Charlottetown, and Shediac) significantly enhanced federal presence in their communities.

65. Overall, all six projects contributed to the optimization of the real property portfolio through the deployment of modern facilities, increased space efficiency, and increased federal exposure.

Strategic Portfolio Management

66. Strategic portfolio management includes alignment, compliance, transformation, and federal presence as key outcomes from the effective design of real property solutions. The six projects examined contributed to RPB's strategic management of the portfolio by:

• Ensuring the projects are aligned with, and implement, RPB and TBS strategies and policies. It is a requirement in the preparation of Investment Analysis Reports (i.e., RPB business cases) that RPB demonstrate how the proposed investment is aligned with key strategies and in compliance with key policies;

• Delivering transformative projects that resulted in the creation of large, strategic assets within the portfolio (Skyline and 3000 Merivale). The relocation of the headquarters complexes of the Canadian Food Inspection Agency, Agriculture and Agri-food Canada, and the Royal Canadian Mounted Police provided modern, affordable, and responsive accommodation for the next 50 years; and

• Favourably reshaping federal presence in smaller communities. The building of large, attractive, technologically-advanced federal Crown-owned structures in the urban cores of Charlottetown and Yellowknife has enhanced federal presence in these communities. The capital lease of Shediac also contributed to the federal presence.

67. Overall, the projects contributed to the strategic management of RPB's real property portfolio through: alignment with departmental and federal policies; creation of valuable, strategic properties; and reshaping the federal presence in smaller communities.

Efficiency and Economy

68. Demonstration of efficiency and economy is defined as an assessment of resource utilization in relation to the production of outputs and progress toward expected outcomes. Efficiency refers to the extent to which resources are used such that a greater level of output is produced with the same level of input or, a lower level of input is used to produce the same level of output. Economy refers to minimizing the use of resources. A program has high demonstrable efficiency and economy when resources maximize outputs at least cost and when there is a high correlation between minimum resources and outcomes achieved.

Efficiency

69. Each project represents a unique set of needs and circumstances. The cost of providing office accommodations to client departments depends on a number of factors. As such, it is not meaningful to compare the total cost of one project over another. A summary of the size and costs associated with each of the six buildings is provided in Exhibit 4 below.

Table Summary

This table lists the size (in square meters), the number of FTEs and the up-front costs ($ millions) for each of the six buildings.

Exhibit 4: Summary of Size and Costs for Large Space Office Accommodation Projects

Project	Square meters	Number of FTEs	Up-front costs ($ millions)
Skyline (NCR)	67,750	3,380	$241.60
3000 Merivale (NCR)	78,411	3,200	$135.00
1600 Star Top (NCR)	19,489	900	$7.20
Jean Canfield Building (Charlottetown, PEI)	12,305	550	$56.40
Michel C. Leger Building (Shediac, NB)	3,912	185	$1.00
Greenstone Building (Yellowknife, NWT)	7,058	200	$28.00

70. The six projects demonstrated that RPB has put in place activities and processes to support greater efficiency. Key stakeholders indicated that RPB applied consistent national accommodation standards that were in effect when the projects were implemented. RPB achieved greater efficiency by taking advantage of business cycles in buying the Skyline property and in renting the Merivale property at less cost. Both the purchase of the Skyline complex and the capital lease for 3000 Merivale illustrate how efficiency can be achieved through leveraging fluctuations in the real estate market  to obtain value for money on new real property acquisitions.

71. In the six projects examined, RPB used one of two delivery options (Crown construction or capital lease) with varying degrees of private sector involvement. The review of these six projects did not reveal a correlation between the delivery option chosen and the degree of achievement of the program's expected outcomes. Of the six projects, three were Crown construction (Skyline, Charlottetown, Yellowknife) and three were capital lease (3000 Merivale, 1600 Star Top, and Shediac).

72. Based on client and stakeholder interviews, the highest ratings of project success in terms of meeting the program outcomes occurred in Shediac (capital lease), 1600 Star Top (capital leases), and Yellowknife (Crown construction). Lower ratings of project success occurred in Skyline (Crown construction), Charlottetown (Crown construction), and 3000 Merivale (capital lease).

73. The delivery option chosen in each case was dependent on circumstances and project objectives. No clear pattern of evidence emerged from stakeholder interviews or document and literature reviews to indicate that one delivery option was preferable over the other.

74. Overall, the evaluation found that it was difficult to develop quantitative measures of the relative efficiency of RPB's large space accommodation projects. The unique characteristics of each project precluded comparisons of the resources utilized to achieve project outcomes. However, through the application of consistent national standards and strategic real estate acquisitions, RPB has made demonstrable efforts to ensure accommodations are provided as efficiently as practicable.

Economy

75. Given the unique nature of the funding model used for Skyline, the evaluation compared two funding models reflected in the six projects: full-funding (Skyline) and split-funding (3000 Merivale, Star Top, Charlottetown, Shediac, and Yellowknife). The strength of the analysis of these two models is limited by the fact that only one project was implemented using the full-funding model.

76. The full-funding model is a method for funding tenant fit-up whereby Treasury Board provides the resources for tenant fit-up to RPB and RPB then manages the allocation and use of those resources. The main objective of the full-funding model is to include all components of the project within a comprehensive funding envelope. The use of the full-funding model for the Skyline project was a unique instance, representing its first use by RPB.

77. The split-funding model is a method for funding tenant fit-up whereby client departments manage the allocation and use of funds for tenant fit-up regardless of whether the funding is from within existing departmental reference levels or from Treasury Board. In some cases the client department will engage RPB as the service provider for a portion of tenant fit-up requirements. The intent of the split-funding model is to logically allocate project accountabilities to both RPB and the tenant organization.

78. The Skyline project provided a unique testing ground for the full-funding model. First, Skyline was one of the largest projects RPB had ever undertaken. Second, Skyline allowed a higher degree of client involvement during project implementation through a detailed Memorandum of Understanding and a strong ongoing partnership between the Canadian Food Inspection Agency, Agriculture and Agri-food Canada, and PWGSC. Third, the Skyline project was intended to be the first large-scale use of the full-service delivery model. The full-service delivery model was a method of project delivery where RPB committed to provide a complete and seamless array of project-related services to client departments through the creation of a dedicated project team. Although Skyline was intended to be the first use of the full-service model, it was not fully implemented, due to a loss of RPB capacity to deliver this type of service when the Skyline project was being implemented. The unique size, governance, and service delivery model all must be taken into consideration when assessing the relative economy of the full-funding and split funding models.

79. The main advantage of the full-funding model lay in greater completeness and transparency of project costs. The full-funding approach increased transparency, enabling identification of cost and financial issues through a single report. It required RPB to track all expenditures and allowed RPB to engage in comprehensive financial reporting. This allowed for greater openness with respect to financial arrangements, and facilitated discussion of more complex project financial issues. RPB was able to improve risk and change management, as well as reinforce RPB's commitment to take responsibility for every aspect of the project. The full-funding model also reduced some of the process burden (e.g., all project costs in one location, no need for billing clients, consistent project tracking of costs), which resulted in savings.

80. The main disadvantages of the full-funding model identified during the Skyline project were the extra project tracking and reporting required for RPB to meet both the requirements of PWGSC financial systems and client departments and the loss of some degree of financial control of expenditures on the part of PWGSC. The additional reporting represented a significant workload for RPB's Skyline project team. In addition, RPB stated that it could not fully control tenant fit-up expenditures during the Skyline project. While RPB had global accountability for the project the client department continued to make decisions on fit-up expenditures and funding for tenant fit-up remained under the control of client departments. As a result, RPB could not fully control client costs and was forced to track the funds separately, creating an additional burden for RPB. One respondent also felt that this created a false accountability, giving the impression of RPB control over expenditures where such control did not exist. A summary of the main advantages and disadvantages that were discerned from this evaluation are outlined below in Exhibit 5.

Table Summary

This table summarizes the main advantages and disadvantages of full-funding and split-funding, identified during this evaluation, for the six projects.

Exhibit 5: Summary of Advantages and Disadvantages for
Full-funding and Split-funding Models and Demonstrated in the Six Projects

	Full-funding	Split-funding
Advantages	Greater completeness and transparency of project costs (e.g., only one set of books) Easier identification of resourcing issues Comprehensive financial reporting Some reduction in process burden for RPB Project implementation not subject to client budget constraints	Accepted, understood, widely used Client is fully responsible and accountable for the costs that are directly related to their program delivery (i.e., client-specific fit-up). Client managed funding means clients are committed and have a stake Respective accountabilities for RPB and client are clear Lower administrative burden for RPB
Disadvantages	Requires significant extra work from RPB to track and report costs in a detailed manner Creates RPB accountability expectations that RPB cannot meet (at present), since client departments continue to play a large decision-making role in fit-up activities Requires clients be willing to submit to RPB's enhanced role, capacity, and accountability requirements	Two systems for tracking and reporting costs Less completeness and transparency of reporting Projects funded from within client reference levels and subject to client budget constraints

81. Stakeholders from RPB were satisfied with the current split-funding model and did not express a desire to migrate toward the full-funding model, as some personnel indicated that the use of the full-funding model was not warranted with small projects.

82. Stakeholders expressed a range of opinions on the merits of the full-funding model for the Skyline project. While some stakeholders agreed that there were benefits to the model, it was not seen as a significant factor by several personnel, indicating that they observed no meaningful difference as a result of the full-funding model. Stakeholders recognized that the full-funding model had both advantages and disadvantages, indicating that the full-funding model was beneficial for the Skyline project given RPB objectives for the full-service delivery model and in terms of the existing Memorandum of Understanding. It was stated that the full-funding model did contribute to achieving project outcomes and that they would use it again.

83. Overall, it was found that the choice of funding model had little appreciable effect on a project's outcome.

84. There is no single delivery option or funding model that is appropriate for all circumstances. Efforts are being undertaken to continually improve the economy of large space office accommodation. A number of delivery strategies, options, and practices could potentially be used to strengthen the cost-effectiveness of RPB project outcomes.

Lessons Learned From Other Jurisdictions

85. Literature from other jurisdictions such as Australia and the United Kingdom indicate that a wide range of viable options in providing large space office accommodation exist. These documents present public-private partnerships as a possible delivery option. They report that one benefit of a public-private partnership is the ability to deliver a project on time, on budget, and within scope. The downside is that material changes to client requirements cannot be accommodated during project implementation without a significant financial penalty.

86. Australia and the United Kingdom recognize that flexibility in the selection of delivery options is essential to effective and efficient project delivery. While public-private partnerships are their preferred approach to large projects, viable options such as management contracting, construction management, and design-build have been identified.

87. The literature also suggests the use of gateway reviews. These are peer reviews conducted by outside practitioners who use their knowledge, experience, and expertise to evaluate the project to determine whether the project is ready to proceed to the next stage.

88. The federal government has undertaken a number of public-private initiatives, including establishing P3 Canada, a Crown corporation with a mandate to administer a $1.25 billion public-private partnership fund to provide alternatives to traditional procurement. The Government is also setting up a federal Public-Private Partnerships Office to facilitate a broader use of public-private partnerships in Canadian infrastructure projects. A new initiative by RPB requires pre-screening for large projects to assess public-private delivery suitability.

Conclusions: Performance

89. Overall, the five completed projects contributed to achieving the program's outcomes. While the five projects were completed on budget—albeit in three projects only by virtue of updating the budget estimates as the costs increased—RPB experienced difficulties in achieving on-time objectives for three projects and scope objectives for one project. Client departments expressed satisfaction with RPB project services on only three projects.

90. Although good management practices were followed on all six projects, RPB experienced difficulty with the challenges, risks, and complexities of large accommodation projects. The projects revealed a number of barriers to achieving success in large-scale office accommodation projects, notably the need to improve front-end planning and risk management. However, RPB's National Project Management System is expected to bring more rigour to the management of RPB's large space accommodation projects.

91. The projects represented prudent expenditure of taxpayer funds and contributed to optimizing the real property portfolio through consolidation, modernization, compliance to standards, and enhancement of federal presence in smaller communities. Three client departments recognized RPB as a good project manager while still indicating that there was room for improvement.

92. Client satisfaction with the accommodation provided was high. The five completed projects contributed to the better strategic management of the full portfolio of federal accommodation assets. Processes and activities were in place to support the achievement of cost-effectiveness. The literature review identifies some strategies that could increase the cost-effectiveness of large space office accommodation projects.

General Conclusions

93. Relevance: RPB's role in large space office accommodation projects is mandated by federal legislation; appropriate to the federal government; linked to federal and departmental priorities; and addresses a demonstrable need.

94. Performance: While RPB has achieved considerable success on three of the six reviewed projects, there was room for improvement in project implementation on the other three, specifically front-end planning and risk management. The National Project Management System is expected to bring increased rigour to the management of RPB's large office space projects.

95. The evaluation was able to provide an assessment of the advantages and disadvantages of the full-funding and split-funding model in the context of the six projects examined. Overall, the funding model was not found to be a major contributor to project outcomes. While some projects demonstrated higher levels of risk and complexity, the evaluation was unable to demonstrate a direct correlation between complexity and lower project success ratings. As Skyline was a pilot project, there is a need to address the conditions for optimal use of the full-funding model in the future.

Management Response

96. The Real Property Branch accepts the evaluation findings and intends to act on the recommendations of the evaluation by implementing their Management Action Plan detailed as follows.

Recommendations and Management Action Plan

Recommendation 1: The Assistant Deputy Minister, Real Property Branch should enhance lessons learned practices to further improve project management practices on large space accommodation projects.

• Management Action Plan 1.1: The National Project Management System Quality Management Area is being updated to enhance lessons learned practices.

  The improved lessons learned practices will be linked to the National Project Management System Continual Improvement process already in-place to ensure that improvement of project management practices will be incorporated into the National Project Management System for the benefit of other project teams

Recommendation 2: In using the full-funding model, the Assistant Deputy Minister, Real Property Branch, should clarify accountability structures.

• Management Action Plan 2.1: The National Project Management System framework requires projects to complete a project charter during the initial stages of a project. The purpose of the project charter is to create a high level agreement between the client department and PWGSC that is intended to establish a framework for the implementation. The intent of the charter is to obtain client agreement on all of the key parameters of the project (scope, time, and cost) prior to Lease Project Approval (LPA) or Preliminary Project Approval (PPA) and to confirm client commitment to expend client funds and resources in developing and/or delivering the defined project. For PWGSC space projects, it is to be utilized in conjunction with the Statement of Requirements to articulate and confirm understanding and agreement related to project goals and objectives, guiding principles, roles and responsibilities, and issue resolution processes. In complex cases, projects will complete a Memorandum of Understanding in addition to a project charter. The Memorandum of Understanding will define additional objectives complete with related roles and responsibilities.

  It is critical that projects identify proper accountabilities, priorities, governance and roles and responsibilities in this document prior to commencement of the project. The full-funding model would require a more comprehensive project charter in order to clarify accountability with regards to processes and procedures for dealing with project issues and reporting.

Recommendation 3: In using the full-funding model, the Assistant Deputy Minister, Real Property Branch, should develop guidelines for the application of this model to address project management issues including: project capacity for cost tracking and reporting; the most appropriate size and complexity for the projects involved; and client management issues, such as client readiness to accept RPB's enhanced role and to utilize RPB processes and systems, and the need for clear accountabilities for both RPB and client departments.

• Management Action Plan 3.1: As stated in Recommendation 2, the National Project Management System framework requires projects to complete a project charter during the initial stages of a project. The purpose of the project charter is to create a high level agreement between the client department and PWGSC that is intended to establish a framework for the implementation. This document should be utilized by project to address any and all roles and responsibilities including clear accountability, priorities and governance for all project parameters.

  When properly used, the project charter would address all issues that occurred during the implementation of the Skyline project with regards to cost reporting, accountabilities, roles and responsibilities and processes.

  PWGSC will leverage existing reporting systems including SIGMA (SAP) to meet specific project reporting requirements. This will ensure that existing financial controls will ensure accurate and reliable reporting.

About the Evaluation

The evaluation examined large space office accommodation projects by using a case study approach to determine the relevance and performance of six of PWGSC's large space office accommodation projects. The objective of the Evaluation of Selected PWGSC Large Space Office Accommodation Projects was to assess the program's relevance: the continued need for the program, its alignment with Departmental and Government priorities, and its consistency with federal roles and responsibilities. The evaluation also determined the program's performance: the program's achievement of its expected outcomes, and its demonstration of efficiency and economy.

Approach

The evaluation was conducted in accordance with the Evaluation Standards of the Government of Canada and the Office of Audit and Evaluation of PWGSC. The evaluation took place between January 1, 2010 and April 15, 2010 and was conducted in three phases: planning, examination, and reporting. To assess the evaluation issues and questions, multiple lines of evidence were used.

Case Studies: PWGSC does not formally define what constitutes 'large space'. For the purposes of the evaluation, case studies of three types of office accommodation projects were selected as they provided a broad coverage of RPB projects. These included large projects in which RPB had a lead role but which did not require Treasury Board approval as they fell below the Treasury Board definition of large projects (i.e., $30 million); projects over $30 million for which Treasury Board approval was required; and projects under $20 million where relevant lessons learned could be gained.

Document Review: More than 200 documents were reviewed in this evaluation. Most contained information and data on the six selected projects. Some RPB documents addressed broader approaches to managing the RPB real property portfolio. The documents included approval documents, project close-out documents, project charters, and client satisfaction surveys. The review also included external client satisfaction surveys conducted by RPB on project management services for Skyline, 3000 Merivale, and Charlottetown.

Literature Review: Over 50 documents were reviewed from international jurisdictions in order to gain a better understanding of delivery options and project delivery best practices. In addition, Canadian documents were reviewed, including the use of a variety of delivery options for large public infrastructure projects in Canada.

Stakeholder Interviews: Thirty-one stakeholders were interviewed for the evaluation. These consisted of stakeholders from TBS (2), as well as the six projects: Skyline (14), Yellowknife (5), Charlottetown (3), Shediac (2), 3000 Merivale (3), 1600 Star Top (2). Interviews were conducted with RPB as well as client departments for all projects except for 1600 Star Top. For this project stakeholder interviews were limited to RPB personnel due to non-availability of DND personnel, many of whom are in rotational positions. The stakeholders were asked a number of questions relating to the relevance, success, and cost-effectiveness of accommodation projects.

Methodological Limitations

The methodology used for this evaluation has a number of significant strengths, but remains subject to several limitations. The key approach to reducing the limitations of certain methodologies was to ensure that data was validated through a multiple-lines-of-evidence approach. The following were the key limitations faced by the evaluation.

First, the (intentionally) small sample size of projects had a considerable diversity of scope, which prevented meaningful extrapolations being drawn to the whole population of large space office accommodation projects.

Second, the passage of time and the relatively high rate of change in RPB posed a challenge to the evaluation. Many of these projects have timelines starting eight or more years ago, and since that time, there have been a number of changes in RPB's approaches to project management, notably through the introduction of the National Project Management System and through pursuing compliance with Treasury Board Policies on Investment Planning and the Management of Projects, both issued in 2007. This limitation limited the ability to draw complete conclusions regarding RPB's current capacity to plan and implement large scale office accommodation projects.

Third, the only quantitative information available came from external surveys of client satisfaction with project management. In addition, this survey information was available for only three projects and only one form was completed per client department.

Reporting

Findings were documented in a Director's Draft Report, which has been reviewed by the Office of Audit and Evaluation's quality assessment function. The program's Director General was provided with the Director's Draft Report and a request to validate facts and comment on the report. A Chief Audit and Evaluation Executive's Draft Report was prepared and provided to the Assistant Deputy Minister, RPB, for acceptance as the Office of Primary Interest. The Office of Primary Interest responded with a Management Action Plan. The Draft Final Report, including the Management Action Plan, was presented to PWGSC's Audit and Evaluation Committee for the Deputy Minister's approval in March 2011. The Final Report will be submitted to the Treasury Board of Canada Secretariat's Centre of Excellence for Evaluation and will be posted on the PWGSC website.

Appendix A

Table Summary

This table represents the evaluation matrix outlines and summarizes the evaluation issues, questions, indicators and identifies the data sources that address the questions. The evaluation matrix is a tool that helps the evaluation team consider the most appropriate and feasible data collection method for each of the questions identified in the evaluation plan.

Evaluation Matrix

Evaluation Issue	Indicators and Responsibility for Data Collection	Data Sources	Methodology
Relevance – Is RPB's role in Large Space Office Accommodation Projects linked to Government of Canada policies and priorities and does it address a continuing need?
1. In the context of large space office accommodation, to what extent is RPB's role linked to PWGSC and Government of Canada (GC) policies and priorities?	Evaluation Team Extent of alignment between activities, outputs and intended outcomes of RPB large space office accommodation projects and PWGSC/GC policies; and priorities	Document Review Legislation Department of Public Works and Government Services Act Federal Real Property and Federal Immovables Act TBS Common Services Policy Policy on the Management of Real Property Policy on the Management of Projects Management of Major Crown Projects PWGSC Framework for Office Accommodation and Accommodation Services PWGSC Report on Plans and Priorities PWGSC Departmental Performance Report Real Property Branch Business Plan Real Property Roles and Responsibilities Action Plan Other relevant studies or reports	Review and analyze key documents to determine the extent the objectives of RPB's large space office accommodation projects are linked to PWGSC/GC policies and priorities, as well as the extent of consistency of the Program's objectives with departmental and governmental policies and priorities
		Stakeholder Interviews TB representatives as required PWGSC/RPB managers Federal organizations' representatives	Interview key stakeholders and RPB staff to determine linkage with departmental and governmental policies and priorities
2. To what extent is RPB meeting the on-going needs of federal organizations in the provision of large space office accommodation?	Evaluation Team Extent to which federal organizations are satisfied with the services and facilities provided by RPB with regard to large space office accommodation projects: Number of large space projects closed out in the past 5 years Number of large space office accommodations delivered on time, on budget and on scope Evidence that RPB provides to federal organizations safe, healthy and affordable large space office accommodations that support their programs and services Number of clients able to move in and deliver programs and services with minimal interruption Extent of balance between client involvement and needs Evidence of timely provision of space within forecast budgets Evidence that RPB large space office accommodation services have evolved to adapt and meet emerging/changing needs Evidence of meeting accommodation standards Evidence of effective project design and change management Extent to which RPB large space office accommodation services address the need for which it was created Evidence that RPB large space office accommodation services are still required	Data and Document Review Client satisfaction surveys Performance/Status reports Statistical data available Feedback reports available Other relevant studies or reports	Review and analyze key documents, data and statistics to determine the ongoing need for RPB's role in large space office accommodation projects
		Interviews/Focus Groups Interview federal organizations' representatives working with RPB for large space office accommodation	Interview key stakeholders and Program management to determine performance
3. To what extent is the RPB role in large space office accommodation projects appropriate to the federal government?	Evaluation Team Existence, usage and effectiveness of alternatives Evidence that federal government organizations comply with relevant policies RPB Evidence of clearly defined, documented and communicated roles for PWGSC and other federal organizations Evidence that management within federal organizations understand the role of RPB Evidence of functioning management and governance structure Evidence that performance is being monitored and reported Evidence that lessons learns have been compiled and utilized Evidence that RPB is fulfilling its responsibility for acquiring large space office accommodation on behalf of federal organizations Evidence that RPB has the capacity to effectively manage for results and deliver its mandate Extent to which RPB supports federal organizations' large space office accommodation needs Extent to which RPB assists federal organizations to comply with the technical aspects of accommodations, e.g. standards, fit-up, etc. Extent of the efficiency and effectiveness of planning and project delivery processes Extent of governance mechanisms within PWGSC and timely decision-making Extent of strategic framework, tools and systems in the provision of office space Level of the quality of working relations with federal organizations Federal Organizations Extent to which federal organizations agree that RPB is fulfilling its responsibilities Extent of RPB responsiveness to the federal organizations Ease of doing business Evidence of processes and supporting tools	Data and Document Review Other jurisdictions Policy mentioned in issue 1 Documents defining roles and responsibilities Performance reports Lessons learned Project delivery process Federal organizations' feedback reports Client testimonials Client satisfaction surveys Other relevant studies or reports	Review and analyze key documents to determine the extent to which the RPB role in large space office accommodation projects is appropriate to the federal government
		Interviews/Focus Groups Interview federal organizations' representatives working with RPB for office accommodation RPB staff Federal organizations	Interview key stakeholders and Program management to determine appropriateness of RPB role
Success - How successful is RPB in achieving its intended outcomes?
4. To what extent is RPB achieving its objectives in delivering large space office accommodations? Meeting the stewardship Needs of Canadians? Meeting operational  needs of the federal organizations? Correlating between success and delivery options (lease, Crown construct, etc.)? Correlating between  success and project Funding models (split- funded, full-service, etc.)?	RPB Meeting operational needs Number of large space office accommodation projects closed out in the past 5 years Number of large space office accommodations delivered on time, on budget and on scope Comparison of actual outcomes to targets Evidence that accommodation standards are met Evidence that RPB provides to federal organizations safe, healthy and affordable large space office accommodations that supports their programs and services Evidence of RPB responsiveness to federal organizations' needs Evidence that federal organization needs are being met Number of clients able to move in and deliver programs and services with minimal interruption Evaluation Team Correlating success with delivery option chosen Identify options (lease, Crown construct) and asses their success in providing safe, health, affordable accommodations on time, budget and scope Comparison of the pros and cons of the different delivery options Correlation between the delivery option chosen and the type and level of client involvement, interaction, collaboration and satisfaction with results Extent of client satisfaction Evaluation Team Correlating success with funding model used Identify the options (split-funded, full-service) and assess RPB success in providing safe, healthy, affordable accommodations on time, budget and scope Comparison of the pros and cons of the different funding models Correlation between the funding model chosen and the type and level of client involvement, interaction, collaboration and satisfaction with results Extent of client satisfaction	Data and Document Review Roles and Responsibilities Accountability Framework Performance reports Operational reports Project delivery processes Client satisfaction surveys Other relevant studies or reports	Review and analyze key documents and data to establish the level of success of RPB processes and mechanisms and to determine the success of the Program's business model in supporting its objectives Review and analyze key performance measures to determine the successful operation of Program processes and mechanisms
		Interviews/Focus Groups PWGSC/RPB managers and staff Clients who have used RPB in the last 5 years for large space office accommodation projects	Interview key stakeholders and RPB personnel to determine how well the Program's business model supports its objectives
5. What are the barriers to successful project delivery of large space office accommodations?	RPB Number of projects that were not delivered on time, on budget and on scope Trends in not delivering on time, on budget and on scope Number of complaints from federal organizations using RPB large space office accommodation services Existence of effective processes for improvements, governance mechanisms and strategic frameworks arising from lessons learned	Data and Document Review Roles and Responsibilities Accountability/Strategic Framework Performance reports Operational reports Continual Improvement Framework Other relevant studies or reports	Review and analyze key documents to determine the barriers to successful delivery of large space accommodation
		Interviews/Focus Groups PWGSC managers and staff including PRB Clients who have used RPB in the last 5 years for accommodation projects (by delivery option and funding model chosen)	Interview key stakeholders to identify the issues and challenges faced by RPB in delivering large space office accommodation to federal organizations
Cost-Effectiveness – Is RPB's role in providing large space office accommodation the most cost-effective way to provide large space office accommodation?
6. Are there more cost-effective alternatives for delivering large space office accommodation projects?	RPB and Evaluation Team Extent to which RPB has accurately identified total accommodation costs Processes in place to identify and minimize non-value added activities Industry best practices employed in project design and delivery Best practices employed in contracting and procurement Best practices in management of consultants and inimplementation of projects Evaluation Team Comparisons to other jurisdictions and industry benchmarks Extent of the existence, usage and effectiveness of potentially less expensive alternatives	Data and Document Review RPB financial statements Costing and Charging Review Project Project Business Management System database Operational reports Financial reports and plans Budgetary allocations Project reports/updates of costs Other relevant studies or reports Literature Review Experiences of other jurisdictions Industry benchmarks used for project costing	Review and analyze key documents to assess the most appropriate and efficient means for achieving cost objectives and outcomes Review literature to identify relevant experiences in other jurisdictions to assess the most appropriate and efficient means for achieving full cost recovery
		Interviews RPB managers Appropriate representatives from other jurisdictions and large organizations	Interview key stakeholders to identify the Program's contribution to cost savings and cost avoidance
7. Is there one delivery option that is more effective than others?	Review large space office accommodation project total accommodation delivery costs Examine the correlation between the total costs of delivering large space office accommodation projects with the delivery option and funding model chosen Within the context of large space office accommodation, extent to which RPB is ensuring best-value to Canadians	Data Document Review Project reports/updates of costs and options Post project reports Literature Review Experiences of other jurisdictions Industry benchmarks used for project costin	Review and analyze key documents to assess the costs and options Review literature to identify relevant experiences in other jurisdictions and industry benchmarks used for project costing
		Interviews RPB project managers	Interview key stakeholders to identify the correlation between the costs of delivering large space office accommodation projects with delivery options and funding model chosen