Evaluation of the Build in Canada Innovation Program (final report)

Evaluation of the Build in Canada Innovation Program (final report) (PDF, 764KB)

May 26, 2017

Main points

i. The Build in Canada Innovation Program aims to help Canadian businesses commercialize their innovative goods and services (i.e. prepare their product for commercial launch) by providing them with the opportunity to demonstrate the application of their innovations on a commercial scale within the federal government. The Build in Canada Innovation Program has two key components through which suppliers can apply for testing: the standard component and the military component. The Build in Canada Innovation Program engages with industry to identify Canadian companies with innovative products or services in the late stages of pre-commercial development and solicits bids from these companies. The program then matches the innovations with federal organizations with a corresponding need, and awards contracts to companies such that their innovations can be used and tested by a federal department or agency. The program then provides the companies with feedback on their innovations and suggestions for how the innovations could be improved prior to taking them to market.

ii. The evaluation examined the ongoing relevance and performance of the Build in Canada Innovation Program. Overall, the evaluation found that the need for both the standard and military components of the Build in Canada Innovation Program persists, given an historical lack of private sector support for the commercialization of late-stage innovations in Canada. Further, the evaluation found that the Build in Canada Innovation Program aligns with several federal priorities, particularly those relating to: investing to build a leaner, more agile, and better-equipped military; investing in innovation; economic growth; and job creation. As the Build in Canada Innovation Program’s role is to foster innovation in Canadian industry on the national level, the evaluation found that it is aligned with federal roles and responsibilities, and could not be devolved to the provincial level or outsourced to the private sector. Consistent, nationwide delivery of a program of this kind by the federal government is most logical given the federal responsibilities associated with the target areas within the standard and military components of the Build in Canada Innovation Program. Further, no similar program for supporting innovation by providing an index buyer, who provides feedback based on testing, was identified. In this context, the Build in Canada Innovation Program can be seen as complementing existing programs that support the development and commercialization of Canadian innovations.

iii. Further, given Public Services and Procurement Canada’s strategic outcome of providing high quality services, including the acquisition of goods and services on behalf of federal departments and agencies, the evaluation noted alignment between the Department’s procurement-related strategic outcome and the Build in Canada Innovation Program’s outcomes related to acquiring innovative goods and services for Federal Government use. The Government also uses its substantial purchasing power as a means to achieve social and economic goals for Canadians and Canadian business. This aligns with the Build in Canada Innovation Program outcomes related to supporting the procurement of innovations as a means to help the successful launch of innovations in the marketplace. The procurement-based delivery model of the program also aligns with Public Services and Procurement Canada’s roles and responsibilities as the common service provider for procurement services in the federal government. This is especially to the case for the military component of the Build in Canada Innovation Program, as Public Services and Procurement Canada has exclusive authority for defence-related contracting in the government of Canada.

iv. The evaluation also noted some degree of alignment between the Build in Canada Innovation Program’s outcomes related to supporting improved performance of innovations in the marketplace and the strategic outcomes for the Department of Innovation, Science, and Economic Development related to enhancing Canada’s innovation performance. The Build in Canada Innovation Program is an example of how Departments with complementary objectives work together to achieve common results for Canadians.

v. With respect to performance, the Build in Canada Innovation Program has made progress towards achieving its intended outcomes. Small and medium enterprises from across the country have expressed interest in the program. Similarly, the evaluation found that an increasing number of federal departments and agencies were interested in participating in the Build in Canada Innovation Program over the course of the evaluation period. Industry and departmental participation in the program have also increased. While the total number of proposals received from industry has declined, the number of pre-qualified proposals has increased, with 72 and 74 pre-qualified proposals received for the fourth and fifth calls for proposals, respectively. These numbers represent an increase of 95% and 100%, respectively, over Call for Proposals 002, which had seen the highest previous number of pre-qualified proposals, at 37.

vi. The number of participating departments has also increased during the two most recent Calls for Proposals. In fact, calls for proposals 004 and 005 saw the highest number of participating departments in the program’s history, with 21 and 19 federal departments participating respectively. These rates of participation represent 50% and 36% increases over Call for Proposals 001, which had seen the highest previous level of participation by federal departments, with 14.

vii. However, while suppliers noted positive experiences with the Build in Canada Innovation Program, such as the testing feedback and having a first time buyer both contributing to the commercialization process for their innovations, they also identified areas which they believe the program could improve, such as its website and progress reports on the status of proposals. In addition, the Build in Canada Innovation Program’s success is also dependent on the extent to which departments and agencies are open to testing innovations to meet their needs, and the evaluation noted opportunities for improvement in relation to this outcome. However, as this obstacle may be related to elements of the culture of the Government of Canada, there are likely contributing factors to it that are beyond the control of the program.

viii. With respect to the Build in Canada Innovation Program’s economy and efficiency, the evaluation found that the program operated economically and efficiently during a period of instability during the transition from being a pilot to a permanent program. Data for the from April 1, 2015 to March 31, 2016 (the 2015 to 2016 fiscal year) (the most recent year for which financial data was available) shows that the Build in Canada Innovation Program spent a greater proportion of its funding on innovations than at any previous point in the program’s history. In that year, 85% of the program’s $25 million budget was spent directly on innovation-related work with companies. The program also demonstrated increased efficiency in the 2015 to 2016 fiscal year, recording its lowest percentage of expenditure on overhead in its history (14.1%) at the same time it was directing a greater proportion of its budget to innovation work. Even greater economy and efficiency may be achievable as the Build in Canada Innovation Program continues its operations with greater security of its permanency and assurance of funding.

Management response

ix. Acquisitions Branch accepts this report and its recommendations.

Recommendations and management action plan

Recommendation 1:

The Assistant Deputy Minister, Procurement should work collaboratively with officials from Innovation, Science and Economic Development Canada (ISED) to maintain coordination of the program with other elements of the innovation agenda, optimize conditions for achieving its expected results, and develop mechanisms to integrate results measurement of the Build in Canada Innovation Program into the results measurement of the overall innovation agenda.

Management action plan 1.1: 

Since summer 2016, Acquisitions Branch has been working closely with Innovation, Science and Economic Development on how the federal government can leverage the procurement of innovation to bolster Canada’s economic growth and global competitiveness. Specifically, officials within Acquisitions Branch have been closely collaborating with Innovation, Science and Economic Development Canada in the context of the Build in Canada Innovation Program renewal and in regards to the design and development of Innovative Solutions Canada, a key component of the Inclusive Innovation Agenda and Budget 2017.

Innovation, Science and Economic Development Canada is finalizing a proposed approach for Innovative Solutions Canada – it is expected the initiative will formalize government support for early-stage technology development with the dual objective of supporting commercialization of Canadian innovations, and driving solutions development that respond to Government of Canada challenges. In this context, Acquisitions Branch will develop program options for innovation procurement and a related results measurement framework that align with Innovative Solutions Canada and other elements of the Innovation, Science and Economic Development Canada-led Innovation Agenda.

Acquisitions Branch will also examine best practices in horizontal management and apply them to the collaboration with Innovation, Science and Economic Development Canada in the context of Build in Canada Innovations program renewal and Innovative Solutions Canada.

Recommendation 2:

The Assistant Deputy Minister, Procurement, should implement strategies to reduce barriers, real and perceived, that are precluding federal organizations from purchasing Canadian innovations.

Management action plan 2.1:

The Build in Canada Innovation Program will continue, through proactive outreach and marketing, to drive both submissions of innovations to the program and the uptake of innovations for testing the public sector. Engagement with horizontal governance bodies (e.g. Science and Technology Advisory Board), federal partners (e.g., Regional Development Agencies, Innovation, Science and Economic Development Canada, and the National Research Council—Industrial Research Assistance Program), targeted regional, seminars and educational opportunities (e.g., innovation showcases), digital engagement and social media will help to build greater awareness and increased interest in how innovative goods and services can support government objectives and challenges.

Further, outreach and educational efforts will be made with provincial, territorial and municipal jurisdictions to enhance the testing opportunities for the Build in Canada Innovation Program innovations. Partnerships between federal departments and other jurisdictions have and continue to take place, and an increase in these opportunities will support greater exposure for innovative goods and services being used in the public sectors, and thus increase the potential uptake for full commercial purchases for future operational use.

More broadly, Public Services and Procurement Canada’s Acquisitions Branch is working with the Treasury Board Secretariat on how procurement policy (through a Treasury Board-led policy suite reset) can best support government objectives, which include socio-economic considerations and benefits when the government buys the goods and services it requires.

Acquisitions Branch, through the procurement modernization initiative, will be assessing how procurement, can more directly encourage and support purchases of innovation, whether the purchases be for research and development purposes (the Build in Canada Innovation Program, other research and development focused-activities), or for supporting full operational needs of government (commercial purchases).

Introduction

1. This evaluation examined the ongoing relevance and performance of the Build in Canada Innovation Program. The evaluation was included in the Public Services and Procurement Canada 2015 to 2018 Risk-Based Audit and Evaluation Plan.

Profile

Background

2. Innovation is a multi-stage process by which individuals, companies and organizations develop, master, and use new products, designs, processes and business methods. The components of innovation include research and development, invention, capital investment, and training and development. Innovation (from idea to commercialization) occurs along a continuum, where technology readiness increases at each stage. In Canada, a gap in support was identified along the innovation continuum, particularly innovations in the later stages of innovation.

3. In recognition of this issue, the federal government launched the Canadian Innovation Commercialization Program in September 2010 as a two year pilot program. The pilot phase concluded in March 2013, and through a Budget 2012 commitment, the program was made permanent and re-launched in April 2013 as the Build in Canada Innovation Program, with the addition of a military component.

4. According to Budget 2010, small and medium-sized enterprises were estimated to account for approximately 98% of all businesses in Canada and employed more than 5 million people, accounting for roughly half of the private sector workforce. It was anticipated that increased innovation among Canadian small and medium-sized enterprises would encourage economic development and create additional “high-value jobs”. Budget 2010 stated that many Canadian companies that had developed new and innovative products struggled to find buyers due to the higher risks associated with untested products.

5. The Build in Canada Innovation Program helps businesses commercialize their innovative goods and services by providing them with the opportunity to prepare their product or service for commercial launch. It does this by procuring pre-commercial Canadian innovations on behalf of interested federal organizations which agree to test the product or service in their environment to determine if it could meet their needs. It also provides companies with the opportunity to demonstrate the successful application of new concepts on a commercial scale by testing their products within the federal government with the benefit of client feedback on the performance of the tested innovations. More specifically, the Build in Canada Innovation Program operates in the following manner:

• industry has a broad range of innovative goods and services
• the Build in Canada Innovation Program engages in outreach to inspire industry to engage with the federal government on their innovations and to inspire government departments to test innovations
• the Build in Canada Innovation Program solicits bids through an open Call for Proposals process for standard component and military component innovations within specified priority areas
• for Calls for Proposals 1 through 5, proposals were received and evaluated to determine the top scored innovations, which were then pre-qualified until funding is exhausted. Beginning with Call for Proposal 6, proposals are received and evaluated against program criteria and are either successful (pre-qualified) or not
• pre-qualified innovations are matched to federal departments through the following process:
  • the program engages pre-qualified companies and testing departments to develop a match for the innovation
  • when a match is confirmed, a memorandum of understanding is signed between the testing department and the Build in Canada Innovation Program, which transfers ownership of the innovation to the testing department
  • a statement of work (the basis for the test plan) is negotiated between the testing department and the supplier based on the supplier’s innovation proposal
  • a contract is established between the supplier and the Build in Canada Innovation Program
• Federal departments test the innovations and provide feedback

6. The objectives of the Build in Canada Innovation Program, which remain the same as those expressed under the Canadian Innovation Commercialization Program, are to promote innovation and support small and medium enterprise development in Canada. Specifically, the Build in Canada Innovation Program aims to:

• create an opportunity for private sector innovators to build on their investments and technologies (their innovations) to better enable their launch into the marketplace
• improve the efficiency and effectiveness of government operations through use of innovations, and the openness of federal procurement, including to small and medium enterprises
• identify entities in Canada that have innovations that are very mature, and connect them with potential users in federal departments
• improve accessibility for doing business with the Government of Canada

7. The program works to achieve these objectives through the support of innovations categorized under two umbrella components: standard and military.

8. The standard component of the program encompasses the same four target areas that were identified as part of the program pilot from 2010 to 2012. Innovations must be usable within the following areas of federal operations:

• environment: goods or services that have been developed either to compete in the environment industry or to compete in another industry while offering clearly demonstrable environmental benefits over existing goods and services
• safety and security: goods or services that have been developed either to compete in the safety and security industry or to compete in another industry while offering clearly demonstrable safety or security benefits over existing goods and services
• health: goods or services that have been developed either to compete in the personal or public health industry or to compete in another industry while offering clearly demonstrable health benefits over existing goods and/or services
• enabling technologies: goods or services that alone or in combination with associated technologies provides the means to generate clearly demonstrable improvements in performance and capabilities for the user

9. As noted above, Budget 2012 established the program as permanent, providing $95 million in additional funding over the next 3 years, as well as an annual budget of $40 million per year thereafter. In addition, Budget 2012, which also dealt extensively with investment in the Canadian military, expanded the program to include a military component which would allow federal organizations with related mandates to benefit specifically from innovative goods and services with a military application. The program’s military component encompasses six target areas:

1. command and support: goods or services that direct defence and security forces towards an objective and are interoperable with those of our allies
2. cyber-security: goods or services that can be used to secure this domain
3. protecting the soldier: goods or services that would serve to protect the soldiers of the Canadian Armed Forces
4. arctic and marine security: goods or services that provide specialized functionality in Canada’s vast maritime boundary and arctic landmass
5. in-service support: goods or services that support and maintain the performance of military equipment throughout its life-cycle, which can include modifications to address changing requirements over the lengthy lifetime of most major equipment
6. training systems: goods or services that prepare Canada’s land, sea, air and civil security forces for the complex situations in which they are called upon to operate. Trainings systems typically have dual-use in both military and commercial applications, but those submitted under the military component should be customizable to the requirements of the Department of National Defence

10. More recently, the program has introduced changes which allow participating federal organizations the option of pursuing follow-on sales after the completion of testing. As such, innovators that received contracts under the program’s fourth and fifth Calls for Proposals were eligible for further testing for up to two years from the date of their pre-qualification or their initial contract, depending on the time of their contact award. Innovators, who started the Contract Award Process but had not received an initial contract were also eligible for testing up to two years from the date of their pre-qualification. These changes are directed at increasing collaboration between federal organizations and Canadian suppliers in support of innovation, as well as an increased ability for federal departments to use innovative goods and services.

Authority

11. The authorities for the Department to undertake the activities of the Build in Canada Innovation Program are provided under section 6 of the Department of Public Works and Government Services Act. This section provides the Minister with the authority for “the acquisition and provision of articles, supplies, machinery, equipment and other materiel for departments” as well as “the acquisition and provision of services for departments.” Further, section 8(1) of the Act states that “the Minister may delegate any of the Minister’s powers, duties or functions under this Act to an appropriate minister for any period and under any terms and conditions that the Minister considers suitable.” This allows for delegation of the Minister’s authorities for the procurement on behalf of government departments.

12. Additionally, with respect to the purchase of military goods and services, section 10 paragraph 2 of the Defence Production Act indicates “the Minister [of PSP] shall have exclusive authority to buy or otherwise acquire defence supplies and construct defence projects required by the Department of National Defence.” The Act further indicates that “the Minister [of PSP] may authorize any person, on behalf and under the control and direction of the Minister, to do any act or thing or to exercise any power that the Minister may do or exercise under [the] Act.” This allows for the delegation of the Minister’s authorities for the procurement of defence supplies.

13. Additional authority for the Build in Canada Innovation Program was provided as part of Budget 2012, which established it as permanent and provided it with the authority to add the military procurement component.

Roles and responsibilities

14. The Office of Small and Medium Enterprises and Strategic Engagement manages the delivery of the Build in Canada Innovation Program. Office of Small and Medium Enterprises and Strategic Engagement is part of Public Services and Procurement Canada’s Acquisitions Branch and is responsible for managing the program from Public Services and Procurement Canada headquarters, as well as delivering it from Public Services and Procurement Canada’s six regional offices. Office of Small and Medium Enterprises and Strategic Engagement regional offices are key partners in conducting the Build in Canada Innovation Program outreach activities, enabling them to leverage partnerships within the small and medium-enterprises community, as well as across federal departments and agencies, formed through the delivery of the Office of Small and Medium Enterprises and Strategic Engagement. Promotional events and seminars are held across Canada to highlight program benefits to stakeholders, and the Build in Canada Innovation Program regularly attends trade shows and similar events across Canada.

15. The Build in Canada Innovation Program is responsible for maintaining an informative website which provides up-to-date news on the program, such as improvements and information on events, the Build in Canada Innovation Program success stories, and information on how to participate in the program. As well, the Build in Canada Innovation Program is responsible for maintaining the procurement process assessment criteria and e-tools. The program maintains the criteria used to evaluate and pre-qualify proposals. Regarding maintaining e-tools, the Build in Canada Innovation Program has introduced a start-to-finish electronic approach to distributing information, receiving and evaluating responses to Call for Proposals, which had traditionally been done through the submission of multiple hard copies.

16. Public Services and Procurement Canada’s Science Procurement Directorate develops the Calls for Proposals, which are posted on Public Services and Procurement Canada’s procurement website (buyandsell.gc.ca) and allows for innovations to be procured through a competitive process. The Science Procurement Directorate is responsible for developing the Calls for Proposals in a way that ensures innovations: are submitted by Canadian bidders; include at least 80% Canadian content; have not been sold commercially; demonstrate the bidder has Intellectual Property ownership or rights; and are valued under $500,000 for the standard component and under $1,000,000 for the military component. The Science Procurement Directorate is responsible for evaluating the proposals received against this mandatory criteria prior to evaluation by the National Research Council’s Industrial Research Assistance Program.

17. Industrial Technology Advisors from the National Research Council’s Industrial Research Assistance Program score proposals based on their testing plan (i.e.: the approach to be taken for testing), level of innovation, commercialization capacity, and benefits (i.e. technical readiness level). This process yields a list of pre-qualified proposals that are eligible to be matched with a testing department, but are not guaranteed a contract.

18. The Innovation Selection Committee (comprised of individuals from private sector and academia with expertise in investment, entrepreneurship, innovation, and commercialization trends) is responsible for conducting a review of the Build in Canada Innovation Program process and providing feedback to the program. The role of this committee and the scope of its review are currently being redefined in the context of changes being made to the delivery of the Build in Canada Innovation Program.

19. There are two entities accountable for the matching process. For the Standard Component, the Build in Canada Innovation Program is responsible for helping connect pre-qualified bidders with potential testing departments. For the Military Component, the Defence Validation Committee (comprised of officials from National Defence) reviews military bids and attempts to find suitable matches with the appropriate sectors within the Department of National Defence/Canadian Forces.

20. Once a match is found for an innovation, the Build in Canada Innovation Program is responsible for establishing an Innovation Transfer and Evaluation Agreement (Memorandum of Understanding) with the testing department, as well as a statement of work based on the supplier’s innovation proposal, which is agreed to by the supplier and the testing department. The Science Procurement Directorate is responsible for the overall contract between the supplier and the Build in Canada Innovation Program.

21. The Build in Canada Innovation Program is responsible for covering all costs related to the procurement of the innovation including purchasing price, shipping and transportation, installation, training and maintenance, and support.

22. Government departments and agencies that test innovations are responsible for adhering to the conditions and requirements outlined within the statement of work they negotiate with the supplier concerning the implementation, utilization, and testing of the innovation. Specific responsibilities include providing the agreed upon testing environment, including necessary equipment and facilities, and adequate staff to be trained and to perform the test. Testing departments and agencies are also responsible for providing feedback on the innovation’s performance according to a mutually agreed upon schedule and in a standard format. Feedback includes, but is not limited to, the overall satisfaction of the testing department with the innovation, the degree to which the innovation met the requirements/expectations, challenges encountered, and the overall impression the testing department had of the Build in Canada Innovation Program process. Testing departments are also responsible for tracking and reporting their in-kind contributions and providing this information to the Build in Canada Innovation Program.

23. The Build in Canada Innovation Program is responsible for providing guidance and facilitating the innovation management process for each testing department, which includes the distribution of testing feedback to suppliers.

Resources

24. The Canadian Innovation Commercialization Program was announced in 2010 as a two year, $40 million pilot program. In Budget 2012, the Build in Canada Innovation Program was made a permanent program with the federal government committing an additional $95 million over three years, starting in April 1, 2013 to March 31, 2014 (the 2013 to 2014 fiscal year) and $40 million per year thereafter. The Build in Canada Innovation Program’s financial requirements in the first three years of permanency (fiscal year 2013 to 2014 to fiscal year 2015 to 2016) have been reduced from $95 million, committed in Budget 2012, to $64 million.

25. In 2011, the federal election froze all outreach activities, and as a result the Build in Canada Innovation Program was unable to use all of the funds allocated for these activities during its pilot phase. Of these funds, $10.4 million was reallocated to other priorities within Public Services and Procurement Canada. Additionally, a six-month moratorium on the Build in Canada Innovation Program’s outreach activities in the 2013 to 2014 fiscal year impacted regional outreach, resulting in a drop in promotional activity during that fiscal year. To ensure future funds are dedicated exclusively to program activities, the program’s funding was established as a special purpose allotment when approval was received to make it a permanent program.

26. Further information on the specifics of the annual funding and expenditures of the Build in Canada Innovation Program are outlined in the Efficiency and Economy section of this report.

Logic model

27. A logic model is a visual representation that links a program’s activities, outputs and outcomes; provides a systematic and visual method of illustrating the program theory; and shows the logic of how a program is expected to achieve its objectives. It also provides the basis for developing the performance measurement and evaluation strategies, including the evaluation Matrix.

28. Given the changes made to the Build in Canada Innovation Program subsequent to its permanency, the logic model was revised by the evaluation Team based on a detailed document review, meetings with program managers and interviews with key stakeholders. It was subsequently validated with program staff. The logic model is provided in Exhibit 1.

Exhibit 1: Logic model

Program activities

29. The Build in Canada Innovation Program’s activities are organized under three pillars (Outreach; Call for Proposals; and Innovation Management). The responsibilities for each activity are outlined in the Roles and Responsibilities section above.

Results of the formative evaluation of the Canadian Innovation Commercialization Program

30. A formative evaluation of the Canadian Innovation Commercialization Program was conducted in 2011. The Canadian Innovation Commercialization Program was the pilot program that ultimately led to the creation of the Build in Canada Innovation Program. The conclusions and recommendations from the formative evaluation were considered in the planning of the current evaluation in order to avoid duplication of effort and to ensure that the current evaluation’s findings are relevant to the program.

31. The 2011 Evaluation of the Canadian Innovation Commercialization Program found that the rationale for the program remained sound and there was a continued need for the unique form of assistance provided by the Canadian Innovation Commercialization Program. It also concluded the program was aligned with government priorities related to support for Canadian innovation and for Canadian small and medium enterprises, as well as Public Services and Procurement Canada’s strategic outcome to deliver high-quality, central programs and services that ensure sound stewardship on behalf of Canadians and meet the program needs of federal institutions.

32. The previous evaluation also concluded that the Canadian Innovation Commercialization Program had been successful in attracting small and medium enterprises and federal departments to the program, as well as the use of innovation by the government, as indicated through multiple matches made between suppliers and government departments. However, the formative evaluation was unable to assess the success of the testing and feedback process as only one innovation was being used at the time of the evaluation.

33. The previous evaluation contained two recommendations aimed at including lessons learned in the performance measurement framework and improving the timeliness of contracting. The program provided a management action plan which has since been fully implemented. The Office of Audit and Evaluation has validated the implementation of these actions.

Focus of the evaluation

34. The objectives of this evaluation were to determine the program’s relevance and its performance in achieving its expected outcomes efficiently and economically. The evaluation assessed the Build in Canada Innovation Program for the period from April 2012 to April 2016.

Approach and methodology

35. The Deputy Minister of Public Services and Procurement Canada approved the conduct of this evaluation as part of the 2013 to 2018 Risk-Based Audit and Evaluation Plan. Planning and research took place between April 2015 and April 2016, in accordance with the Standard on Evaluation for the Government of Canada and in conformity with the Public Services and Procurement Canada Policy on Evaluation.

36. While a formative evaluation was conducted on the pilot of this program (Canadian Innovation Commercialization Program), the scope of the program has changed in that a military component has been added. As such, the current evaluation focused on the core questions relating to relevance and performance of both the standard and military components; however, it did not examine aspects relating to the implementation of the program which were examined in the formative evaluation. Given the potential for changes in federal priorities and departmental mandates following the October 2015 federal election, the relevance of the Build in Canada Innovation Program as a whole was examined irrespective of the conclusions of the formative evaluation.

37. More information on the approach and methodologies used to conduct this evaluation can be found in the About the Evaluation section at the end of this report.

Findings and conclusions

38. The findings and conclusions below are based on multiple lines of evidence that were collected during the evaluation. They are presented according to the Core Issues to be Addressed in Evaluations (relevance and performance), as outlined in the former Treasury Board Directive on the evaluation Function, which was in effect during the conduct of this evaluation.

Relevance

39. The assessment of relevance involved an examination of the extent to which the program addressed a continuing need, was aligned with federal priorities and departmental strategic outcome and was an appropriate role and responsibility for the federal government.

Continuing need

40. The evaluation assessed the extent to which there continues to be a need for the Build in Canada Innovation Program. The evaluation found that the need persists for both the standard and military components of the program given the lack of investment from the private sector for innovations at the pre-commercialization stage. From the perspective of the continuing need for the military component of the Build in Canada Innovation Program, the six target areas correspond with the six Key Industrial Capabilities, which are the priority areas for planned defence procurement. The military component also supports the increased focus by the Canadian Armed Forces on the development of original products domestically.

41. As noted in the 2011 evaluation report, the original rationale for the program was that the ability of a business, particularly a small or medium enterprise, to innovate is critical to Canada's economic future. The program was designed to address a gap in federal support for Canadian innovators at the pre-commercialization stage, with a view towards stimulating innovation by Canadian companies through the provision of financial and non-financial support for companies with products and/or services at this critical point in their development.

42. On the financial level, there has been an ongoing shortage of investors for pre-commercialization innovations due to the high level of risk associated with this stage of the innovation continuum. In 2011, an expert panel reported to the Minister of State for Science and Technology that “innovative start-up firms need access to risk capital to build a bridge between their new ideas and commercial viability,” and its belief that “the key to addressing Canada’s well-documented business innovation challenges - including the significant commercialization gap - is to strategically target efforts to support the growth of innovative firms.” The Build in Canada Innovation Program continues to fill this gap in private sector investment.

43. As well, research conducted by the Conference Board of Canada in 2013 ranked Canada fourteenth among fifteen peer countries in investment in innovation. In its 2015 report, the Conference Board reported that Canada had become stronger in investment in innovation, ranking ninth of sixteen peer countries, though there was significant disparity between provinces.

44. On the non-financial level, the continuing need for the Build in Canada Innovation Program is also demonstrated by the strong support for federal procurement as a mechanism to promotion innovation development in the supplier community. Specifically, the evaluation found that federal procurement offers advantages to suppliers that direct financing does not. These advantages to suppliers include establishment of an index client which can be used as a reference in driving future sales as well as real world testing of their innovations. Federal procurement allows the federal government to stimulate business innovation using procurement as a demand-pull instrument (i.e.: creating demand to stimulate growth). By targeting areas of priority for the federal government, the demand-pull effect on innovation also assists the federal government in using Canadian innovation to meet its policy needs.

45. In 2011, the expert panel reported to the Minister of State for Science and Technology that “the government should make better use of its substantial purchasing power to create opportunity and demand for leading-edge goods, services and technologies from Canadian suppliers. This will foster the development of innovative and globally competitive Canadian companies connected to global supply chains, while also stimulating innovation and greater productivity in the delivery of public goods and services.” The report, Innovation Canada: A Call to Action, singled out the Build in Canada Innovation Program as one of Canada’s opportunities to use procurement (as opposed to strictly financing through a grant or contribution program) to enhance industrial innovation. Further, a 2015 survey of 35 member nations conducted by the Organisation for Economic Co-operation and Development (OECD) determined that 80% of responding countries were similarly engaged in some level of innovation procurement.

46. For the military component specifically, a February 2013 special report to the Minister of Public Works and Government Services entitled Canada First: Leveraging Defence Procurement Through Key Industrial Capabilities, indicated that while many allied countries were experiencing either a shift or decrease in military spending in recent years, “Canada continues to re-equip its land, sea and air forces with sustained expenditures on equipment and readiness…” Further, $240 billion has been forecasted between 2008 and 2027 to re-equip all three branches of the Canadian Forces (i.e.: the Canadian Army, the Royal Canadian Navy, and the Royal Canadian Air Force). However, the Panel responsible for the Canada First report indicated that with most of the key decisions on these procurements set to occur in the next few years, the window of opportunity to leverage defence procurement is closing rapidly. As such, and given the lack of available support to innovators, the Panel noted that Canada runs the risk of Canadian entrepreneurs not being in a position to capitalize on this opportunity. The Build in Canada Innovation Program provides an avenue through which Canadian entrepreneurs can simultaneously capitalize on this opportunity while contributing to Canada’s mitigation of this risk.

47. The Canada First report noted that in 2011, the Canadian defence-related industry accounted for more than two thousand companies with more than 70,000 employees, representing an estimated $12.6 billion in annual revenues. The report further noted that many defence-related companies get their start with one or a few initial customers, where a government is almost always one of these first customers. However, the report suggested that domestic military procurement opportunities could be significantly improved with a view towards greater investment and more globally competitive industrial capabilities in Canada, and it endorsed the implementation of Key Industrial Capabilities^{Footnote 1} as a means of fully leveraging economic opportunities for Canadians as a result of planned defence procurement. Further, it encouraged the government to reduce its reliance on Industrial Regional Benefits^{Footnote 2} through the achievement of better balance within the portfolio of procurement options, which includes developing original products domestically. The six Key Industrial Capabilities identified in the report are the same six priority areas identified for the military component of the Build in Canada Innovation Program (outlined in paragraph 6). Given this alignment, the Build in Canada Innovation Program appears to be addressing the issues noted by the Panel responsible for the Canada First report.

48. From the perspective of the key the Build in Canada Innovation Program stakeholders, a March 2016 survey conducted by Public Services and Procurement Canada’s Office of Audit and Evaluation sought the views of officials from other federal entities regarding, among other things, the relevance of the military component of the program. The results of the survey indicated that other federal departments and agencies believed there is a continuing need for the Build in Canada Innovation Program. More specifically, the survey found:

• 92% of respondents agreed or strongly agreed that there was a continuing need for the military component of the Build in Canada Innovation Program
• 81% of respondents agreed or strongly agreed that the Build in Canada Innovation Program was supportive of the objectives of the Defence Acquisition Guide
• 76% of respondents agreed or strongly agreed that the increased use of innovations through the Build in Canada Innovation Program has had a positive impact on the Government’s ability to acquire the right equipment for the Canadian Armed Forces
• 75% of respondents agreed or strongly agreed that the Build in Canada Innovation Program provides the Canadian military with the ability to use innovative products/services
• 71% of respondents agreed or strongly agreed that the Canadian military is open to using more innovative products/services instead of products/services already commercially available

49. Suppliers who had participated in the program also expressed strong support for the continuation of the Build in Canada Innovation Program. Case studies conducted by Public Services and Procurement Canada’s Office of Audit and Evaluation in March 2016 sought the views of companies that had tested innovations through the Build in Canada Innovation Program. The participating innovators were largely supportive of the program, particularly in relation to the appropriateness of the Build in Canada Innovation Program’s procurement-based delivery model.

Alignment with federal priorities and departmental strategic outcome

50. The evaluation assessed the extent to which the Build in Canada Innovation Program is aligned with government-wide and departmental priorities. The evaluation found that the Build in Canada Innovation Program aligns with federal priorities and with the Public Services and Procurement Canada’s strategic outcome for providing high-quality programs and services that meet the needs of federal institutions. The evaluation also noted some degree of alignment between the Build in Canada Innovation Program’s longer term outcomes and the strategic outcomes identified for the Department of Innovation, Science, and Economic Development.

Alignment with federal priorities

51. Budget 2016 indicated the Government’s view that, to achieve economic growth and prosperity, Canada must invest more in innovation. It is the Government’s position that for Canada to become a global innovation leader, more of the country’s creative entrepreneurs and innovative companies will need to seize global market opportunities to grow their businesses. Throughout Budget 2016, economic growth is repeated as a key priority, along with job creation. The Prime Minister linked these goals to mandate commitments of several Ministers, including the Minister of Public Services and Procurement whose Mandate Letter outlines the expectation that the Minister will “ […] modernize procurement practices so that they […] include practices that support our economic policy goals.” As a program that aims to promote innovation growth, which is linked to job creation, the Build in Canada Innovation Program aligns with these priorities.

52. With respect to the Government’s priorities related to the Canadian Armed Forces specifically, one of the commitments outlined in the 2015 Speech from the Throne was to invest in building a leaner, more agile, better-equipped military. This commitment was supported in Budget 2016, which stated that the Government is committed to providing greater security for Canadians, which includes ensuring that the Canadian Armed Forces have the equipment and personnel required to protect Canadian sovereignty, defend North America, provide disaster relief, conduct search and rescue, support United Nations peace-keeping operations, and contribute to the security of allies and to coalition operations abroad. To this end, the investment in innovations through the Build in Canada Innovation Program supports the Government’s identified priorities related to innovation development to seize global opportunities as well as ensuring the Canadian Forces have the right equipment.

Alignment with the Public Services and Procurement Canada strategic outcome

53. The Build in Canada Innovation Program’s alignment with Public Services and Procurement Canada’s strategic outcome for procurement-related services to federal departments was previously established in the 2011 Evaluation of the Canadian Innovation Commercialization Program. Public Services and Procurement Canada’s strategic outcome of providing high quality services, including the acquisition of goods and services, on behalf of federal departments and agencies aligns with the Build in Canada Innovation Program’s outcomes related to acquiring innovative goods and services for Federal Government use. The Government also uses its substantial purchasing power as a means to achieve social and economic goals for Canadians and Canadian business. This aligns with the Build in Canada Innovation Program outcomes related to supporting the procurement of innovations as a means to help the successful launch of innovations in the marketplace. The procurement-based delivery model of the program also aligns with Public Services and Procurement Canada’s roles and responsibilities as the common service provider for procurement services in the federal government. This has been reconfirmed as part of recent corporate reporting documents, such as the 2014 to 2015 Public Services and Procurement Canada Departmental Performance Report and the Acquisitions Branch Business Plan, which highlight the Build in Canada Innovation Program as one of the ways in which the Department is delivering innovative, effective and efficient procurement services to federal organizations. The alignment between the Department’s strategic outcome and the Build in Canada Innovation Program is also applicable to the military component of the Build in Canada Innovation Program, as Public Services and Procurement Canada’s strategic outcome is equally applicable to the delivery of military procurement.

54. The evaluation also noted that some of the intended outcomes of the Build in Canada Innovation Program align with the strategic outcomes of the Department of Innovation, Science, and Economic Development. Specifically, program outcomes related to supporting improved performance by better positioning Canadian companies to commercialize their innovations and Canadian innovations being successfully brought to market appear to align with that department’s strategic outcomes related to enhancing Canada’s innovation performance. A feature of Minister’s Mandate letters is a communicated desire to work horizontally to achieve benefits for Canadians. The Build in Canada Innovation Program is an example of how Departments with complementary objectives work together to achieve common results for Canadians.

Alignment with federal and departmental roles and responsibilities

55. To determine if the program is an appropriate role and responsibility for the federal government and Public Services and Procurement Canada, the evaluation reviewed the extent to which the Build in Canada Innovation Program could be devolved to the private sector or the provinces and whether the program complements, duplicates or overlaps with other federal government functions. Additionally, the evaluation examined the extent to which the outcomes of the Build in Canada Innovation Program are best aligned with the roles and responsibilities of Public Services and Procurement Canada.

56. The evaluation found that given the national scale of this program, its delivery by the federal government is most logical. Additionally, the Build in Canada Innovation Program appears to complement existing Government of Canada programs which support innovation development and commercialization. The evaluation found a strong preference among suppliers for the procurement-based program delivery model over a grants/contribution/loan approach.

Extent to which the Build in Canada Innovation Program is a federal role and responsibility

57. As outlined in the section above regarding the continuing need for the program, the raison d’être for the Build in Canada Innovation Program relates to the lack of private sector investment in pre-commercialization innovations across Canada. Given this, the evaluation found that the Build in Canada Innovation Program is aligned with federal roles and responsibilities as the program is directed at stimulating an element of the national economy.

Further, the evaluation concluded that, based on the evidence supporting the continuing need for this program, outsourcing the responsibility for the Build in Canada Innovation Program to the private sector would not be feasible. Given the Build in Canada Innovation Program’s intended outcomes of stimulating as aspect of the economy in which the private sector itself is deficient, and the need to ensure nationwide consistency for such an initiative, the evaluation concluded that federal government delivery of the program is most appropriate. As well, the program aims to increase the use of innovations in the federal government, which is a responsibility owned by the Government of Canada. Similarly, with respect to the viability of devolving the program to the provinces, the nature of the target areas within both the standard and military components of the Build in Canada Innovation Program are such that the Build in Canada Innovation Program would not align with the mandates of the provincial governments. Additionally, per the National Defence Act, the defence of Canada is a federal role and responsibility which cannot be devolved to the provincial level, meaning the military component of the Build in Canada Innovation Program must reside with the federal government. Further, the provinces would not be in a position to ensure uniform implementation of this type of programming across the country and, as investment in innovation varies considerably across provinces, the need for a nationally-focused strategy to ensure consistency is indicated. As such, the evaluation concluded that devolution of the program to the provinces would also not be feasible.

Extent to which the Build in Canada Innovation Program complements, duplicates or overlaps with other federal government functions

58. The Build in Canada Innovation Program appears to complement existing federal programing for innovators. Case studies of eight suppliers who had completed testing with the program found that while all suppliers had used some form of federal funding (predominantly the National Research Council’s Industrial Research Assistance Program), there were no other programs like the Build in Canada Innovation Program that offered the benefits of large-scale client testing and a first time buyer. All suppliers interviewed also noted that the procurement-based program design of the Build in Canada Innovation Program is preferable to a grant or contribution approach.

59. A jurisdictional review conducted by the Office of Audit and Evaluation examined 13 federal programs and found that the Build in Canada Innovation Program did not overlap or duplicate existing federal programming for innovation commercialization. The review found that the Build in Canada Innovation Program was the only program providing a setting in which pre-commercialized innovations could be used and tested, whereas other existing programming provided either funding by way of a grant, contribution or loan, or provided general business mentorship. None of the programs examined provided a forum for suppliers to test their innovations.

Extent to which the Build in Canada Innovation Program is aligned with the roles and responsibilities of the Department

60. The Build in Canada Innovation Program’s procurement-based delivery model aligns with the mandate and accountabilities of the Minister of Public Services and Procurement for the acquisition of goods and services on behalf of other federal entities as laid out in the Department of Public Works and Government Services Act and the Defence Production Act.

61. The Department of Public Works and Government Services Act establishes Public Services and Procurement Canada as the common service agency for the Government of Canada, and provides the foundation for the Department’s roles in providing the Government of Canada’s departments, boards and agencies with common services in support of their programs. This support includes, but is not limited to, acquisitions on behalf of federal organizations, which is a key component of the Build in Canada Innovation Program. Similarly, the Treasury Board of Canada’s Common Services Policy establishes roles and responsibilities for Public Services and Procurement Canada in acquisition services for federal organizations. As a program that is concerned with the acquisition and use of innovations by federal organizations, the Build in Canada Innovation Program is well-aligned to these elements of Public Services and Procurement Canada’s roles and responsibilities.

62. Further, the Defence Production Act provides the foundation for supplementary roles and responsibilities for the Minister of Public Services and Procurement with regards to the Build in Canada Innovation Program’s military component. Under the Defence Production Act the Minister of Public Services and Procurement is vested with authorities specific to defence supplies and contracts required by federal departments, as well as exclusive authority for defence supplies and projects required by the Department of National Defence. As such, there is strong alignment between the military component of the Build in Canada Innovation Program and the roles and responsibilities granted the Minister under the Defence Production Act.

Conclusions: relevance

63. Overall, the evaluation found that the program remains relevant, and there is a continuing need for the Build in Canada Innovation Program. The evaluation found that the Build in Canada Innovation Program aligns with the roles and responsibilities and priorities of the federal government and complements the existing suite of federal programs directed at supporting innovation in the Canadian economy.

64. The evaluation found that there is alignment between the program and Public Services and Procurement Canada’s strategic outcome. The Build in Canada Innovation Program’s procurement-based delivery model produces activities and outputs which are aligned with the Department’s roles and responsibilities as the common service organization responsible for procuring goods and services on behalf of the Government of Canada. This is additionally the case as relates to the military component of the Build in Canada Innovation Program, as Public Services and Procurement Canada has more specific roles and responsibilities for the procurement of defence-related supplies flowing from authorities granted to the Minister under the Defence Production Act. The Build in Canada Innovation Program’s outcomes related to companies and federal departments partnering to support a culture of innovation and related to federal departments using innovations also align to Public Services and Procurement Canada’s strategic outcome. As noted above, the evaluation also found that some program outcomes appear to align with the strategic outcomes of the Department of Innovation, Science, and Economic Development. The Build in Canada Innovation Program is an example of how Departments with complementary objectives work together to achieve common results for Canadians.

Performance

65. Performance assesses the extent to which the Build in Canada Innovation Program was able to achieve its objectives and the degree to which it was able to do so in a cost-effective manner that demonstrated efficiency and economy.

Outcome achievement

66. The evaluation examined the degree to which the Build in Canada Innovation Program achieved its intended immediate, intermediate, and final outcomes, as outlined in the logic model in Exhibit 1. The intended outcomes of the program are identified below, followed by an assessment of the extent to which they were achieved.

Immediate outcome #1: Small and medium enterprises are interested in engaging with the federal government.

67. The evaluation examined the extent to which industry, particularly small and medium enterprises, was interested in the Build in Canada Innovation Program during the period examined. The evaluation found that there is a relatively consistent level of interest by suppliers, particularly small and medium enterprises, to participate in the Build in Canada Innovation Program outreach activities.

68. One of the primary activities of the Build in Canada Innovation Program is to conduct outreach; the Build in Canada Innovation Program uses outreach activities to raise awareness of and garner interest in the program with both potential suppliers and potential testing departments. As such, much of the success of the program in meeting its objectives relies on the effectiveness of the Build in Canada Innovation Program’s outreach activities.

69. The Build in Canada Innovation Program uses a number of outreach activities, which are primarily focused on small and medium enterprises. These include a website, an information line, and participation at trade shows and industry events. The most proactive of these are the events, nearly 1,600 of which the Build in Canada Innovation Program has participated in since 2010. As Exhibit 2 indicates, the annual participation by the Build in Canada Innovation Program in outreach events has varied, from as low as 98 events in 2013 to as high as 540 events in 2011. In 2014, the most recent year for which data was available, the Build in Canada Innovation Program participated in 318 events and assisted nearly 11,000 suppliers. The vast majority (97%) of the attendees at events were small and medium enterprises.

70. Overall, the data (Exhibit 2) suggests that the Build in Canada Innovation Program outreach activities continue to generate supplier interest in the program, particularly by small and medium enterprises, and that there is ongoing interest by suppliers in attending events to learn more about the Build in Canada Innovation Program. While 2013 saw a decrease in the number of events attended and suppliers assisted, this was largely attributable to the moratorium placed on the Build in Canada Innovation Program’s outreach activities, and therefore beyond the program’s control. Conversely, the number of suppliers attended per event rose in that year, indicating that the Build in Canada Innovation Program continued to work towards its intended outcomes despite the constraints that were placed on it. Additionally, while it appears that the Build in Canada Innovation Program may have been operating more efficiently in 2013 by averaging 83 suppliers assisted per event, the average dollars spent per supplier assisted is within the variable averages over the period of 2010 and 2014. As well, it should be understood that to effectively reach suppliers across Canada as part of the national delivery of the program, the Build in Canada Innovation Program has to participate in events outside of urban centers, where the population base for the events will be smaller and the costs associated with travel may be higher.

Source: Build in Canada Innovation Program Annual Reports to the Treasury Board of Canada

71. The Build in Canada Innovation Program also uses its website and information line to engage with suppliers. In the spring of 2015, the Build in Canada Innovation Program conducted strategic consultations to obtain industry feedback on key areas in which the program could improve. One of the primary concerns raised by industry stakeholders who participated in the strategic consultations was the need for improved clarity of information and process transparency, with respondents indicating a need for improvements to the website; access to resources such as webinars; information sessions and bulletins; and updates on the progress of their application or proposal. Several respondents suggested offering case studies and examples of past successful applicants to assist with both the application processes and the Build in Canada Innovation Program’s promotion to businesses.

72. In response to these concerns, key improvements have been made to streamline the process to apply and access the Call for Proposal information essential to the Build in Canada Innovation Program’s intake process. Most notably the Build in Canada Innovation Program has reduced the number of ‘clicks’ to the application submission form. This has led to increased access to key documentation. The Build in Canada Innovation Program also hosts webinars with bidders interested in the program. Further, the Office of Small and Medium Enterprises and Strategic Engagement regional offices have held reverse-trade shows/forums that bring government departments together with potential and past innovators to discuss opportunities, showcase innovations, and identify some of the priorities and needs of government organizations. The Build in Canada Innovation Program has also developed videos, infographics and learning e-module tools to better educate potential innovators on the Build in Canada Innovation Program, and support its social media presence.

73. Overall, the evaluation found that suppliers, largely small and medium enterprises, remain interested in engaging with the federal government as evidenced by the ongoing participation by industry at events where the Build in Canada Innovation Program continues to assist a relatively consistent number of suppliers. Stakeholder consultations have suggested areas for improvement, and the program has indicated that it is working to address the identified areas.

Immediate outcome #2: Federal government departments have knowledge of and take interest in testing innovations.

74. The evaluation examined the extent to which federal government departments have knowledge of and take interest in testing innovations. The evaluation found that, as with suppliers, the Build in Canada Innovation Program’s achievement of this outcome is largely dependent on the outreach activities undertaken by the program. In the Office of Audit and Evaluation’s March 2016 survey of key federal government stakeholders, nearly half of respondents indicated their involvement with the program began as a result of the Build in Canada Innovation Program outreach.

75. Participating innovators have found the lack of participation by departments and agencies in the program to be a challenge. Respondents to the Build in Canada Innovation Program’s strategic consultations with industry highlighted challenges in finding government testing departments, with 24% of respondents considering this to be an area for improvement. This reinforces the importance of the Build in Canada Innovation Program’s outreach to federal organizations. However, part of the challenge in attracting additional departments and agencies, so as to provide more matching options for innovations, could be related to a lack of experience in participating in a procurement process by government officials. The March 2016 survey of key federal government stakeholders conducted as part of this evaluation found that the Build in Canada Innovation Program was the first experience in a federal procurement process for 40% of respondents. Overall, 62% of respondents indicated that the Build in Canada Innovation Program could improve upon its outreach to government organizations.

76. The Build in Canada Innovation Program is working to increase its engagement with federal departments and agencies. To date, the Build in Canada Innovation Program has established agreements with 30 departments and agencies and has engaged with all remaining federal organizations identified in Schedule I, I.1, and II of the Financial Administration Act. In addition, all regional development agencies are engaged through the Office of Small and Medium Enterprises and Strategic Engagement regional offices and the Build in Canada Innovation Program team.

77. While actual participation by federal departments is discussed in more detail below, the evaluation found that the Build in Canada Innovation Program’s outreach is key to ensuring departments are knowledgeable about the program. The evaluation also found that there may be areas for improvement in the Build in Canada Innovation Program’s outreach so as to garner more awareness and interest by departments in participating in the program.

Immediate outcome #3: Private sector innovators use the program to test their innovations.

78. The evaluation examined the extent to which industry used the Build in Canada Innovation Program to test their innovations. The evaluation found that suppliers matched with departments are completing testing. Additionally, more innovations are pre-qualifying and therefore more suppliers are able to participate in the program to test their innovations. The evaluation also found that the geographical distribution of suppliers participating in the Build in Canada Innovation Program across Canada is representative of population distribution across Canada.

79. The evaluation noted that suppliers who have been matched with a participating organization are completing the testing of their innovations (Exhibit 3). However, as the testing period of each innovation is largely dependent on the nature of the innovation itself, and the fact that there is no standardized timeframe for the completion of testing, this measure is potentially misleading. Additionally, Calls for Proposals 004 and 005 were recently released and therefore the percentage of matched innovations having completed testing is understandably lower than the previous three Calls for Proposals. Further, the extent to which suppliers have control over their ability to use the Build in Canada Innovation Program to test their innovations is impacted by the extent to which federal departments are willing to test innovations.

Source: Program data

80. To address this, the evaluation examined the extent to which suppliers attempt to participate in the Build in Canada Innovation Program to test their innovations through the submission of proposals. However, this metric is not without deficiency. The ongoing changes being implemented to improve the Build in Canada Innovation Program make it difficult to assess year over year participation by industry. For example, it would be expected that there should be an increase in the number of proposals received by the Build in Canada Innovation Program as supplier awareness of it increases, yet the number of proposals received per Call for Proposals has generally declined year over year. This is because the technology readiness level required of proposals has changed over time, with a view to attracting innovations that are at a more advanced stage of development and ready to test in an operational environment.

81. Consequently, the evaluation found that with the exception of the third Call for Proposals, the number of pre-qualified proposals overall has increased year over year, indicating that more suppliers are involved in the program and more innovations are being tested. In addition, the percentage of prequalified proposals relative to the total proposals received is also increasing, indicating that the Build in Canada Innovation Program is receiving more high quality proposals whose technology readiness level makes the related innovations more appropriate selections for testing.

Source: Build in Canada Innovation Program 2014 to 2015 Report to the Treasury Board of Canada and Program Data

82. However, not all proposals received, pre-qualified, and/or matched with a testing department are from first time participants in the Build in Canada Innovation Program. Some companies have submitted proposals for testing several different innovations as part of subsequent Calls for Proposals, while others have submitted the same proposal several times. As such, the actual number of suppliers participating in the Build in Canada Innovation Program is less than the number of proposals received.

83. Since the Build in Canada Innovation Program is a national program, it is important that suppliers across Canada are able to participate in it. As such, the evaluation examined whether suppliers across Canada are attempting to test their innovations through the Build in Canada Innovation Program by comparing the number of proposals submitted and pre-qualified to the general population distribution of Canada.

84. In terms of regional distribution of the engagement by industry in the Build in Canada Innovation Program, while two-thirds of proposals received and pre-qualified by the program are from Ontario and Quebec, the percentage distribution of proposals received and pre-qualified by the Build in Canada Innovation Program for the most part mirrors the percentage distribution of population across the provinces and territories, as outlined in Exhibit 5. As such, it would appear that the program has been effective in attracting relatively representative supplier participation across Canada.

Exhibit 5: Geographical distribution - proposals received and pre-qualified relative to population
(Calls for Proposals 001 through 005)

Sources: Build in Canada Innovation Program 2014 to 2015 Report to the Treasury Board of Canada and Statistics Canada, CANSIM 051-0005 (2015 Q4)

85. While a perfectly even distribution of suppliers submitting proposals and being pre-qualified should not be expected vis-à-vis the distribution of Canada’s population, it should be expected that the Build in Canada Innovation Program outreach is conducted on a national scale. Per Exhibit 5, it would appear that relative participation in the program across the Prairie provinces (Alberta, Saskatchewan, and Manitoba) is lacking relative to other provinces, while relative participation by suppliers in Ontario is higher than in other provinces.

86. Since the Build in Canada Innovation Program allows companies to apply for testing for multiple products, the evaluation team examined whether the distribution of unique suppliers also aligned with the general population distribution of the Canadian population. As outlined in Exhibit 6, the percentage of unique suppliers submitting proposals to test their innovations through the program generally aligned with the Canadian population distribution.

Exhibit 6: Geographical distribution – unique companies applying and pre-qualified relative to population
(Calls for Proposals 001 through 005)

Sources: Program data and Statistics Canada, CANSIM 051-0005 (2015 Q4)

87. Based on the above, the evaluation concluded that the program has made progress towards the achievement of this outcome, as evidenced by the fact that suppliers from across Canada are participating in the program, through the submission of proposals, with an increasing number and proportion of proposals pre-qualifying for testing. Additionally, suppliers matched with participating departments are completing testing.

Immediate outcome #4: Government departments test innovations

88. The evaluation examined the extent to which federal government departments participate in the testing of innovations. The evaluation found that participation by federal departments and agencies is increasing over time, with some participating departments having conducted testing of multiple innovations.

89. The degree of participation by federal departments has varied for each Call for Proposals, with the two most recently completed Calls for Proposals saw the highest level of participation by departments to date. The percentage of pre-qualified proposals being matched has been relatively strong, with the exception of the third Call for Proposals. While the proposals received under the sixth Call for Proposals^{Footnote 3} appears low, this is a result of the fact that it was the most recent call to close, and that it takes, on average, 181 business days to match an innovation with a department. As such, there are innovations that are still being matched under the sixth Call for Proposals.

Source: Program data

90. As Exhibit 7 demonstrates, the Build in Canada Innovation Program has increased the number of participating departments in recent years which should help address this issue. Additionally, the program has maintained an average ratio of around 2.5:1 for matched innovations to participating departments, which indicates that the program is relying on participating departments to test multiple innovations. However, it should be understood that a disproportionate reliance will remain on the Department of National Defence to test many of the innovations within the military component of the program.

91. Per the above, the program has made progress towards the achievement of this outcome. While the program is relying on departments to test multiple innovations, the nature of the Build in Canada Innovation Program, more specifically the military component, means that there will be a disproportionate reliance on the Department of National Defence to test innovations.

Intermediate outcome #1: Canadian companies are better prepared to enter the marketplace with their innovations due to feedback gained from the department and the prior purchase of the innovation

92. The evaluation examined the extent to which participation in the Build in Canada Innovation Program has helped better prepare Canadian companies to enter the marketplace with their innovations. The evaluation found that the Build in Canada Innovation Program has been effective in contributing to the preparedness of participating suppliers to commercialize their innovations, both from the provision of testing feedback and from the benefits gained from having a first time buyer.

Usefulness of testing feedback

93. Throughout the matching and testing process, the Build in Canada Innovation Program administers surveys to the matched suppliers and testing departments. Respondents to the post-innovation testing surveys between September 2014 and March 2016 noted that the Build in Canada Innovation Program testing led or contributed to multiple benefits, including improved product functionality/usability (73%). Overall, 97% of respondents indicated they had used the feedback received from the Build in Canada Innovation Program testing to further product development or for other applications.

94. An examination of eight companies that have completed the Build in Canada Innovation Program testing found that the feedback provided through the Build in Canada Innovation Program has led to improvements in their innovations. Three of the suppliers noted that the feedback received from the testing department significantly reduced their time to market, with one supplier noting that the feedback reduced their time to market from five years to two. While companies generally noted that their innovations were market-ready or very close to market-ready prior to participation in the Build in Canada Innovation Program, some companies indicated they experienced challenges relating to technical issues in the development of their innovation which necessitated the need for testing of the product by a client.

95. In the spring of 2015, the Build in Canada Innovation Program conducted a request for information to obtain feedback from industry and federal organizations on the program, in which the majority of respondents (82%) were from industry. The request for information found that many respondents believe the Build in Canada Innovation Program is an important mechanism to test innovations and generate feedback for innovators. Almost two thirds of respondents (62%) believed that the program was of greater or equal importance compared to other innovation support programs they had experience with; only 13% believed the program was less important. Respondents that felt the Build in Canada Innovation Program was more important than other programs often noted that the program fills a “commercialization gap” whereby innovators are ready to commercialize but have difficulty finding suitable testing grounds. Further, 92% of respondents to the post-innovation testing surveys administered by the program between September 2014 and March 2016 indicated that the Build in Canada Innovation Program testing experience was important or very important to the commercialization of their innovations.

Impact of Government of Canada being a first-time buyer

96. Multiple respondents to the Build in Canada Innovation Program’s spring 2015 request for information indicated that they saw the program as an opportunity to gain validation for their innovation or that having the opportunity to identify the federal government as a first buyer or reference sale for their product was useful in marketing the product. Per the post-innovation testing surveys administered by the program between 2014 and 2016, suppliers noted the Build in Canada Innovation Program led or contributed to: enhanced credibility with private and/or public sector clients (67%); and the visibility/exposure needed to launch the product (60%). Overall, 91% of respondents have used their Build in Canada Innovation Program contract as a reference for other sales.

97. Additionally, an examination of eight companies that had completed the Build in Canada Innovation Program testing confirmed that the Build in Canada Innovation Program was valued as a way to get a reference customer that they could use to market their innovation and attract additional sales. Many of the suppliers noted that they include the Build in Canada Innovation Program testing reference in sales presentations and in discussions with potential clients.

98. Suppliers consulted as part of the case studies for this Evaluation noted that there was valuable information to obtain beyond the specific testing feedback. For example, for some this was the first time experiencing a large scale deployment in a defined commercial setting providing valuable “real world” experiences and lessons learned. When asked whether an alternative form of delivery for the program (i.e.: provision of a grant, contribution, or loan) would be more beneficial, none of the suppliers interviewed identified a preferred delivery model.

99. Overall, the evaluation found that the program is progressing towards the achievement of this outcome. Suppliers found the Build in Canada Innovation Program to be unique in its delivery in that it provided a reference customer and filled the need for testing support for commercialization.

Intermediate outcome #2: Federal departments support innovative Canadian businesses

100. The evaluation examined the extent to which the Build in Canada Innovation Program contributes to federal departments supporting innovative Canadian businesses. The evaluation found that participating departments were supportive of the program and believed the feedback provided to suppliers was beneficial in supporting commercialization. However, as previously noted, the evaluation also found that the Build in Canada Innovation Program could improve its achievement of intended outcomes by taking steps to increase the participation of federal departments and agencies in the program.

101. There are multiple ways in which the federal government supports innovative Canadian businesses. For the Build in Canada Innovation Program specifically, the program needs federal departments to participate to progress towards the achievement of this outcome. As previously noted, participation by departments has increased each year. However, future participation by departments is, to a certain extent, dependent on the degree to which departments believe their efforts in testing innovations are yielding positive results, both for the department and for suppliers.

102. The March 2016 survey of federal government stakeholders who had participated or expressed interest in the Build in Canada Innovation Program found a considerable amount of support for the program. In particular, the two most prevalent motivations federal government stakeholders chose for participating in the Build in Canada Innovation Program were to identify and assess innovative goods or services of potential value to their organization (40%) and the desire to support innovation amongst Canadian companies or industries (24%). As Exhibit 8 outlines, federal government stakeholders generally held favourable views of the impact of the Build in Canada Innovation Program on innovators.

Source: March 2016 Office of Audit and Evaluation Survey of Federal Government Stakeholders

103. With respect to the specific feedback provided to suppliers during testing, federal stakeholders believed the feedback would support the commercialization of the tested innovations in a variety of ways, as outlined in Exhibit 9, the most notable relating to the performance validation and technical improvements of the innovation.

Source: March 2016 Office of Audit and Evaluation Survey of Federal Government Stakeholders

104. Overall, the evaluation found that departmental participation in the program is increasing and that federal stakeholders believe the Build in Canada Innovation Program is providing value in many ways to suppliers. As such, the evaluation concluded that the Build in Canada Innovation Program provides a mechanism through which federal departments can support innovative Canadian businesses.

Final outcome #1: Canadian innovations are successfully brought to market

105. An assessment of the Build in Canada Innovation Program’s longer term outcomes is particularly challenging given the relatively short period of time in which the program has been in operation and the fact that the time it takes to commercialize a good or service is not standardized. Nevertheless, the evaluation team examined the degree to which Canadian innovations that participated in the Build in Canada Innovation Program were successfully brought to market. The evaluation found that many of the suppliers had commercialized their innovations since participating in the program. For those who had yet to commercialize, the indication from suppliers was that participation in the program was valuable in progressing their innovations towards ultimate commercialization.

106. As previously noted in this report, suppliers found that the lack of participation by departments and agencies in the program was a challenge. Despite this, the post-innovation testing surveys administered by the program between September 2014 and March 2016 found that 83% of respondents indicated their innovation had been commercialized since participating in the program. Additionally, respondents indicated that following their Build in Canada Innovation Program contract, there have been additional sales of their innovations to several sectors, as outlined in Exhibit 10, with the private sector being the primary sector purchasing innovations after the Build in Canada Innovation Program testing.

Source: Program data

107. The evaluation team’s examination of eight completed innovations found that five of the eight innovations (63%) had commercialized since participating in the Build in Canada Innovation Program. The three remaining companies had not yet commercialized their products, though these companies indicated that the information they received from their testing experience with the Build in Canada Innovation Program led to improvements in either the products themselves or the core technology used for similar innovations. Two of the three companies whose innovations had not yet commercialized indicated that, pending further product development in light of feedback received from participating in the Build in Canada Innovation Program, they intended to commercialize in the next year or two.

108. Based on the available information, it can be concluded that the Build in Canada Innovation Program is progressing towards the overall achievement of this outcome. While not all companies examined have commercialized, participation in the program provided a valuable contribution towards the commercialization goals of participating companies.

Final outcomes #2 and #3: Companies and federal departments partner to support a culture of innovation and federal departments use innovations

109. The evaluation examined the degree to which companies and federal departments partner to support a culture of innovation and the extent to which federal departments use innovations. There are many factors that impact culture, and it could take considerable time for a program to influence culture change. Further, the extent to which federal departments use innovations is linked, to a certain extent, to organizational culture. As the amount of influence the Build in Canada Innovation Program has to effect change in the organizational culture of the Government of Canada is minimal, the evaluation examined the achievement of the two final outcomes from a proof of concept perspective.

110. The evaluation found that the existence of perceived barriers to using innovations could be limiting the program’s ability to contribute to a culture of innovation in the federal government. While suppliers are interested in partnering with the federal government, as outlined earlier in this report, the achievement of these outcomes was limited by the extent to which federal departments and agencies were open to leveraging innovations to meet their needs.

111. Feedback from industry indicated that finding departments with which to match was a challenge. When asked about the extent to which federal organizations would be open to purchasing early-stage innovations outside the Build in Canada Innovation Program, federal government stakeholders were mixed in their perceptions: 28% indicated they would be open to testing innovations outside the program, while 21% indicated they would not be open to testing innovations outside the program (the remaining were neutral). Additionally, the Office of Audit and Evaluation’s March 2016 survey of federal government stakeholders found there were multiple perceived barriers that could be precluding federal departments and agencies from using innovations to meet their programming needs, which are outlined in Exhibit 11. It is important to note, however, that these are not perceived barriers to participating in the Build in Canada Innovation Program; rather these are perceived barriers to procuring innovative goods and services in general. As such, these results should be seen in the context of the culture of innovation in the federal government as a whole rather than an as indicator of the Build in Canada Innovation Program’s performance.

Source: March 2016 Office of Audit and Evaluation Survey of Federal Government Stakeholders

112. While the above data must be taken in the context of varying degrees of understanding of the requirements of federal procurement address their needs is noteworthy. While it is understood that the program has improved on the issues experienced during its pilot phase (Canadian Innovation Commercialization Program), particularly with respect to the time it took the program to establish contracts, the prevalence of stakeholders maintaining the perception that the procurement process is a barrier merits attention by the program.

113. As outlined in Exhibit 10, slightly less than one third of suppliers (31%) reported sales to the Government of Canada following completion of the Build in Canada Innovation Program testing despite the funds invested by the program and the in-kind contributions invested by testing departments. The spring 2015 request for information conducted by the program found that one of the most important reasons participants applied to participate in the Build in Canada Innovation Program was the potential for future government contracts. Additionally, 43% of respondents believed that supporting additional follow-on sales to the federal government after a Build in Canada Innovation Program test was important and should be considered as part of an increase to the mandate of the Build in Canada Innovation Program.

114. The program has recently made changes to allow departments the option of pursuing follow-on sales after completion of testing. This means innovations that received a contract under the Build in Canada Innovation Program during the fourth and fifth Calls for Proposals were eligible for further testing for up to two years from the date of their pre-qualification or their initial contract, depending on the time of their contact award. Innovators, who started the Contract Award Process but had not received an initial contract were also eligible for testing up to two years from the date of their pre-qualification. In theory, this should increase the extent to which federal departments and Canadian suppliers partner for a culture of innovation and the ability of federal departments to use innovations for those directly involved in the program.

Conclusions: outcome achievement

115. The evaluation found that the Build in Canada Innovation Program has made progress towards achieving its intended outcomes. Small and medium enterprises across Canada were interested in the program and supplier participation in the program remains strong, with an increasing proportion of proposals being prequalified and matched with testing departments and completing testing. Further, the Build in Canada Innovation Program has been effective in contributing to the preparedness of participating suppliers to commercialize their innovations. Though suppliers have indicated that the Build in Canada Innovation Program could improve its provision of information on the program and the progress of their proposals.

116. Similarly, interest and participation by federal departments has increased over time, with some participating departments having tested multiple innovations. Participating departments are supportive of the program and believe the feedback provided to suppliers is beneficial in supporting commercialization. However, outreach to federal organizations could be improved to provide more options for matching innovations. The evaluation found that the existence of potential barriers to using innovations in the federal government could be limiting the program’s ability achieve its intended outcomes and contribute to a culture change in the federal government.

117. The program also appears to be progressing towards the achievement of its final outcomes. The majority of the suppliers examined as part of the case studies for this evaluation had commercialized their innovations since participating in the program, with many indicating that the program had provided them with a valuable reference client which led to additional sales. Those who had yet to commercialize indicated that participation in the program was valuable in progressing their innovations towards ultimate commercialization. As noted above, however, perceived barriers to using innovations within the federal government may be limiting the program’s ability to more fully contribute to commercialization of innovations, which in turn may preclude the Build in Canada Innovation Program from fully achieving its final outcomes related to partnering to support a culture of innovation and federal departmental usage of innovations. Recent changes to the program, such as the ability for federal departments to partner with industry for follow-on sales after completion of testing, may help address these perceived barriers.

Economy and efficiency

118. Demonstration of economy and efficiency is defined as an assessment of resource utilization (i.e. program inputs) in relation to the production of outputs. Economy refers to minimizing the use of program inputs. Efficiency refers to the extent to which resources are used such that an equivalent or greater level of output is produced using the same amount or fewer resources. A program has high economy and efficiency when financial resources maximize outputs and minimize inputs.

Economy

119. The evaluation examined the extent to which the program was being delivered economically, measured by an examination of the Build in Canada Innovation Program’s use of its inputs. The evaluation found that the Build in Canada Innovation Program has improved its ability to expend its funds economically.

120. The relative recency of the Build in Canada Innovation Program, combined with the funding instabilities it experienced in its first few years, makes a conclusive determination of program economy challenging. Nevertheless, the evaluation found that the Build in Canada Innovation Program has improved in its ability to expend the allocated funds on innovations (Exhibit 12). From April 1, 2014 to March 31, 2015 (the 2014 to 2015 fiscal year) saw a considerable lapse of funds – the result of issues beyond the program’s control – which caused delays in the release of the fourth Call for Proposals. This resulted in the fourth and fifth Calls for Proposals being released within six months of one another and a strain on resources, which were required to match innovations from two Calls for Proposals at the same time. However, the subsequent increase in stability in the Build in Canada Innovation Program’s operating environment has resulted in more funding being used on innovations as a percentage of total budget.

121. In the 2015 to 2016 fiscal year, 85% of the program’s total budget was dedicated to spending on innovation-related work, demonstrating an economical use of resources to fund a higher amount of innovations (see Exhibit 13 for percentage breakdown of funding expenditures). This represented the highest proportional spending on innovations in the program’s history. It should be noted that the 2015 to 2016 fiscal year period had significantly more pre-qualified proposals than in previous years due to the delay in the release of the fourth Call for Proposals.

Return to table note *   Source: Program Data reallocated funds are not included in the total budget

122. The Build in Canada Innovation Program is staffed with 14 full-time-equivalent positions, many of which have historically been filled on a temporary basis by students and casual employees. While using students and casual staff does provide some flexibility for managing operating budgets, leveraging temporary staff to fill approximately 25% of the program’s available full-time-equivalent positions can also have a negative impact on productivity. Specifically, higher turnover rates associated with temporary resources tend to result in a loss of corporate memory (with the exception of the few students that have become indeterminate employees), lost time as new employees follow the learning curve associated with any new position, and a higher expenditure of management’s time and effort on recruitment and training. With more stable funding, there should be a decreased need for the Build in Canada Innovation Program to maintain flexibility in staffing at the expense of its productivity.

Efficiency

123. The evaluation examined the extent to which the Build in Canada Innovation Program was financially and operationally efficient. The evaluation found that greater financial efficiencies were found when there was greater stability in the program’s operations. Additionally, the program was meeting its target of eight months to award contracts following bid submission.

124. The Build in Canada Innovation Program experienced instability in its operating environment in the early years of its existence. As a pilot program, the degree to which long term planning could have been effectively undertaken was relatively limited. While the program was made permanent in Budget 2012, there were delays beyond the control of the program which limited the Build in Canada Innovation Program’s ability to progress with the fourth Call for Proposals in the 2013 to 2014 fiscal year. The overhead was higher in 2013 to 2014 fiscal year because the Build in Canada Innovation Program was granted permanent funding late in that fiscal year and were only allotted a certain amount of funding for innovation management. Consequently, the amount that could be spent on funding innovation testing in that fiscal year was limited, and a higher percentage of funding was expended on overhead (outreach, salaries, operations and management) than otherwise would have been the case. However, as indicated in Exhibit 13, greater stability in the program has resulted in a higher proportion of funds being spent on innovations. In the 2015 to 2016 fiscal year, the Build in Canada Innovation Program demonstrated the highest level of efficiency in its history, with just 14.1% of total budget spent on overhead, with most of the remaining budget (85.2%) being directed to innovation spending.

Source: Program data

125. In the evaluation of the Canadian Innovation Commercialization Program (the pilot program for the Build in Canada Innovation Program), it was noted that while the program aimed to establish contracts within eight weeks from the close of the Call for Proposals, this timeline was later revised to eight months. The evaluation of the pilot program noted that by the eight month mark, only two contracts had been established while 25 were awaiting finalization. Data provided by the program indicated that the average time it took to establish a contract for an innovation under the sixth Call for Proposal was 180.7 business days (roughly eight and a half months), which was aligned with the efficiency target established during the implementation of the pilot program.

Conclusions: economy and efficiency

126. The evaluation found that, to the extent possible during a period of instability, the program has operated economically and efficiently. While a conclusive determination on economy and efficiency was made challenging by the program’s recency and the initial instability of its operating environment and funding, the Build in Canada Innovation Program has demonstrated improvements in both economic and efficient utilization of resources. This suggests that greater economy and efficiency may be achievable as the program continues its operations with greater security of its permanency and stable funding.

Conclusions: performance

127. The evaluation found that the Build in Canada Innovation Program has progressed towards achieving its intended outcomes. Industry remains interested in participating in the program; suppliers that have matched with participating departments are completing their testing. However, suppliers have indicated to the program that there are opportunities for improving the information it makes available to industry. Interest and participation in the program by federal organizations has increased over time, with participating departments having tested multiple innovations; however outreach to federal organizations could be improved upon to help provide more options for matching innovations. The Build in Canada Innovation Program was effective in contributing to the preparedness of participating suppliers to commercialize their innovations. Participating departments were supportive of the program and believed the feedback provided to suppliers was beneficial in supporting commercialization, as was the experience from having a first-time buyer for their innovations. However, the existence of perceived barriers to using innovations in the federal government could be limiting the program’s ability contribute to a culture of innovation in the federal government.

128. The evaluation found that many of the suppliers examined as case studies for this Evaluation had commercialized their innovations since participating in the program, with many indicating that the program had provided them with a valuable reference client which led to additional sales. For those who had yet to commercialize, the indication from suppliers was that participation in the program was valuable in progressing their innovations towards ultimate commercialization. As noted above, however, barriers to using innovations within the federal government may be limiting the program’s ability to more fully contribute to commercialization of innovations. While suppliers appear interested in partnering with the federal government, as evidenced by the increasing participation of suppliers outlined above, the extent of achievement of these outcomes is limited to the extent to which federal departments and agencies are open to leveraging innovations to meet their needs.

129. The evaluation also found that, to the extent it could control during a period of initial instability, the program is operating economically and efficiently. It should be expected that greater economy and efficiency may be achievable as the program continues its operations with greater security of its permanency and assurance of funding.

General conclusions

130. Overall, the evaluation found there is a continuing need for the Build in Canada Innovation Program. The program aligns with the priorities of the federal government and with the strategic outcome of Public Services and Procurement Canada. The Build in Canada Innovation Program is also well-aligned with the Department’s responsibilities as the common service provider for government procurement, especially as relates to the military component of the program. The Build in Canada Innovation Program complements the existing suite of federal programs supporting the commercialization of innovations. We also noted some degree of alignment between the Build in Canada Innovation Program’s outcomes and the mandate of the Minister of Innovation, Science, and Economic Development. The Build in Canada Innovation Program is an example of how Departments with complementary objectives work together to achieve common results for Canadians.

131. The program has progressed towards achieving its intended outcomes, with suppliers from across the country and an increasing number of federal departments and agencies participating in the program. While suppliers were having positive experiences with the testing feedback contributing to their innovation commercialization process, the degree to which the program will be successful is ultimately dependent on the extent to which departments and agencies are open to leveraging innovations to meet their needs. As this is primarily an obstacle related to the overall culture of the Government of Canada, there are many contributing factors that are beyond the control of the program, though the evaluation noted opportunities for the program to reduce potential barriers to participation.

132. The evaluation found that many of the suppliers examined as case studies for this evaluation had commercialized their innovations since participating in the program, with many indicating that the program had provided them with a valuable reference client which led to additional sales. Federal departments who have participated in program were generally supportive of the Build in Canada Innovation Program and believe the feedback provided to suppliers is beneficial in supporting commercialization.

133. The evaluation also concluded that, to the extent possible during a period of instability, the program has operated economically and efficiently in recent years. Greater economy and efficiency may be achievable as the program continues its operations with the greater security afforded by its permanency and the stability of its funding.

Recommendations and management action plan

Recommendation 1:

The Assistant Deputy Minister, Procurement should work collaboratively with officials from Innovation, Science and Economic Development Canada to maintain coordination of the program with other elements of the Innovation Agenda, optimize conditions for achieving its expected results, and develop mechanisms to integrate results measurement of the Build in Canada Innovation Program into the results measurement of the overall Innovation Agenda.

Management action plan 1.1:

Since summer 2016, Acquisitions Branch has been working closely with Innovation, Science and Economic Development on how the federal government can leverage the procurement of innovation to bolster Canada’s economic growth and global competitiveness. Specifically, officials within Acquisitions Branch have been closely collaborating with Innovation, Science and Economic Development Canada in the context of the Build in Canada Innovation Program renewal and in regards to the design and development of Innovative Solutions Canada, a key component of the Inclusive Innovation Agenda and Budget 2017.

Innovation, Science and Economic Development Canada is finalizing a proposed approach for Innovative Solutions Canada – it is expected the initiative will formalize government support for early-stage technology development with the dual objective of supporting commercialization of Canadian innovations, and driving solutions development that respond to Government of Canada challenges. In this context, Acquisitions Branch will develop program options for innovation procurement and a related results measurement framework that align with Innovative Solutions Canada and other elements of the Innovation, Science and Economic Development Canada-led Innovation Agenda. Acquisitions Branch will also examine best practices in horizontal management and apply them to the collaboration with Innovation, Science and Economic Development Canada in the context of Build in Canada Innovations program renewal and Innovative Solutions Canada.

Recommendation 2:

The Assistant Deputy Minister, Procurement, should implement strategies to reduce barriers, real and perceived, that are precluding federal organizations from purchasing Canadian innovations.

Management action plan 2.1:

The Build in Canada Innovation Program will continue, through proactive outreach and marketing, to drive both submissions of innovations to the program and the uptake of innovations for testing the public sector. Engagement with horizontal governance bodies (e.g. Science and Technology Advisory Board), federal partners (e.g., Regional Development Agencies, Innovation, Science and Economic Development Canada, and the National Research Council—Industrial Research Assistance Program), targeted regional, seminars and educational opportunities (e.g., innovation showcases), digital engagement and social media will help to build greater awareness and increased interest in how innovative goods and services can support government objectives and challenges.

Further, outreach and educational efforts will be made with provincial, territorial and municipal jurisdictions to enhance the testing opportunities for Build in Canada Innovation Program innovations. Partnerships between federal departments and other jurisdictions have and continue to take place, and an increase in these opportunities will support greater exposure for innovative goods and services being used in the public sectors, and thus increase the potential uptake for full commercial purchases for future operational use.

More broadly, Public Services and Procurement Canada’s Acquisitions Branch is working with the Treasury Board Secretariat on how procurement policy (through a Treasury Board-led policy suite reset) can best support government objectives, which include socio-economic considerations and benefits when the government buys the goods and services it requires.

Acquisitions Branch, through the procurement modernization initiative, will be assessing how procurement, can more directly encourage and support purchases of innovation, whether the purchases be for research and development purposes (Build in Canada Innovation Program, other research and development focused-activities), or for supporting full operational needs of government (commercial purchases). Target implementation date of February 2018.

About the evaluation

Authority

The Deputy Minister for Public Services and Procurement Canada approved the conduct of this evaluation as part of the 2015 to 2018 Risk-Based Multi-Year Audit and Evaluation Plan.

Evaluation objectives

The evaluation examined the Build in Canada Innovation Program, delivered by the Office of Small and Medium Enterprises and Strategic Engagement Sector within the Acquisitions Branch. This evaluation had two objectives:

• to determine the relevance of the program: the continued need for the program, its alignment with governmental priorities and its consistency with federal roles and responsibilities
• to determine the performance of the program: the achievement of its expected outcomes and a demonstration of the efficiency and economy of the program

Approach

The evaluation was conducted in accordance with the Standard on Evaluation for the Government of Canada. The evaluation took place between April 2015 and October 2016 and was conducted in three phases: planning, examination and reporting. To assess the evaluation issues and questions, the following lines of evidence were used.

Document review: An initial document review provided an understanding of the program and its context to assist in the planning phase. Documents reviewed included documents provided by the program, as well as documents written about the program and the Government of Canada’s priorities. These documents included those pertaining to federal priority setting (e.g.: mandate letters, speeches from the throne, federal budgets, etc.), Departmental priority setting (e.g.: business plans, reports on plans and priorities, performance reports, etc.), and program documents (e.g.: reports to the Treasury Board of Canada Secretariat, presentations, feedback surveys, program-led studies etc.).

Literature review: A literature review was conducted to: contextualize the program both nationally and internationally; provide theoretical background for the program model; and provide baseline data against which the program could be assessed.

Jurisdictional review: An analysis of federal and provincial programs was conducted to provide information on the degree to which the Build in Canada Innovation Program complements and/or overlaps with other federal and provincial programming.

Survey: The evaluation conducted a survey sent to departments and agencies that had participated/were currently participating in the matching/testing process, as well as departments and agencies that had not yet participated in the matching process but had shown interest in the program. Survey invitations were sent to 601 people; 74 of the 601 invitations failed to deliver (email addresses were no longer valid). Therefore, a total of 527 invitations were successfully received, of which 88 successfully completed the survey for a 16.7% response rate. The survey assessed satisfaction with experiences with the Build in Canada Innovation Program, perspectives on the relevance and performance of the program, and areas for program improvement.

Financial analysis: Financial data related to the program’s budgets, expenditures, and staff resources were reviewed.

Case studies: The focus of the case studies was to examine the extent to which the program had a meaningful and measurable impact on the commercialization of innovations tested through the program. To do so, the evaluation team conducted case studies of 8 companies that had completed testing with the Build in Canada Innovation Program, which represents 10% of all companies that have participated in the program. The companies were chosen at random with two companies selected from each of the first four Calls for Proposals. Case studies included interviews with senior representatives and a review of their case files and supporting documentation.

Limitations of the methodology

Document review: Given the change in policy direction subsequent to the October 2015 federal election, documentation on current federal priorities was relatively limited and excluded prior federal priorities.

Survey: A total of 119 responses were received to the survey. Of the 119, 26 responses were incomplete (the survey was closed prior to being completed) and 5 declined to participate.

Case studies: Two companies initially contacted for interviews declined to participate and were replaced with other companies randomly selected from the same Call for Proposals. While the evaluation team requested financial and other related performance data, the majority of the respondents declined to provide this commercially sensitive data.

Reporting

Findings were documented in a Director’s Draft Report, which was reviewed by the Office of Audit and Evaluation’s Quality Assurance function. The program’s Director General was provided with the Director’s Draft Report and asked to validate facts and comment on the report. A Chief Audit and Evaluation Executive’s Draft Report was prepared and provided to the Assistant Deputy Minister, Acquisitions Branch, for acceptance as the Office of Primary Interest. The Office of Primary Interest was requested to respond with a Management Action Plan. The Draft Final Report, including the Management Action Plan, was presented to Public Services and Procurement Canada’s Audit and Evaluation Committee for the Deputy Minister’s approval in January 2017. The Final Report was submitted to the Treasury Board Secretariat and posted on the Public Services and Procurement Canada website.

Project team

The evaluation was conducted by employees of the Office of Audit and Evaluation, overseen by the Director of Evaluation and under the overall direction of the Chief Audit and Evaluation Executive. The evaluation was reviewed by the Quality Assessment function of the Office of Audit and Evaluation.