2011-610 Evaluation of the Canadian Innovation Commercialization Program (Final report)

2011-610 Final report (PDF, 522KB)

December 23, 2014

Main points

What was examined

1. The Canadian Innovation Commercialization Program (CICP) was a two-year pilot procurement project launched in 2010 as part of the Government of Canada's commitment to promote Canada's economic growth^Footnote 1. The program is unique in that it targets a broad spectrum of Canadian industries while using the procurement of goods and services as a method to support innovation. Through CICP, Public Works and Government Services Canada (PWGSC) purchases innovative products and services developed by Canadian companies that federal departments and agencies use, test and provide feedback.
2. The evaluation assessed CICP's relevance and performance. At the time of the evaluation, CICP was in the early stages of implementation, having completed its first and second rounds of Calls for Proposals as a pilot program. As the program had not yet completed all of the activities planned for the pilot initiative, the assessment of performance in this report is limited and preliminary. The evaluation examined the program's organizational context, implementation, and delivery of activities to identify strengths and opportunities for improvements.
3. The Department of Public Works and Government Services Canada's Acquisitions Branch is responsible for the program, with CICP being delivered by the Office of Small and Medium Size Enterprises (OSME). To provide its services, the CICP has a budget of $40 million for the duration of the two-year pilot phase.

Why it is important

4. Announced in the 2010 Budget, CICP assists in addressing the "pre-commercialization gap" by purchasing and testing Canadian innovations. Innovation occurs along a continuum of activities ranging from research to commercialization. Innovation in Canada is supported publicly and privately at the earlier stages of the continuum. The term "pre-commercialization gap" refers to a shortage in capital that is often experienced by innovators in the development stage occurring between government or corporate funded research and the preparation of a product for the marketplace.

What we found

5. The rationale for the program remains sound and there is a continued need for the unique form of assistance provided by CICP. The continued relevance of CICP has been articulated in the 2012 Federal Budget.
6. The program is aligned with government priorities related to support for innovation and for small and medium enterprises and PWGSC's departmental strategic outcome. CICP is aligned with federal roles and responsibilities related to procurement. As the principal procurer of goods and services for the Government of Canada, PWGSC is uniquely positioned to foster innovation in the area of goods and services required by the federal government.
7. The program has been successful in attracting Small and Medium Enterprises (SMEs) and federal departments to the program. Initial targets set for participation were greatly exceeded. Enlisting government departments in the testing phase of the program has proven a greater challenge than obtaining their participation in outreach activities.
8. The program is achieving success in supporting the use of innovation by government, as indicated through multiple matches made between suppliers and other government departments. The evaluation was unable to assess the success of use and feedback as only one innovation was being used at the time of this assessment.
9. Progress made to date in the implementation of the program and early results position the program well for the achievement of the intended intermediate and ultimate outcomes.
10. The program activities examined by the evaluation were implemented successfully. In the first round of the program, actual contracting for qualified innovations took place at a slower pace than expected, resulting in delayed implementation of testing and feedback activities.

Recommendations and management action plan

Recommendation 1: The Assistant Deputy Minister (ADM), Acquisitions Branch, should revisit the performance measurement strategy (including logic model, performance indicators and targets) for the program to incorporate lessons learned in the implementation of the pilot procurement initiative and to reflect expectations for an on-going program.

• Management action plan 1.1: A review of the current program logic model and lessons learned from the Pilot will be gathered culminating in a Lessons Learned Report.
• Management action plan 1.2: The Lessons Learned Report will inform the development of a new performance measurement strategy. This new strategy will detail a logic model, performance indicators and targets that are commensurate with a mature on-going program. A Draft Performance Measurement Strategy will be produced and distributed for general comment prior to finalization.
• Management action plan 1.3: A final Performance Measurement Strategy for ADM approval will be produced before March 31, 2013.

Recommendation 2: The Assistant Deputy Minister, Acquisitions Branch, should assess the time required to establish contracts for qualifying innovations and, where warranted, revise current standards with the objective of improving the overall timeliness of the implementation of the program's activities.

• Management action plan 2.1: An assessment will be made of the time required to establish contracts for qualifying innovations. If warranted, changes will be made to existing standards to improve timeliness and this will be reflected in the Performance Measurement Strategy.

Introduction

1. This report presents the results of the evaluation of the Canadian Innovation Commercialization Program. The Audit and Evaluation Committee of Public Works and Government Services Canada approved this evaluation as part of the 2011-16 Risk-Based Audit and Evaluation Plan.

Profile

Background

2. In response to the issue of a gap in support along the innovation continuum, the federal government created CICP (announced in the 2010 Budget): a $40-million, two-year pilot program, through which PWGSC purchases innovative products and services developed by Canadian companies for testing and deployment by federal departments and agencies.
3. Innovation is a multi-stage process by which individuals, companies and organizations develop, master, and use new products, designs, processes and business methods. The components of innovation include research and development, invention, capital investment, and training and development. Innovation (from idea to commercialization) occurs along a continuum, where technology readiness increases at each stage. Appendix A contains the Technology Readiness Scale used by CICP, outlining the stages of innovation.
4. The Government of Canada has a long commitment to publicly-funded academic research and development at universities and colleges, contributing approximately $2.8 billion in 2008-09. In addition to supporting academic research, the Government of Canada has many programs and services designed to assist innovative companies; however, many of these programs are directed at the early stages of the innovation continuum (i.e., fundamental and applied research stages). Innovators in the pre-commercialization stage of product development, however, lack support to begin testing their innovations in an operational environment and to prepare their product for the marketplace. The lack of support for the pre-commercialization phase has been identified as one reason why overall innovation in Canada lags behind other major industrialized countries, as noted by the following examples:
   • The Conference Board of Canada (2011) awarded Canada a "D" grade in its support of innovation and ranked Canada 14th out of 17 developed countries in the provision of such support.
   • The Certified General Accountants Association of Canada (2010) concluded that innovation in Canada is "struggling."
   • The Government of Canada's Science, Technology and Innovation Council (2008) noted that low overall research, development and commercialization by business has been a defining characteristic of Canada's economy for the past four decades.
5. The Canadian Innovation Commercialization Program assists Canadian businesses by purchasing, testing, and providing feedback on their innovative goods and services before the company takes them to the general marketplace. Proposed innovations must fall within one of the four priority areas: the environment, safety and security, health, and enabling technologies. CICP purchases the innovative product or service from the company, which provides the business with increased cash flow and a reference for future sales. As well, through use of its innovative goods and services by federal departments and agencies, the company receives critical testing and feedback for incorporation into further refinement of the product or service before it is offered in the general marketplace.
6. The CICP uses a unique approach to procurement to select and purchase innovative goods and services. The process contains six elements; the first is the Call for Proposals process by which Canadian businesses propose their innovative product or services to the government. The evaluation phase is a two step process where proposals are assessed against the program criteria and successful bidders are ranked based on the available budget for that round of Call for Proposals as pre-qualified innovations. The pre-qualified innovations are matched with testing departments that could benefit from their use. The innovations are purchased and the departments begin using the products. Testing departments provide feedback once the testing has been completed
7. By the end of 2011-12, the program conducted 780 outreach events and 21,873 supplier interactions to promote the program. The first Call for Proposals occurred between October 5, 2010 and November 16, 2010, and a second Call for Proposals occurred between July 13, 2011 and August 18, 2011. The first Call for Proposals resulted in 375 proposal submissions, the second round resulted in 337 proposal submissions, and a third Call for Proposals was released in February 29, 2012. Following the recommendations of the Innovation Selection Committee, the program pre-qualified 27 innovations after the evaluation of the first round of submissions, and pre-qualified 37 innovations after the evaluation of the second round of submissions. As of March 31, 2012, 23 contracts had been awarded stemming from the activities of first and second Calls for Proposals, and one innovation had concluded the testing phase of the program.

Authority

8. The Department of Public Works and Government Services Act grants PWGSC the authority to undertake the activities of the program. Section 6 (Paragraphs a and b) of the Act detail the powers of the Minister with regards to the "acquisition and provision of articles, supplies, machinery, equipment and other materiel for departments" and "the acquisition and provision of services for departments."

Roles and responsibilities

9. The Office of Small and Medium Enterprises manages the delivery of CICP. OSME is part of PWGSC's Acquisitions Branch and is comprised of four directorates within PWGSC headquarters as well as six regional offices.
10. OSME regional offices are responsible for conducting CICP outreach activities, enabling them to leverage partnerships within the SME community formed as delivery result of the OSME program.
11. PWGSC's Services and Specialized Acquisitions Management Sector is responsible for the procurement and contractual aspects of the program.
12. The National Research Council of Canada's Industrial Research Assistance Program participates in the proposal review process. During the first round of Call for Proposals, the Industrial Research Assistance Program team consisted of 140 trained industrial technical advisors with competencies in the four priority areas.
13. The Innovation Selection Committee is the final validation stage of the program prior to the pre-qualification of proposals under a Call for Proposals, meaning that they can be selected by a federal department or agency for testing. The committee is composed of members from government (up to 30% of members) and the private sector (at least 70% of members) with experience and knowledge in such areas as entrepreneurship, innovation and commercialization, and international business practices.
14. Federal departments and agencies participate in the CICP through Memoranda of Understanding with respect to the use and testing of innovative products or services procured by the program. Participating departments and agencies are responsible to carry out testing as well as providing PWGSC with a completed feedback form.
15. CICP is responsible for providing companies whose product or service has been tested by a federal department or agency with feedback on the strengths and weaknesses of the innovation and how the innovation could be improved prior to commercialization.

Stakeholders

16. The program's main stakeholders are Canadian companies that are involved in developing innovative products, as well as federal government departments and agencies interested in participating in the program.

Resources

17. Of the total program budget of $40 million, $8 million has been set aside to promote CICP over a period of three years The remainder of the funds ($32 million) has been reserved for innovation management. Administrative costs for the program, for fiscal years 2010-11 and 2011-12, were $6.2 million^Footnote 2.
18. CICP's human resource complement is 16 Full Time Equivalent (FTEs). CICP's headquarters complement is made up of 10 Full Time Equivalents. In addition, an OSME staff member in each regional office is dedicated to the program for a total of six regional FTEs.

Logic model

19. A logic model is a visual representation that links a program's activities, outputs, and outcomes; provides a systematic and visual method of illustrating the program theory; and shows the logic of how a program, policy or initiative is expected to achieve its objectives. It also provides the basis for developing the performance measurement and evaluation strategies, including the evaluation matrix.
20. The Evaluation used a logic model prepared by the program which described CICP's activities, outputs, and expected outcomes for the pilot. The evaluation team updated the logic model to better reflect CICP's current expected outcomes. It was subsequently validated by program staff. The logic model is presented in Exhibit 1.

Exhibit 1: Logic model

Program activities

21. CICP has three main activity streams: outreach to suppliers, procurement of innovation and use of innovation. As an evaluation of a pilot initiative, activities are described as part a review of the implementation of the program, located within the Performance section of this report.

Focus of the evaluation

22. The evaluation scope included activities conducted between September 2010 and March 2012.
23. The objective of this evaluation was to determine the program's relevance and performance in achieving its planned outcomes in accordance with the Treasury Board Policy on Evaluation. In addition to the assessment of the Treasury Board's core issues for program evaluation, the evaluation also addressed the issue of implementation.
24. An evaluation matrix (including evaluation issues, questions, indicators and data sources) was developed during the planning phase (see Appendix B). Multiple lines of evidence were used to evaluate the program based on the evaluation matrix. These include:
25. Document review: A review of 45 documents was conducted to situate the program both nationally and internationally to assess the program model, provide baseline data against which the program could be assessed, and identify alternative delivery models through an analysis of other jurisdictions.
26. Interviews: The evaluation team conducted 18 interviews with key program and PWGSC managers. In addition, the evaluation team conducted interviews with program partners.
27. Performance data analysis: Performance data was analyzed in order to measure the program's outcomes. The evaluation team used financial and non-financial data from reports and other documents provided by CICP.
28. Client survey: A survey was developed by the evaluation team to capture client perspectives on the relevance and performance of the program. The population surveyed was the suppliers who submitted bids to CICP's first Call for Proposals. Three hundred and twenty-five suppliers who had submitted a bid under the Call for Proposals were asked to participate in the survey, and ninety-five of them chose to respond to the survey (providing a 29.2% response rate). Eighteen percent (n =17) of survey respondents reported that their innovation had qualified for the program.
29. Financial analysis: Financial data related to the program's budget and expenditures were reviewed and analyzed to assess the efficiency and economy of the program. All financial information was provided by the program.
30. More information on the approach and methodologies used to conduct this evaluation can be found in the About the Evaluation section at the end of this report.

Findings and conclusions

31. The findings and conclusions below are based on multiple lines of evidence used during the evaluation. They are presented by evaluation issue (relevance and performance).

Relevance

32. Relevance is measured as the extent to which the program or initiative addresses a continuing need, is aligned with federal priorities and departmental strategic outcomes and is an appropriate role and responsibility for the federal government.

Continuing need

33. Continuing need assesses the extent to which the program continues to address a demonstrable need and is responsive to its clients. Lines of evidence utilized to evaluate continuing need included: the continued relevance of the original program rationale; legislative or policy requirements; and uptake of the program's services. Based on these criteria, the evaluation found that there is a continuing need for CICP.
34. The CICP's original rationale is that the ability of business, particularly Small and Medium Enterprises, to innovate is critical to Canada's economic future, and that there was a gap in federal programming at the stage (called the pre-commercialization stage) at which innovation by companies occurs.
35. The validity of the original rationale at the program's conceptualization is supported by the evaluation's literature review. In 2005, The National Angel Organization concluded that "the bottleneck holding back the benefits of the government's large investments in Research and Development (R&D)"^{Footnote 3} was a shortage of investors for pre-commercialization activities due to the high level of risk present at this stage of the innovation continuum. As well, the Conference Board of Canada concluded in 2010 that "Canada does not take the steps that other countries take to ensure science can be successfully commercialized and used as a source of advantage for innovative companies seeking global market share."^{Footnote 4}
36. In response to the original rationale, CICP was designed to support innovation by Canadian companies through the provision of financial and non-financial support for companies with a products and/or services at a critical point in their development: the pre-commercialization phase.
37. The original rationale for the program remains valid as there is a continuing need for Canadian businesses to innovate and conduct research and development, and a need for support at this stage of the innovation continuum. In its report Innovation Canada: A Call to Action, the Review of Federal Support to Research and Development Expert Panel (2011) recommended CICP be made a permanent program following evaluation of the pilot phase. This view is supported by the Standing Committee on Government Operations and Estimates in its November 2011 report, where it recommends that PWGSC "consider making the Canadian Innovation Commercialization Program permanent in order to further stimulate innovation in Canada." Furthermore, the ongoing relevance of the CICP is also supported by findings of the client survey: When asked "how important is a government program like CICP that supports the pre-commercialization of innovations, to your company?", 85% of respondents to the Office of Audit and Evaluation (OAE) survey said it was important to very important.
38. There is no legislated requirement to deliver the program. Authority to provide the procurement-based program derives from Section 6 of the Department of Public Works and Government Services Act.^{Footnote 5}
39. The demand for services by beneficiaries is illustrated by the higher than anticipated rate of participation by Canadian businesses. The program anticipated that the first round of Call for Proposals would result in 100 proposals being submitted. The first round resulted in 375 proposals. Program management stated that such a volume of interest was unexpected: Budget 2010 had committed support for up to 20 demonstration projects.
40. In conclusion, there is a continuing need for the program based on the on-going validity of the original rationale and the higher than anticipated rate of participation in the program by Canadian businesses.

Alignment with federal priorities and departmental strategic outcomes

41. Program alignment is determined by assessing program linkages with federal government priorities and with departmental strategic outcomes. The evaluation found that CICP contributes to the achievement of federal government priorities and to the strategic outcome of the department.
42. CICP is aligned with two federal government priorities:
    • Support for Canadian small- and medium-sized businesses
    • Innovation for economic growth
43. The 2010 Federal Budget announced that the "Government will support innovation in Canada's small business sector by launching a new Small and Medium-sized Enterprise Innovation Commercialization Program."^{Footnote 6} The resulting Canadian Innovation Commercialization Program was opened to all Canadian companies regardless of size. However, given that responsibility for the program was assigned to the Office of Small and Medium Enterprises, implementation of the program by this entity helps insure a focus is on SME participation and support.
44. In October 2010, the Government of Canada announced a review of federal support for business research and development to determine how to strengthen the impact of federal investments in support of a more innovative economy. Led by an independent expert panel, the review was completed in 2011 with a call for procurement programs geared towards acquiring innovative products. To this end, the panel concluded that:

    "The government's procurement and related programming must be used to create opportunity and demand for leading-edge goods, services and technologies from Canadian suppliers, thereby fostering the development of innovative and globally competitive Canadian companies...."^{Footnote 7}

45. The 2011 Speech from the Throne articulated the Government's continuing priority of fostering optimal conditions for economic growth by supporting "innovation and new technologies."^{Footnote 8}
46. The 2011 Federal Budget, which focused on stimulating job growth in the economy, included an emphasis on "fostering commercialization and business innovation."^{Footnote 9} The government recognized that, for knowledge and innovation to be drivers of economic success, Canada must "improve the commercialization of research."^{Footnote 10} CICP was designed to target innovations and support government priorities related to the environment, safety and security, health, and enabling technologies.
47. The 2012 Federal Budget re-committed government support for innovation for economic growth. To this end, CICP received funding to make the program permanent and to add a military procurement component. The budget allocated $95 million to CICP for 2013-14 through 2015-16 and then $40 million per year after that period. The 2012 Budget provided the following rationale for the continuation of the program:

    "By selling to the federal government, businesses can demonstrate the value of their products and services, increase the scale of their operations, and generate future sales to non-federal customers."^Footnote 11

48. CICP aligns with PWGSC's Strategic Outcome to provide "high-quality, central programs and services that ensure sound stewardship on behalf of Canadians and meet the program needs of federal institutions."^{Footnote 12} Procurement is one of these services. The 2010-11 Report on Plans and Priorities establishes procurement as a priority for PWGSC by stating the department is to provide timely, value-added acquisitions and related common services to Canadians and the federal government. The 2011-12 Report on Plans and Priorities further supports this procurement priority by saying the department must improve access to procurement opportunities and that "one way of improving access is through the Canadian Innovation Commercialization Program."^{Footnote 13} Furthermore, the CICP is identified in the 2011-12 Report and Plans on Priorities as a means of improving SME access to procurement opportunities^{Footnote 14}. The 2011-12 Report and Plans on Priorities notes CICP's affiliation with OSME.
49. In conclusion, CICP aligns with federal priorities related to innovation. Its delivery by OSME further aligns with this priority as it supports Canadian small and medium size businesses. Given PWGSC's responsibility for procurement CICP also aligns with the Department's strategic outcome.

Appropriate role and responsibility for the federal government

50. To determine if the program is an appropriate role for the federal government, three elements were reviewed: decentralization to other federal departments and agencies, devolution of program responsibility to another level of government, and outsourcing to the private sector. Based on these criteria, the evaluation found the services provided by CICP are an appropriate role and responsibility for the federal government and a good fit within PWGSC.
51. Decentralization to other federal departments or agencies is not feasible due to the program's focus on supporting innovation specifically through the procurement process - a process which is the responsibility of PWGSC. Support for pre-commercialization of innovations by other federal government departments targets specific industries aligned with that department's mandate, for example Agriculture Canada and the agriculture industry. Further, these other programs often take the form of grants and/or contributions and not a purchase of goods or services like CICP. While a department such as Industry Canada could deliver a broadly-targeted innovation program, PWGSC (as the legislated agent to procure goods and services for the Government of Canada) is best suited to deliver a program designed to support innovation through the procurement process. Furthermore, CICP is unique: It is the only federal government program that makes use of Government of Canada investment in procurement to promote innovation on a national level.
52. The devolution of CICP to another level of government or outsourcing to the private sector would be inappropriate because the program currently purchases innovations to be used and tested within the federal government. Neither another level of government nor the private sector has accountability to the Government of Canada for procurement activities supporting multiple federal departments and agencies across Canada. Finally, outsourcing to the private sector could create a conflict of interest.
53. CICP aligns with federal roles and responsibilities. The program's distinguishing characteristic as a procurement-based initiative makes it a logical fit within PWGSC. As the federal department mandated to procure goods and services for the Government of Canada, PWGSC is well-equipped to design and manage the delivery of CICP.

Conclusions: Relevance

54. The rationale for the program remains sound and there is a continuing need for the unique form of assistance provided by CICP. The continued relevance of CICP to government priorities has been affirmed by the 2012 Federal Budget.
55. CICP is aligned with government priorities related to support for small business and innovation for economic growth as well as PWGSC's departmental strategic outcome.
56. CICP aligns with federal roles and responsibilities related to procurement. The program's distinguishing characteristic as a procurement-based initiative makes it a logical fit within PWGSC. As the federal department mandated to procure goods and services for the Government of Canada, PWGSC is well-equipped to design and manage the delivery of CICP.

Performance

57. Performance is the extent to which a program or initiative is successful in achieving its objectives and the degree to which it is able to do so in a cost-effective manner that demonstrates efficiency and economy. As CICP was a pilot project at the time of the evaluation, the implementation of the program was reviewed as part of the assessment of performance. As the evaluation was conducted concurrently with the implementation of initial program activities, the outcomes envisioned for later stages (i.e. intermediate and ultimate outcomes) could not be fully assessed as part of this evaluation.

Implementation

58. Implementation is concerned with the extent to which the program's activities were delivered as planned; and the extent to which alternative design and delivery options have been explored so as to inform potential improvements to the program following the pilot stage.
59. The evaluation reviewed implementation activities conducted between September 2010 and March 2012.

Outreach to suppliers

60. Outreach activities were conducted with suppliers to provide them with an opportunity to learn more about the CICP program and help them decide if participating in a Call for Proposals would be the right thing for their business. Outreach activities consisted of developing a program website, as well as organizing Supplier outreach events such as innovation forums, supplier seminars, booths at tradeshows, speaking opportunities at conferences and other events, and holding meetings with industry associations. CICP outreach events were delivered by OSME regional offices. The total planned funding for outreach activities was $8 million over three fiscal years. The funding budget was prepared with the intention that unspent funds would be reallocated towards purchasing additional innovations.

Supplier outreach events

61. In September 2010, the program initially set a target of 400 supplier interactions and 40 outreach events per fiscal year. By the end of 2010-11, the program had exceeded initial targets with 8,044 supplier interactions and 253 events, and in 2011-12, CICP held 527 events with 13,989 supplier interactions.
62. Suppliers report positive feedback on the outreach events. The survey of suppliers conducted by the Office of Audit and Evaluation as part of this evaluation confirmed program management's assessment of its outreach events. Survey results indicate that 88% of respondents gained knowledge of CICP and 60% gained knowledge of the government procurement process through attending outreach events.
63. According to the program, the most effective results across all regions came from establishing partnerships with business associations, a critical success factor facilitated by leveraging on the existing activities of OSME. Through these partnerships, the program was able to more effectively target their key audience.
64. Following the release of the first Call for Proposals, the program hosted regional bidders' conferences in Montreal and Ottawa. Although the conferences were well received, suppliers from outside these areas expressed concern over the fairness of some regions receiving information ahead of others. In an effort to improve the process, the program hosted an online conference, a first of its kind within PWGSC. The online conference was well received by suppliers, with 83% of respondents to the Office of Audit and Evaluation survey indicating that they found the online conferences to be "useful to very useful". Due to the success of the online conference, the program intends to deliver information sessions only through webinars as part of subsequent Calls for Proposals.
65. In conclusion, supplier outreach events served to increase awareness of the program. Leveraging the outreach activities of OSME regional offices was an effective method to implement the program's supplier outreach events.

Program website

66. The CICP website was designed and implemented to provide general information about the program and directions on how to participate. For suppliers, the site contains all the links and forms required to submit a proposal including descriptions of the priority areas as well as financial and certification requirements. For government departments, the conditions and requirements to become a testing department are outlined. The site also publishes a listing of pre-qualified innovations resulting from each Call for Proposals.
67. As of March 31, 2012, the website's homepage has received more than 25,000 unique page views. Prior to the website's launch, the CICP Performance Measurement Strategy targeted about 800 visits per round of Call for Proposals. Thus the number of visits substantially exceeded the target.
68. Suppliers responding to the Office of Audit and Evaluation survey accessed the website for general information about CICP (85% had done so) and for information about the Call for Proposals (89% had done so). Sixty four percent of the survey respondents indicated that they were "satisfied to very satisfied" with the website.

Procurement of innovations

69. The procurement of innovations under the CICP is a multi-step process which involves the participation of other government departments and external advisors. The procurement process was developed in partnership between OSME and PWGSC's Services and Specialized Acquisitions Management Sector (SSAMS), the department's specialist in R&D procurement, with input from the National Research Council of Canada's Industrial Research Assistance Program (NRC-IRAP). The two groups worked in close collaboration to development a competitive procurement approach for CICP based on a Call for Proposals (CFP) process.
70. The procurement process required the development of assessment criteria and tools; the conduct of Calls for Proposals to solicit innovations for consideration by the program; evaluation and of proposals from suppliers by technical advisors and an assessment of their qualification for the program by a committee; and contracting the purchase of innovations by PWGSC from qualified suppliers. CICP paid for the services received from NRC-IRAP. Private sector members of the Innovation Selection were paid through contracts for their participation in the program.

Assessment criteria and e-tools

71. The assessment of the proposals was developed by the program to include a technical assessment and then consideration by the Innovation Selection Committee. In the case of the technical assessment, a Technology Readiness Levels scale was identified as the appropriate criteria by which to determine the innovation's maturity. In order to be eligible for CICP, it was decided that innovations must score a Technology Readiness Levels that is between 7 and 9. Appendix A provides descriptions of innovation maturity levels for a nine-stage continuum.
72. One of CICP's unique aspects was its innovative use of e-tools, more specifically the introduction of a start-to-finish electronic approach to distributing information, placing, and evaluating Call for Proposals responses. It was a first of its kind for the federal government as the traditional approach for a Call for Proposals requires the submission of multiple hard copies. Developed for the CICP, the online process proved to be efficient and environmentally friendly. The system saved document storage space, paper (the first Call of Proposals would have required approximately 200,000 printed sheets of paper), printing, and movement of documents.
73. As detailed in the section below on Calls for Proposals, the program received exceedingly more proposals than it anticipated. Program managers indicated that if the online system had not been in place, it would have been impossible to review and evaluate the proposals within expected timeframes.
74. When asked to provide suggestions or comments regarding the online proposal process as part of the Office of Audit and Evaluation supplier survey, a small number of respondents (13.3%) suggested independently that the proposal process would be greatly improved if the website permitted the attachment of visual aids. Survey respondent feedback included comments such as "for complex innovations such as software you need video/pictures to see the real advances" and "you should be allowed to post pictures and charts in the proposal so that a project concept can be easily understood by the reviewer." This feature was added by the program to the online proposal submission tool for the second and all future Calls for Proposals.

Calls for proposals

75. The Calls for Proposals were posted on the MERX procurement system. The first Call for Proposals was posted between October 5, 2010 and November 16, 2010, and the second round between July 13, 2011 and August 18, 2011, and a third round was posted February 29, 2012. The first Call for Proposals resulted in 375 proposals being submitted, and the second round produced 337 proposal submissions.
76. Supplier feedback made to the program during the first round of Call for Proposals was generally positive; however, some suppliers commented the documentation was difficult to understand and reported the Call for Proposals document as long and complicated. For the second Call for Proposals, documents were revised to address complexity issues identified by users, provide supplementary information, and improve integration with the program's website. For the third Call for Proposals, documents were further revised to make the application approach more streamlined. In response to user feedback, proposal submission forms were made more straightforward and the length of the English documents was reduced from 87 pages to 42 pages, and further reduced to 14 pages for the third Call for Proposals.^{Footnote 15}

Proposal evaluation and selection

77. Proposals were evaluated by the NRC-IRAP: the top-ranked technically compliant proposals were then referred to the Innovation Selection Committee for validation prior to pre-qualification. The first Call for Proposals was evaluated between November 2010 and January 2011, and the second Call for Proposals was evaluated between August 19, 2011 and December 4, 2011.
78. The evaluation of the first Call for Proposals required more effort and personnel than had been anticipated. The NRC-IRAP required 140 industrial technical advisors, rather than the 75 planned to evaluate the 375 submitted proposals. The initial evaluation process required that proposals be fully evaluated before being placed on a list of technically compliant bids to be referred to the Innovation Selection Committee. This was a labour intensive, time consuming process since over 50% of the bids did not merit such an in depth analysis. Following the first round, the NRC-IRAP identified numerous areas for improvement in the evaluation process which were addressed for the second round.
79. Of the 375 proposals received during the first Call for Proposals, 40 of the highest ranked proposals that were deemed technically compliant by NRC-IRAP were submitted to the Innovation Selection Committee. Based on the validation of the Innovation Selection Committee, the program pre-qualified 27 innovations. According to the interviewees, the Committee brought a private sector perspective to the program that greatly enhanced the selection process: the Committee provided valuable feedback to the program through their ability to question disparities, challenge the evaluators' assumptions and provide confirmation from a group of qualified stakeholders.
80. The evaluation of second round proposals was more efficient according to the NRC-IRAP due to the implementation of a filtering process. This improvement greatly reduced the workload of NRC-IRAP. Proposals were first screened through a first stage which consisted of three mandatory requirements. Of the 337 proposals submitted, two proposals were withdrawn leaving 335 to be screened through the first stage. Of these, 166 proposals did not meet the screening criteria, leaving 169 proposals to be technically evaluated. Following the technical evaluations, 52 proposals were referred to the Innovation Selection Committee. From the 52 proposals, the program pre-qualified 37 innovations.

Contracting

81. PWGSC's Services and Specialized Acquisitions Management Sector conducts contracting activities on behalf of CICP. Contracts are established for the purchase of innovations by CICP and delivery of the good or service to the testing department.
82. At initiation, the program envisioned innovations to cost approximately half a million dollars each which would result in the purchase of approximately 20 innovations over the course of the pilot phase, as per the expectations of Budget 2010. Overall, the purchase prices requested by suppliers were lower than anticipated and the innovation purchase target would be exceeded as a result of the first Call for Proposals.
83. In the first Call for Proposals of the program, contracting for qualified innovations was slower than anticipated due to unforeseen complexities in matching and contracting for innovations. An initial target of eight weeks was established for the issuing of the contract from the closing of the first Call for Proposals; however this target was not met. The program revised the initial target for the establishment of a contract to eight months. By the eighth month of the first Call for Proposals (July 2010), only two contracts had been established, while the remaining 25 contracts were awaiting finalization.
84. The total value of potential contracts from the first Call for Proposals was a minimum of $7.7 million. Average potential contract value was $383,248, with the lowest potential contract value estimated at $38,335 and the highest at $584,200. Below, Exhibit 2 provides an overview of potential contract values for the 20 innovations for which a potential contract value had been determined. As of March 31, 2012, the potential value of the remaining 7 pre-qualified innovations had not been determined. The evaluation noted that there were 5 contract estimates at values over $500,000, although Call for Proposal requirements had limited costs to under that amount.

85. The total value of potential contracts from the second Call for Proposals was a minimum of $11.4 million. Average potential contract value was $308,463, with the lowest potential contract value estimated at $32,850 and the highest at $500,000. Below, Exhibit 3 provides an overview of potential contract values for the 37 innovations for which a potential contract value had been determined.

86. As of March 2012, 19 contracts have been awarded stemming from the first and second Calls for Proposals. Further discussion of contracting activities and outputs is found in analysis of immediate and intermediate results.

Use of innovation

87. CICP promotes the use of innovation by soliciting and facilitating the testing of the pre-commercial goods and services it purchases by government departments and/or agencies.

Relationship building

88. CICP, in order to meet its objectives, has actively built relationships with other government departments and agencies in order to encourage their participation in the matching and testing activities of the program.
89. Relationship building with other government departments took place through various means, including presentations to other government department representatives, client groups and program partners. Further, it was determined that 1,636 representatives of other government departments participated in the outreach events discussed earlier in this section.

Matching process

90. For qualified innovations to be eligible for purchase, a government department or agency must be identified as a potential user of the innovation.
91. As of March 31, 2012 and resulting of the first and second rounds of Call for Proposals, 23 innovations have been contracted and a total of 12 departments and agencies identified as testing departments. Innovations covered all priority areas, with Safety and Security having the greatest number of innovations (n =8).
92. Seventy-seven percent of respondents to the Office of Audit and Evaluation survey of CICP participants who had suggested a testing department in their proposal said they were matched with that department.

Use of innovation and preparation of testing feedback

93. Once the innovation is purchased, the testing department is responsible the testing of the innovation and preparing feedback based on its use.
94. At the time of the evaluation, only one innovation had completed testing within a federal department.

Distribution of testing feedback to suppliers

95. Following the testing period, the testing department provides feedback on the innovation's use to the CICP. The CICP provides that feedback to the supplier with the intention that the information will assist the supplier in advancing their innovation.
96. At the time of the evaluation, testing activities were not yet fully implemented.

Design and delivery alternatives

97. The Expert Panel Report (2011) describes the CICP pilot as a "supply-push," as applicants currently submit proposals to provide innovative solutions for trial and testing, not as responses to explicitly identified needs of a particular department or agency. The panel suggests that the government be proactive by providing incentives for solving operational problems identified by departments. The panel's report also recommends that the Government of Canada "identify a lead minister responsible for innovation" with a stated mandate to put innovation at the centre of the government strategy to improve Canada's economic performance. The report goes on to say that Canada needs a "whole of government" innovation policy that includes research, development, commercialization, and business support strategies. The lead minister should also be charged with developing outcome-oriented performance objectives to enable comparisons of programs across all federal departments. A "centre of government" approach to innovation could see relocating CICP to that centre, while leveraging PWGSC's expertise in procurement.
98. In addition, one alternative that has been suggested by interviewees would be to include provincial governments in the testing phase. This would apply specifically to technologies where provinces have a high degree of expertise and the facilities to test the innovations, such as health focused products. Seeking opportunities for collaboration with other levels of government could increase efficiency by having a jurisdiction with the requirement for, and expertise test the innovations.

Conclusions: Implementation

99. Outreach activities surpassed initial targets and served to increase awareness of the program. Leveraging the outreach activities of OSME regional offices was an effective method to implement the program's outreach events.
100. The program implemented an online electronic process that was effective and innovative.
101. The program was effective at identifying problems with the application process, tools, and requirements in the first Call for Proposals, and in making appropriate changes prior to the second Call for Proposals. The re-design of program tools and eligibility criteria (making the Call for Proposals document shorter, adding information to the website, and changing the required technology readiness level) contributed to making the process more efficient during the second Call for Proposals.
102. The program surpassed the Performance Measurement targets for the number of proposals submitted by suppliers. Initial targets set for issuing contracts had to be revised due to the large volume of proposals received.
103. The evaluation found evidence of alternative design and delivery options for the program.

Outcome achievement

104. The evaluation examined the degree to which the program achieved its intended immediate, intermediate, and ultimate outcomes. CICP outcomes are identified in italics below, followed by an assessment of the extent to which they have been achieved.

Immediate outcome: SMEs and government departments are attracted to CICP

105. In order for CICP to achieve its objectives related to innovation, both Canadian businesses and other government department participation in the program is required. An immediate outcome related to participation was included due to the program being new and undertaking an approach not used before.
106. Outreach activities delivered from September 2010 to March 2012 by OSME regional offices included 21,873 participants^{Footnote 16} from the supplier community and 1636 departmental representatives. Exhibit 4 and 5 below provides details on participation by outreach activity type.

Source: CICP Results Fiscal Year 2010-11, OSME

Source: CICP Results Fiscal Year 2011-12, OSME

107. Participation by SMEs was not tracked as part of the monitoring of outreach events, as CICP is available to all Canadian businesses including large companies (characterized by more than 500 employees).^{Footnote 17} However, given that responsibility for CICP was assigned to OSME and its focus is on SMEs, most participants at CICP outreach activities were likely SMEs. Furthermore, since a minimum of 49 of the 64 (76.5%) innovations pre-qualified under the program in the first and second round of Call for Proposals were from SMEs^{Footnote 18}, it can be concluded that CICP has been successful in attracting SMEs to the program.
108. One of the program's strategies was to build a strong networking relationship with government departments. One means of accomplishing this was through the participation of client departments at outreach events. According to CICP's Performance Measurement Strategy, the program set a target of 200 departmental representatives attending outreach events per fiscal year. From September 2010 to March 2012, the program reached 1,636 departmental representatives (Exhibit 4, Exhibit 5).
109. According to stakeholders, some departments were on board from the beginning and eager to participate, specifically departments and agencies that already provide innovative companies with grants and services. These departments participated in outreach activities to help promote CICP and their own programs. Suppliers who attended these outreach events and learned that CICP did not meet their current needs nevertheless had the opportunity to network with representatives of other programs in support of innovation.
110. In conclusion, CICP has been successful in achieving its immediate outcome of attracting SMEs and other government departments to the program.

Immediate outcome: Successful use of innovations by government.

111. Successful use of innovations by government is indicated by matches made between the program and other government departments and as well as product/service testing and feedback.
112. As noted in the Implementation section, a total of 64 innovations were pre-qualified as a result of the first (n =27) and second (n =37) Calls for Proposals. As of March 2012, matches with testing departments had been confirmed for 73% (47) of all pre-qualified innovations. Seventy-seven percent of respondents to the Office of Audit and Evaluation survey of CICP participants who had suggested a testing department in their proposal said they were matched with that department.
113. Of the innovations matched with testing departments after the first Call for Proposals, contracts had been awarded to 63% (n =17) of the innovations as of the end of March 2012. In total, 15 departments and agencies are participating in the testing of innovations. Both PWGSC and Defence Research and Development Canada were each to test two innovations as a result of the first Call for Proposals.
114. Of the innovations matched with testing departments after the second Call for Proposals, two contracts have been awarded, two additional contracts were in development, and 17 statements of work/Innovation Transfer Agreement were in development as of the end of March 2012.
115. At the time of the evaluation it was too early in the program's implementation to assess achievement of the testing and feedback component of the program, as only one innovation had completed the testing phase.
116. The program noted that enlisting government departments in the testing phase of the program was more of a challenge than obtaining their participation in outreach activities. Program managers stated that other federal government departments need to be more engaged in the program. Government departments generally do not have a mandate for innovation in their procurement activities and are not knowledgeable of the benefits of innovative products and services. The proper contacts within the departments and departmental champions need to be established and departments need to be made more aware of the benefits of the program and encouraged to participate.
117. In conclusion, CICP has shown progress towards meeting this outcome, as indicated through multiple matches made between suppliers and other government departments, but has not yet implemented activities to an extent where a complete assessment can be made.

Intermediate outcome: Canadian companies are better prepared to enter the marketplace

118. CICP managers were unanimous in indicating that they believed the program better prepared companies to enter the marketplace. They indicated that even the suppliers whose submissions were not selected still derived critical business knowledge by going through the application process. In the course of putting together proposals, businesses must prepare commercialization strategies, market assessments, business plans, and financial impacts, all of which are value-adding exercises regardless of whether the proposal is successful. However, survey results of businesses that participated in the first round of Call for Proposals indicated that participants did not feel that participating in the CICP helped prepare them to commercialize their innovation. Only 36% of the respondents "strongly agreed or agreed" that they were better prepared.
119. Many of the program managers interviewed claimed that educating the suppliers on the governmental procurement process is an important part of CICP's mandate. One way to achieve this might be for the program to engage the suppliers that did not qualify to "close the loop" and ensure they have gained knowledge on how to enter into the federal marketplace through their participation in CICP.
120. In conclusion, it is too early in the implementation of the CICP to fully assess this outcome. Initial findings suggest that few of surveyed participants believed they are better prepared to participate in the marketplace as a result of the program, however, none have yet benefited from the use or feedback from government departments.

Intermediate outcome: Federal departments contribute to a culture of innovation in Canada

121. Government stakeholders interviewed agreed that the government has a role to play in the commercialization of innovations and that the program creates a better understanding of innovation within government. One stakeholder noted that "the program brings people together to share knowledge and meet government departments and others researchers." Suppliers, however, were less unanimous on this subject. Of the suppliers who responded to the Office of Audit and Evaluation survey, 55% said that they "strongly agreed or agreed" with the statement that CICP and the government contributed to the culture of innovation in Canada.
122. In conclusion, it is too early in the implementation of the CICP to fully assess this outcome. Initial findings suggest that half of surveyed participants believe federal departments are contributing to a culture of innovation in Canada.

Ultimate outcome: Canadian innovations are commercialized successfully as a result of the Program

123. Since the program is in the pilot phase and has just begun matching and testing innovations with other federal government departments, the evaluation was unable to conclude on the achievement of this intended outcome.

Conclusions: Outcome achievement

124. The program has been successful in attracting SMEs and federal departments to the program. Enlisting government departments in the testing phase of the program, however, has been more of a challenge and took longer than obtaining their participation in outreach activities.
125. The program is achieving success in the use of innovations by government, as indicated through multiple matches made between suppliers and other government departments.
126. Progress made to date in the implementation of the program and early results within the domain of immediate outcomes has well-positioned the program for achieving the intended intermediate and ultimate outcomes.

Efficiency and economy

127. Demonstration of efficiency and economy is defined as an assessment of resource utilization in relation to the production of outputs and outcomes. Efficiency refers to the extent to which resources are used such that a greater level of output is produced with the same level of input or, a lower level of input is used to produce the same level of output. Economy refers to minimizing the use of resources. A program has high demonstrable economy and efficiency when resources maximize outputs at less cost and when there is a high correlation between minimum resources and outcomes achieved.
128. The evaluation found that the program is demonstrating efficient use of resources in the early stage of its implementation, particularly through leveraging OSME regional offices, online outreach strategies and through modifications by NRC-IRAP to the approach taken in the technical assessment of proposals. The impact on the cost-efficiency of the program due to revised completion targets for the contracting element of the program have not been assessed by the evaluation.
129. The program was able to reallocate funds originally budgeted for outreach to purchase innovations. Since CICP used OSME's network of regional offices to provide outreach, economies and efficiencies for these activities were realized.
130. Increased efficiencies in its outreach activities were achieved following the first Call for Proposals. After the first Call for Proposals the program concluded that in-person bidders conferences were time consuming and cost-prohibitive. They also provided some bidders (those who were able to attend locally) with information before bidders in other parts of the country. In response to these issues, the program established online conferences. The resulting "webinars" proved cost-effective since they can be broadcast from one location to attendees across the country, reducing program travel and accommodation costs. The program was able to provide information and respond to over 300 questions in the course of the first online conference. The process was repeated successfully for the second Call for Proposals.

Conclusions: Efficiency and economy

131. The program has demonstrated a commitment to efficient program delivery by reducing the cost of its outreach activities by leveraging OSME's outreach activities and replacing in-person bidders' conferences with more accessible webinars.
132. Through its reallocation of funds and re-profiling, the program has shown that it is aware of the risks associated with unspent funds. It has revised its timelines for purchasing innovations in order to prevent the lapsing of funds.

General conclusions

133. The rationale for the program remains sound and there is a continued need for the unique form of assistance provided by CICP. The continued relevance of CICP has been articulated in the 2012 Federal Budget.
134. The program is aligned with government priorities related to support for innovation and for small and medium enterprises and PWGSC's departmental strategic outcome. CICP is aligned with federal roles and responsibilities related to procurement. As the principal procurer of goods and services for the Government of Canada, PWGSC is uniquely positioned to foster innovation in the area of goods and services required by the federal government.
135. The program has been successful in attracting Small and Medium Enterprises and federal departments to the program. Initial targets set for participation were greatly exceeded. Enlisting government departments in the testing phase of the program has proven a greater challenge than obtaining their participation in outreach activities.
136. The program is achieving success in supporting the use of innovation by government, as indicated through multiple matches made between suppliers and other government departments.
137. Progress made to date in the implementation of the program and early results position the program well for the achievement of the intended intermediate and ultimate outcomes.
138. The program activities examined by the evaluation were implemented successfully. In the first Call for Proposals of the program, actual contracting for qualified innovations took place at a slower pace than expected, resulting in delayed implementation of testing and feedback activities.

Recommendations and management action plan

Recommendation 1: The Assistant Deputy Minister, Acquisitions Branch, should revisit the performance measurement strategy (including logic model, performance indicators and targets) for the program to incorporate lessons learned in the implementation of the pilot procurement initiative and to reflect expectations for an on-going program.

• Management action plan 1.1: A review of the current program logic model and lessons learned from the Pilot will be gathered culminating in a Lessons Learned Report.
• Management action plan 1.2: The Lessons Learned Report will inform the development of a new performance measurement strategy. This new strategy will detail a logic model, performance indicators and targets that are commensurate with a mature on-going program. A Draft Performance Measurement Strategy will be produced and distributed for general comment prior to finalization.
• Management action plan 1.3: A final Performance Measurement Strategy for ADM approval will be produced before March 31, 2013.

Recommendation 2: The Assistant Deputy Minister, Acquisitions Branch, should assess the time required to establish contracts for qualifying innovations and, where warranted, revise current standards with the objective of improving the overall timeliness of the implementation of the program's activities.

• Management action plan 2.1: An assessment will be made of the time required to establish contracts for qualifying innovations. If warranted, changes will be made to existing standards to improve timeliness and this will be reflected in the Performance Measurement Strategy.

About the evaluation

Authority

The Audit and Evaluation Committee of Public Works and Government Services Canada approved this evaluation as part of the 2011-12 to 2015-16 Risk-Based Multi Year Audit and Evaluation Plan.

Evaluation objectives

The evaluation examined the Canadian Innovation Commercialization Program, delivered by the Office of Small and Medium Enterprises within Acquisitions Branch. This evaluation had two objectives:

• To determine the relevance of the program: the continued need for the program, its alignment with governmental priorities, and its consistency with federal roles and responsibilities.
• To determine the performance of the program: the program's implementation, its preliminary outcomes, and its demonstration of efficiency and economy.

Approach

The evaluation was conducted in accordance with the Evaluation Standards of the Government of Canada and those of the Office of Audit and Evaluation at PWGSC. The evaluation took place between May 2011 and April 2012 and was conducted in three phases: planning, examination, and reporting. To assess the evaluation issues and questions, the following lines of evidence were used.

Document review: An initial document review provided an understanding of the program and its context to assist in the planning phase. Documents reviewed included documents provided by the program, as well as documents written about the program. Documents included past Speeches from the Throne, PWGSC's Reports on Plans and Priorities, Business Plans, and Program Reports.

Literature review: A literature review was conducted to contextualize the program both nationally and internationally; provide theoretical background for the program model; provide baseline data against which the program could be assessed and identify alternative delivery models through an analysis of other jurisdictions.

Interviews: The evaluation team conducted interviews with key program and PWGSC managers. In addition, the evaluation team conducted interviews with representatives from client departments and agencies, program partners, and business associations. The qualitative analysis of the interviews provided information about the program's activities, outputs, and expected outcomes. Interview guides were used.

Survey: A survey was developed by the evaluation team to capture client perspectives on the relevance and performance of the program. The population surveyed were the suppliers who submitted bids to the first Call for Proposals of the program. The results of the surveys were incorporated where relevant and form a benchmark for future evaluations.

Financial analysis: Financial data related to the program's budget and expenditures were reviewed and analyzed to comment on the efficiency and economy of the program. All financial information was provided by the program.

Limitations of the methodology

As the scope of the evaluation was intended to cover the first round of procurement, due to delays in the process, the program was unable to complete all of the contracts by the time the evaluation had commenced. As such, the evaluation was not able to comment on the early results with regard to innovation usage within departments.

Reporting

Findings were documented in a Director's Draft Report, which was internally cleared through the Office of Audit and Evaluation's Quality Assurance function. The Program's Director General will be provided with the Director's Draft Report and a request to validate facts and comment on the report. A Chief Audit and Evaluation Executive's Draft Report will be prepared and provided to the Assistant Deputy Minister, Acquisitions Branch, for acceptance as the Office of Primary Interest. The Office of Primary Interest will be requested to respond with a Management Action Plan. The Draft Final report, including the Management Action Plan, will be presented to PWGSC's Audit and Evaluation Committee for the Deputy Minister's approval in November 2012. The Final report will be submitted to the Treasury Board Secretariat and posted on the PWGSC website.

Project team

The evaluation was conducted by employees of the Office of Audit and Evaluation, overseen by the Director of Evaluation and under the overall direction of the Chief Audit and Evaluation Executive. The evaluation was reviewed by the Quality Assessment function of the Office of Audit and Evaluation.

The Director of Evaluation was consulted as a member of the sub-committee of evaluation during the consultation and research phases of the Review of Federal Support to Research and Development - Innovation Canada: A Call to Action. The consultations addressed design and measurement of innovation programs. The consultation process did not have an impact on the conduct of the evaluation or its findings.

Appendix A: Technology readiness levels

Appendix B: Evaluation matrix