Evaluation of the Document Imaging Services Sub-program (1.7.4) (final report)

Introduction

1. This report presents the results of the Evaluation of the Document Imaging Services sub-program located as item 1.7.4 on the 2014 to 2015 Program Alignment Architecture (PAA) for Public Works and Government Services Canada (PWGSC). This program first appeared in the PWGSC's PAA in 2008.

2. The Document Imaging Solutions Center (DISC) (the program) is situated in the Cheque Redemption Control Directorate (CRCD) located in the Accounting, Banking and Compensation Branch (ABCB).

3. This evaluation was included in the 2013 to 2018 Risk-Based Audit and Evaluation Plan. The Evaluation provides an assessment of the relevance and the performance of the Program for the period April 2012 to September 2014. Planning and research took place between May and December 2014, in accordance with the Standard on Evaluation for the Government of Canada and in conformity with the PWGSC Policy on Evaluation. More information on the approach, methodologies and limitations of this evaluation can be found in Appendix A: About the evaluation. 

4. The Evaluation Directorate completed a risk-based calibration assessment to determine the best approach for the conduct of this evaluation. The assessment identified Document Imaging Services as an appropriate candidate for a streamlined evaluation for the following reasons:

• the Program represents less than 1% of PWGSC's annual spending
• the Program operates within a centralized delivery mechanism
• the Program experiences low sensitivity to public scrutiny

Why it is Important

5. In 2014, the Government of Canada launched "Digital Canada 150", a plan to guide its digital future. This future includes the digitization of government documents. For example, as of 2017, Library and Archives Canada will only accept from federal organizations, digital records and their associated metadata for archival purposes. It is estimated that up to 93.3 million pages of documentation are produced annually by federal departments and agencies. Digitization can assist federal departments and agencies in meeting their recordkeeping obligations. For these reasons, as well as to meet PWGSC's obligations under the Treasury Board Policy on Evaluation, it is important to periodically evaluate the relevance and performance of PWGSC's Document Imaging Services to support decision-making.

About the program

6. In 2001, using modern imaging technology, CRCD implemented an automated system in support of the Receiver General for Canada program that electronically reconciled paper cheques received from financial institutions. This automated system included a process of creating a digital copy (image) of the cheques (document) for the purposes of performing an online reconciliation within the accounting system. CRCD continues to perform this activity in-house for the Receiver General of Canada.

7. In 2002, CRCD began offering its document imaging services on a cost-recovery basis to other PWGSC programs, as well as to other federal departments and agencies, creating the standalone program called DISC. Document imaging (also referred to as electronic imaging or imaging in this report) is a technological process by which users scan hard copy documents (such as cheques, letters, forms, pictures, and/or microfilms) into a computer system and store them in digital format. The process includes indexing of the digital files and/or entering metadata^{Footnote 1} about the digital files into a computer system and saving the digital files on some form of storage technology (CDs, hard drives, servers) for query or retrieval purposes. The Program employs advanced imaging technology.

8. CRCD and DISC activities are performed in Matane, Quebec, using a common infrastructure of high volume production scanners. DISC is overseen by the Director General for CRCD and the Director located in Gatineau, Quebec, where there is a small project office for client engagement and partnership management.

9. DISC provides federal departments and agencies with a range of support services in the area of document imaging which are referred to as "Document Imaging Solutions". The document imaging process employed by DISC is detailed in Appendix B: Document imaging process. DISC initially works with its clients to identify the areas of support required (the business requirement definition phase, the analysis and configuration phase and the testing phase), and then provides the services selected by the client (the production phase).

10. DISC presents its services to its clients under the following 4 categories:

1. Core: under this service level, there are no document preparation services provided in advance of scanning. Digital file processing is limited to a data capture of up to 8 characters. No quality assurance process is applied. Digital files are returned to the client by external media or an electronic file transfer
2. Basic: under this service level, there are limited document preparation services provided in advance of scanning. There is greater data capture (that is file number and file name) than at the core service level. A quality assurance process is employed. DISC returns digital files to the client by external media or an electronic file transfer
3. Enhanced: under this service level, the same services are provided as at the Basic level however there is also the possibility of receiving the client's documents from multiple platforms (scanner, fax, etc.). The documents go through a similar document preparation, scanning and quality assurance processes as under the Basic service level, however, more data is captured and data validation steps can be performed against data extracts provided by the client. At this service level, DISC can manage the digital files on behalf of the client whereby DISC gives authorized users access to the digital files via a secure web portal
4. Comprehensive: under this service level, the same services are provided as at the enhanced level, however DISC can also provide support to its clients in the management of their own users. At the front end of its service delivery for this level, DISC can manage the receipt of documents from different technologies and from different users: documents can be received by DISC on behalf of the clients through a centralized PO Box, email and/or fax. At the tail end of its service delivery, DISC can interface with its clients' information technology (IT) when delivering digital files, a feature whereby DISC can directly enable its clients to manage their own client base

11. DISC has 3 established price points for 3 of its 4 service levels (basic, enhanced and comprehensive). DISC also offers other custom services for secret and archival documents for which a price-estimate can be provided.

12. Information on the type and volume of services conducted by the Program on behalf of its clients is detailed later in this report.

13. A logic model (Appendix C: Logic model) for DISC was prepared by the Evaluation team in consultation with the program. The logic model identifies DISC's intended outcomes and the activities undertaken by the program in support of their achievement. The following provides a brief description of the activities identified in the logic model:

1. Outreach: consists primarily of:
   1. participation (through information kiosks or booths) at various government fairs and other events to promote PWGSC's services in this area
   2. maintenance of a presence in the federal government's record and information management community
   3. liaison with client service groups as services are included in Departmental Service Agreements and service standards are posted on the PWGSC internet website
2. Client, costs, and performance management: consists mainly of:
   1. the conduct of client need assessments to assist federal departments in identifying their imaging requirements
   2. the development and management of the Memorandum of Understanding (MOU) between client departments and PWGSC
   3. the development and maintenance of the cost-recovery model according to the PWGSC cost recovery framework, the processing of account payables and receivables
   4. the development and monitoring of the service standards and performance measures
   5. the production of reports for different stakeholders
3. Operations: includes processes, procedures and systems used to carry out the document imaging and related activities. The processes, procedures and systems are independently certified and DISC maintains an internal quality assurance function for its operations. The activities, which vary depending on the service levels chosen by the client, encompass the following categories:
   1. Centralized inbound mail management (including regular mail, courier and fax): receive correspondence and select, record track, open and prepare documents for scanning
   2. Scanning: scan documents into the file format selected by the client and check the quality of images with automated and manual tools
   3. Automated and manual data extraction and classification: index, classify and use automated software to extract key information (that is., metadata) from paper records
   4. Access, archiving and disposal: archive and provide record management services on a secure system. Images can be accessed through integration with client IT systems and online access can be provided using the government intranet. On-site physical document storage and shredding are also available to the client

14. In fiscal year 2013 to 2014, $11.45M of revenues and $11.42M of expenditures were reported; this included the salaries and associated costs of the program's 141 employees (approximately 107 Full-time employee (FTE)s).

Authority

15. DISC, as an in-house activity of the CRCD, was initiated under the authority of the Receiver General for Canada. No additional authority was sought for its provision of imaging services to other federal departments and agencies. PWGSC has the authority to charge and recover costs for services under the Department of Public Works and Government Services Act and the Treasury Board Common Services Policy. PWGSC's authority to spend the revenue generated by DISC is derived from the Treasury Board Policy on Special Revenue Spending Authority.

Evaluation reporting

16. The results of the Evaluation research and analysis are presented in table format (see Section 1: Relevance and Section 2: Performance). The tables present the findings, evidence, conclusions and recommendations by evaluation issue. The indicators and data sources used to assess the evaluation issues are included in the table as well. This reporting format meets the requirements for evaluation reporting set out in Section 6.4.1a (i-ix) of the Treasury Board Standard on Evaluation for the Government of Canada.

Evaluation conclusions

17. There is a continuing need for DISC, and a growing need for advice and guidance in identifying document imaging and data capture needs in the emerging domain of document digitization. There is a strong current and projected need for document imaging services on the part of federal departments and agencies to meet their business and record keeping needs. DISC has responded to that need and is nearing maximum capacity.

18. Although some other government departments do their own imaging in-house, there are no other 'standalone' programs such as DISC within the Government of Canada. While similar services are available from the private sector, none of the firms on the Standing Offer provide the entire suite of services provided by DISC. Furthermore, the program is highly complementary with PWGSC's Receiver General for Canada and the Pension Programs. DISC's experience in defining imaging requirements and ability to provide an entire suite of services, combined with its secure operating environment, has laid the ground work for a new multi-channel enterprise solution to support the increased digitalization of Government of Canada documentation. The Evaluation found that DISC is currently in the process of identifying and analysing the business needs of federal departments with the intent of better defining its value proposition of the services it offers to clients.

19. While objectives for DISC have yet to be formalized the program was found to be aligned with government and departmental priorities, in particular those related to the achievement of efficiencies, the safeguarding of information and regional development. The provisions within the Public Works and Government Services Act provided a basis for the continuation of DISC as well as for the establishment of a common service within the Department to provide document imaging and data capture advice and services to other federal departments and agencies.

20. Performance results to-date for DISC are mixed. DISC services are compliant with international quality standards. The quality management system is International Organisation for Standardization (ISO) compliant and undergoes yearly audits to maintain the certification. The Evaluation determined that DISC has generally provided timely service to its clients; however, the Program has not collected data on all service standards related to timeliness. The Evaluation determined that DISC encountered challenges providing competitively priced services. Efficiency gains were observed in 2011 to 2012 and 2012 to 2013 fiscal years, however, we observed an increase in its average cost per page in 2013 to 2014. Program management is aware of its current barriers to competitiveness and is taking action to increase its efficiency in the future.

21. The Evaluation did not assess the extent to which there have been savings (economy) to the Government of Canada as a result of departments' and agencies' use of DISC service, nor through the provision of a common infrastructure for document imaging and data capturing services to other government departments and agencies.

Management response

22. Public Works and Government Services Canada (PWGSC) intends to transition from its current optional direct document imaging service fulfillment to an enterprise model that primarily uses the private sector to meet imaging needs, thus ensuring cost containment, best value for money as well as compliance with required Government of Canada standards and policies. Towards that end, a Request for Proposal (RFP) has recently been posted.

Recommendations and Management Action Plans

Recommendation 1: the Assistant Deputy Minister for Accounting, Banking and Compensation Branch (ABCB) should enhance the ongoing relevance of the Program by:

1. formalizing the rationale and objectives for PWGSC's on-going provision of document imaging services to federal departments and agencies
2. considering the value proposition offered by DISC in the context of the complementary and duplicative aspects of its services with the private sector

• Management action plan 1.1: A business case detailing the approach, rationale, objectives and benefits of the Enterprise Solution for Document Imaging has been developed. Communication materials highlighting the benefits and relevance of the Enterprise Document Imaging Solution will be developed
• Management action plan 1.2: ABCB will limit use of DISC internal services, conducting further analysis on those services once results of the RFP are known and vendors are on board

Recommendation 2: The Assistant Deputy Minister for Accounting, Banking and Compensation Branch should enhance the ability of the Program to achieve its outcomes related to pricing and efficiency by examining the potential for alternative delivery approaches.

• Management action plan 2.1: ABCB will be engaging client departments and confirming imaging volume/dollar commitments, driving volume price discounts
• Management action plan 2.2: ABCB to launch Enterprise Solution for Document Imaging that primarily uses the private sector to meet imaging needs. The RFP has been structured to enable competitive pricing and ensure departments pay only for the services they need. If approved, the mandatory aspect of the service will ensure volume discounts

Recommendation 3: The Assistant Deputy Minister for Accounting, Banking and Compensation Branch should enhance the performance measurement practices of the Program by streamlining the number of service standards in Memorandums of Understanding with clients and ensuring the on-going data collection and assessment of those found to be the most useful.

• Management action plan 3.1: ABCB will streamline the number of service standards in Memoranda of Understanding with clients, in support of the ongoing data collection and assessment of the service standards found to be the most useful
• Management action plan 3.2: ABCB will implement a quarterly dashboard to report on service standards

Section 1: Relevance

Continued need: The extent to which federal departments and agencies have a need for the type of services provided by Document Imaging Services.

Federal priorities and departmental strategic outcome: The extent to which the objectives of Document Imaging Services are aligned with federal government priorities and with PWGSC's Strategic outcome

Appropriate Role and Responsibility for the Federal Government: Does the provision of DISC services align with federal roles and responsibilities?

Section 2: Performance

Outcome achievement: Immediate outcome—The extent to which DISC is offering federal institutions competitively priced document imaging and data capturing solutions.

Outcome achievement: Immediate outcome—The extent to which DISC offers document imaging and data capturing solutions in compliance with international quality standards.

Outcome achievement: Intermediate outcome—The extent to which Federal institutions receive efficient and timely document imaging and data capturing solutions.

Efficiency and economy:  is defined as the extent to which resources are utilized in relation to the production of outputs and outcomes. Efficiency refers to the extent to which resources are used such that a greater level of output is produced with the same level of input or, a lower level of input is used to produce the same level of output. Economy refers to minimizing the use of resources in the pursuit of outcomes. A program has high demonstrable economy and efficiency when resources maximize outputs at least cost and when there is a high correlation between minimum resources and outcomes achieved. Efficiency and economy was assessed through multiple lines of evidence including document review, data review and interviews.

In the course of evaluating DISC, efficiency and economy were evaluated as core components of the Program's immediate and intermediate outcomes. It was found that DISC had made progress towards providing federal institutions with efficient document imaging and data capturing solutions when it has been able to maintain economies of scale in its operations (efficiency) but had encountered challenges providing competitively priced services (economy). The Evaluation, however, did not identify the savings (economy) to the Government of Canada through the use of DISC services by government departments and agencies (such as through related to reduced or eliminated paper file storage costs as a result of document imaging and data capturing solutions).

DISC has achieved a level of efficiency as a result of applying economies of scale: it has reduced its average cost per page from $.70 in 2010 to 2011 to $.43 in 2013 to 2014.

DISC demonstrated a level of economy at the time of its establishment (pre-evaluation scope) through its re-use of existing government resources to implement this program for a cost-avoidance equivalent of $3.5 million (from 2003 to 2008), however, the Evaluation was unable to determine the extent of savings to the Government of Canada through the provision of a common infrastructure for document imaging and data capturing services. The program operates under the PWGSC Cost Management Framework, using the net voting funding mechanism, and therefore has the obligation to cost recover and break even at each fiscal year end. The Evaluation found that over the last 5 years, the Program has had a deficit once and a surplus on 4 occasions as show in Exhibit 4: Document Imaging Solutions Center Revenues, Expenditures and Balances, 2009 to 2010 through 2013 to 2014 (appendix D); this is impacting DISC's ability to demonstrate economy as well as indicating the impact of inconsistent unit pricing, an issue discussed within this report under the assessment of Immediate outcome 1.

Appendix A: About the evaluation

Authority

This evaluation was included in the 2013 to 2018 Risk-Based Audit and Evaluation Plan.

Evaluation objectives

The Evaluation examined Document Imagining Services, delivered by the Cheque Redemption and Control Directorate within PWGSC. This evaluation had 2 objectives:

1. To determine the relevance of the program: the continued need for the program, its alignment with governmental priorities and departmental outcomes and its alignment with federal roles and responsibilities
2. To determine the performance of the program: the achievement of its expected outcomes and a demonstration of efficiency and economy by the Program

Approach to the evaluation of Document Imaging Services

The Evaluation Directorate completed a risk-based calibration assessment to determine the best approach for the conduct of this evaluation. The assessment identified DISC as an appropriate candidate for a streamlined evaluation for the following reasons:

• the program represents less than 1% of PWGSC's annual spending
• the program operates within a centralized delivery mechanism
• the program experiences low sensitivity to public scrutiny

The level of effort allocated to the conduct of the Evaluation reflects the above factors. An evaluation matrix was developed using the Treasury Board's 5 core issues to be addressed in evaluations. The following data sources were used to inform the lines of evidence to answer the questions in the evaluation matrix:

• Program document review: the preliminary document review provided information on the Program and its context to assist in the planning of the Evaluation. A more comprehensive program document review was conducted to collect information to answer the evaluation questions. The program document review consisted of primary documents that pertained to DISC: client files (including agreements, contracts, invoicing and payment records); program administration, monitoring and reporting documents (such as internally- and externally-focused reports); meeting minutes, client surveys, program-led studies; and financial records
• Literature review: the preliminary literature review provided information on the Program and its context to assist in the planning of the Evaluation. A more comprehensive literature review was conducted to collect information to answer the evaluation questions. The literature review consisted of secondary documents that pertained to DISC: departmental documents (such as the Report on Plans and Priorities and the Departmental Performance Report); government documents (such as legislative and policy documents); and external publications
• Financial analysis: financial data related to the Program's budgets, revenues, and expenditures were reviewed, including contract information and service fee structures. This information informed the analysis of program efficiency and economy
• Interviews: a total of 14 structured interviews, using an interview guide, were held with key program staff (n=3), clients (n=9), and officials with other federal organizations (n=2)

Limitations of the evaluation

The planning of the Evaluation applied a risk-based assessment to determine the most appropriate approach and level of effort to be applied in order to ensure the availability of timely and objective information to meet the needs of senior management. In the application of the approach, several issues were encountered. None of these issues were significant enough to prevent evaluation reporting.

• Literature review: the Evaluation was unable to locate data for the total value of imaging work provided internally by other federal departments and agencies and the Evaluation had limited access to data for the work provided externally by the private sector. the parameters of the Evaluation did not permit the research necessary to establish such data. The Evaluation was also unable to find benchmarks to compare the quality of DISC and private sector services. Consequently, the Evaluation relied primarily on the views of the Program's clients to provide an indication of the relative quality of services
• Interviews: the evaluation's interview invitations had a 37% response rate (9 out of 24)
• Surveys: the parameters of the evaluation did not include the administration of a survey. The evaluation relied on existing data available through 2 Program-directed surveys conducted prior to this evaluation
• Financial analysis: documents on program revenues were provided at an aggregate level, which impeded the Evaluation's ability to analyze aspects of the Program's performance in a more detailed manner. Moreover, the Program changed its methodologies for the calculation of costs and revenues over the period of 2011 to 2014 which made comparisons difficult

Reporting

Based on the lines of evidence, the Evaluation team prepared a Director's Draft Report presenting the findings, conclusions, and recommendations. This draft report was reviewed by the Quality Assessment and Practice Improvement Directorate within the Office of Audit and Evaluation and by the Chief Audit and Evaluation Executive. Subsequently, it was sent to senior program management for review and comments. Validation of facts and any other comments on the Director's Draft Report were assessed and, where appropriate, incorporated in the Chief Audit and Evaluation Executive's Draft Report. This report was sent to the Office of Primary Interest for final acceptance. A management response to the report and a management action plan in response to the recommendations was requested. The draft Final report, management response, and management action plan were tabled at the Audit and Evaluation Committee for the Deputy Minister's approval in March 2015. The Final report will be submitted to the Treasury Board Secretariat and posted on the PWGSC website.

Project team

The Evaluation was conducted by employees of the Office of Audit and Evaluation, overseen by the Director of Evaluation and conducted under the overall direction of the Chief Audit and Evaluation Executive. The evaluation was reviewed by the Quality Assessment and Practice Improvement Directorate of the Office of Audit and Evaluation.

Appendix B: Document imaging process

Appendix C: Logic model

Appendix D: Report exhibits

Exhibit 1: Document Imaging Solutions Center services in 2013 to 2014 (in volume and percentage)

Exhibit 2: Document Imaging Solutions Center clientele distribution between Public Works and Government Services and other government departments

Exhibit 3: Total costs and imaging volumes compared to cost per page for fiscal years 2010 to 2011 to 2013 to 2014

Exhibit 4: Document Imaging Solutions Center Revenues, Expenditures and Balances, 2009 to 2010 through 2013 to 2014