Digital Initiatives at LAC

Digital Policies, Guidelines and Tools

Library and Archives Canada
Local Digital Format Registry (LDFR)
File Format Guidelines for Preservation and Long-term Access
Version 1.0

Table of Contents | Next

1. Introduction

1.1 Purpose

This document identifies the file formats that Library and Archives Canada (LAC) will be supporting within the Trusted Digital Repository (TDR). The formats are identified as:

• Recommended; or
• Acceptable for transfer.

“Recommended” formats are those that LAC believes will be sustainable over a long period of time, whereas the formats considered “acceptable for transfer” are those formats that LAC considers to be most representative of commonly used formats (formats in widespread use) in the collections that LAC will be preserving in the TDR (e.g., most commonly used formats in digital publications and Government of Canada (GoC) electronic records).

The list of file formats to be supported will evolve over time, particularly as new formats are introduced or older formats become obsolete. It should be noted that for any given collection submitted for preservation within LAC’s TDR, file formats that do not fall within the category of “recommended” or “acceptable for transfer” will be evaluated on the basis of their content: where the content is deemed of preservation value, the content will be normalized/migrated to a “recommended” preservation format¹.

1.2 Background

1.2.1 Preserving digital information

Canadians have been generating digital information for decades. Our books, music, movies and the records of our private and public organizations are increasingly being created in digital formats. The preservation of this digital information is a problem that touches all sectors – academic, government, private and non-profit – and ultimately all Canadians.

By its very nature, digital information is fragile. Digital bits can be preserved, but our ability to use the information is at risk if the computer hardware and software needed to interpret/render the information are no longer available, or the format specifications are not accessible (e.g., the format is proprietary, is subject to intellectual property rights, or the specifications are no longer available). Preserving digital information is complicated. It involves the active commitment of organizations, the development of appropriate policies and plans, and the implementation of sound practices. It requires all organizations with an interest in preserving digital information to share expertise, advice and best practices.

Among these best practices, the identification and use of appropriate file formats is critical for preserving digital information. Due to a mix of technical and practical issues, certain file formats are more suitable for digital preservation. This document identifies and describes digital formats which LAC is recommending for long-term preservation and access to digital information.

These recommendations are contextualized within LAC’s Digital Preservation Policy² and the development of LAC’s TDR. The TDR is LAC’s digital preservation infrastructure supporting secure acquisition, storage, management and continuing access to Canada’s digital memory.

1.2.2 Digital content preservation strategy

LAC has adopted the following strategy for preserving digital content:

• When digital content is first accepted/approved for preservation in the TDR (that is, the content has been evaluated by LAC and deemed to be of preservation value), a preservation master is created (termed a “preservation master (0)” or PM(0));
• As part of the acceptance/approval process, the digital content is normalized as required (that is, migrated from the submitted/transferred format to one of the appropriate recommended preservation formats), thereby creating a new preservation master (termed a “preservation master (+1)” or PM(+1));
• From the current preservation master (i.e., PM(0) or PM(+1)), a copy of the digital content is created to service access requests by internal and external users (termed a “service copy”)³;
• The service copies can be presented using LAC-supported play-out services as well as client-based play-out services where needed or desired (an example of a play-out service would be an Apache server for HTML pages combined with a browser on the client, or a video streaming server; on the client, the Adobe Reader is an example of a client-based play-out service).

1.3 Target audience and use

LAC has developed these guidelines for a broad audience including the public, academic and private sectors. Whether it is a government department producing a budget or a citizen self-publishing, this document is intended to provide guidance on which digital file formats are most suitable for preservation and long-term access.
These guidelines also serve as the policy foundation for LAC’s Local Digital Format Registry (LDFR), the underpinning set of guidelines for file format normalization/migration services within LAC’s TDR.

1.4 Scope

These guidelines and recommendations are concerned with media-independent content; that is digital content that is managed as file types and is not inextricably linked to a physical storage medium (in contrast to videotape which is dependent both on the physical carrier and the playback equipment). These guidelines do not address recommendations for physical preservation media⁴.

The file formats covered in this document have been clustered into the following content types:

• Text
• Audio
• Digital video
• Still images
• Web archiving 
• Geospatial 
• Structured data, including:
  • Databases
  • Statistical and Qualitative Analysis Data
  • Scientific Data
• Computer Aided Design (CAD):
  • Technical drawings
  • Computer-aided Software Engineering (CASE)

This document consists of file format recommendations based on LAC’s experience in collecting and preserving digital content as well as international best practices.

1.5 Summary of recommendations

1.5.1 Definition of file formats

Generally speaking, file formats are specific patterns or structures which organize and define data. Some formats contain only one ‘stream’ of uncompressed data, others may contain codecs to encode and compress the data⁵, and others still may support several ‘streams’ of media.

In addition to file formats, there are also ‘container’ or ‘encapsulating’ formats. These formats can contain and support various types or layers of audio, video, still imagery, and their associated metadata. Each of these formats may be handled by different programs, processes, or hardware; but for the multimedia data stream to be interpreted properly, the information must be encapsulated together. Library of Congress define three types of container formats:

• “wrapper” format: wrapper is often used by digital content specialists to name a file format that encapsulates its constituent bitstreams and includes metadata that describes the content within. Archetypal examples include WAVE and TIFF. Files that are instances of these wrappers are distinguished in terms of their underlying bitstreams, e.g., WAVE files may contain (a) linear pulse code modulated (LPCM) audio, (b) highly compressed audio as used for digital telephony, or (c) other representations of sound. Meanwhile, the self-describing, content-declaring feature of a wrapper is typified by the familiar TIFF header. Relatively more complex and facile wrappers like QuickTime may contain multiple objects, e.g., one or more video streams and separate audio streams;
• “simple bundling” formats: these formats encapsulate their constituent files and, save for a directory that provides the filenames, do not describe the content and the relationships that may exist between files. Archetypes include ZIP, StuffIt, and TAR, the latter associated with the UNIX operating system. Simple bundling formats tend to be generic, i.e., they may be used for a wide range of content types;
• “self-describing bundling” formats: these formats are employed to represent the bundle of files that comprise a complex digital work, e.g., a book text with supporting illustrations or a movie with multiple segments and sound tracks in different languages. Self-describing bundling formats list the component parts and their relationships (information about the relationships is often called structural metadata) and may indicate how the work as a whole can be rendered or used. Bundling formats often incorporate technical details about each component, since a single object may include a mix of texts, sound, images, etc. They may or may not encapsulate their constituent files. They include metadata that describes their content and the relationships between files. Archetypes for this subcategory include METS (Metadata Encoding and Transmission Standard) and MPEG-21 (Multimedia Framework).

For further information on formats, see the working definition⁶ on the Library of Congress Web site on Sustainability of Digital Formats.

There are thousands of file types now in existence: LAC’s guidelines specify only the file formats that will be supported in the TDR. For a more complete registry please refer to PRONOM⁷, the Unified Digital Format Registry⁸ or the Library of Congress Web site on Sustainability of Digital Formats⁹.

1.5.2 Evaluating the sustainability of file formats

In developing these guidelines, LAC has attempted to balance the requirements for quality, stability, potential longevity and industry acceptance. Where possible, a preference has been placed on the selection of non-proprietary national and international standards, or failing the availability of non-proprietary standards on, de facto standard industry formats. De facto standard formats are widely used and recognized formats that have become industry standards because of their ubiquitous use and support, and not because they have been formally approved by a standards organization. LAC has also reserved the right to select formats that it believes will become more widely adopted by the preservation community in the near future (e.g., SIARD).

Based on a review of criteria published by Library of Congress, the National Archives (UK), and the National Library of the Netherlands¹⁰, Library and Archives Canada has established the following criteria for evaluating file formats for long-term preservation and access.

Table 1, below, summarizes the evaluation scheme used, whereas Table 2, following, provides a definition for each evaluation criterion along with the rating to be assigned based on the degree to which the criterion has been met.

Table 1: Rating Scheme

Rating
Symbol	Description
√	Evaluation criterion fully met
√$	Evaluation criterion fully met, however a cost is associated with meeting the criterion (e.g., to acquire the specification)
*	Evaluation criterion partially met
x	Evaluation criterion not met
√/x	Evaluation criterion met in one sector (e.g., for Government of Canada content) but not met in another sector (e.g., for non-government / commercial content)
√/*	Evaluation criterion met in one sector (e.g., for Government of Canada content) but not met / partially met in another sector (e.g., for non-government / commercial content)

1.5.3 File format recommendations

Table 3, following, summarizes the files formats that LAC recommends for the preservation of and long term access to digital content, and also identifies the file formats that are acceptable for the transfer of digital content to LAC.

Please note that there is no implied migration path from the “acceptable for transfer” formats and the “recommended” for preservation formats. The selection of a preservation format will be based on the degree to which the significant properties of the source format (and of individual instances of the format) are retained in the target preservation format (and the relative importance (or weigthing) of specific properties).

Table 4 summarizes the ratings of LAC’s recommended file formats against the criteria identified in Section 1.5.2, whereas Appendix A – Recommended Preservation Format Evaluation provides detailed rating information. Please note that there is no implied order of preference / precedence in the list of formats.

Appendix B – Applying the Guidelines to LAC Preservation Policies, graphically demonstrates the mapping of the recommended preservation formats to LAC’s preservation strategy (outlined in Section 1.2.2).

Table 2: Evaluation Criteria Definition and Rating

Criterion	Evaluation Basis	Rating
Openness/Transparency	Specifications available from one or more of the following: a) Open membership organization (such as the W3C (World Wide Web Consortium), the OMG (Object Management Group)) b) International standards organization (such as the ISO) c) Industry-based open membership organization	√ Evaluation criterion fully met
	Specifications available only at cost	√$ Evaluation criterion fully met, however a cost is associated with meeting the criterion
	Specifications potentially available from multiple sources (could not be confirmed)	* Evaluation criterion partially met
	Specifications only available from / under the control of a single vendor or small group of vendors	x Evaluation criterion not met
Adoption as a preservation standard	The majority of the organizations investigated use/are planning to use the format as a preservation standard (50% or more of the organizations)	√ Evaluation criterion fully met
	Some of the organizations investigated use/are planning to use the format as a preservation standard (less than 50% of the organizations)	* Evaluation criterion partially met
	None of the organizations investigated use/are planning to use the format as a preservation standard	x Evaluation criterion not met
Stability/Compatibility
a) degree of forward/backward compatibility	A format is backward compatible if it provides all of the functionality of a previous release or version of the format A format is forward compatible if it has the ability to gracefully accept content intended for later versions of the format (that is, software designed to interpret / render a prior version of a format can also interpret / render the current version of the format) Forward/backward compatibility:
	a) High compatibility: A format is both forward and backward compatible	√ Evaluation criterion fully met
	b) Medium compatibility: A format is backward compatible only	* Evaluation criterion partially met
	c) Low compatibility: A format is neither forward nor backward compatible	x Evaluation criterion not met
b) degree of protection against file corruption	Corruption protection: Resilience to random bit-level/byte-level changes in content
	a) High resilience: Changes have little or no impact to renderability/interpretability / uses methods for detecting/recovering from changes	√ Evaluation criterion fully met
	b) Medium resilience: Changes affect renderability but not interpretability / some ability to recover from changes	* Evaluation criterion partially met
	c) Low resilience: Any change affects the ability to interpret and render the format	x Evaluation criterion not met
c) frequency of version releases	Format stability demonstrated by the number of version releases and/or extensions; format’s use in derivatives and/or industry-specific applications
	High format stability	√ Evaluation criterion fully met
	Medium format stability	* Evaluation criterion partially met
	Low format stability	x Evaluation criterion not met
Dependencies/Interoperability	Low dependency / High interoperability Low dependency / Medium interoperability Medium dependency / High interoperability	√ Evaluation criterion fully met
	Low dependency / Low interoperability Medium dependency / Medium interoperability Medium dependency / Low interoperability	* Evaluation criterion partially met
	High dependency / Low interoperability High dependency / Medium interoperability High dependency / High interoperability	x Evaluation criterion not met
Standardization	Format follows a formal process enacted by any of the following: a) Open membership organization (such as the W3C (World Wide Web Consortium), the OMG (Object Management Group)) b) International standards organization (such as the ISO) c) Industry-based open membership organization	√ Evaluation criterion fully met
	Format is subject to documented processes implemented by a single vendor or small group of vendors or no documented process	x Evaluation criterion not met

Table 3: Recommended and Acceptable for Transfer File Formats

Content Type	Recommended	Acceptable for transfer
Text	EPUB for electronic books Extensible Hypertext Markup Language (XHTML) Extensible Markup Language (XML) Hypertext Markup Language (HTML) Multipurpose Internet Mail Extensions (MIME) Open Document Format (ODF) PDF for long-term preservation (PDF/A) Rich Text Format (RTF) Standard General Markup Language (SGML) Text (plain text)	Office Suites: Microsoft Office including: Word Document Format, Excel Spreadsheet Format, Powerpoint Presentation Format WordPerfect Suite including: WordPerfect Document Format, Quattro Pro Spreadsheet Format, Corel Presentations Format Lotus Smartsuite including: WordPro Document Format, 1-2-3 Spreadsheet Format, Freelance Graphics Format Portable Document Format (PDF)
Audio	Broadcast Wave Format (BWF) (for newly digitized content (i.e., creating)) Waveform Audio Format (WAV) (for migrating born digital audio content)	Audio Interchange File Format (AIFF) Mpeg-1 layer-3, Mepg-2 layer-3 (MP3) Mpeg-4 aac – advanced audio coding (AAC) Musical instrument digital interface (MIDI) Window media audio (WMA)
Digital Video	Motion JPEG 2000	Audio video interleave (AVI) Moving pictures expert group (MPEG-2) Moving pictures expert group (MPEG-4) Quicktime (MOV) Windows media video (WMV)
Still Images	Joint photographic experts group (JPEG) Joint photographic experts group jpeg 2000 (JP2) Tagged image file format (TIFF) TIFF - GeoTIFF	Digital imaging and communications in medicine (DICOM v. 3.0) Encapsulated postscript (EPS) Graphics interchange format (GIF) Portable network graphics (PNG)
Web Archiving	Internet archive format (ARC) Web archive format (WARC)
Structured Data - Databases	Software Independent Archiving of Relational Databases (SIARD) Delimited Flat file with DDL	dBase Format (DBF)
Structured Data – Statistical and Qualitative Analysis	Data Documentation Initiative (DDI) Version 3.0 Data Exchange and Conversion Utilities and Tools (DExT) Statistical Data and Metadata Exchange (SDMX) Delimited Flat File with Variable Descriptions	SAS SPSS
Structured Data – Scientific	XML Container
Geospatial11	ISO 19115 Geographic Information – Metadata (NAP – Metadata) (North American Profile)	Canadian Council on Geomatics Interchange Format (CCOGIF) Digital Elevation Model (DEM) Digital line graphics – level 3 (DIG-3)* Environmental systems research institute (ESRI) export format – (E00)* Environmental systems research institute (ESRI) shape file format (SHP)* International Hydrographic Organization (IHO) S-57, Edition 3.1*
Computer Aided Design –Technical Drawing	Drawing Interchange File Format/Data eXchange Format (DXF)	Computer Graphics Metafile (CGM)
Computer Aided Design – CASE	XML Metadata Interchange (XMI)
Source Code and Scripts	XML Container	Text

---

¹ Note: Within the TDR, automatic normalization will be performed on the “acceptable for transfer” formats identified in the guidelines (conversion or migration to a “recommended” format): all other formats will be addressed on an individual case basis. Should the format prove to be a commonly used format, automated normalization/migration will be considered for future submissions.

²  www.collectionscanada.gc.ca/digital-initiatives/012018-2000.01-e.html

³ A service copy may be created as part of the acceptance/approval process or may be produced dynamically.

⁴ A policy addressing storage media for use in preservation is currently under development.

⁵ Please see Appendix C: Concepts and Definitions - Codecs.

⁶ www.digitalpreservation.gov/formats/intro/format_eval_rel.shtml#what

⁷ www.nationalarchives.gov.uk/pronom/

⁸ www.gdfr.info/udfr.html

⁹ www.digitalpreservation.gov/formats/content/content_categories.shtml

¹⁰ See Gillesse et al 2008; Rauch, Carl et al. 'File-Formats for Preservation: Evaluating the Long-Term Stability of File-Formats." Proceedings ELPUB2007 Conference on Electronic Publishing : Vienna, Austria , 2007. http://elpub.scix.net/data/works/att/122_elpub2007.content.pdf; National Archives (UK). "Selecting File Formats for Long-Term Preservation." (2003). www.nationalarchives.gov.uk/documents/selecting_file_formats.rtf; Library of Congress. "Sustainability of Digital Formats: Planning for Library of Congress Collections." (2007). www.digitalpreservation.gov/formats/sustain/sustain.shtml.

¹¹ For geospatial information, the “acceptable for transfer” formats with asterisks will be preserved as is (not migrated) until such time as the adoption rate of the Treasury Board Secretariat (TBS) standard (identifying ISO 19115), and the avalaibility of tools supporting the standard is more fully understood (exception to preservation strategy for the near future).

Table of Contents | Next