Introducing Front Porch Digitals Archive eXchange Format

Ever since the Tower of Babel fell, human beings have sought a common way to communicate with one another and with future generations. Technological innovations — writing, for example — did a lot to preserve communication but at the same time posed new problems. Today the diversity of discrete formats for processing and storing digital media makes communication among different production and storage systems complicated, expensive, and sometimes downright impossible. The goal of establishing a common language among all the world’s people has proved elusive, but the goal of establishing a common format for digital media doesn’t have to be. In fact, over the past several years, we at Front Porch Digital have developed such a specification in the Archive eXchange Format, AXF. What’s required now is general agreement that this “common language” is a good idea.
AXF Defined
At the most basic level, AXF is a file container that can encapsulate any number and any type of files in a fully self-describing package. AXF’s defining attribute is its innovative embedded file system approach whereby the encapsulated package actually contains its own file system abstracting the underlying operating system, storage technology, and file system. This approach allows AXF to be both content and storage agnostic. In other words, because the AXF object itself contains the file system, it can exist on any generation of data tape, spinning disk, flash, optical media, or other storage technology that exists today or may exist tomorrow.
The AXF specification has been architected so that each AXF Object originates with an AXF Object Header — a structure containing descriptive XML metadata such as the AXF Object’s unique identifier (UMID/GUID), creation date, object provenance, and file-tree information including file permissions. Following the AXF Object Header are any number of optional AXF Generic Metadata packages, which are self-contained, open metadata containers in which applications can include AXF Object-specific metadata. This metadata can be structured or unstructured, open or vendor-specific, binary, or XML.
The next part of the AXF Object construct is any number of File Data + File Padding + File Footer triplets — or the AXF File Payload, the actual byte data of the files encapsulated in the object. File padding, which ensures alignment of all AXF Object elements on storage medium block boundaries, is key to the AXF specification. The File Footer structure contains the exact size of the preceding file, along with an optional file level checksum designed to be processed on-the-fly by the application during restore operations.
The final portion of an AXF Object is the AXF Object Footer, which repeats the information contained in the AXF Object Header and adds information captured during the AXF Object’s creation, including per-file checksums and structure block positions. The AXF Object Footer is important to the resiliency of the AXF specification because it allows efficient re-indexing by foreign systems when the media content is not previously known. This adds self-describing characteristics to the AXF medium itself, ensuring full recoverability in the case of application data loss or movement to a foreign system, regardless of the storage technology selected.
Supporting High-Volume Storage
For us at Front Porch Digital, a prime motivator for developing an advanced open file specification like AXF was to meet the needs of users attempting to leverage linear data tape storage in demanding environments. To meet this need, AXF is not only tuned for performance, it can scale from simple IT-centric disk-based, single file applications through to the most demanding media operations, those with petabytes of content stored in long-term archives on data tape. In all cases, AXF offers an abstraction layer that hides the complexities of the storage technology from the higher-level applications, while it also offers fundamental encapsulation, provenance, fixity, portability, and preservation characteristics. And the same self-describing AXF format can be used on disk storage as well.
Because AXF is media and storage agnostic, it can support the modern generation of data tape formats such as LTO5 and T10000C, which can exist with or without a file system — namely the Linear Tape File System, or LTFS. AXF brings to LTFS all of the benefits described previously, which are not part of the LTFS design itself. AXF and LTFS are certainly not mutually exclusive but users should understand the limitations and transport-centric nature of LTFS prior to selecting it as their archive or preservation format. In fact, AXF provides all of the features offered by LTFS with many additional benefits. Further, because AXF does not rely on features of modern data tape technologies, users with existing infrastructures do not have to upgrade, saving significant cost.
Advantages of an Embedded File System
The AXF specification’s embedded file system enables AXF to translate between any generic set of files and logical block positions on any storage medium, whether or not the medium has its own file system. This gives AXF an advantage over formats that rely on storage technology elements — file marks on data tape, for example — because that reliance hinders both storage capabilities as well as performance. Likewise, these alternative formats are ineffective for complex file collections with millions of related elements and for extremely large individual files, which may span more than one storage medium — some of the limitations of LTFS.
A further benefit of AXF over legacy container formats, as well as some modern operating and file systems, is its ability to scale without limits, encapsulating any number of related component files of any size. AXF thus supports large-scale archive and preservation systems as well as simple, standalone applications. The benefit to users is easier encapsulation or wrapping of content, longer term protection, and the ability to transport content between systems from different vendors without regard for the underlying storage technology, file system, or operating system.
Conclusion
According to the Biblical account of the Tower of Babel, a people with one language are capable of accomplishing anything they can imagine. While an open standard for storage and preservation of media assets may not be quite the panacea that implies, its ability to support interoperability among systems helps ensure long-term accessibility to valued assets. And to keep up with evolving storage technologies offers profound present and future benefits for any enterprise that uses media — from heritage institutions, to schools, to broadcasters. Front Porch Digital has incorporated AXF into the latest release of its DIVArchive Content Storage Management (CSM) system, which was launched at the 2011 NAB Show, and today the company is committed to working closely with standards bodies such as SMPTE for ratification and standardization of AXF.
More information on AXF and SMTPE standards body activities is available at OpenAXF.org and smpte.org.