Why fast production storage is just not enough

Martin Bennett

Author: Martin Bennett

Published 1st July 2015

by Martin Bennett Issue 102 - June 2015

UHD, 4k, 6k, 8k¦the proliferation of high-resolution formats and files along with the growing number of skillsets that can collaborate on projects push storage bandwidth and capacities beyond their limits. If we just take a look at the potential storage requirements for a single 8K raw digital cinema format then we can kiss goodbye to 86,000 GB per hour! Then add to this the predicted increase in general video content IP video traffic is expected to be 79% of all consumer internet traffic in 2018 - this continual increase in media will test the storage strategies of all facilities, big and small.
Producing packages using best image quality from thousands of assets are what clients and audiences want. But without the right production storage infrastructure, this multitude of assets and potentially large workgroups can severely tax your storage infrastructure, creating multiple issues including poor streaming performance, interrupted access, and in many cases lost data. This is a real market concern. Facilities, media companies and broadcasters lose time, money and potentially clients and audiences.
With millions of assets available to potentially hundreds of creatives, resource contention across storage and lack of sophisticated media management tools become a major issue. The current crop of production storage solutions will not meet these demands, and cloud storage with its requirements for expensive dedicated access will not be the way forward for many years either. Efficient, reliable and high-availability media storage, normally reserved for enterprise-class applications, is fast becoming the requirement across all levels of the production and post. But is there an option that is affordable to solve these growing storage pains for all?
At EditShare, we have designed a scale-out media system that would address these data demands not just for the large enterprise operation, but for production and post facilities of all sizes. The core technology is based on the new EditShare File System (EFS), a parallel file system with clever file distribution and smart contention management that uniquely addresses the needs of emerging media workflows and high-resolution file formats. Taking it one step further, we included a media asset management application to better manage the increase in content and the growing number of people with access to it. And the key is its affordability.

An Intelligent Storage Environment
To illustrate the benefits, I will use the EditShare File System (EFS) as an example (see diagram below). In most EFS applications, client devices are non-linear editing systems (NLEs) or other media processing workstations and each client device is equipped with driver software that enables it to communicate with the EFS.
When a client device requests to write a file, EFS instructs the client to break the file into a number of data blocks, to calculate the parity data associated with those blocks, and to write the data blocks and parity, in parallel, to all of the storage nodes of the system. Thus, the designation of the EFS as a parallel file system.
A key performance benefit is the fact that when distributed in this manner, the resulting bandwidth of the write operation is the sum of the bandwidth available from all the storage elements (i.e. all disk drives) in the system.
The number of data blocks is determined by the XOR configuration of the storage system. The system illustrated in Figure 1 has an XOR 2 configuration; data blocks will be written to 2 of the nodes and the associated parity will be written to the remaining node. In the general case, the EFS always reserves one node for parity. Thus, an EFS system will have an XOR (nodes -1) configuration. There is no limit to the number of storage nodes in an XStream EFS Shared Storage System.

Reading Stored Data
When a client device requests to read a file from storage, it, again, communicates with EFS and is given the information it needs to locate the stored data associated with the requested file. The client retrieves those blocks and reassembles the file for use in the client application.
This type of file system is designed for media production environments where multiple editors share project spaces and media assets. Typically, each editor uses an NLE (client). It is a natural consequence of the environment that due to factors such as concurrent requests for the same data, the nature of the write process or simply bad luck, as more media files are distributed across the storage nodes and as requests to read files are made, contention for access to the storage nodes will increase. And in this environment, contention is unacceptable because of the potential to slow down or delay access to a given file and the potential to interrupt processes like live playout or cause video or audio frames to be dropped when content is being rendered.
Swift Read Technology
To deal with the issue of contention, the file system monitors the performance of the storage nodes and detects which ones are responding more slowly and may delay read requests. Armed with this critical information, it can instruct a client device to read all of the data blocks associated with the requested file or, if a node containing data is experiencing contention and is likely to delay a read request, to skip that node and reconstruct the requested data using the parity information instead. By reconstructing instead of waiting for a slow resource, data can be retrieved at a faster rate than is normally possible in the presence of contention for storage resources.
An added benefit of this type of file system is that the same performance monitor that detects slow responses from a storage node can also detect when a storage node has completely stopped working, as in the case of a hardware failure. Thus your file system is able to continue operation even if an entire storage node discontinues operating, offering you an unprecedented level of fault tolerance.

Efficient Use of Storage
The final benefit of a parallel file system is the efficient way in which it makes use of storage resources and how that efficiency increases as system size increases. In the case of the example in Figure 1, two storage nodes are used for data storage and one node is used for parity. Thus, the XOR 2 configuration illustrated uses 2/3 of its storage resources for data and 1/3 for parity. In an XOR 3 configuration, three nodes (3/4 of storage resources) are used for data and one is used for parity. An XOR 4 configuration uses 4/5 nodes for data, XOR 5 uses 5/6 nodes for data and so on.
Other high performance shared storage systems simply make duplicate copies of data to avoid contention and improve read speeds. This approach, known as mirroring, can at best use just 50% of storage resources.
Make Sure You Look Under the Hood When It Comes to Selecting Storage
So another step change is taking place in storage. A high-availability distributed file system for the masses was once inconceivable, but with the introduction of EFS it seems there is a solution on the market to allow facilities to grow their internal storage infrastructure securely and profitably. It is important for facilities and engineers to take a closer look under the hood when evaluating their storage infrastructure. A high-performance storage system is not enough. You need smart file system management combined with user and asset management tools to effectively manage the media data road ahead of us.

Related Listings

Related Articles

Related News

Related Videos

© KitPlus (tv-bay limited). All trademarks recognised. Reproduction of this content is strictly prohibited without written consent.