Building a MultiâTier Storage Plan
Building a Multi‐Tier Storage Plan
Considerations for Designing Multi‐Tier Fwefe
Storage Networks for Video Erevr
Vbev
CONTENTS What Makes Storage For Video Different? ................................................................................................... 2 The Value of a Storage Plan .......................................................................................................................... 3 What is a Multi‐Tier Storage Network? ........................................................................................................ 4 Tier 1 ‐ Online ................................................................................................................................................ 5 Tier 2 – Nearline ............................................................................................................................................ 5 Tier 3 ‐ Archival ............................................................................................................................................. 6 Data Flow in a 3‐Tier Video Storage Environment ........................................................................................ 7 Common Mistakes ........................................................................................................................................ 8 Tips for Purchasing & Implementation ......................................................................................................... 8 Use Your Storage Plan to Drive Revenue ...................................................................................................... 9 About ProMAX Systems ................................................................................................................................ 9
Copyright ©2014 ProMAX Systems Page | 1
WHAT MAKES STORAGE FOR VIDEO DIFFERENT? The process of Video Production places high demands on technology and IT infrastructure. Video storage systems designed to deliver streaming performance are significantly different than what you might find in a traditional IT environment. In most traditional IT environments users run software applications that sporadically request small amounts of data at a time, either from local storage or servers on the network. These traditional network servers are designed to support large numbers of users and 100’s and 1000’s of requests at a time. To support this type of workload their storage systems are configured for a high number of IOPS (Inputs/Outputs per Second) and generally serve low bandwidth to connected users. In contrast, users in video production environments run creative software applications that consistently request large amounts of data at a time. The process of playing back video is called streaming and this requires a high bandwidth, low latency connection to storage in order to playback the video effectively and without dropped frames. The larger the video files, the more bandwidth that’s required to facilitate streaming it. Unlike traditional IT, network video servers are designed to support smaller numbers of users, and fewer requests, but must be able to deliver high bandwidth to each. Their storage systems are configured to generate a large pool of disk bandwidth that is then shared out to the connected users. Depending on the video formats in play a network storage system may have to deliver up to 90MB/s to users over Gigabit Ethernet and up to 500MB/s over 10 Gigabit Ethernet. When developing a storage plan for video production environments it is important to understand the unique requirements of working with video and select systems that will support your workflow both now and in the future.
Copyright ©2014 ProMAX Systems Page | 2
THE VALUE OF A STORAGE PLAN How well protected are you from data loss? How quickly can you restore data in the event of a disaster? How easily can you find and access data? How cost‐effective are your storage systems? Whether you’re a business owner, a workgroup manager or an independent video editor the answers to these questions are important, and they should concern you as your content is your most valuable intellectual property. Storage plans can range from simple to complex and the right one is what’s best for your business and your workflow. In this report we will examine the benefits of a Multi‐Tier Storage Plan and look at some of the considerations for designing your own custom plan. Here are some of the biggest benefits of having an effective multi‐tier storage plan:
Improve ROI and Cost Effectiveness. Increase cost effectiveness by moving media from high performance storage to less expensive systems. High performance storage can then be more efficient and deliver a greater ROI. Increase Uptime & Data Protection. Data distribution and RAID management are key to improving system uptime and overall data protection. Get Faster Access to Media. A Multi‐Tier Storage plan when combined with Digital Asset management can greatly improve your ability to search for and retrieve media regardless of where it is located. Improve Restore & Recovery Times. A well‐constructed storage plan in conjunction with reliable management software can greatly improve restore and recovery times.
Copyright ©2014 ProMAX Systems Page | 3
WHAT IS A MULTI‐TIER STORAGE NETWORK? Multi‐Tier storage is an underlying principle of ILM (Information Lifecycle Management), and is a strategy to increase performance while reducing the overall cost of storage. It is a storage networking methodology whereby data is stored on various types of media based on performance, availability and recovery requirements. For example, data intended for restoration in the event of data loss or corruption could be stored locally ‐‐ for fast recovery ‐‐ while data for regulatory purposes could be archived to lower cost disks. In today's Video Production environments, tiered storage infrastructures range from simple two‐tier architecture consisting of fast local storage with a Nearline disk backup to more complex infrastructures featuring tape libraries and data management systems that in some cases are comprised of five‐to‐six tiers. Regardless of the size or complexity of the workflow more organizations are looking to implement multi‐tier storage networks as a way to lower cost and improve operational efficiency. In this report we will focus on a 3‐Tier Storage Network, a very common architecture for video production environments.
Copyright ©2014 ProMAX Systems Page | 4
TIER 1 ‐ ONLINE Tier 1 is the highest performance storage available to users and usually the most expensive. Also known as “Online” storage, this is the primary active workspace that will be accessed by your creative applications. As an individual or in smaller workgroups this could be the SSD in your workstation or a fast, locally attached storage device such as a SAS or Thunderbolt RAID array. In larger environments this is often a high performance SAN or NAS system. Tier 1 (Online) storage systems most often utilize high performance SAS or SSD drives. Tier 1 storage systems are usually configured with RAID levels that emphasize performance over protection. When planning for and purchasing Tier 1 Storage the most important considerations are:
Speed & Performance – Streaming video requires high bandwidth and low latency. The less compressed your codecs are the higher the bandwidth requirements. Make sure that your online storage can meet both the disk bandwidth and network throughput required for your workflow. Ease of Use – Tier 1 is the storage that your creative users interface with the most and so it needs to be easy to access and easy to use. If this is shared storage, be sure to consider how your team will use the system to collaborate and how easy it is to manage and maintain. Software Application Requirements – When working with video there are many creative applications that have specific requirements for storage systems. Makes sure to check that the software you use will be supported on your Tier 1 storage system.
TIER 2 – NEARLINE Tier 2 or Nearline storage is lower cost, low performance storage commonly used as short‐medium term backup or as temporary holding pen before archive to tape. For an individual workstation where the SSD is Tier 1, the Tier 2 storage device maybe a locally attached USB/FireWire drive or even a cloud storage service such as Amazon S3. In larger facilities Tier 2 storage systems are often dense, rack mounted RAID arrays that utilize lower cost SATA drives. Nearline storage systems utilize RAID levels that emphasize uptime and availability over performance and should ideally have a fast connection to the Tier 1 storage system for moves and copies. When planning for and purchasing Tier 2 Storage the most important considerations are:
Cost – Tier 2 storage is typically less feature rich, lower performance and uses less expensive media, so even though the total storage capacity is often much larger than Tier 1 the average cost per terabyte should be lower. Cost per terabyte (Total Cost of System / Total Storage Capacity TBs) is one of the best ways to compare Tier 2 storage systems. Uptime & Availability – The primary purpose of Tier 2 storage is to provide reliable backup and easy access to offline files that can be quickly pulled back to Tier 1 for online operations. So your Tier 2 storage should include features that ensure a high level of uptime and availability. Data Management – Good Tier 2 Storage systems benefit from a fast connection to Tier 1 storage and a strong software layer that can manage moves, copies and automated backup schedules.
Copyright ©2014 ProMAX Systems Page | 5
TIER 3 ‐ ARCHIVAL Tier 3 Storage in our 3 tier model represents long term archival. It has the lowest cost per terabyte but is the slowest to access if you need to retrieve data. This is where you store projects and files that will not be used on a regular basis and do not need to be readily available, and is often used for storing data for regulatory purposes. Tier 3 storage is extremely durable and the media is designed to last for an unlimited amount of time on the shelf. For individuals with smaller amounts of data to archive this might be optical media such as CDs or DVDs. For most creative groups though LTO (Linear‐Tape‐Open) magnetic tape is the most common Tier 3 media as it has the lowest cost per terabyte and is extremely durable. As LTO technology has advanced over the years the compression and amount of data that can be stored on a single tape has increased. When LTO1 was released in 2000 a single cartridge could hold 100GB of data. In 2012, the release of LTO6 improved this capacity to 2.5TB per cartridge. When planning for and purchasing Tier 3 Storage the most important considerations are:
Which Media to Use – For most people the best choice is LTO tape due to cost and durability. If you do not have a large amount of data LTO4 or LTO5 may be used as they are cheaper than the current standard LTO6. It is also possible to archive to regular SATA hard drives although you should be aware they are not as durable as tape and after long periods of time on the shelf may not spin up when powered back on. Which Format to Use – Different archival devices and software utilize different file formats for writing to tape. When choosing the right system for your workflow it is important to understand the advantages and limitations of these formats. For example, certain customers may require delivery on tape in a specific format, or your production company may require that all projects be archived to tape in a specific format. The most important thing here is to understand the requirements and how it will be used. If the requirements are not clear then it is best to choose a universal format such as TAR or LTFS (Linear‐Tape‐File‐System) as these can be restored from many different devices. Amount of Data – Consider the amount of data you need to archive and the frequency for which you need to archive it. An LTO6 drive can write to tape at approximately 165MB/s. If you are in a large facility or working on a big production you may need multiple LTO drives to be able to write the data fast enough that it can be completed before the next Archive is scheduled. For these types of jobs automated LTO tape libraries are used. These devices can utilize many LTO drives and automate the moves and management of hundreds of tapes. Time to Restore – When building your plan it’s important to remember that Tier 3 Archival is for long term data retention and the process to locate and restore a project or file can take hours or even days if the tape has been stored off site.
Copyright ©2014 ProMAX Systems Page | 6
DATA FLOW IN A 3‐TIER VIDEO STORAGE ENVIRONMENT The following illustration represents possible data flow in a video production company that is utilizing a 3‐Tier storage network. In this example, new footage is ingested directly to Tier 1 storage where the post production team can get fast access to it. Leveraging the high performance Tier 1 storage the creative team is able to stream and edit the video before completing and delivering the project to the customer. Once completed the project and media are moved from Tier 1 to the lower performance, lower cost Tier 2 storage. While changes to the project cannot be made directly on the Tier 2 storage it is accessible and easy enough to copy back to Tier 1 if changes are needed. After a period of time when the customer has signed off on the completed project the content is moved to Tier 3 storage for long‐ term archival. Once archived to LTO tapes the project is no longer easily accessible but can be located and restored in the event that it is needed.
Copyright ©2014 ProMAX Systems Page | 7
COMMON MISTAKES
Under‐Analyzing – It’s important to dedicate time and resources into building a great plan. This can mean sitting down with one or more colleagues and really digging in on the details. Amount of data, size of projects, time to restore, cost, legal requirements, power requirements; there are many factors to consider when building your plan and any systems administrator will tell you, it’s better to take a little more time and get it right the first time. Overestimating the power of RAID protection ‐ Never overestimate RAID protection. Even with multiple RAID controllers and layers of protection disks and controllers can fail and data can be corrupted. Disks and server systems have many hundreds of moving parts that all need to work in harmony for the RAID system to be effective. If one of these fails then you have a problem. Backing data up to other media and locations is extremely important. Overinvesting – When building a 3 Tier storage system it’s sometimes easy to focus on Tier 1 as this is the storage that you interact with the most. It’s the highest performance and the most expensive. But don’t make the mistake of putting all your budget into Tier 1 and neglecting 2 and 3. It’s important to have balance in your plan. Remember, you can save money on expensive Tier 1 storage by managing and moving your data to other Tiers. This is the advantage of a multi‐tier storage network. Assuming it’s all Too Difficult – Don’t make the mistake of thinking a multi‐tier storage system is only for the bigger organizations. The methodology of moving data between different systems based on cost, performance and availability is an effective strategy for creative groups of all sizes and can be executed very inexpensively. Take an individual editor for example: o Tier 1 – Workstation, onboard 2 disk RAID 1. Approx. $400 o Tier 2 – 2 Portable USB drives for backup. Approx. $200 o Tier 3 – External LTO2 tape drive. Approx. $1500 Learning the Hard Way – Hindsight is 20/20 as they say. Don’t make the mistake of not investing in your storage plan or waiting until something bad happens to implement one. Lost data can lead to lost revenue, lost clients and damaged reputations. With the right plan in place you can greatly minimize the risk of data loss and ensure your business is operating efficiently.
TIPS FOR PURCHASING & IMPLEMENTATION While the concept of multi‐tier storage networks is here to stay, the technology used to implement them is changing constantly. When building your storage plan it is advisable to work with a consultant or vendor that specializes in designing video storage workflows and has knowledge of and access to the latest technology. Some media technology companies such as ProMAX Systems can assist with all aspects of your storage plan, from consultation and design, to production and implementation. ProMAX Platform Storage Systems are used in multi‐tier storage networks around the world.
Copyright ©2014 ProMAX Systems Page | 8
USE YOUR STORAGE PLAN TO DRIVE REVENUE It’s easy to look at storage networks, backup and archival systems as a difficult cost of doing business. You know you need them but you hate paying for them. Most recently though more companies are not only looking at their storage systems as investment in the future of their business but as a real asset and value proposition. Here are two great ways you can use your multi‐tier storage network to drive revenue. 1. Offering LTO Archive as a Service. It is becoming more common for Video Production companies, either through partnership or directly, to offer some type of LTO Archive service. This could be as simple as providing a copy of their project on an LTO tape as part of delivery, or charging the customer to maintain an archived copy at the production facility. 2. Make Your Storage Plan a Selling Point. This is one of the easiest ways to leverage your investment in storage technology. Customers love data that’s safe. Knowing that their project and their investment is being kept safe is extremely important and can make a difference when choosing a company to work with. So build your storage plan into your value proposition and promote it as a pillar of strength for your organization.
ABOUT PROMAX SYSTEMS ProMAX develops and manufactures innovative media technology solutions; specializing in shared storage servers, high‐end editing workstations and storage devices for video and media content creators around the world. ProMAX has been providing expertly devised and supported end‐to‐end workflow solutions to the media and entertainment, corporate, education, house of worship and government sectors since 1994. CEO Jess Hartmann acquired ProMAX in 2008, and rapidly expanded the company’s product development and manufacturing operations; breaking into new areas of industry leadership by building powerful systems to meet the performance needs of collaborative media creation and demanding post production processes. For more information, visit www.promax.com, follow us on Twitter, and see more ProMAX happenings on Facebook and YouTube. [email protected] | +1 (800) 977‐6629
Copyright ©2014 ProMAX Systems Page | 9
