Storage Solutions in the AWS Cloud
Storage Solutions in the AWS Cloud Miles Ward – Enterprise Solutions Architect
Traditional Storage On-Premise Storage Options
SAN – network-attached block devices: LUNs DAS – local block devices (disks) NAS – network attached file systems, filers Differ in, Performance, Durability, and Cost Often used to form a Storage Hierarchy
Cloud Storage AWS offers multiple cloud-based storage options. • Amazon S3 (Simple Storage Service)
• Amazon EBS (Elastic Block Storage) • Amazon EC2 Local Instance Store (Ephemeral volumes) Each has a unique combination of performance, durability, cost, and interface Often used to form a Storage Hierarchy
Other Storage options? Amazon S3, EBS, and EC2 Instance Store volumes are the basic storage building blocks, but for more specialized uses, or managed resources, we have other options as well:
Cloudfront Amazon RDS SimpleDB SQS Relational Databases on EC2
Storage by the Numbers S3
EBS
Local
Scale
Unlimited Storage 5TB objects 100 name spaces (buckets)
1TB per volume ~44TB per server
147 – 1690GB 1-5 volumes
Durability
99.999999999% (standard) 99.99% (RRS)
.1-.5% AFR
Ephemeral design requires applicationlevel redundancy
Latency
High
Low
Very Low
Management
Zero Maintenance
Snapshots, RAID, Consistency
Manual Redundancy
Locality
Region-Centric
AZ-Specific
In-machine
Interface
REST or SOAP Web Services
Block Device
Block Device
S3 (Simple Storage Service) Pay for exactly what you use Highly Durable Limitlessly scalable
Natively On-Line
S3 Best Practices Replicate Metadata Parallel IO for max speed Multipart Upload
Sorted Writes Resource-level IAM permissions
S3 Use Cases and Users Media Serving Media Sharing File Storage / Backup
Static Content Big Data
Amazon EBS Pay for what you provision POSIX compliant network attached block device Native Redundancy and Write Cache; optimized for random writes and reads.
EBS Best Practices Pre-warming Snapshots Striping (RAID 0)
Block-device-mapping
EBS Use Cases and Users Application hosting
Envoy Media Group Case Stud
Storage for multimedia content delivery on Amazon W
Data Processing
Introd
Establi that of media
Database
Large Data
and po channe implem
Envoy moved their IT infrastructure from a traditional co-location advantage of -as-youto share, and promptly delivered, Envoy selected Gluster in conce
Challenges
Envoy currently hosts their entire web presence on Amazon Web which has helped them meet the needs of their growing business base. One of their biggest challenges was ensuring the IT infrastru to respond to the highly dynamic nature of demand such as when launch and then grow exponentially. From a data storage standpo needed a solution that integrated easily with AWS and provided h Storing the data locally on EC2 instances is not suffcient since that persistent and would be lost if a server reboots. Amazon EBS volu to one instance at a time a problem if your business requires hu needed to ensure their data was easily shared.
EC2 Instance Store or Ephemeral Volumes Free with your EC2 Instance Zero Network Overhead; local, direct attached resource. High performance for sequential reads and writes Temporary; goes away when your instance does
Instance Store Best Practices App-level replication Striping (RAID 0, 1) Block-device-mapping
Instance Store Use Cases and Users Stateless Web Hosts Transcoding Caching
HPC DNFS
Better Together: Cross-system patterns EBS S3: Snapshots
S3 SimpleDB: meta-data store
S3 Local Storage: Caching
Short Detour: S3 vs EBS AMIs Two types of EC2 AMIs and Instances: Instance store (“S3-backed”) Original EC2, boot from ephemeral storage Can Start and Terminate only All data is ephemeral (unless separate EBS volume attached)
EBS root (“boot from EBS”) Newer, boot from EBS volume Can Start, Stop, Create Image, and Terminate Ephemeral storage still available, but not exposed by default Use instance Block Device Mapping (command line and API only)
Additional EBS volumes can be attached to either type
So, which should I choose? Might be all of them! S3: Static Online Content, Media Serving, Backups, Big Data, Centralized Resources EBS: Applications that require POSIX, Databases, Data Warehouses Ephemeral: Cache/Swap/Temp, Applications that require minimum latency
Test! It costs pennies to evaluate these technologies against your use-case, and making the right choice could save you millions.
How do I get it to the Cloud? AWS Import/Export Faster than Internet transfer and more cost effective than upgrading your connectivity Use cases: data migration, offsite backup, direct data interchange, disaster recovery
WAN Optimization Asperatech, Riverbed, Tsunami-UDP, etc
Read the originals! Storage Options in the AWS Cloud: https://media.amazonwebservices.com/AWS_Storage_O ptions.pdf Storage Use Cases in the AWS Cloud: https://media.amazonwebservices.com/AWS_Storage_U se_Cases.pdf
Thanks!
http://aws.amazon.com
Traditional Storage On-Premise Storage Options
SAN – network-attached block devices: LUNs DAS – local block devices (disks) NAS – network attached file systems, filers Differ in, Performance, Durability, and Cost Often used to form a Storage Hierarchy
Cloud Storage AWS offers multiple cloud-based storage options. • Amazon S3 (Simple Storage Service)
• Amazon EBS (Elastic Block Storage) • Amazon EC2 Local Instance Store (Ephemeral volumes) Each has a unique combination of performance, durability, cost, and interface Often used to form a Storage Hierarchy
Other Storage options? Amazon S3, EBS, and EC2 Instance Store volumes are the basic storage building blocks, but for more specialized uses, or managed resources, we have other options as well:
Cloudfront Amazon RDS SimpleDB SQS Relational Databases on EC2
Storage by the Numbers S3
EBS
Local
Scale
Unlimited Storage 5TB objects 100 name spaces (buckets)
1TB per volume ~44TB per server
147 – 1690GB 1-5 volumes
Durability
99.999999999% (standard) 99.99% (RRS)
.1-.5% AFR
Ephemeral design requires applicationlevel redundancy
Latency
High
Low
Very Low
Management
Zero Maintenance
Snapshots, RAID, Consistency
Manual Redundancy
Locality
Region-Centric
AZ-Specific
In-machine
Interface
REST or SOAP Web Services
Block Device
Block Device
S3 (Simple Storage Service) Pay for exactly what you use Highly Durable Limitlessly scalable
Natively On-Line
S3 Best Practices Replicate Metadata Parallel IO for max speed Multipart Upload
Sorted Writes Resource-level IAM permissions
S3 Use Cases and Users Media Serving Media Sharing File Storage / Backup
Static Content Big Data
Amazon EBS Pay for what you provision POSIX compliant network attached block device Native Redundancy and Write Cache; optimized for random writes and reads.
EBS Best Practices Pre-warming Snapshots Striping (RAID 0)
Block-device-mapping
EBS Use Cases and Users Application hosting
Envoy Media Group Case Stud
Storage for multimedia content delivery on Amazon W
Data Processing
Introd
Establi that of media
Database
Large Data
and po channe implem
Envoy moved their IT infrastructure from a traditional co-location advantage of -as-youto share, and promptly delivered, Envoy selected Gluster in conce
Challenges
Envoy currently hosts their entire web presence on Amazon Web which has helped them meet the needs of their growing business base. One of their biggest challenges was ensuring the IT infrastru to respond to the highly dynamic nature of demand such as when launch and then grow exponentially. From a data storage standpo needed a solution that integrated easily with AWS and provided h Storing the data locally on EC2 instances is not suffcient since that persistent and would be lost if a server reboots. Amazon EBS volu to one instance at a time a problem if your business requires hu needed to ensure their data was easily shared.
EC2 Instance Store or Ephemeral Volumes Free with your EC2 Instance Zero Network Overhead; local, direct attached resource. High performance for sequential reads and writes Temporary; goes away when your instance does
Instance Store Best Practices App-level replication Striping (RAID 0, 1) Block-device-mapping
Instance Store Use Cases and Users Stateless Web Hosts Transcoding Caching
HPC DNFS
Better Together: Cross-system patterns EBS S3: Snapshots
S3 SimpleDB: meta-data store
S3 Local Storage: Caching
Short Detour: S3 vs EBS AMIs Two types of EC2 AMIs and Instances: Instance store (“S3-backed”) Original EC2, boot from ephemeral storage Can Start and Terminate only All data is ephemeral (unless separate EBS volume attached)
EBS root (“boot from EBS”) Newer, boot from EBS volume Can Start, Stop, Create Image, and Terminate Ephemeral storage still available, but not exposed by default Use instance Block Device Mapping (command line and API only)
Additional EBS volumes can be attached to either type
So, which should I choose? Might be all of them! S3: Static Online Content, Media Serving, Backups, Big Data, Centralized Resources EBS: Applications that require POSIX, Databases, Data Warehouses Ephemeral: Cache/Swap/Temp, Applications that require minimum latency
Test! It costs pennies to evaluate these technologies against your use-case, and making the right choice could save you millions.
How do I get it to the Cloud? AWS Import/Export Faster than Internet transfer and more cost effective than upgrading your connectivity Use cases: data migration, offsite backup, direct data interchange, disaster recovery
WAN Optimization Asperatech, Riverbed, Tsunami-UDP, etc
Read the originals! Storage Options in the AWS Cloud: https://media.amazonwebservices.com/AWS_Storage_O ptions.pdf Storage Use Cases in the AWS Cloud: https://media.amazonwebservices.com/AWS_Storage_U se_Cases.pdf
Thanks!
http://aws.amazon.com
