AWS Module 7 - Storage

Lien de la note Hackmd

Section 1: Amazon Elastic Block Store (Amazon EBS)

Storage

• Provides persistent block storage volumes with Amazon EC2 instances
• Called non-volatile storage
• Replicated within AZ

AWS Storage options: block storage vs object storage

What if you want to change one character in a 1-GB file ?

Amazon EBS

Amazon EBS enables you to create individual storage volumes and attach them to an Amazon EC2 instance

• Amazon EBS offers block-level storage
• Volumes are automatically replicated within its AZ
• Can be backed up automatically to Amazon S3 through snapshots
• Uses include
  • Boot volumes and storage for Amazon Elastic Compute Cloud (Amazon EC2) instance
  • Data storage with a file system
  • Database hosts
  • Enterprise app

Amazon EBS volume types

Amazon EBS

• Snapshots
  • Point-in-time snapshots
  • Recreate a new volume at any time
• Encryption
  • Encrypted Amazon EBS volumes
  • No additional cost
• Elasticity
  • Increase capacity
  • Change to different types

Volumes, IOPS and pricing

1. Volumes
   • Amazon EBS volumes persist independently from the instance
   • All volume types are charged by the that is provisioned per month
2. IOPS
   • General Purpose SSD
     • Charged by the amount that you provision in GB per month until storage is released
   • Magnetic
     • Charged by the number of requests to the volume
   • Provisioned IOPS SSD
     • Charged by the amount that you provision in IOPS (multiplied by the percentage of days that you provision for the month
3. Snapshots
   • Added cost of Amazon EBS snapshots to Amazon S3 is per GB-month of data stored
4. Data transfer
   • Inbound data transfer is free
   • Outbound data transfer accross Regions incurs charges

Section 2: Amazon Simple Storage Service (Amazon S3)

Storage

Amazon S3 is object-level storage.

• If want to change part of a file, must do the change and repload the entier file

Amazon S3 overview

• Data stored as objects in buckets
• Virtually unlimited storage
  • Single object is limited to 5 TB
• Designed for 11 9s of durability
• Granular access to bucket and objects
• Data private per default
• Can set up notification
  • When object is added
  • When object is deleted

Amazon S3 stroage classes

Amazon S3 offers a range of object-level storage classes that are designed for different use cases

• Amazon S3 standard
  • High availability
  • High durability
  • Perfomance
  • Frequently access data
• Amazon S3 Intelligent-Tiering
  • Optimize cost
  • Moving data to the most cost-effective access tier
  • long-live data with unpredictable access pattern
• Amazon S3 Standard-Infrequent Access (Amazon S3 Standard-IA)
  • Data accessed less frequently
  • long-term storage
• Amazon S3 One Zone-Infrequent Access (Amazon S3 One Zonw-IA)
  • Data accessed less frequently
  • Stores data in a single availbility zone
• Amazon S3 Glacier
  • Secure
  • Durable
  • low cost
  • data archiving
  • three retrieval options
    • min to hours
• Amazon S3 Glacier Deep Archive
  • Lowest cost
  • long-term detention
  • retrieved once or twice a year

Amazon S3 bucket URLS (two styles)

To upload your data:

1. Create a bucket in an AWS Region
2. Upload almost any number of objects to the bucket

Bucket path-style URL endpoint: https://s3.ap-northeast-1.amazonaws.com/bucket-name

Bucket virtual-hosted-style URL endpoint https://bucket-name.s3-ap-northeast-1.amazonaws.com

Data is redundantly stored in the Region

Prevent data loss

Designed for seamless scaling

Amazon S3:

• automatically manage the storage
• scales to handle high volume of request
• billed for what you use

Access the data anywhere

• AWS CLI
• AWS Management Console
• SDK

Bucket names must be globally unique and DNS compliant: all lowercase, only letters, numbers and dashes

Amazon S3 common scenarios

• Backup and storage
• Application hosting
• Media hosting
• Software

Amazon S3 pricing

• Pay for what you use
  • GBs per month
  • Transfer OUT to other Regions
  • PUT, COPY, POST, LIST and GET requests
• You do not pay for
  • Transfers IN to Amazon S3
  • Transfers OUT from Amazon S3 to Amazon CloudFront or Amazon EC2 in the same region

Amazon S3: Storage pricing

To estimate Amazon S3 costs:

1. Types of storage classes
   • Standard storage is for
     • 11 9s of durability
     • 4 9s of availability
   • S3 Standard-Infrequent Access (S-IA) is for
     • 11 9s of durability
     • 3 9s of availaibility
2. Amount of storage
   • The number and size of objects
3. Requests
   • Number of requests (GET, PUT, COPY)
   • Type of requests
     • Different rates for GET requests
4. Data transfer
   • Pricing based on amount of data transferred ou of Amazon S3 Region
     • Data transfer in is free, but incur charges for data transferred out

Section 3: Amazon Elastic File System (Amazon EFS)

Storage

Implements storage for EC2 instances

Features

• File storage in the AWS Cloud
• Works well for big data and analystics, media processing workflows, content management, web serving and home directories
• Petabyte-scale, low-latency file system
• Shared storage
• Elastic capacity
  • Gigabytes to petabytes of data
• Supports Network File System (NFS) versions 4.0 and 4.1 (NFSv4)
• Compatible with all Linux-based AMIs for Amazon EC2
• Pay for what you use

Amazon EFS architecture

Amazon EFS implementation

1. create your Amazon EC2 resources and launch your instance
2. Create your Amazon EFSfile system
3. Create your mount targets in the appropriate subnets
4. Connect your Amazon EC2 instances to the mount targets
5. Verify the resources and protection of your AWS account

Amazon EFS resources

• Mount target
  • Subnet ID
  • Security gorups
  • One or more per file system
  • Create in a VPC subnet
  • One per AZ
  • Must be in the same VPC
• Tags
  • Key0value pairs

Section 4: Amazon S3 Glacier

Storage

Secure, durable and extremely low-cost data archiving.

• Archive
  • Any object such as photo, video, file or document stored in Amazon S3 Glacier
  • Bas unit of storage
  • unique ID
• Vault
  • Container for storing archive
  • Specifies vault name
  • Premissions access policy
  • Vault lock policy

Amazon S3 Glacier review

• Designed to provide 11 9s of durability for objects
• Supports encryption of data in transit/at rest through Secure Sockets Layr (SSL) or Transpor Layer Security (TLS)
• Vault lock: enforces compliance through a policy
• Extremely low-cost for long-term archiving
  • Three options: expedited, standard or bulk
  • Retrieval times from a few minutes to hours

Amazon S3 Glacier

• Storage service for low-cost data archiving and long-term backup
• Configure lifecycle archiving Amazon S3 content to Amazon S3 Glacier
• Retrieval options
  • Standard: 3-5 hours
  • Bulk: 5-12 hours
  • Expedited: 1-5 min

Amazon S3 Glacier use cases

• Media asset archiving
• Healthcare info archiving
• Regulatory and compliance archiving
• Sicentific data archiving
• Digital preservation
• Magnetic tape replacement

Using Amazon S3 Glacier

• RESTful web services
• Java or .NET SDKs
• Amazon S3 with lifecycle policies

Lifecycle policies

Amazon S3 lifecycle policies enable you to delete or move objects based on age.

Amazon S3 storage classes

Storage comparison

Server-side encryption

Server-side encryption:

• SSE S3
  • each objects has unique key
  • AES 256
• SSE-C
  • Own encryption keys
• AWS Key Management Service
  • Scaled for the cloud
  • Customer master keys
  • IAM Console or API
    • Access keys
    • How keys can be used

Security with Amazon S3 Glacier

• Controle access with IAM
• Amazon S3 Glacier encrypts your data with AES-256
• Amazon S3 Glacier manages your keys for you

Wrap-up

A company wants to store data that is not frequently accessed. What is the best and cost-effective solution that should be considered ?

1. Amazon S3 Storage Gateway
2. Amazon S3 Glacier
3. Amazon EBS
4. Amazon S32

Answer

keyword:

• not frequently accessed
• cost-effective solution

Answer: 2.