Cloud Notes: AWS S3

Overview

"Amazon Simple Storage Service (Amazon S3) is an object storage service that offers industry-leading scalability, data availability, security, and performance". At a high level S3 has the following characteristics:

• Object based storage for static files that do not change. EG: JPEG, MOV, etc..
• Objects are stored in "buckets"
• Files from 0 Bytes - 5TB in size
• S3 has a universal namespace across all tenants and must be unique
• S3 buckets and objects have a HTTPS endpoint associated to them
• Successful uploads to an S3 buckets return a HTTP 200 Ok message

Features

S3 has the following features.

• Globally resiliant service
• Tiered storage classes
• Lifecycle management
• Object versioning
• Encryption
• Security Policies

Objects

Objects are just files stored as key / value pairs

{
  "Key": "Name",
  "Value": "FileData",
  "VersionID": "VersionNumber",
  "Subresources": [
    "ACLs",
    "Torrent"
  ]
}

Data Consistency

There are two methods to maintain data consistency.

Read After Write Consistency

• For PUTs of a new object
• Can immediately read object after PUTing it to S3

Eventual Consistency

• For overwrite PUTs and DELETEs
• Can take some time to propagate changes
• May be reading the old file on GETs or still see the old file on DELETEs

Storage Classes

There are 6 storage classes for S3 services. See link for further details on each tier.

S3 - Standard

• The original S3 service
• 99.99% uptime
• 11 x 9s durability

S3 - IA (Infrequently Accessed)

• Infrequent accessed data but requires rapid access when needed

S3 - One Zone - IA

• Lower cost IA in a single availability zone

S3 - Intelligent Tiering

• Uses AI to automatically move data to the most cost effective storage tier

S3 - Glacier

• Low cost storage with configuratable retreival times from minutes to hours

S3 - Glacier Deep Archive

• Lowest cost storage with a minimum of 12 hours retrieval time

Cost

Cost of the service is calculated on a number of factors.

• Storage volume
• Number of requests
• Storage management
• Data transfer

Transfer acceleration

• Uses AWS cloudfront to upload files to an AWS pop close to the user
• The file is then transferred to the S3 bucket across the AWS backbone

Cross region replication

• Users upload a file to a "primary" bucket the file is then replicated to a "secondary" bucket in another AWS region

Transfer acceleration

• Uses AWS cloudfront to upload files to an AWS pop close to the user
• The file is then transferred to the S3 bucket across the AWS backbone

Cross region replication

• Users upload a file to a "primary" bucket the file is then replicated to a "secondary" bucket in another AWS region

Security

S3 buckets can be secured in the following ways.

Bucket Policies

• Applies to the whole bucket

Object Policies

• Applies to individual files

IAM Policies for Users and Groups

• Applies to users and groups