Introduction to

Amazon S3

Ashay Shirwadkar
Agenda
What is Amazon S3?
Storage Classes
Namespace
Security
Server Side encryption
Access Control
S3 APIs
But before that...
What?
- Cloud computing, also on-demand computing, is a kind of
Internet-based computing that provides shared processing
resources.
- Resources being
- Networks
- Servers
- Storage
- Applications and services

Why?
- The term cloud is used as a metaphor for the Internet.
- So it means nothing. Just a nice word and now it's hot....
Types of storage

Client Server
Object

NFS/SMB/rsync
REST APIs
iSCSI/AoE/Fiber Channel
What is amazon S3?
Amazon S3 is acronym stands for Simple Storage Service .

S3 is web store, not a file system, it’s simple write once, read many (WORM) object
store having eventual consistency.
“Write once” means that an object cannot be changed after it is written, and “read many” means that
multiple copies of the object are made across different availability zones.

S3 is secure, durable & highly-scalable. It is accessed Via API’s (SOAP and REST)
Server side encryption
Data is stored with 99.999999999% durability
Stores data ranging from 1B to 5TB

A bedrock architectural component for many applications

Dropbox, Bitcasa, and Tahoe-LAFS-on-S3, among others, use S3 for online backup and synchronization
services.Tumblr, Spotify, and Pinterest host media on S3.
Consistancy
Durability

Load Balancers Load Balancers

Web Servers Web Servers

Indexing Indexing Storage

Storage
Availability Zone Availability Zone
Region
Cloud Storage Classes
Reduced
Standard Redundancy Glacier
Storage
● Designed to provide high ● Designed to provide lesser ● Suitable for archiving data,
durability and high availability redundancy with availability. where data access is infrequent
and retrieval time of several
● Designed to sustain ● Reduces cost by storing data at hours is acceptable.
concurrent loss of data in two lower level of redundancy than
availability zone. in standard storage ● Uses very low cost amazon
● Objects you want to have ● Objects you can afford to lose glacier service, but managed
high durability. or can recreate. through s3.
● E.g. Master copy of a movie ● E.g. Different encoding of movie ● Objects you want to put in
media media. archive ( Rare Use).
● E.g. Digital archive of old movie
media.
Namespaces
The S3 consists of Buckets and Objects. In a single bucket we can have multiple
Objects.

Globally Unique
bucket name + object name (key) => Uniquely identify each object in a S3 cloud. Every object can be
addressed through bucket and key combination.

Buckets are similar to a directories

Object Name has to be unique within the bucket

Max 1024 bytes UTF-8
Can have ‘path’ prefix
Namespaces
Amazon s3

johns-docshare userdocs src

drafts/rpt.doc style.css img/icon.ico swf/flash.swf user/foo.c user/bar.c

Security
S3 provides regional service
Data never leaves region unless you move it

Server Side Encryption

Automatic encryption of data at rest
Strong AES-256
Enabled using simple PUT Header
Self managed i.e no need to manage key store
Server Side Encryption

Encrypted Object
Object Encrypted
Bucket
Data

Encrypted
per-object key

per-object
key

Key Management
(monthly rotated)
Master Key
Access Control
S3 provides Policies, ACL’s and IAM (Identity and Access Management)
Use these to define rules for sharing objects or buckets

IAM Bucket Policies ACL’s

● Fine Grained ● Fine Grained ● Coarse Grained
● Provide Role based ● Apply Policies on bucket from ● Apply access control at object
Access AWS console. or bucket level.
● Apply policies at role, ● Incorporate user restrictions
user and group level. without using IAM

Allow Allow
Allow
Bob, John
Bob, John
Actions:
PutObject Actions:
Actions:
PutObject
Read
Resource:
arn:aws:s3:::mybucket/* Resource:
arn:aws:s3:::mybucket/*

Bob John My bucket My bucket My Object

S3 API
Accessible through SOAP and REST API’s
In S3, The operations can be divided into 3 categories
- Operations on Service/s.
Get list of all buckets owned by the authenticated sender of the request.
- Operations on Bucket/s.
- Operations on Object/s.

User must have Access Key and Secret Access Key

- Provide Temporary access to services
- Keys can be generated through IAM.

String to sign
- Every request has different string to sign.
S3 Authentication - Client side
● Create Request String to Sign Format
String to Sign StringToSign = HTTP-Verb + "\n" +
Request GET\n Content-MD5 + "\n" +
GET /foo/bar.jpg HTTP/1.1 \n Content-Type + "\n" +
Host: johnsmith.s3.amazonaws.com \n Date + "\n" +
Date: Mon, 26 Mar 2007 19:37:58 +0000 Mon, 26 Mar 2007 19:37:58 +0000\n Canonicalized Amz
/johnsmith/foo/bar.jpg Headers +
CanonicalizedResource;

String to Sign
...
HMAC
● Create HMAC-SHA1 calculation
Your
Signature
Signature and
Base64
Encoding
Secret Access
Key

Request
GET /foo/bar.jpg HTTP/1.1
Host: johnsmith.s3.amazonaws.com
● Send Request Date: Mon, 26 Mar 2007 19:37:58 +0000

Authentication: AWS Access Key:Signature

S3 Authentication - Server side
Request
GET /foo/bar.jpg HTTP/1.1
● Retrieve Access Host: johnsmith.s3.amazonaws.com
Get Access Get Secret Secret Access
Date: Mon, 26 Mar 2007 19:37:58 +0000
Key key Access key Key
Authentication: AWS Access Key:Signature

String to Sign
...
HMAC Calculated
● Create HMAC-SHA1 calculation Signature
Signature and
Base64
Encoding
Secret Access
Key

Calculated Your
Signature Signature
● Compare Two Yes: Request is authenticated
Signatures No: Request authentication fails
Operations on Buckets
Standard Operations
Put Bucket - Creates bucket if does not exist.
Get Bucket - List all the objects within the bucket.
Delete Bucket - Deletes the bucket. All the objects within the buckets must be deleted.

Other operations
Bucket lifecycle configuration - Set the lifecycle of objects within the bucket
Bucket policies - Set policies on bucket
Bucket location - Set the location of bucket
Bucket notification - Receive notifications when certain events happen in your bucket
Bucket logging - Enable logging for a bucket
Bucket request Payment - Returns the request payment configuration of a bucket.
Bucket versioning - Enable versioning of objects within the bucket
Operations on Objects
Standard Operations
Put Object - Creates Object.
Post Object- POST is an alternate form of PUT that enables browser-based uploads.
Get Object - Gets object along with its metadata.
Head object - Gets only metadata.
Delete Object - Deletes the Object.

Multipart Upload
Upload a single object as a set of parts.
Each part is a contiguous portion of the object's data.
Upload for objects from 5 MB to 5 TB in size.

Other operations
Object ACL’s - set the ACL permissions for an object that already exists in a bucket.
Object Copy - Creates a copy of an object
Multipart Upload
Initiate Multipart Upload
Initiates a multipart upload and returns an upload ID.
Provide this upload ID in each subsequent upload part requests.

Responce
Request
HTTP/1.1 200 OK
POST /example-object?uploads HTTP/1.1
x-amz-id-2: Uuag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg==
Host: example-bucket.s3.amazonaws.com
x-amz-request-id: 656c76696e6727732072657175657374
Date: Mon, 1 Nov 2010 20:34:56 GMT
Date: Mon, 1 Nov 2010 20:34:56 GMT
Authorization: authorization string
Content-Length: 197
Connection: keep-alive
Server: AmazonS3

<?xml version="1.0" encoding="UTF-8"?>

<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<Bucket>example-bucket</Bucket>
<Key>example-object</Key>
<UploadId>VXBsb2FkIElEIGZvciA2aWWpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9hZA</
UploadId>
</InitiateMultipartUploadResult>
Multipart Upload
Upload Part
Uploads a part in a multipart upload.

Request Responce
PUT /example-object?partNumber=1&
HTTP/1.1 200 OK
uploadId=VXBsb2FkIElEIGZvciA2aWWpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9h
x-amz-id-2:
ZA
Vvag1LuByRx9e6j5Onimru9pO4ZVKnJ2QRPfTaOFg==
HTTP/1.1
x-amz-request-id: 656c76696e6727732072657175657374
Host: example-bucket.s3.amazonaws.com
Date: Mon, 1 Nov 2010 20:34:56 GMT
Date: Mon, 1 Nov 2010 20:34:56 GMT
ETag: "b54357faf0632cce46e942fa68356b38"
Content-Length: 10485760
Content-Length: 0
Content-MD5: pUNXr/BjKK5G2UKvaRRrOA==
Connection: keep-alive
Authorization: authorization string
Server: AmazonS3
***part data omitted***
Multipart Upload
Complete Multipart Upload
Completes a multipart upload by assembling previously uploaded parts.

Request Responce
POST /example-object? HTTP/1.1 200 OK
uploadId=VXBsb2FkIElEIGZvciA2aWWpbmcncyBteS1tb3ZpZS5t x-amz-id-2:
MnRzIHVwbG9hZA HTTP/1.1 Uuag1LuByRx9e6j5Onimru9pO4ZVKnJ2Qz7/C1NPcfTWAtRPfTaOFg==
Host: example-bucket.s3.amazonaws.com x-amz-request-id: 656c76696e6727732072657175657374
Date: Mon, 1 Nov 2010 20:34:56 GMT Date: Mon, 1 Nov 2010 20:34:56 GMT
Content-Length: 391 Connection: close
Authorization: authorization string Server: AmazonS3

<CompleteMultipartUpload> <?xml version="1.0" encoding="UTF-8"?>

<Part> <CompleteMultipartUploadResult xmlns="
<PartNumber>1</PartNumber> http://s3.amazonaws.com/doc/2006-03-01/">
<ETag>"b54357faf0632cce46e942fa68356b38"</ETag>
</Part> <Location>http://Example-Bucket.s3.amazonaws.com/Example-Object</Location>
<Part> <Bucket>Example-Bucket</Bucket>
… <Key>Example-Object</Key>
</Part> <ETag>"3858f62230ac3c915f300c664312c11f-9"</ETag>
</CompleteMultipartUpload> </CompleteMultipartUploadResult>
Multipart Upload
Abort Multipart Upload
Request
DELETE /example-object?uploadId=VXBsb2FkIElEIGZvciBlbHZpbmcncyBteS1tb3ZpZS5tMnRzIHVwbG9hZ HTTP/1.1
Host: example-bucket.s3.amazonaws.com
Date: Mon, 1 Nov 2010 20:34:56 GMT
Authorization: authorization string

Get Object

Request
GET /example-object HTTP/1.1
Host: example-bucket.s3.amazonaws.com
Date: Mon, 1 Nov 2010 20:34:56 GMT
Authorization: authorization string
API Operations-
API Operations contd.
DEMO
Create Bucket
#!/bin/bash -x
bucket=$1
Request < HTTP/1.1 200 OK
< x-amz-id-2: Responce
if [ -z "$1" ] jWt9BVmZkL1eU/i1gRoUrsB19/RHYwHGJZdst5ttGlLx7IvFPzHDSSNFluRyDRrCewG4xoFi
oJA=
then
< x-amz-request-id: 9FC84EB054B018F9
echo "usage: ./bucket_put <bucket_name>" < Date: Wed, 12 Aug 2015 11:52:02 GMT
exit 1 < Location: /casoft
fi < Content-Length: 0
resource="/${bucket}/" * Server AmazonS3 is not blacklisted
dateValue=`date -R` < Server: AmazonS3
stringToSign="PUT\n\n\n${dateValue}\n${resource}" <
s3Key=’Your Access Key’ * Connection #0 to host calsoft.s3.amazonaws.com left intact
s3Secret=’Your Secret Access Key’
signature=`echo -en ${stringToSign} | openssl sha1 -hmac $
{s3Secret} -binary | base64`
curl -v -X PUT \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
http://${bucket}.s3.amazonaws.com/
List Bucket
#!/bin/bash -x Request < HTTP/1.1 200 OK Responce
bucket=$1 < x-amz-id-2:
if [ -z "$1" ] nBbw0yclRZ4jPzPEECEKI0oCRMQrdIihEXbCVuUvYdXl75CGYH3/IcsPu/jxkCJb
< x-amz-request-id: 90A0E3312B09453A
then
< Date: Wed, 12 Aug 2015 11:55:48 GMT
echo "usage: ./bucket_list <bucket_name>" < x-amz-bucket-region: us-east-1
exit 1 < Content-Type: application/xml
fi < Transfer-Encoding: chunked
resource="/${bucket}/" * Server AmazonS3 is not blacklisted
dateValue=`date -R` < Server: AmazonS3
s3Key=’Your Access Key’ <
s3Secret=’Your Secret Access Key’ <?xml version="1.0" encoding="UTF-8"?>
* Connection #0 to host calsoft.s3.amazonaws.com left intact
signature=`echo -en ${stringToSign} | openssl sha1 -hmac $
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-
{s3Secret} -binary | base64` 01/"><Name>calsoft</Name><Prefix></Prefix><Marker></
curl -v -X GET \ Marker><MaxKeys>1000</MaxKeys><IsTruncated>false</IsTruncated></
-H "Host: ${bucket}.s3.amazonaws.com" \ ListBucketResult>
-H "Date: ${dateValue}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
http://${bucket}.s3.amazonaws.com/
Delete Bucket
#!/bin/bash -x
bucket=$1
Request < HTTP/1.1 204 No Content
< x-amz-id-2: Responce
if [ -z "$1" ] 4uuTflJqeUnewAYGmgghfiaBf/yfdja3DE7GmC9+e0QmBmE9T+2c/Ylt19jcndrM
< x-amz-request-id: 96AEC8E1A534EC3E
then
< Date: Wed, 12 Aug 2015 12:06:44 GMT
echo "usage: ./bucket_delete <bucket_name>" * Server AmazonS3 is not blacklisted
exit 1 < Server: AmazonS3
fi <
resource="/${bucket}/" * Connection #0 to host calsoft.s3.amazonaws.com left intact
dateValue=`date -R`
s3Key=’Your Access Key’
s3Secret=’Your Secret Access Key’
signature=`echo -en ${stringToSign} | openssl sha1 -hmac $
{s3Secret} -binary | base64`
curl -v -X DELETE \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/
Create Object
#!/bin/bash -x
bucket=$1 Request < HTTP/1.1 100 Continue
Responce
* We are completely uploaded and fine
file=$2 < HTTP/1.1 200 OK
if [[ (-z "$1") || (-z "$2") ]]
then
< x-amz-id-2:
echo "usage: ./object_upload <bucket_name> <object_name>" OzR2U7CgsWwtHVbF8qcTiIpezFk5FVt9PxoFq9Px2QP8y7L0kOR2gQysfu9/EFNjUBdqIrzs
exit 1 E2o=
fi < x-amz-request-id: 301904328CA5B6FF
resource="/${bucket}/${file}" < Date: Wed, 12 Aug 2015 12:08:01 GMT
contentType="application/text" < ETag: "78d5333e735ae15f5f19f2e76838b728"
dateValue=`date -R` < Content-Length: 0
stringToSign="PUT\n\n${contentType}\n${dateValue}\n${resource}"
s3Key=’Your Access Key’
* Server AmazonS3 is not blacklisted
s3Secret=’Your Secret Access Key’ < Server: AmazonS3
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -binary | <
base64` * Connection #0 to host calsoft.s3.amazonaws.com left intact
curl -v -X PUT -T "${file}" \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Content-Type: ${contentType}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${file}
List Object
#!/bin/bash -x
bucket=$1 Request < HTTP/1.1 200 OK
< x-amz-id-2: Responce
file=$2 7yX1htWSKHVn+ssy32emeQoaF8WgFjRQuEio4PNzSyxjoJiPnmjcsmJvbJFZTTDx
if [[ (-z "$1") || (-z "$2") ]] < x-amz-request-id: 1786B98B373C737E
then < Date: Wed, 12 Aug 2015 12:11:42 GMT
echo "usage: ./object_download <bucket_name> < Last-Modified: Wed, 12 Aug 2015 12:08:01 GMT
<object_name>" < ETag: "78d5333e735ae15f5f19f2e76838b728"
exit 1 < Accept-Ranges: bytes
fi < Content-Type: application/text
resource="/${bucket}/${file}" < Content-Length: 12
dateValue=`date -R` * Server AmazonS3 is not blacklisted
stringToSign="GET\n\n\n${dateValue}\n${resource}" < Server: AmazonS3
s3Key=’Your Access Key’ <
s3Secret=’Your Secret Access Key’ Hello,World
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} - * Connection #0 to host calsoft.s3.amazonaws.com left intact
binary | base64`
curl -v -X GET \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${file}
Delete Object
#!/bin/bash -x
bucket=$1 Request < HTTP/1.1 204 No Content
< x-amz-id-2: Responce
file=$2 swMz88s6IV8i3dCwP6fSuklrubABX0O7XV1jBt7fUZtCP2x86IPozq+5Usy5wE7x
if [[ (-z "$1") || (-z "$2") ]] < x-amz-request-id: 247E152DAB64A000
then < Date: Wed, 12 Aug 2015 12:14:58 GMT
echo "usage: ./object_delete <bucket_name> * Server AmazonS3 is not blacklisted
<object_name>" < Server: AmazonS3
exit 1 <
fi * Connection #0 to host calsoft.s3.amazonaws.com left intact
resource="/${bucket}/${file}"
dateValue=`date -R`
stringToSign="DELETE\n\n\n${dateValue}\n${resource}"
s3Key=’Your Access Key’
s3Secret=’Your Secret Access Key’
signature=`echo -en ${stringToSign} | openssl sha1 -hmac ${s3Secret} -
binary | base64`
curl -v -X DELETE \
-H "Host: ${bucket}.s3.amazonaws.com" \
-H "Date: ${dateValue}" \
-H "Authorization: AWS ${s3Key}:${signature}" \
https://${bucket}.s3.amazonaws.com/${file}
Q&A