Working with Amazon Web Services (AWS) can sometimes be a bit confusing especially when it comes to the Simple Storage Service (S3)
To explain, S3 is an object store which is to say you can place objects into S3 for access where ever you want. Now this in of itself is not confusing but the way you work with this data can be a bit of a mine field, for instance folders in S3 whilst supported, are inadvisable due the permissions hierarchy of the service.
Permissions are set on the object and S3 sees everything within a bucket (The AWS S3 term for storage root) as an object including folders. So when setting permissions you need to set them on everything within the bucket and at all levels e.g.
If you want the
myobject.file to be publicly accessible then you need to ensure that the Access Control List (ACL) setting
public is applied to each element in the bucket, so in this case
So here is the big question, if folders are just objects, are they needed or can the process of object storage be approached differently? for example codifying the object name rather than placing it in a folder structure.
s3://mybucket/myfolder/mysubfolder/myobject.file could become
s3://mybucket/myfolder-mysubfolder-myobject.file and just held in the root of the bucket?
Certainly in working with a client over the last couple of months that have over 420K objects I have come to the conclusion that this is the way that the client should approach object storage.
And so we have planned to move from this storage methodology
Thereby making a codified object name which is held in the root of the client bucket and making the process of management of the objects far simpler.
Code Snips for Permissions
I have had occasion recently to need to manipulate the permissions on an S3 bucket holding a large amount of objects and unfortunately the AWS console, whilst excellent for most tasks doesn’t seem to be quite up to the task of ACL settings in S3.
Therefore I have gone back to the trusty console for help and I am using the AWS CLI (Command Line Interface) with a python library called s3cmd. to install the AWS CLI on your mac use Homebrew and simply run
brew install python this will install python and pip then install the was cli using pip
pip install awscli then lastly install s3Cmd
pip install s3cmd
The last step here is to configure s3Cmd to hold your settings run s3cmd --configure and follow the steps, you will need your AWS Key and Secret to do this, and also be sure to set your default location for me this was EU.
S3Cmd has some fantastic tools available but by far the most useful in my option is the ability to manipulate the ACL for all objects from the command line.
Set public ACL for all objects in bucket:
s3cmd setacl s3://bucket/ --acl-public --recursive
Set private ACL for all objects in bucket:
s3cmd setacl s3://bucket/ --acl-private --recursive
And so on, look at the manpage for s3cmd by running
s3cmd --help and you will be able to see all of the great things that can be done by just using the terminal window.
Everything above can also be used on a Windows or Linux machine if thats your bag ;)