S3 Bucket – EC2 Directory Sync using AWS Lambda

Allow only AWS CLI commands for the login user: $visudo ubuntu ALL=(ALL:ALL) /usr/local/bin/aws   Write the lambda function Prepare the deployment package $mkdir /python-packages $virtualenv -p /usr/bin/python /python-packages $source /python-packages/bin/activate $pip install pycrypto $pip install paramiko Deactivate Virtualenv $mkdir /deployment-package $copy the GITHUB URL file here $cp -r /python-packages/lib/python2.7/site-packages/* deployment-package/ $cp -rf /python-packages/lib/python2.7/site-packages/.libs_cffi_backend deployment-package/ $chmod […]

Full Access to a specific S3 Bucket except DeleteObject

{ “Statement”: [ { “Effect”: “Allow”, “Action”: [ “s3:GetBucketLocation”, “s3:ListAllMyBuckets” ], “Resource”: “arn:aws:s3:::*” }, { “Effect”: “Allow”, “Action”: [ “s3:ListBucket” ], “Resource”: [ “arn:aws:s3:::testbucket-unni” ] }, { “Effect”: “Allow”, “Action”: [ “s3:PutObject”, “s3:GetObject” ], “Resource”: [ “arn:aws:s3:::testbucket-unni/*” ] } ] } Details: Get and List actions given to “arn:aws:s3:::*” to enable console view List action […]

IAM Users Only for Bucket Access

Ideally we have to IAM Roles if the access credentials is used by an App hosted in EC2, else the following can be setup : Create an IAM Bucket say unni-test Create an IAM User with the same name as bucket say – unni-test Now we can use IAM Variables (here aws:username) to create just […]

s3cmd Elaborated…

Use –rr option (reduced redundancy) for every put and sync commands !!!.  Use –bucket-location option to mention nearest geographical location to avoid latency. To view contents inside a bucket #s3cmd ls s3://bucketname To copy/sync a directory into a bucket #s3cmd sync Desktop/check s3://bucket_name To view all contents of all buckets one level down (only non […]

Copy S3 Buckets across AWS accounts

 This procedure will mess up the metadata of files and will not be able to set permissions from the new location.     Amazon S3 bucket names are UNIQUE accross all AWS Accounts. For example, suppose your first account username is acc1@gmail.com and second is acc2@gmail.com. #s3cmd –configure : Configure for acc1 aws account and […]

Install & Use s3cmd for S3 Storage

Amazon S3 is a reasonably priced data storage service. Ideal for off-site backups, archiving and other data storage needs. It is generally more reliable than your regular web hosting for storing your files and images. Check out About Amazon S3 section to find out more. S3cmd is a command line tool for uploading, retrieving and […]