Downloading an S3 bucket

Since I’m curious about how often files are downloaded from S3, I enabled logging on the buckets serving them and directed the logs into a bucket which I created to hold them. Then I wanted to move everything on that logs bucket to my local machine, so I could poke around in the logs and ascertain the best way to turn them into useful information.

Install the CLI

Stackoverflow pointed me toward the awscli Python module. Since I don’t keep a system pip, I installed it to a virtualenv:

$ virtualenv2 v
$ source v/bin/activate
$ pip install awscli

Tell the CLI your credentials

It just needs your Access Key ID and Secret Access Key.

$ aws configure

Download the desired bucket

Decide where you want to save the bucket contents to, then sync it:

$ aws s3 sync s3://bucket-of-logs ~/bucket-logs/

That’s it!

It’s surprisingly easy to download the contents of an entire S3 bucket, considering that the documentation on how to do it is spread out over several different sources and buried in comprehensive reference guides.