Streaming upload to S3

Here’s a short recipe of how to transmit files from an external source to an S3 bucket, without downloading the whole source and hence unnecessarily allocating memory:

from botocore.vendored import requests
import boto3
def main(event, context):
s3 = boto3.resource('s3')
bucket = s3.Bucket('mybucket')
destination = bucket.Object('path/to/destination')
url = 'https://foobar.com'
with requests.get(url, stream=True) as response:
destination.upload_fileobj(response.raw)

It’s taking advantage of request’s stream capability.

Even with files over 2 GB in size, the Lambda container consumed only about 120 MB of memory. Pretty sweet. Of course, this approach is applicable to any platform, not just Lambda.