Here’s a short recipe of how to transmit files from an external source to an S3 bucket, without downloading the whole source and hence unnecessarily allocating memory:
from botocore.vendored import requests | |
import boto3 | |
def main(event, context): | |
s3 = boto3.resource('s3') | |
bucket = s3.Bucket('mybucket') | |
destination = bucket.Object('path/to/destination') | |
url = 'https://foobar.com' | |
with requests.get(url, stream=True) as response: | |
destination.upload_fileobj(response.raw) |
It’s taking advantage of request’s stream capability.
Even with files over 2 GB in size, the Lambda container consumed only about 120 MB of memory. Pretty sweet. Of course, this approach is applicable to any platform, not just Lambda.