Here’s a short recipe of how to transmit files from an external source to an S3 bucket, without downloading the whole source and hence unnecessarily allocating memory:
|from botocore.vendored import requests|
|def main(event, context):|
|s3 = boto3.resource('s3')|
|bucket = s3.Bucket('mybucket')|
|destination = bucket.Object('path/to/destination')|
|url = 'https://foobar.com'|
|with requests.get(url, stream=True) as response:|
It’s taking advantage of request’s stream capability.
Even with files over 2 GB in size, the Lambda container consumed only about 120 MB of memory. Pretty sweet. Of course, this approach is applicable to any platform, not just Lambda.