Here’s a short recipe of how to transmit files from an external source to an S3 bucket, without downloading the whole source and hence unnecessarily allocating memory:
| from botocore.vendored import requests | |
| import boto3 | |
| def main(event, context): | |
| s3 = boto3.resource('s3') | |
| bucket = s3.Bucket('mybucket') | |
| destination = bucket.Object('path/to/destination') | |
| url = 'https://foobar.com' | |
| with requests.get(url, stream=True) as response: | |
| destination.upload_fileobj(response.raw) |
It’s taking advantage of request’s stream capability.
Even with files over 2 GB in size, the Lambda container consumed only about 120 MB of memory. Pretty sweet. Of course, this approach is applicable to any platform, not just Lambda.
