Here’s a short recipe of how to transmit files from an external source to an S3 bucket, without downloading the whole source and hence unnecessarily allocating memory:
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| from botocore.vendored import requests | |
| import boto3 | |
| def main(event, context): | |
| s3 = boto3.resource('s3') | |
| bucket = s3.Bucket('mybucket') | |
| destination = bucket.Object('path/to/destination') | |
| url = 'https://foobar.com' | |
| with requests.get(url, stream=True) as response: | |
| destination.upload_fileobj(response.raw) |
It’s taking advantage of request’s stream capability.
Even with files over 2 GB in size, the Lambda container consumed only about 120 MB of memory. Pretty sweet. Of course, this approach is applicable to any platform, not just Lambda.
BaseException 