Amazon S3 and AWS Lambda are two powerful services in the AWS ecosystem, and integrating them can lead to the creation of powerful serverless applications. By using Amazon S3 events to trigger Lambda functions, you can perform real-time processing on data as it enters your S3 buckets.
Here's a brief overview of how the integration works and some potential use cases:
Integration Overview:
Setting Up an S3 Bucket: Before integrating, you should have an S3 bucket in place. Amazon S3 is a scalable object storage service, where you can store any amount of data in bucket containers.
Amazon S3 Events: You can configure your S3 bucket to send events when certain actions occur. This includes object creation, deletion, and more.
Lambda Function: AWS Lambda lets you run code without provisioning or managing servers. You write a function in one of the supported languages (e.g., Python, Node.js), and Lambda executes the function in response to events.
S3 Trigger for Lambda: In the AWS Lambda configuration, you can set up a trigger that points to your S3 bucket. Once set, specific events in the S3 bucket (like object creation or deletion) will trigger the Lambda function.
Execution: When the defined event occurs in S3, it captures the event details and sends it as a payload to the connected Lambda function. The Lambda function then processes this event in real-time.
Use Cases:
Image Processing: When a new image is uploaded to an S3 bucket, a Lambda function can be triggered to process the image. This could include resizing, applying filters, or extracting metadata.
Data Transformation: When new data files (CSV, JSON, etc.) are uploaded to S3, trigger a Lambda to transform the data format, cleanse it, or enrich it.
Real-time Analytics: Analyze incoming data immediately upon upload. For example, analyzing log files as they're generated and sent to S3.
Backup and Archiving: Automatically copy new objects to another S3 bucket or send them to AWS Glacier for long-term archiving.
Notification System: Send notifications, for instance, using Amazon SNS, when a new object is added or deleted from the bucket. This could be used for monitoring or alerting purposes.
Machine Learning: Trigger ML models to make predictions based on new data that's uploaded to S3. For instance, a Lambda function might call Amazon SageMaker with new data for predictions.
Content Validation: Check incoming content for compliance or standards. If a file doesn't meet certain criteria, it could be flagged or removed.
Database Updates: When new data is added to S3, trigger a Lambda function to update a database, for instance, Amazon DynamoDB or RDS.
Video Transcoding: Convert video files to different formats or resolutions using AWS Lambda and services like Amazon Elastic Transcoder as soon as they're uploaded to S3.
Audit and Logging: Track actions performed on the objects in the S3 bucket and store logs or send them to monitoring services.
To fully leverage the potential of S3 and Lambda integration, developers often combine other AWS services (e.g., Amazon DynamoDB, Amazon SNS, or AWS Step Functions) to create comprehensive serverless workflows.
Tutorial: Image Resizing with Lambda using Two S3 Buckets
Below is a step-by-step tutorial to resize images using Amazon S3 and AWS Lambda:
Code files are available at https://github.com/Brain2life/aws-cookbook/tree/main/s3-trigger-lambda
Prerequisites:
An AWS account.
AWS CLI installed and configured with necessary permissions or you can use the AWS Management Console.
Steps:
1. Create two S3 Buckets:
Navigate to the Amazon S3 service in the AWS Management Console.
Click on 'Create bucket', provide a unique name for your bucket, and proceed with the default settings. Click 'Create'.
Create a second S3 bucket for storing resized images
2. Create an IAM Role for Lambda:
Navigate to the IAM service.
Go to 'Roles' and click 'Create role'.
Choose 'Lambda' as the service that will use this role and click 'Next: Permissions'.
Attach the
AWSLambdaExecute
policy (it provides permissions for Lambda to run and for S3 object read/write actions).Review and create the role by providing a name.
3. Create a Lambda Function:
Navigate to the AWS Lambda service.
Click 'Create function'.
Choose 'Author from scratch'. Provide a function name.
For the runtime, select a suitable version (e.g., Python 3.8).
Under 'Execution role', select 'Use an existing role' and choose the IAM role you created in step 2.
Click 'Create function'.
4. Add the Image Resizing Code:
In the Lambda function dashboard, scroll to the 'Function code' section.
Use the following sample Python code which uses the
PIL
(Pillow) library:
import boto3
from PIL import Image
import io
s3_client = boto3.client('s3')
def resize_image(image_content):
with Image.open(io.BytesIO(image_content)) as image:
image.thumbnail((128, 128))
buffer = io.BytesIO()
image.save(buffer, 'JPEG')
buffer.seek(0)
return buffer
def lambda_handler(event, context):
bucket_name = event['Records'][0]['s3']['bucket']['name']
file_key = event['Records'][0]['s3']['object']['key']
# Fetch the image from the original-images-bucket
response = s3_client.get_object(Bucket=bucket_name, Key=file_key)
image_content = response['Body'].read()
# Resize the image
resized_image_buffer = resize_image(image_content)
# Save the resized image to the resized-images-bucket
output_key = 'resized_' + file_key
s3_client.put_object(Bucket='resized-images-566571321054', Key=output_key, Body=resized_image_buffer, ContentType='image/jpeg')
return {
'statusCode': 200,
'body': f"Image {file_key} successfully resized and saved to resized-images-bucket"
}
Before deploying, we need the
Pillow
library. To do this:Locally, create a new directory.
Install Pillow and Boto3 SDK:
pip install \ --platform manylinux2014_x86_64 \ --target=package \ --implementation cp \ --python-version 3.8 \ --only-binary=:all: --upgrade \ pillow boto3
Create a .zip file containing your application code and the Pillow and Boto3 libraries.
cd package zip -r ../lambda_function.zip . cd .. zip lambda_function.zip lambda_function.py
Upload this
lambda_function.zip
to your Lambda function using the AWS Console or AWS CLI.
5. Add an S3 Trigger:
In the Lambda function dashboard, click on '+ Add trigger'.
Choose 'S3' and then select your bucket.
Set the event type to 'PUT' (or 'All object create events').
Configure other settings as desired and click 'Add'.
6. Testing:
Upload an image to your S3 bucket.
After a few moments, check the resized-images S3 bucket. You should see a resized version of the image with the prefix 'resized_'.
Original "minion.jpg" image size properties:
Resized "minion.jpg" image size properties:
This is a basic example of how you can set up image resizing using AWS Lambda and S3. Depending on your needs, you might want to enhance this with error handling, logging, different storage structures, or integration with other services.