Get started with Amazon S3 Event Notifications

Amazon S3 (Simple Storage Service) event notifications are a feature that allows you to receive notifications when certain events happen in your S3 buckets. These events include creating, deleting, or modifying objects in a bucket. The notifications can be sent to various AWS services for further processing or actions. Here's how it integrates with different AWS services:

Amazon Simple Notification Service (SNS):
- S3 can send event notifications to SNS topics.
- When an event occurs in S3, a message is published to an SNS topic.
- Subscribers to this topic (like email addresses, AWS Lambda functions, or HTTP/S endpoints) receive the notification message.
Amazon Simple Queue Service (SQS):
- S3 event notifications can be sent to SQS queues.
- This allows you to decouple the processes that produce events from the systems or processes that consume them.
- You can have SQS trigger other services or worker processes that pull these messages from the queue for processing.
AWS Lambda:
- S3 can trigger a Lambda function directly in response to events.
- This is useful for running serverless computing actions like image resizing, file transformation, or data processing directly after an object is uploaded to S3.
Amazon EventBridge:
- S3 events can be routed to EventBridge, a serverless event bus that connects applications using events.
- EventBridge can then route these events to various AWS services or custom applications based on rules you define.

Currently, Amazon S3 can publish notifications for the following events:

New object created events
Object removal events
Restore object events
Reduced Redundancy Storage (RRS) object lost events
Replication events
S3 Lifecycle expiration events
S3 Lifecycle transition events
S3 Intelligent-Tiering automatic archival events
Object tagging events
Object ACL PUT events

For full descriptions of all the supported event types, see Supported event types for SQS, SNS, and Lambda.

Use Cases

Automated Image or Video Processing:
- When a new image or video is uploaded to an S3 bucket, an event notification is sent to a Lambda function, which processes the image (like resizing, watermarking) or video (like transcoding).
Data Backup and Replication:
- S3 event notifications can trigger a process to back up newly uploaded data to another bucket or a different storage service.
Real-time Data Processing:
- For applications like log analysis or real-time analytics, event notifications can trigger processes that analyze new data as soon as it arrives in S3.
Workflow Automation:
- Notifications can initiate workflows in other AWS services or custom applications. For example, when a new file is uploaded, it might trigger a data validation process or update a database.
Monitoring and Alerts:
- Event notifications can be used to monitor bucket activity and send alerts via SNS for unusual or unauthorized access patterns.
Queue-based Processing:
- Using SQS, you can queue events for batch processing, which is useful for handling high volumes of events or for processing that can be deferred.

These integrations and use cases demonstrate the versatility of Amazon S3 event notifications in building efficient, scalable, and automated cloud-based applications and workflows.

Tutorial: Create Amazon S3 Event Notification with Amazon SQS for object create events via AWS CLI

To test the concept of Amazon S3 event notification with Amazon SQS using the AWS Command Line Interface (CLI), you'll need to perform a series of steps. These steps include setting up an SQS queue, configuring an S3 bucket for event notifications, and then testing the setup by uploading a file to the S3 bucket.

Prerequisites

Ensure you have the AWS CLI installed and configured with the necessary permissions.
Have an AWS S3 bucket and an IAM resource policy with the necessary permissions for SQS.

Create an SQS Queue

Create a new SQS queue:

 aws sqs create-queue --queue-name MyS3EventsQueue

Get the Queue URL:

 aws sqs get-queue-url --queue-name MyS3EventsQueue

Note the Queue URL for later use.

Configuring access permissions for SQS:

Configuring the permissions for the SQS queue to accept events from S3 is a crucial step. This involves setting the correct access permissions for the SQS queue so that it can receive messages from the S3 bucket.

Step 1: Create resource policy for the SQS Queue

Create a policy JSON file:

Create a file named sqs-s3-policy.json with the following content:

  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Principal": {
                  "Service": "s3.amazonaws.com"
              },
              "Action": [
                  "SQS:SendMessage"
              ],
              "Resource": "arn:aws:sqs:Region:account-id:queue-name",
              "Condition": {
                  "ArnLike": {
                      "aws:SourceArn": "arn:aws:s3:*:*:awsexamplebucket1"
                  },
                  "StringEquals": {
                      "aws:SourceAccount": "bucket-owner-account-id"
                  }
              }
          }
      ]
  }

Replace arn:aws:sqs:Region:account-id:queue-name with your actual SQS queue ARN, arn:aws:s3:*:*:awsexamplebucket1 with your actual S3 bucket ARN and bucket-owner-account-id with your actual account ID.

💡

To create S3 bucket use the following command: aws s3 mb s3://your-bucket-name

Attach the resource policy to SQS queue using AWS CLI:

To avoid errors you need to escape double quotes and provide JSON file as a string input to the command.

 aws sqs set-queue-attributes --queue-url [Your Queue URL] --attribu
 tes Policy="'{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Service\":\"s3.amazonaws.c
 om\"},\"Action\":[\"SQS:SendMessage\"],\"Resource\":\"arn:aws:sqs:Region:account-id:queue-name\",\"Condition\":
 {\"ArnLike\":{\"aws:SourceArn\":\"arn:aws:s3:*:*:awsexamplebucket1\"},\"StringEquals\":{\"aws:SourceAccount
 \":\"bucket-owner-account-id\"}}}]}'"

Step 2: Proceed with the S3 Bucket Notification Configuration

Now that your SQS queue has the appropriate permissions, you can proceed with configuring the S3 bucket to send notifications to the SQS queue.

This completes the setup, where the SQS queue is now configured to receive messages from the S3 bucket when specified events occur.

Configure S3 Event Notifications to Send Messages to SQS

Get the ARN of the SQS queue:
```
 aws sqs get-queue-attributes --queue-url [Your Queue URL] --attribute-names QueueArn
```
Replace [Your Queue URL] with the URL obtained from the previous step. Note down the Queue ARN.
Create a bucket notification configuration file (json format): Create a JSON file (e.g., s3-notification.json) with the following content:
```
 {
   "QueueConfigurations": [
     {
       "QueueArn": "arn:aws:sqs:region:account-id:MyS3EventsQueue",
       "Events": ["s3:ObjectCreated:*"]
     }
   ]
 }
```
Replace arn:aws:sqs:region:account-id:MyS3EventsQueue with your actual Queue ARN.

Add the notification configuration to your S3 bucket:

 aws s3api put-bucket-notification-configuration --bucket your-bucket-name --notification-configuration file://s3-notification.json

Replace your-bucket-name with your S3 bucket name.

Test the Setup

Upload a file to your S3 bucket:
```
 aws s3 cp local-file.txt s3://your-bucket-name/
```
Replace local-file.txt with the path to a local file and your-bucket-name with your S3 bucket name.
Check the SQS queue for messages:
```
 aws sqs receive-message --queue-url [Your Queue URL]
```
This should show a message indicating that a new file was uploaded to your S3 bucket.

Clean Up

Remember to delete the SQS queue and remove the S3 bucket notification configuration after testing to avoid incurring unnecessary charges.

This tutorial demonstrates how to set up and test S3 event notifications with Amazon SQS using the AWS CLI. The process involves creating an SQS queue, creating and adding resource policy to SQS queue, configuring S3 bucket notifications, and then testing the setup by uploading a file to S3 and checking the SQS queue for the event message.