Setting up S3 Event Notification to invoke AWS Lambda using AWS CLI — 2024

AWS S3 event notifications allow you to receive notifications when certain events occur in your S3 bucket. You can configure notifications for various events such as object creation, deletion, or restoration. These notifications can be sent to Amazon Simple Notification Service (SNS), Amazon Simple Queue Service (SQS), or AWS Lambda, enabling you to integrate S3 events with other AWS services or your applications.

In this article, we will discover how to set up an S3 trigger that invokes an AWS Lambda function from the CLI.

We’ll show you how to do this in the following steps:

• Step 0: Prepare the required environment variables
• Step 1: Save the existing S3 notification configurations
• Step 2: Allow S3 to invoke the lambda function
• Step 3: Add the new notification config
• Step 4: Configure the S3 Bucket Notification

Architecture

Here’s an architecture diagram of what we’ll implement:

Architecture Diagram

Steps:

Step 0: Prepare the required environment variables

Before we begin, ensure you have the following variables configured:

# variables
LAMBDA_FUNCTION_NAME="{lambda-function-name}"
LAMBDA_FUNCTION_ARN="arn:aws:lambda:{region}:{account-id}:function:{lambda-function-name}"
S3_ARN="{s3-arn}"
ACCOUNT_ID="{account-id}"
S3_BUCKET_NAME="{s3-bucket-name}"
EVENT_TYPE="s3:ObjectCreated:*"
FILTER_PREFIX="{s3-path}" # in our example it’s set to: “files/”
FILTER_SUFFIX="{s3-object-suffix}"

Step 1: Save the existing S3 notification configurations

In order to avoid overriding any existing or manually created notification configurations in your S3 bucket, we will save the existing configuration first in a variable.

echo "get all the S3 notification Events"
CURRENT_S3_CONFIG=$(aws s3api get-bucket-notification-configuration --bucket $S3_BUCKET_NAME --output json)


echo "check if there are existing notification configurations"
if [ -z "$CURRENT_S3_CONFIG" ] || [ "$CURRENT_S3_CONFIG" == "{}" ]; then
  echo "No existing notifications found. Initializing a new configuration."
  CURRENT_S3_CONFIG='{"LambdaFunctionConfigurations": []}'
else echo $CURRENT_S3_CONFIG
fi

Step 2: Allow S3 to invoke the lambda function

The lambda function must have a resource-based policy that allows S3 to publish notifications to our lambda. Otherwise, we will get the following error message:
“Unable to validate the following destination configurations when creating an Amazon S3 Event Notification”.

echo "grant the lambda function the permissions to be invoked by S3"
aws lambda add-permission \
--function-name $LAMBDA_FUNCTION_NAME \
--principal s3.amazonaws.com \
--statement-id s3invoke \
--action "lambda:InvokeFunction" \
--source-arn $S3_ARN \
--source-account $ACCOUNT_ID

>> Result: a new resouce-based policy was added to our lambda function

the “s3invoke” resouce-based policy was added to the lambda

Step 3: Add the new notification config

At this stage, we will append the new configuration to the existing S3 config that we previously stored in a variable.

echo "appending a new notification configuration."
UPDATED_S3_CONFIG=$(echo $CURRENT_S3_CONFIG | jq \
  --arg arn "$LAMBDA_FUNCTION_ARN" \
  --arg event "$EVENT_TYPE" \
  --arg prefix "$FILTER_PREFIX" \
  --arg suffix "$FILTER_SUFFIX" \
  '.LambdaFunctionConfigurations+=[{"Id":"PutEventNotification","LambdaFunctionArn":$arn,"Events":[$event],"Filter":{"Key":{"FilterRules":[{"Name":"prefix","Value":$prefix},{"Name":"suffix","Value": $suffix}]}}}]')

Step 4: Configure the S3 Bucket Notification

This is the final step, where we set the S3 trigger with the help of s3api command.

echo "updating bucket notification configuration for bucket: $S3_BUCKET_NAME"
aws s3api put-bucket-notification-configuration \
--bucket $S3_BUCKET_NAME \
--notification-configuration "$UPDATED_S3_CONFIG"

>> Results:

After running the command, a new trigger is going to be added to our lambda and end up looking like the following:

AWS Console — lambda function: “lambda-test”

Conclusion

In this article we’ve successfully set up an S3 trigger that invokes an AWS Lambda function using the AWS CLI. This setup is useful for automating workflows and integrating various AWS services based on events occurring in your S3 buckets. You can extend this setup by adding more complex logic to your Lambda function to handle different types of events and perform various actions.

References

• AWS s3api CLI command Put-bucket-notification-configuration:
  https://docs.aws.amazon.com/cli/latest/reference/s3api/put-bucket-notification-configuration.html
• Invoking Lambda with events from other AWS services:
  https://docs.aws.amazon.com/lambda/latest/dg/lambda-services.html
• Working with resource-based policies in Lambda:
  https://docs.aws.amazon.com/lambda/latest/dg/access-control-resource-based.html