Monitor Your Lambda Logs and Get Notified on Errors — AWS CLI - 2024

Introduction:

In the fast-paced world of cloud computing, ensuring that your AWS Lambda functions are running smoothly is crucial. However, even the most well-designed functions can encounter issues. When errors occur, you need to be alerted immediately so that you can take prompt action and minimize downtime. This is where monitoring your Lambda logs becomes invaluable. By setting up log monitoring and error notifications, you can stay informed of any issues in real time, allowing you to respond quickly and keep your applications running optimally.

In this guide, we’ll walk you through the process of monitoring your AWS Lambda logs and configuring notifications to alert you when errors happen. Whether you’re a seasoned cloud engineer or new to AWS, this step-by-step approach will help you ensure that your Lambda functions are always performing at their best. Let’s get started!

We’ll show you how to do this in the following steps:

• Step 0: prerequisites
• Step 1: Create a CloudWatch Metric Filter
• Step 2: Create an SNS Topic for Notifications
• Step 3: Subscribe to the Topic (with your email)
• Step 4: Create a CloudWatch Alarm
• Step 5: Test the alarm

Architecture

Here’s an architecture diagram of what we’ll implement:

Architecture Diagram

Key Concepts:

Before we dive into coding, let’s review some essential CloudWatch terminology.

• Log events:

The log event record that CloudWatch Logs understands contains two properties: the timestamp of when the event occurred, and the raw event message.

Example: lambda function’s code

lambda function’s code

After running the lambda function, we get the following log events:

CloudWatch Log events

• Log streams:

A log stream is a sequence of log events that share the same source. For a Lambda function, a log stream is automatically created in CloudWatch after each invocation.

CloudWatch Log streams

• Log groups:

Log groups define groups of log streams that share the same retention, monitoring, and access control settings.

CloudWatch Log groups

Now that we’ve cleared up the CloudWatch terminology, let’s dive into the coding.

Steps:

Step 0: prerequisites

To follow along with this article, ensure the following:

1. AWS Lambda Function: You should have an existing AWS Lambda function. If you haven’t created one yet, please do so before continuing. In this article, we are using a Lambda function named “my-test-lambda” as an example.
2. CloudWatch Log Group Name: Obtain the CloudWatch Log Group name associated with your “my-test-lambda” function. This is necessary for monitoring and logging purposes as we will reference it later in the article.

Step 1: Create a CloudWatch Metric Filter

An AWS CloudWatch Metric Filter is a feature that allows you to extract metric data from log events in Amazon CloudWatch Logs. When the specified pattern is found in the lambda’s logs, the filter automatically creates a metric. This metric can be used like any other CloudWatch metric, allowing us to monitor it, set alarms, or trigger automated responses.

aws logs put-metric-filter --log-group-name /aws/lambda/my-test-lambda --filter-name ErrorFilter --filter-pattern '"ERROR"' --metric-transformations metricName=ErrorCount,metricNamespace=MyApp,metricValue=1

> Result:

if we go to the console and under the log group: /aws/lambda/my-test-lambda we would find the new metric filter added.

CloudWatch Metric Filter: “ErrorFilter”

Step 2: Create an SNS Topic for Notifications

AWS SNS is a fully managed service that provides message delivery and notification capabilities. It allows you to send messages, notifications, and alerts to a large number of recipients using different communication protocols.In this example, we will create an SNS topic named: “error-notifications”

aws sns create-topic --name error-notifications

Step 3: Subscribe to the Topic (with your email)

Subscriptions define how and where messages should be delivered when a topic receives a message. SNS supports multiple subscription protocols, including Email, SMS, HTTP/HTTPS, Amazon SQS, Lambda functions, and more.

In this example, we will send an email notification when a new message is published to our SNS topic.

aws sns subscribe --topic-arn arn:aws:sns:$(aws configure list | grep region | awk '{print $2}'):$(aws sts get-caller-identity --query 'Account' --output text):error-notifications --protocol email --notification-endpoint myemail@example.com

> Result:

From the AWS Console, we can see that a new subscriber was added. The email subscription requires the recipient to click a confirmation link. Until confirmed, the subscription remains in a pending state.

SNS Subscriptions

Check your INBOX for a new subscription confirmation email :

AWS Notification — Subscription Confirmation Email

Step 4: Create a CloudWatch Alarm

AWS CloudWatch Alarms enable you to monitor metrics and automatically perform actions based on specified thresholds. The AWS CLI command below, is used to create a CloudWatch alarm named “ErrorAlarm” that monitors the metric “ErrorCount” and sends a notification to the specified SNS topic when the alarm state changes to “In alarm”.

aws cloudwatch put-metric-alarm --alarm-name ErrorAlarm --metric-name ErrorCount --namespace MyApp --statistic Sum --period 60 --threshold 1 --comparison-operator GreaterThanOrEqualToThreshold --evaluation-periods 1 --alarm-actions arn:aws:sns:$(aws configure list | grep region | awk '{print $2}'):$(aws sts get-caller-identity --query 'Account' --output text):error-notifications

Step 5: Test the alarm

In order to test the alarm, we need to create a new log stream in CloudWatch Logs named: my-log-stream

CloudWatch Log Streams

Next, we add a new log event as if the lambda’s code raised an error. This new log entry will make the CloudWatch Alarm state change to “In alarm”.

aws logs put-log-events --log-group-name /aws/lambda/my-test-lambda --log-stream-name my-log-stream --log-events timestamp=$(date +%s000),message="This is an ERROR log entry"

Check your INBOX for a notification email:

AWS Email Notification

Conclusion

In this article, we explored a robust approach to monitoring AWS Lambda logs and getting notified when errors occur. By setting up CloudWatch Logs and configuring alarms, you can ensure that your Lambda functions are running smoothly and that you are promptly alerted to any issues. This proactive monitoring strategy helps maintain the reliability and performance of your serverless applications, enabling quick responses to potential problems.

References

• Amazon CloudWatch concepts:
  https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/cloudwatch_concepts.html
• What is Amazon CloudWatch Logs?
  https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/WhatIsCloudWatchLogs.html
• Amazon Simple Notification Service Examples:
  https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/sns-examples.html
• What is AWS Lambda?
  https://docs.aws.amazon.com/lambda/latest/dg/welcome.html