Automating EMR Step Monitoring with EventBridge and AWS CLI (2024)

Introduction

In today’s data-driven landscape, monitoring and automation are essential for ensuring the smooth operation of data pipelines. As data engineers, we often find ourselves managing complex workflows on AWS, particularly with services like Amazon EMR (Elastic MapReduce), which plays a pivotal role in big data processing. One powerful way to enhance your EMR workflows is by integrating AWS EventBridge to automatically monitor and respond to the status of individual EMR steps.

In this guide, we’ll walk through how to create an EventBridge rule using the AWS CLI to track the status of your EMR steps.

We’ll show you how to do this in the following steps:

• Step 0: Prerequisites
• Step 1: Create an SNS Topic
• Step 2: Subscribe to the SNS Topic
• Step 3: Create an EventBridge Rule to Monitor EMR Step Status
• Step 4: Add Permissions to EventBridge to Publish to SNS
• Step 5: Create a Target to Send Notifications via SNS
• Step 6: Test the EventBridge Rule

Architecture

Here’s an architecture diagram of what we’ll implement:

Architecture diagram

Concepts

Before diving into the practical steps, it’s important to familiarize yourself with key Amazon EMR and EventBridge terminologies. This will help ensure clarity as we move through the guide:

• Amazon EMR (Elastic MapReduce): A cloud service that simplifies the processing and analysis of large datasets using open-source tools like Apache Hadoop, Apache Spark, and HBase. EMR allows for distributed computing, making it essential for big data operations.
• EMR Cluster: A group of Amazon EC2 instances that work together to process data. The cluster includes a master node, core nodes, and optionally task nodes. Each node plays a distinct role in managing tasks and processing data.
• EMR Step: An action or unit of work in an EMR cluster, such as a Spark job, Hadoop task, or a script execution. Steps are executed in sequence, and the cluster tracks their status (pending, running, completed, failed).
• Step Status: Each EMR step goes through different statuses — starting from PENDING, moving to RUNNING, and ending with either COMPLETED or FAILED. Monitoring these statuses is critical for automating responses based on step outcomes.
• EventBridge Rule: A configuration in AWS EventBridge that listens for specific events (like a change in step status) and triggers automated actions, such as sending notifications or starting another workflow.

Understanding these concepts will make it easier to follow along as we set up monitoring for EMR step statuses in the following sections.

Steps:

Step 0: Prerequisites

Prepare the environment variables you’ll need in a later step.

EMAIL_ADDRESS=enter_your_email

Step 1: Create an SNS Topic

This SNS topic will be used to send notifications when the EMR steps’ status changes. The command below creates a new SNS topic named: “EMRStepStatusTopic”.

aws sns create-topic --name EMRStepStatusTopic

> Output:

AWS CloudShell — create sns topic

Take note of the TopicArn from the output, we’ll need it in future steps.

Step 2: Subscribe to the SNS Topic

To receive notifications from AWS SNS, we need to subscribe to the topic. The email subscription requires the recipient to click a confirmation link. Until confirmed, the subscription remains in a pending state.

aws sns subscribe \
 --topic-arn place_SNS_topic_arn_here \
 --protocol email \
 --notification-endpoint $EMAIL_ADDRESS

Confirm the subscription via the email sent to your inbox.

AWS Notification — Subscription Confirmation Email

> From the AWS Console, we can see that a new subscriber was added.

SNS Subscriptions list

Step 3: Create an EventBridge Rule to Monitor EMR Step Status

We need to create an EventBridge rule that monitors the status of EMR steps. This rule will trigger the SNS notification whenever a step status changes (e.g., failed, completed, or cancelled).

aws events put-rule \
--name EMRStepStatusChangeRule \
--description "Monitor EMR step status changes" \
--event-pattern '{
"source": ["aws.emr"],
"detail-type": ["EMR Step Status Change"],
"detail": {"state": ["COMPLETED", "FAILED", "CANCELLED"]}
}'

> Output:

AWS CloudShell — create eventbridge rule

Step 4: Add Permissions to EventBridge to Publish to SNS

The aws sns add-permission command applies a resource-based policy to an Amazon SNS topic. It allows us to specify who can interact with the topic and what actions they are permitted to perform. In this example, we will grant EventBridge permission to publish notifications to the SNS topic we’ve previously created.

aws sns add-permission \
--topic-arn place_SNS_topic_arn_here \
--label AllowEventBridgeToPublish \
--aws-account-id $(aws sts get-caller-identity --query 'Account' --output text) \
--action-name Publish

Step 5: Create a Target to Send Notifications via SNS

Next, we need to create a target for the EventBridge rule. The target will be the SNS topic, and EventBridge will publish messages to it when a matching event is triggered.

aws events put-targets \
 --rule EMRStepStatusChangeRule \
 --targets '[
     {
     "Id": "1",
     "Arn": "place_SNS_topic_arn_here"
     }
 ]'

> Output:

AWS CloudShell —specify targets for EventBridge

Step 6: Test the EventBridge Rule

Finally, we can add a new step to the running EMR cluster, and the EventBridge rule will capture its status changes and send an SNS notification.

Whenever the step status changes (COMPLETED, FAILED, or CANCELLED), an SNS notification will be triggered.

Conclusion

In this article, we’ve explored how to leverage AWS EventBridge and the AWS CLI to monitor the status of individual EMR steps, a crucial aspect of maintaining robust and efficient data pipelines. By creating an EventBridge rule, you can automate monitoring and take timely actions based on the success or failure of your EMR steps.

For data engineers, this workflow offers significant advantages: it reduces manual intervention, improves the reliability of data processing, and enables greater automation in your AWS ecosystem. As you continue to scale your EMR workloads, integrating EventBridge for event-driven monitoring will provide flexibility and enhance operational efficiency.

Armed with the knowledge from this guide, you’re now equipped to optimize your EMR workflows, making your data engineering processes not only smarter but also more resilient.

References

• What is Amazon EMR?
  https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-what-is-emr.html
• Submit work to a cluster
  https://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-work-with-steps.html
• Creating rules that react to events in Amazon EventBridge:
  https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-create-rule.html
• Adding EMR steps to a cluster with the AWS CLI
  https://docs.aws.amazon.com/emr/latest/ManagementGuide/add-step-cli.html
• What is an Event-Driven Architecture?
  https://aws.amazon.com/event-driven-architecture/