A Step-by-Step Guide to Creating Your Own Custom Lambda Layer (NO PIP INSTALL REQUIRED)

Pandas library plays an important role in data engineering due to its ability to efficiently handle data manipulation and transformation tasks. When combined with AWS Lambda, it allows data engineers to perform serverless, on-demand data processing in the cloud.

By default, Pandas is not included in AWS Lambda’s Python environment. That is to say if you try to import Pandas in your code and run it. You will get this error message:

lambda function failing to import Pandas

Using Pandas with AWS Lambda requires creating a Lambda Layer, which is essentially a package of code, libraries, or dependencies that you can deploy and use across multiple Lambda functions. Instead of including the shared code directly in each Lambda function’s deployment package, you can place it in a Layer and reference that Layer in multiple functions. This helps reduce the size of your function packages and promotes code reusability.

In this article, we will walk you through the step-by-step process of creating your own custom Lambda layer, enabling you to leverage the full potential of Pandas in your data manipulation and analysis tasks.

Step 1: Use The Correct Folder Structure

In order for your lambda function code to access the layer content, you need to properly organize the folders that will include Python libraries.

When you add a layer to a function, Lambda loads the layer content into the /opt directory of that execution environment. For each Lambda runtime, the PATH variable already includes specific folder paths within the /opt directory.

For Python, create the following folders:

YOUR_LOCAL_DIR\python\lib\python3.9\site-packages\

Note: In this tutorial, we’re using Python 3.9. If you’re using a different runtime, you could change the folder name from python3.9 to python{your_runtime}

Step 2: Download the wheel files

Pandas library requires two additional libraries to properly function: Numpy and Pytz. With that being said, we’re going to download them all from Python Package Index.

Get Pandas wheel file

• Go to https://pypi.org/project/pandas/1.3.5/#files
• Select the wheel file for the corresponding python version you’re looking for. In our example, we’re using python 3.9 so we’ll select the following wheel file: pandas-1.3.5-cp39-cp39-win32.whl (9.1 MB view hashes)

pandas whl file

• Extract the downloaded files into python\lib\python3.9\site-packages\

Get Numpy wheel file

• Go to https://pypi.org/project/numpy/1.21.5/#files
• Select the wheel file for the corresponding python version you’re looking for. In our example here, we’re using python 3.9 so we’ll select the following wheel file: numpy-1.21.5-cp39-cp39-win32.whl (11.7 MB view hashes)

numpy whl file

• Extract the downloaded files into python\lib\python3.9\site-packages\

Get Pytz wheel file

• Go to https://pypi.org/project/pytz/2021.3/#files
• Select the wheel file for the corresponding python version you’re looking for. In our example here, we’re using python 3.9 so we’ll select the following wheel file: pytz-2021.3-py2.py3-none-any.whl (503.5 kB view hashes)

pytz whl file

• Extract the downloaded files into python\lib\python3.9\site-packages\

Step 3: Package The Layer’s Libraries

At this stage, the site-packages folder should contain all the needed libraries and look like this:

content of site-packages folder

Next, navigate to the ‘python’ directory and generate a ZIP archive. It’s important to place ‘python’ as the root folder of your layer. Otherwise, the libraries won’t be recognized by your lambda function.

python as root folder of the ZIP archive

Step 4: Create an AWS Layer

Navigate to AWS Management Console:

• Go to the AWS Lambda service in the AWS Management Console.
• Click on “Layers” in the left sidebar.

AWS Management Console —AWS Lambda

• Click on “Create layer”

AWS Management Console — Layers

• Provide a name, description, and compatible runtime for your layer.
• Upload the ZIP archive containing your layer code.
• Click on “Create” to publish the layer.

AWS Management Console — Create Layer

Step 5: Attach the Custom Layer to Your Lambda Function

After successfully creating the Lambda layer, you can now attach it to your Lambda functions.

Creating a custom AWS Lambda layer enables you to manage your shared code and dependencies efficiently, leading to cleaner and more maintainable Lambda functions. By following the steps outlined in this guide, you can now harness the power of AWS Lambda layers and enhance the development and scalability of your serverless applications. Happy coding!