Is it possible to debug Lambdas locally?
Why would you want to debug your Lambda functions locally? To answer that question, we need to go back to the basics by asking why local debugging is so highly valued and why debugging is even done in the first place.
Some developers want to debug to simply get rid of these pesky creatures that inhabit the dark corners of their lambdas. By using a debugger, you make an attempt to understand the origin of unexpected behaviors and remediate their root causes. For other developers, “disinsection” doesn’t quite cut it. Rather than just getting rid of the bugs, these people would like to understand the flow of data in an attempt to better grasp what’s going on. Last but not least, there are curious minds who would boldly wonder, “What happens if I tweak this?” Having a debugging toolchain in place allows them to extract more knowledge from these experiments, resulting in a higher quality product that is delivered faster.
In this article, we’ll present several tools that may help you with debugging your code. Whether you want to fix unexpected behavior or simply learn more about your data flow, these products can help you.
Let’s face an uncomfortable truth: email generally works, HTTP performs without a struggle, and AWS is doing just fine; yet, end users keep coming up with bug reports. How come?
Most bugs happen at the business logic layer and can be thought of as mismatches between the domain model and reality. As development progresses, we tend to focus more on the technical qualities of the code and less on business needs. This underlying issue can be remedied by fighting code with code. In order to satisfy business needs, you should create a suite of automated tests to verify that your software behaves in an expected manner.
Testing is a double-edged sword, though. If you aren’t careful, you risk developing tests that are very hard to write, yet are so brittle that they break during even the tiniest of refactors. Creating these tests isn’t the best use of an engineer's time, and having to do so can be avoided by proper separation of IO and domain code:
In the above snippet of Python code, the HTTP request is mixed with business logic. Testing it in its current form would require either monkey patching or actually performing the request. The former would make the tests brittle, while the latter would make them slow. However, a simple trick can turn the tables:
Now, in the test code, you can pass a function that returns the predefined payload and then formulate assertions based on it, while the production code actually makes the request. This idea can be further generalized into dependency injection along with advanced tooling if needed. These techniques can make testing fun again!
Embrace Function as a Service (FaaS)
You’d like to use Lambdas as a service? Then treat them like one! Setting up proper instrumentation will help you to gather application insights both locally and in production. Since the lifespan of a function is short (a maximum of 15 minutes, typically the time of one request), monitoring might not make much sense for individual functions. However, aggregates can provide information about the overall state of the system. A great way to learn about your architecture is to check out the architecture view in Thundra, which you can do in a separate article dedicated to this feature.
On the other hand, centralized tracing and logging (with proper log levels) are absolutely necessary for complex functions. Going through the call graph and lifecycle events will help you discover root causes for defects as well as performance bottlenecks. The tracing and logging tools may also surprise you with their features. Jaeger’s ability to graph the service mesh from spans is invaluable in the case of Lambdas since the connections between lambdas are defined on per function basis and lack a global overview. You can get a global overview by using Thundra’s full tracing capabilities as well.
Debugging: Possible Approaches
There are several ways to approach the problem of debugging. Before digging into tooling, it’s worth checking out the solutions that don’t require additional utilities. You might find that after implementing instrumentation and proper testing, debugging is no longer required.
If debugging is required, you’ll have to decide if it should be done in the cloud or in the comfort of your local machine. Debugging your lambdas while they sit on AWS can be a troublesome process, in part because there’s no debugger there. On the other hand, the downside of running lambdas locally is their lack of integration with the rest of the cloud. In production, this integration is transparently handled by the provider via lambda settings.
From Containers with Love
The most straightforward way to get started with debugging is to use docker-lambda, which spins up a Docker container with your lambda inside. How is this different from putting a lambda in a container yourself? The provided images aim to faithfully replicate the function execution environment. You can take a peek inside the /base folder if you’re curious about what exactly is available to a lambda.
If your code happens to be interacting with other AWS services, you can pass the appropriate credentials via environment variables, and the event can be injected either as an argument to the invocation or through stdin pipe. Upon invocation, a log is printed onto the console with the exact same information you can expect from going through CloudWatch. This UNIX-friendly interface, combined with its perfect coverage of runtime engines, makes this tool suitable for CI. The comprehensive invocation output will help immensely when estimating running time and required resources.
Serverless Application Model
In the context of debugging, the Serverless Application Model (SAM) can be considered an extension of docker-lambda. Yet, the goal of SAM is to provide much more coarse-grained lambda management services via the command line tool aws-sam-cli. It also uses Docker under the hood. SAM handles the whole lifecycle of a function, from development through packaging and deployment. The program works well with CloudFormation. Using the two together, you can provide the necessary backing services for your mesh of lambdas to thrive. SAM integrates seamlessly with text editors like Visual Studio Code to provide a focused development experience.
AWS on Your Machine
For debugging more complex AWS flows, you might turn your eyes towards localstack. It’s a toolchain for setting up the whole AWS stack directly on your laptop. Once you’ve integrated localstack, the possibilities are endless. In addition to gaining speed from not provisioning remote resources and saving money on cloud bills, you can also set up advanced chaos experiments to test the resilience of your lambda-based system.
Since there is no such thing as a free lunch, there are certain drawbacks associated with using localstack. Configuring it to your preferences might not be as straightforward as it is with the tools discussed previously. In localstack, there are many settings to tweak. Additionally, its parity with AWS is not 100%. At the time of writing, AWS Athena was missing. Bearing these limitations in mind, localstack can be a great aid in your debugging journey.
Debugger on AWS
How about reversing the model a bit, and, instead of running functions locally, bringing a debugger to AWS? With Thundra that’s possible. After a little generic instrumentation, you can direct Thundra’s tracing effort towards the method of your choice. Since it utilizes the environment variables, there is no need to change the code or redeploy the lambda. If you’re looking for a fine-grained approach to tracing, Thundra’s line-by-line tracing is the answer. Below, you can see an example trace chart from a Thundra console showing its line-by-line tracing. It also shows the source code and values of local variables at each line.
All the other efforts for debugging are simulating the AWS-like environment in your local. ServerlessDebugger lets you remote-debug your AWS Lambda functions running on their real AWS environment. With it, you are able to see the values of local variables and stacktrace at each line. In this way, you’ll get rid of any concerns of mocking any resources.
Debugging Lambdas locally is by no means a trivial process. The myriad tools and techniques available (including no debugging at all) for doing so can be a bit overwhelming. The wide spectrum of available options does allow you to select the best tool for the job at any system size and FaaS adoption stage, however. Most of the tools have a reasonable learning curve, so try them out, and then stick to the one that fits your use case best.
ServerlessDebugger lets you debug your serverless apps like you debug any code that you host on your computer with native debugging actions, enabling you to see the values of local variables and the entire stack trace. Take your free account and start debugging today. We are more than happy to answer your questions and get your feedback through our Discord community, contact us page or firstname.lastname@example.org