6 releases (3 breaking)
0.4.0 | Oct 16, 2024 |
---|---|
0.3.0 | May 22, 2024 |
0.2.1 | Apr 16, 2024 |
0.1.1 | Apr 4, 2024 |
#1495 in Network programming
201 downloads per month
31KB
619 lines
AWS Lambda Log Proxy
Filter / transform / forward logs from AWS Lambda functions before they are sent to CloudWatch Logs.
[!NOTE] This is a library for developers. If you are looking for a binary executable, see AWS Lambda Log Filter.
[!CAUTION] Possible data loss if you write tons of logs and return immediately. See possible data loss below.
Usage
Installation
cargo add aws-lambda-log-proxy
Examples
See examples for more details.
A real world case: AWS Lambda Log Filter.
Documentation
FAQ
Why We Need This?
We need a solution to realize the following features without modifying the code of your existing AWS Lambda functions:
- Reduce the volume of logs to lower the costs.
- Forward logs to other destinations (like Elasticsearch) in real time.
- But keep the EMF lines untouched so Amazon CloudWatch can still retrieve the metrics.
How It Works?
You need a wrapper script to start the handler process (with the _LAMBDA_TELEMETRY_LOG_FD
environment variable removed) and pipe the logs to the proxy. The proxy will filter / transform / forward the logs before they are sent to CloudWatch Logs.
We use AWS Lambda Runtime Proxy to intercept AWS Lambda runtime API requests so we can suppress the invocation/next
request to process logs as much as possible. You need to override the AWS_LAMBDA_RUNTIME_API
environment variable for your handler function to point to the runtime proxy in the wrapper script.
Why Not Just Grep
Actually you could just create a shell scripts with the content env -u _LAMBDA_TELEMETRY_LOG_FD "$@" 2>&1 | grep --line-buffered xxx
and use a wrapper script to realize a simple filter. However we need this lib to realize the following features:
- Suppress the
invocation/next
request to process logs as much as possible. This is very important if you want to stream the logs to external services in real time. - Identify the log lines with EMF so we can keep them untouched.
- Implement custom filters / transformers / forwarders in Rust.
Possible Data Loss
Though we tried our best to suppress the invocation/next
request to process logs as much as possible, if you write tons of logs (more than thousands of lines) and return immediately, there might be some logs not processed.
As a best practice, it is your responsibility to do a thorough benchmark test against your own use case to ensure the logs are processed as expected.
CHANGELOG
Dependencies
~6–15MB
~187K SLoC