Dash0 Lambda Extension

An extension for capturing observability data from AWS Lambda invocations and shipping to Dash0.

Table of Contents

• Layer ARNs
• Supported Regions
• Configuration
  • Required
  • Optional
  • Secret Masking
• Manual Instrumentation
• Enrichment Attributes
  • Span Attributes
  • Log Attributes
  • Metrics
• Serverless Framework Plugin
• Dockerized Lambdas

This extension has five main functionalities:

1. Enable auto-instrumentation for supported runtimes, which currently include Python, Node, Java.
2. Receive traces from auto/manual instrumentations, enrich with data acquired in the extension, and send to Dash0.
3. Detect runtime errors such as timeout or out of memory and create synthetic traces for them
4. Collect all logs and send to Dash0, correlated with the trace id of the invocation.
5. Create metrics for invocation duration, cold start duration, billed duration, and memory used.

Layer ARNs

See the release page for the latest ARNs of the extension layers for each runtime.

Supported Regions

The extension layers are published to the following AWS regions:

Region	Location
us-east-1	US East (N. Virginia)
us-east-2	US East (Ohio)
us-west-1	US West (N. California)
us-west-2	US West (Oregon)
ap-south-1	Asia Pacific (Mumbai)
ap-northeast-1	Asia Pacific (Tokyo)
ap-northeast-2	Asia Pacific (Seoul)
ap-northeast-3	Asia Pacific (Osaka)
ap-southeast-1	Asia Pacific (Singapore)
ap-southeast-2	Asia Pacific (Sydney)
ca-central-1	Canada (Central)
eu-central-1	Europe (Frankfurt)
eu-west-1	Europe (Ireland)
eu-west-2	Europe (London)
eu-west-3	Europe (Paris)
eu-north-1	Europe (Stockholm)
il-central-1	Israel (Tel Aviv)
sa-east-1	South America (São Paulo)

Configuration

Required

• AWS_LAMBDA_EXEC_WRAPPER=/opt/wrapper - This environment variable must be set in order to enable tracing. If this environment variable will not be set, only logs will be collected.

• DASH0_ENDPOINT - The integration endpoint for you organization in Dash0, i.e. https://ingress.eu-west-1.aws.dash0.com:4318.

• Either DASH0_TOKEN or DASH0_TOKEN_SECRET_ARN must be set (see below).

Optional

• DASH0_TOKEN - The API token for your Dash0 project. Either this or DASH0_TOKEN_SECRET_ARN must be set.

• DASH0_TOKEN_SECRET_ARN - The ARN of an AWS Secrets Manager secret containing the Dash0 API token. Either this or DASH0_TOKEN must be set. If both are set, DASH0_TOKEN takes precedence.

• DASH0_TOKEN_SECRET_KEY - The JSON key within the secret to extract the token from. Required when the secret stored in DASH0_TOKEN_SECRET_ARN is a JSON object. If not set, the entire secret value is used as the token.

• DASH0_DISABLE_AUTO_INSTRUMENTATION - Auto-instrumentation can be turned off by this environment variable, which will result in creating synthetic traces by the extension for all invocations.

• DASH0_SEND_ON_INVOCATION_END - The extension has two modes of sending to the backend, either on invocation end or on the next invocation. The default is true. Sending on invocation end will increase the billed duration of the lambda, but not the response time. Sending on next invocation will decrease the billed duration since the sending will take place in parallel of the regular execution, but might delay the sending up to 7 minutes in case of last invocation in the container.

• DASH0_EXTENSION_LOG_LEVEL - Log level for the extension itself. Valid values: trace, debug, info, warn, error. Default: warn.

• DASH0_DISTRO_DEBUG - When set to true, additional logs related to tracing and auto-instrumentation will be emitted. Default: false.

• DASH0_REQUEST_TIMEOUT - Timeout in milliseconds for HTTP requests to the backend. Default: 2000.

• DASH0_CREATE_PAYLOAD_LOG_RECORDS - When set to true (the default), the extension creates log records containing the request and response payloads for the lambda invocation and each client call. Set to false to disable. Default: true.

• DASH0_DISABLE_TELEMETRY_LOG_COLLECTION - When set to true, disables collecting logs from the Lambda Telemetry API. Default: false.

• DASH0_DISABLE_TELEMETRY_METRICS - When set to true, disables emission of the supplementary FaaS metrics (faas.invoke_duration, faas.init_duration, dash0.faas.billed_duration, faas.mem_usage). Default: false.

• DASH0_DISABLE_TELEMETRY_TRACES - When set to true, disables both auto-instrumentation and any synthetic spans created by the extension (including error-path synthetic traces). Stronger than DASH0_DISABLE_AUTO_INSTRUMENTATION, which only disables auto-instrumentation while still emitting synthetic spans. Default: false.

• DASH0_DATASET - When set, the extension adds a Dash0-Dataset header to all OTLP export requests, routing telemetry to the specified dataset in the Dash0 backend.

• DASH0_XRAY_TRACES_ENABLED - When set to true, the extension preserves the original X-Ray trace context instead of creating supplementary spans. Use this when AWS X-Ray active tracing is enabled on the Lambda function. Default: false.

The four API_GATEWAY variables below also apply to Application Load Balancer-triggered invocations; the names are kept for backwards compatibility.

• DASH0_ENABLE_API_GATEWAY_SPAN_NAME - When set to true, renames the handler span for an HTTP-triggered invocation (API Gateway REST API v1, HTTP API v2 proxy integration, or ALB) to <method> <route>, e.g. GET /pets/{id}. ALB reports no route, so those spans are named after the bare method (e.g. GET). Off by default, so existing span names don't change under you. Default: false.

• DASH0_API_GATEWAY_REQUEST_HEADERS_TO_CAPTURE - Comma-separated, case-insensitive list of request header names to capture as http.request.header.<name> span attributes for API Gateway- and ALB-triggered invocations. Empty by default: headers may carry PII, auth tokens, or cookies, so none are captured unless named here.

  Example: DASH0_API_GATEWAY_REQUEST_HEADERS_TO_CAPTURE=content-type,x-request-id

• DASH0_API_GATEWAY_RESPONSE_HEADERS_TO_CAPTURE - Same as above, but for the handler's response headers, captured as http.response.header.<name>.

• DASH0_CAPTURE_API_GATEWAY_QUERY_STRING - When set to true, captures the request query string as url.query for API Gateway- and ALB-triggered invocations. Off by default: query strings can carry signed-URL tokens or other sensitive values. Default: false.

• DASH0_DISABLE_PYTHON_DEPENDENCY_CHECK - Python only. On startup, the Python distribution checks whether its own dependencies conflict with the versions installed in the function, and skips loading the auto-instrumentation if they do. When set to true, that check is skipped and the distribution loads regardless. Use this if the check reports a false positive; note that a real conflict may cause the function to fail at runtime. Default: false.

• DASH0_JAVA_SDK_SELF_METRICS_ENABLED - Java only. When set to true, exports the OpenTelemetry SDK's self-metrics (otlp.exporter.seen, otlp.exporter.exported, queueSize, processedSpans, processedLogs). Default: false.

• DASH0_JAVA_RUNTIME_METRICS_ENABLED - Java only. When set to true, exports the JVM runtime metrics (jvm.*). Default: false.

Secret Masking

The extension automatically masks sensitive data in traces payloads. By default, any JSON key matching these patterns (case-insensitive) will have its value replaced with ****:

• .*pass.*
• .*key.*
• .*secret.*
• .*credential.*
• .*passphrase.*

This is applied to:

• Lambda event payloads
• Lambda response payloads
• Any http request/response payloads captured by the auto-instrumentation

Custom masking rules:

• DASH0_MASK_RULES - JSON array of regex patterns to customize which keys are masked. When set, this replaces the default patterns.

  Example: DASH0_MASK_RULES='[".*token.*", ".*auth.*", ".*private.*"]'

• DASH0_MASK_ENV_VARS - JSON array of regex patterns specifically for masking environment variables captured in traces. When not set, falls back to using DASH0_MASK_RULES (or the defaults).

  Example: DASH0_MASK_ENV_VARS='[".*PASSWORD.*", ".*API_KEY.*"]'

Secret masking in HTTP request and response payloads:

The following environment variables allow fine-grained control over secret masking in HTTP payloads captured by the auto-instrumentation. Each accepts a JSON array of regex patterns. When not set, they fall back to DASH0_MASK_RULES (or the defaults).

• DASH0_MASK_REQUEST_BODY - Regex patterns for masking keys in HTTP request bodies.

  Example: DASH0_MASK_REQUEST_BODY='[".*credit_card.*", ".*ssn.*"]'

• DASH0_MASK_REQUEST_HEADERS - Regex patterns for masking HTTP request header names.

  Example: DASH0_MASK_REQUEST_HEADERS='[".*authorization.*", ".*cookie.*"]'

• DASH0_MASK_RESPONSE_BODY - Regex patterns for masking keys in HTTP response bodies.

  Example: DASH0_MASK_RESPONSE_BODY='[".*token.*", ".*session.*"]'

• DASH0_MASK_RESPONSE_HEADERS - Regex patterns for masking HTTP response header names.

  Example: DASH0_MASK_RESPONSE_HEADERS='[".*set-cookie.*"]'

• DASH0_MASK_QUERY_PARAMS - Regex patterns for masking HTTP query parameter names.

  Example: DASH0_MASK_QUERY_PARAMS='[".*api_key.*", ".*token.*"]'

Manual Instrumentation

If you prefer to set up OpenTelemetry instrumentation yourself instead of relying on the extension's auto-instrumentation, you can use the manual layer and point your OTLP exporters to the extension's local endpoint. The extension will receive the telemetry, enrich it, and forward it to Dash0.

1. Add the manual layer to your Lambda function: arn:aws:lambda:<region>:115813213817:layer:dash0-extension-manual:<version>.
2. Configure your OTLP trace exporter to send to http://127.0.0.1:9009/v1/traces.
3. If exporting metrics, configure your OTLP metric exporter to send to http://127.0.0.1:9009/v1/metrics.
4. Make sure to flush all telemetry before the Lambda invocation completes (e.g., in a response hook or before returning the response).

Enrichment Attributes

The extension enriches telemetry data with additional attributes beyond what the auto-instrumentation provides.

API Gateway and Application Load Balancer

When a Lambda is invoked over HTTP, the extension recognizes the trigger shape and adds HTTP semantic-convention attributes to the handler span:

• REST API (v1), proxy integration - detected by a top-level httpMethod field alongside requestContext.
• HTTP API (v2), proxy integration - detected by requestContext.http alongside a top-level rawPath field.
• Application Load Balancer - detected by a requestContext.elb object. ALB events also carry a top-level httpMethod, so this check runs first to keep them from being reported as REST API (v1).

Lambda Function URLs are also covered, with no extra detection logic needed: AWS defines the Function URL invocation event as the same shape as an HTTP API (v2) proxy integration event (requestContext.http + top-level rawPath), so it's picked up by the v2 path above. The one difference is http.route, which is always $default for Function URLs, since they have no route concept of their own - requestContext.routeKey is always $default for this trigger type.

This runs independently of the in-function OpenTelemetry SDK: the extension already parses the raw invoke event and the raw return payload for every invocation (see Manual Instrumentation for how telemetry reaches the extension). As a result, these attributes appear for every supported Lambda runtime (Node.js, Python, Java, .NET, Go), and even when auto-instrumentation is disabled and the extension builds a synthetic trace itself.

Attribute	REST API (v1) source	HTTP API (v2) source	ALB source
http.request.method	httpMethod	requestContext.http.method	httpMethod
url.path	path	rawPath	path
url.scheme	always https	same	the x-forwarded-proto header
http.route	resource, as-is (e.g. /pets/{id})	requestContext.routeKey with the leading method stripped	not available
server.address / server.port	requestContext.domainName / always 443	same	the host header, split on a trailing port; port falls back to x-forwarded-port
client.address	requestContext.identity.sourceIp	requestContext.http.sourceIp	first entry of the x-forwarded-for header
network.protocol.version	parsed from requestContext.protocol (e.g. HTTP/1.1 -> 1.1)	parsed from requestContext.http.protocol	not available
http.response.status_code	statusCode from the handler's proxy-integration return payload ({ statusCode, headers, body })	same	same

ALB specifics

ALB reports no route template (there is no resource or routeKey field), so http.route is left unset rather than filled in with the high-cardinality request path. Likewise there is no protocol field, so network.protocol.version is omitted. Because ALB listeners serve plain HTTP as well as HTTPS, url.scheme follows x-forwarded-proto instead of being hardcoded to https as it is for API Gateway.

Target groups with the lambda.multi_value_headers.enabled attribute send multiValueHeaders and multiValueQueryStringParameters instead of headers and queryStringParameters, in both directions. Both shapes are handled. A header that appears multiple times is captured as a single comma-separated value, so http.request.header.<name> stays a string in either configuration.

Health-check invocations (user-agent: ELB-HealthChecker/2.0) carry no host and no forwarding headers, so they yield just http.request.method and url.path.

Request attributes and the response status code are captured unconditionally, since none of them carry PII. Three behaviors are opt-in via environment variables, each defaulting to off so nothing changes for existing users until you ask for it. The variables are named after API Gateway for backwards compatibility, but they apply to ALB-triggered invocations too:

• Renaming the span from the default handler name to <method> <route> (e.g. GET /pets/{id}) - DASH0_ENABLE_API_GATEWAY_SPAN_NAME. For ALB, where no route is available, the span is named after the bare method (e.g. GET), per the HTTP semantic conventions.
• Capturing specific request or response headers as http.request.header.<name> / http.response.header.<name> - DASH0_API_GATEWAY_REQUEST_HEADERS_TO_CAPTURE / DASH0_API_GATEWAY_RESPONSE_HEADERS_TO_CAPTURE, each an explicit allow-list rather than "capture all headers", since headers can carry auth tokens or cookies.
• Capturing the request query string as url.query - DASH0_CAPTURE_API_GATEWAY_QUERY_STRING, since query strings can carry signed-URL tokens or other secrets.

See Configuration for the exact syntax of each variable.

Span Attributes

The following attributes are added to spans by the extension (if relevant):

Attribute	Type	Description
faas.invocation_id	string	The AWS request ID of the current invocation.
faas.trigger	string	The event source that triggered the Lambda (e.g., aws:sqs, aws:dynamodb, aws:event_bridge). Extracted from the event payload.
faas.init_duration	double	The cold start initialization duration in milliseconds. Only present on cold start invocations.
dash0.faas.record_count	int	The number of records in a batch event (SQS, DynamoDB Streams, Kinesis, SNS).
dash0.faas.trigger_arn	string	The ARN of the event source (e.g., SQS queue ARN, DynamoDB stream ARN, SNS topic ARN).
dash0.faas.event_bridge_source	string	The source field from an EventBridge event.
dash0.faas.event_bridge_detail_type	string	The detail-type field from an EventBridge event.
http.request.method	string	The HTTP method. API Gateway- and ALB-triggered invocations only.
url.path	string	The request path. API Gateway- and ALB-triggered invocations only.
url.scheme	string	Always https for API Gateway; from x-forwarded-proto for ALB.
http.route	string	The matched route template, as reported by API Gateway (e.g. /pets/{id}). API Gateway-triggered invocations only - ALB reports no route.
server.address	string	The API Gateway domain name, or the host header for ALB.
server.port	int	Always 443 for API Gateway; from the host header or x-forwarded-port for ALB.
client.address	string	The caller's source IP. API Gateway- and ALB-triggered invocations only.
network.protocol.version	string	The HTTP protocol version (e.g. 1.1). API Gateway-triggered invocations only - ALB reports no protocol.
http.response.status_code	int	The statusCode from the handler's return payload. API Gateway- and ALB-triggered invocations only.

See API Gateway and Application Load Balancer above for how these are derived per trigger type, and which related attributes are opt-in.

Resource Attributes (Spans)

These attributes are added to the resource of span data:

Attribute	Type	Description
service.name	string	The service name, from OTEL_SERVICE_NAME or defaults to unknown_service.
process.environment_variable.<KEY>	string	Lambda environment variables (with sensitive values masked). Added to the span resource.

Log Attributes

The following attributes are added to log records by the extension (if relevant):

Attribute	Type	Description
faas.invocation_id	string	The AWS request ID, used to correlate logs with the invocation span.
dash0.faas.payload_type	string	The type of payload log record. Values: lambda_event, lambda_return_value, http_request_body, http_response_body. Only present on payload log records.

Resource Attributes (Logs)

These attributes are added to the resource of log data:

Attribute	Type	Description
cloud.platform	string	Always set to aws_lambda.
cloud.resource_id	string	The full ARN of the Lambda function.
cloud.account.id	string	The AWS account ID.
service.name	string	The service name, from OTEL_SERVICE_NAME or defaults to unknown_service.

Metrics

The extension creates the following histogram metrics for each Lambda invocation:

Metric	Unit	Description
faas.invoke_duration	ms	Duration of the invocation.
faas.init_duration	ms	Duration of the cold start initialization. Only present on cold start invocations.
dash0.faas.billed_duration	ms	Billed duration of the invocation.
faas.mem_usage	MB	Memory used by the invocation.

Custom Metrics

Besides the metrics created by the extension, functions can record their own metrics through the OpenTelemetry API. How this works depends on the runtime.

Java

With auto-instrumentation, Java functions can record their own metrics through the OpenTelemetry API (for example GlobalOpenTelemetry.getMeter("my-app").counterBuilder("orders.processed").build().add(1)). The agent exports them to the extension with an OTLP exporter that is enabled by default, and flushes them when the handler returns. Add io.opentelemetry:opentelemetry-api to your function as a regular runtime dependency (implementation in Gradle, the default compile scope in Maven); the agent only bundles a private copy. Set OTEL_METRICS_EXPORTER=none to turn metric export off.

The agent's JVM runtime metrics (jvm.*) are disabled by default. To enable them, set DASH0_JAVA_RUNTIME_METRICS_ENABLED=true. The OpenTelemetry SDK's own self-metrics about its exporter and span processor (otlp.exporter.seen, otlp.exporter.exported, queueSize, processedSpans, processedLogs) are also not exported by default; set DASH0_JAVA_SDK_SELF_METRICS_ENABLED=true to export them.

Resource Attributes (Metrics)

These attributes are added to the resource of metric data:

Attribute	Type	Description
cloud.platform	string	Always set to aws_lambda.
cloud.resource_id	string	The full ARN of the Lambda function.
cloud.account.id	string	The AWS account ID.
service.name	string	The service name, from OTEL_SERVICE_NAME or defaults to unknown_service.

Serverless Framework Plugin

For Serverless Framework users, the serverless-dash0 plugin can automatically add the Dash0 layer and configuration to your Lambda functions. See the plugin README for setup instructions and configuration options.

Dockerized Lambdas

For containerized Lambda functions, use the provided Docker images in a multi-stage build. The extension images are available for Node.js, Python, and Java runtimes.

Node.js

FROM public.ecr.aws/lambda/nodejs:20

# Copy extension from Dash0 image
COPY --from=public.ecr.aws/r1r2x3i6/extension-node:latest /opt /opt

# Enable tracing
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/wrapper
ENV DASH0_TOKEN=your-token-here

# Copy your function code
COPY index.js ${LAMBDA_TASK_ROOT}

CMD ["index.handler"]

Python

FROM public.ecr.aws/lambda/python:3.12

# Copy extension from Dash0 image
COPY --from=public.ecr.aws/r1r2x3i6/extension-python:latest /opt /opt

# Enable tracing
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/wrapper
ENV DASH0_TOKEN=your-token-here

# Copy your function code
COPY app.py ${LAMBDA_TASK_ROOT}

CMD ["app.handler"]

Java

FROM public.ecr.aws/lambda/java:21

# Copy extension from Dash0 image
COPY --from=public.ecr.aws/r1r2x3i6/extension-java:latest /opt /opt

# Enable tracing
ENV AWS_LAMBDA_EXEC_WRAPPER=/opt/wrapper
ENV DASH0_TOKEN=your-token-here

# Copy your function code
COPY target/my-function.jar ${LAMBDA_TASK_ROOT}

CMD ["com.example.Handler::handleRequest"]