Skip to content

Forward each invocation's logs before Lambda freezes, with an E2E of the AWS Lambda docs - #5

Merged
PetrHeinz merged 14 commits into
mainfrom
claude/lambda-docs-e2e
Oct 19, 2026
Merged

PetrHeinz merged 14 commits into
mainfrom
claude/lambda-docs-e2e

Conversation

@PetrHeinz

@PetrHeinz PetrHeinz commented Oct 16, 2026 •

Copy link
Copy Markdown
Member

Logs sent through this layer reached Better Stack one or two invocations late, and the last invocation's logs before the execution environment shut down never arrived. This forwards each invocation's logs before Lambda freezes the environment, and adds an E2E workflow that follows the AWS Lambda docs article exactly as written to prove it.

Why logs were late:

  • The Telemetry API hands the extension an invocation's logs only when its TIMEOUT_MS batching window closes, but Lambda freezes the environment as soon as the runtime and the extension ask for the next event. An invocation's logs reached the extension only when the next invocation thawed it.
  • The extension forwarded its queue the moment it received INVOKE, before those logs arrived, so they waited one invocation more. On a quiet function that is minutes.
  • On SHUTDOWN the extension forwarded before Lambda handed over the last logs, so those were lost.

The fix:

  • The extension also subscribes to platform events, and the HTTP listener tells platform.runtimeDone apart from the function's logs; other platform events are dropped. A batch with a malformed function log is still rejected.
  • On INVOKE, the extension waits for that invocation's platform.runtimeDone (events arrive in order, so its logs are queued by then), forwards them in one request, and only then asks for the next event. If the report never comes, it forwards anyway one second before the invocation's deadline, so the extension never makes an invocation time out.
  • The request to Better Stack times out after one second. An endpoint that does not answer holds an invocation for at most that long, and the logs stay queued for the next forward.
  • An invocation now stays open until Lambda hands over its logs, up to TIMEOUT_MS after the handler finished, plus the request to Better Stack. That time is billed, and a synchronous caller waits for it, so TIMEOUT_MS now defaults to 25 ms, Lambda's minimum, instead of 1000 ms. It is how long Lambda buffers telemetry before the hand-off inside the sandbox, not how often the extension sends to Better Stack, which stays one request per invocation. In the E2E, a handler logging two lines took 344 to 692 ms with the old extension, 1366 to 1428 ms with a 1000 ms window and 435 to 778 ms with 25 ms.

The E2E (.github/workflows/e2e.yml):

  • Reads the article's code blocks from the live page and fails when their shape changes. Everything it deploys comes from them: the layer download and publish commands, the environment variables (placeholders filled in as for a signed-in reader), the logging snippets, the pip packages.
  • Layer: the daily schedule downloads the release ZIP the article links. Pull requests and pushes to main build the checkout the way releases.yml does, so a PR tests its own changes. The workflow also understands the older build-from-source shape of the article.
  • Functions: one per runtime and architecture the article's publish command lists (nodejs22.x and nodejs24.x on x86_64 and arm64), plus python3.11 and python3.14, each with its own source. Each is invoked once as a cold start, read back for 90 s, then invoked four more times 60 s apart. Every logged line must be stored once per invocation, with its level and fields, a dt inside that invocation, in order, and received within 60 s of the invocation returning. The summary says during which invocation Better Stack received each line and lists each invocation's duration.
  • AWS: a throwaway account shared with the CloudWatch Lambda E2E of logtail/logtail-aws-lambda. The CI role of this repository may only touch functions, layers and log groups named e2e-lambda-docs-*. Every job deletes what it created and stale leftovers of its own runtime and architecture.

The tests for the fix came first in their own commits (red in CI), then the fixes (green), in #7 and #8, which this PR now contains.

🤖 Generated with Claude Code

PetrHeinz and others added 8 commits October 16, 2026 15:18
Follows https://betterstack.com/docs/logs/aws-lambda/ as written: the
article's code blocks are read from the page, the layer is built with the
article's pnpm commands (latest pnpm, pnpm 9 when that fails) and published
with its publish-layer-version command, and one function per runtime and
architecture it lists runs its logging snippet with its environment
variables. Python functions are packaged from its pip lines and run its
setup and logging snippets. Each line must be stored once, as logged, with
dt inside the invocation, in order, and received within 60 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The article's first block may now either build this repository or
download the latest release ZIP. The schedule downloads what the article
links; pull requests and pushes to main build the checkout the way
releases.yml does. The layer job prints the ZIP's file modes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A job that failed before its credentials step had its cleanup fail on
every AWS listing. Credentials now come first; without them the job
created nothing in AWS and the cleanup says so.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
One invocation never got a line stored in 15 minutes: Lambda freezes the
execution environment as soon as the invocation returns, so the layer
only hands over its lines when the environment thaws, and Python's
background flush never runs. The first read-back now waits 90 s for the
first invocation's lines, a second invocation follows, and the second
read-back waits up to 15 minutes for both invocations' lines. Rows are
assigned to the invocation their dt falls into.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
To tell a delay of the environment's initialization from one that every
invocation has, the function is now invoked once (a cold start, whose
init duration the summary shows), read back for 90 s, then invoked four
more times 60 s apart. The final read-back says for every line during
which invocation, or after which one, Better Stack received it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
PetrHeinz and others added 2 commits October 19, 2026 15:01
The article's Python step is about to show the logging calls inside
`def lambda_handler(event, context):`, ending with a flush of the Better
Stack handler. The docs job now takes either shape: bare logging calls
still become the body of lambda_handler, and a handler function is used
as written, with its name as the Lambda handler. A `<name>.flush()` call
is allowed among the logging calls.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The extension now subscribes to platform events, waits for each invocation's platform.runtimeDone (or until a second before its deadline), forwards that invocation's logs, and only then asks for the next event. Tests first in b467311 (red), fix in 97ca7c8 and a378acb.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PetrHeinz PetrHeinz changed the title E2E workflow: the AWS Lambda docs article, Node.js layer and Python Forward each invocation's logs before Lambda freezes, with an E2E of the AWS Lambda docs Oct 19, 2026
PetrHeinz and others added 3 commits October 19, 2026 15:28
A request Better Stack never answers held the extension, and so the
invocation, until the function's timeout. The forwarder must give up
after its timeout and keep the logs queued.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The forward had no timeout, so an endpoint that never answered held the
invocation until the function's timeout. The request now aborts after
REQUEST_TIMEOUT_MS (1000 ms) and the logs stay queued for the next
forward. The fallback before an invocation's deadline uses the same
margin, so a forward always ends before the deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The article's Python handler is about to wrap its logging calls in try
and flush the Better Stack handler in finally. The docs job unwraps a
plain try/finally (no except, no else) and reads the calls in both
parts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The extension waits for Lambda's telemetry hand-off at the end of every invocation, and Lambda hands the last batch over only when TIMEOUT_MS expires, so the window is added to every invocation. 25 ms keeps that wait short; Better Stack still gets one request per invocation.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@PetrHeinz
PetrHeinz marked this pull request as ready for review October 19, 2026 13:53
@PetrHeinz
PetrHeinz merged commit c86267c into main Oct 19, 2026
4 checks passed
@PetrHeinz
PetrHeinz deleted the claude/lambda-docs-e2e branch October 19, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant