Skip to content

fix(propeller): clamp ArrayNode subNode delta timestamps instead of panicking - #8270

Open
BhavayGoyal wants to merge 1 commit into
flyteorg:masterfrom
BhavayGoyal:fix/array-node-delta-timestamp-overflow
Open

BhavayGoyal wants to merge 1 commit into
flyteorg:masterfrom
BhavayGoyal:fix/array-node-delta-timestamp-overflow

Conversation

@BhavayGoyal

Copy link
Copy Markdown

Tracking issue

Closes #8144

Why are the changes needed?

The ArrayNode handler stores each subNode's start offset (seconds since the ArrayNode's LastAttemptStartedAt) in SubNodeDeltaTimestamps, a CompactArray sized by MaxDeltaTimestamp (3 days by default, so 18-bit items, max 262143). The value was written unclamped:

deltaDuration := uint64(subNodeStartedAt.Time.Sub(startedAt.Time).Seconds())
arrayNodeState.SubNodeDeltaTimestamps.SetItem(index, deltaDuration)

So SetItem panics (Value [267605] is too big. Max value is [262143].) when:

  • a subNode starts more than ~3 days after the ArrayNode (big fan-out with low parallelism), or
  • the delta is negative because the ArrayNode attempt was retried/recovered and its start time moved past the subNode's; casting a negative float to uint64 gives a huge value.

The value is never stored, so the panic repeats on every round until the workflow runs out of system retries.

What changes were proposed in this pull request?

  • Added deltaTimestampSeconds in array/handler.go, which returns the delta in whole seconds saturated to [0, 2^ItemSize - 1] (the limit CompactArray.SetItem actually checks). It uses the array's own ItemSize, so it stays correct if max-delta-timestamp is changed after the array was created.
  • The only effect of clamping is that a clamped subNode's reconstructed LastAttemptStartedAt in node events is less precise. A delta of 0 already means "not set" and gets recomputed on the next round.

How was this patch tested?

  • Added TestDeltaTimestampSeconds (same time, in range, largest storable value, the 267605s value from the issue, subNode started before the ArrayNode). It checks the result and that SetItem doesn't panic on an array built with the default 3-day max. With the old expression the "beyond range" and "negative" cases panic; with the fix all pass.
  • go test ./pkg/controller/nodes/array/ passes, and golangci-lint run ./pkg/controller/nodes/array/ reports 0 issues.

Labels

  • fixed

(Written with Claude, working with @BhavayGoyal.)

…anicking

The delta between the ArrayNode and a subNode start time was stored in
SubNodeDeltaTimestamps unclamped. A subNode starting more than
MaxDeltaTimestamp after the ArrayNode, or a negative delta after the
ArrayNode attempt was retried or recovered (wrapping to a huge uint64),
made CompactArray.SetItem panic on every evaluation, so the workflow
could never make progress again.

Saturate the delta to the range the array item can hold.

Signed-off-by: Bhavay Goyal <bhavaygoyal4b@gmail.com>
@github-actions github-actions Bot added the flyte label Oct 28, 2026
@codecov

codecov Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 57.35%. Comparing base (00b50bb) to head (32d56da).

Additional details and impacted files
@@            Coverage Diff             @@
##           master    #8270      +/-   ##
==========================================
- Coverage   57.43%   57.35%   -0.09%     
==========================================
  Files         925      931       +6     
  Lines       57772    58346     +574     
==========================================
+ Hits        33184    33463     +279     
- Misses      21583    21832     +249     
- Partials     3005     3051      +46     
Flag Coverage Δ
unittests-datacatalog 53.62% <ø> (ø)
unittests-flyteadmin 53.26% <ø> (ø)
unittests-flytecopilot 48.05% <ø> (?)
unittests-flytectl 64.11% <ø> (ø)
unittests-flyteidl 76.63% <ø> (ø)
unittests-flyteplugins 60.64% <ø> (ø)
unittests-flytepropeller 53.91% <100.00%> (+0.02%) ⬆️
unittests-flytestdlib 64.40% <ø> (-0.03%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant