Skip to content

Improve mempool retry resilience and SECP256k1 test setup - #3830

Merged
mkoura merged 4 commits into
masterfrom
fix/mempool-submission-backoff
Nov 6, 2026
Merged

mkoura merged 4 commits into
masterfrom
fix/mempool-submission-backoff

Conversation

@mkoura

@mkoura mkoura commented Nov 6, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Replace fixed submit-api retries with configurable exponential backoff, jitter and a retry budget. Preserve rejection diagnostics and propagate transport failures.
  • Cache SECP256k1 payment addresses with refill thresholds in both raw and build tests. Separate issuer funding from expected script failures.
  • Restore the missing PlutusV3 rejection assertion and update stale error expectations.
  • Add configurable resource monitoring, defaulting to 60-second snapshots, with node-thread scheduling counters, cgroup throttling and available GC/CPU metrics.

Testing

  • All 48 SECP256k1 E2E cases passed across the raw run and corrected build-module rerun.
  • 526 unit tests/doctests passed.
  • Linters passed.
  • Monitoring smoke-tested with legacy/modern endpoint configs and live four-node metrics.
  • Sustained CPU-contention reproduction was not performed.

Replace fixed retry counts with exponential backoff and equal jitter.
Use a configurable elapsed-time budget and maximum delay, retry only
explicit MempoolTxTooSlow rejections, and retain the last rejection in
diagnostics. Transport failures continue to propagate.

Cover recovery, deadline exhaustion, backoff caps, invalid settings,
and non-retryable responses with deterministic unit tests.
Cache payment and issuer addresses with source refill thresholds in
both raw and build tests. Fund fresh minting inputs in fixtures so
funding failures cannot be caught as expected script rejections.

Keep mempool timeouts outside relaxed error-message assertions and
restore the missing PlutusV3 assertion with current CEK error messages.
All 48 cases passed across the raw run and corrected build rerun.
Exercise changing rejection messages during submit-api retries and
assert that exhaustion reports the latest rejection, not the first.
Check warning logs retain each rejection and its selected backoff.

The framework unit tests and utility doctests passed.
Sample resources at a configurable interval, defaulting to 60 seconds.
Record node thread scheduling counters, visible cgroup quotas and
throttling, and available GC/CPU metrics from configured endpoints.
Support legacy hasPrometheus and modern PrometheusSimple backends.

Validate the interval before destructive setup and document controlled
lower-concurrency reruns without changing global concurrency limits.
Validate monitoring with smoke checks and live four-node snapshots.
Linters passed.
@mkoura
mkoura requested a review from saratomaz as a code owner November 6, 2026 16:12
@mkoura
mkoura merged commit 14c7203 into master Nov 6, 2026
3 checks passed
@mkoura
mkoura deleted the fix/mempool-submission-backoff branch November 6, 2026 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant