[satran004]

Backport of #1353 to release/2.0.x. Fixes #1352 on 2.0.x. Part of #1339.

Clean cherry-pick -x of both main commits. release/2.0.x already has the job-cancellation code (activeJob, StaleAdaPotJobException) that the change builds on.

./gradlew :aggregates:adapot:test: 47 tests, 0 failures (JDK 21).

The 2.0.x ledger-state fault run found the bug (scenario L03: Postgres down during the epoch 36 reward calculation, preprod). I'll rerun L03 and L18 with this change.

Problem

If the database went down while an AdaPot reward job was running, saving the job's failure state also failed. The exception escaped processJob, and AdaPotJobManager.processJobs() moved on to the next job:

• the failed epoch's job was dropped from the queue (its row stayed STARTED, its adapot row kept reserves = 0);
• the next epoch was calculated from those zero reserves;
• when that failed, processJobs() returned, and no later job ran until a restart.

Past the end of the bundled expected-values file, nothing would catch the wrong reserves, and the next epoch would be saved as COMPLETED. Details and logs are in #1352.

Change

• AdaPotJobManager:
  • runUntilDone(job) runs a job until it completes, becomes stale, or is cancelled by a rollback.
  • A failure, exception or error (including e.g. StackOverflowError) is retried with backoff: 10 s, doubling, capped at 30 min. The loop no longer exits on a failure, so jobs stay in epoch order: epoch N+1 can't start before epoch N completes.
  • The job stays active while it waits, so handleRollback can still cancel it. The wait checks for a cancel every second.
  • The job thread is interrupted on shutdown (@PreDestroy).
• AdaPotJobProcessor:
  • It makes one attempt per call. Its own loop (up to 4 full runs, 5 s apart) is removed, since the manager now retries.
  • Saving the failure state is best-effort: if it fails, processJob returns false instead of throwing.

A job that keeps failing for a reason other than an outage (for example a real calculation mismatch) is now retried at most every 30 min, with an error logged each time, instead of stopping the job thread. adapot_jobs.error_message shows the last error.

Not changed here

Found in review. These existed before this change; a restart reruns the job the same way.

• With balance pruning enabled (off by default), a job that keeps failing while sync continues can later rebuild its stake snapshot from balances that pruning has already removed.
• The partition cache in PostgresPartitionManager marks an epoch's partition as created before the snapshot transaction commits. If that transaction rolls back, retries in the same JVM write to the default partition until a restart.
• A rollback that lands between take() and activateJob(), or whose database deletes then fail, isn't fully coordinated with the in-memory cancel.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MJrQUCWhJpLCQR5Asv7nDb

[beanbot-gh]

Initial automated review · Aster

Hi @satran004, thanks for the PR! 🙏 This is an automatic first check by an AI reviewer. It does not replace maintainer review and may be wrong.

Reviewed commit: e08408f9897baeaa3e0e1a1155e80068aefbd6ea

Issues found:

• aggregates/adapot/src/main/java/com/bloxbean/cardano/yaci/store/adapot/job/AdaPotJobManager.java:168 — If the database commits COMPLETED but the reply is lost, the next attempt can overwrite that epoch’s treasury and reserves with zero. Could we check whether the same job already completed before retrying?
• aggregates/adapot/src/main/java/com/bloxbean/cardano/yaci/store/adapot/job/AdaPotJobManager.java:181 — A rollback can leave the replacement job waiting up to five minutes because cancellation only sets a flag. Could we make the retry wait end on cancellation?

Not tested: I reviewed the backport and CI without running builds, tests, or an outage simulation.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[satran004]

End-to-end check of this backport, using the jar built from test/recovery-combined-2.0.x + this PR:

• L03 (Shelley, epoch 4) and L18 (Conway, epoch 163) both recover inside the same JVM, with no restart.
• The job retries at 10/20(/40) s, and every later epoch matches db-sync.

Details: #1353 (comment)

🤖 Generated with Claude Code

https://claude.ai/code/session_01MJrQUCWhJpLCQR5Asv7nDb