[satran004]

Fixes #1352. Part of #1328.

Problem

If the database went down while an AdaPot reward job was running, saving the job's failure state also failed. The exception escaped processJob, and AdaPotJobManager.processJobs() moved on to the next job:

• the failed epoch's job was dropped from the queue (its row stayed STARTED, its adapot row kept reserves = 0);
• the next epoch was calculated from those zero reserves;
• when that failed, processJobs() returned, and no later job ran until a restart.

Past the end of the bundled expected-values file, nothing would catch the wrong reserves, and the next epoch would be saved as COMPLETED. Details and logs are in #1352.

Change

• AdaPotJobManager:
  • runUntilDone(job) runs a job until it completes, becomes stale, or is cancelled by a rollback.
  • A failure, exception or error (including e.g. StackOverflowError) is retried with backoff: 10 s, doubling, capped at 30 min. The loop no longer exits on a failure, so jobs stay in epoch order: epoch N+1 can't start before epoch N completes.
  • The job stays active while it waits, so handleRollback can still cancel it. The wait checks for a cancel every second.
  • The job thread is interrupted on shutdown (@PreDestroy).
• AdaPotJobProcessor:
  • It makes one attempt per call. Its own loop (up to 4 full runs, 5 s apart) is removed, since the manager now retries.
  • Saving the failure state is best-effort: if it fails, processJob returns false instead of throwing.

A job that keeps failing for a reason other than an outage (for example a real calculation mismatch) is now retried at most every 30 min, with an error logged each time, instead of stopping the job thread. adapot_jobs.error_message shows the last error.

Not changed here

Found in review. These existed before this change; a restart reruns the job the same way.

• With balance pruning enabled (off by default), a job that keeps failing while sync continues can later rebuild its stake snapshot from balances that pruning has already removed.
• The partition cache in PostgresPartitionManager marks an epoch's partition as created before the snapshot transaction commits. If that transaction rolls back, retries in the same JVM write to the default partition until a restart.
• A rollback that lands between take() and activateJob(), or whose database deletes then fail, isn't fully coordinated with the in-memory cancel.

Tests

• AdaPotJobManagerTest:
  • through the real job thread: epoch N+1 isn't started while epoch N fails and is retried;
  • the same job is retried after an exception, a false and an Error;
  • a rollback stops the retries, both while the job throws and during the backoff (it ends the wait early);
  • interrupt ends the loop;
  • the backoff schedule and cap.
• AdaPotJobProcessorTest:
  • one attempt per call;
  • false instead of an exception when the failure state can't be saved, or when the job lookup fails.
• ./gradlew :aggregates:adapot:test: 59 tests, 0 failures (JDK 21). The job tests passed 3 more reruns.

The fault-injection scenarios L03 and L18 (#1338) cover this end to end. I'll rerun them with this change on the 2.0.x backport.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MJrQUCWhJpLCQR5Asv7nDb

[beanbot-gh]

Initial automated review · Aster

Hi @satran004, thanks for the PR! 🙏 This is an automatic first check by an AI reviewer. It does not replace maintainer review and may be wrong.

Reviewed commit: 09c2c0f140c41608c9ab56d83e94c8f8e35c45aa

Issues found:

• aggregates/adapot/src/main/java/com/bloxbean/cardano/yaci/store/adapot/job/AdaPotJobManager.java:168 — If the database saves COMPLETED but the connection drops before confirming it, the retry can reset that epoch’s AdaPot to zero. Could we skip the retry when the same job is already complete?
• aggregates/adapot/src/main/java/com/bloxbean/cardano/yaci/store/adapot/job/AdaPotJobManager.java:181 — A rolled-back job can block its replacement for up to five minutes because cancellation does not wake the retry wait. Could we check for cancellation before waiting and wake the wait on rollback?

Not tested: I read the code and tests without running a build or simulating an outage.

[satran004]

Review round 1 (before requesting review) · head 9ced88d

Three independent reviews of 09c2c0f: two Claude review agents (one on the retry loop with rollbacks and interrupts, one on whether rerunning a half-finished job is safe) and Codex. None found a blocker.

The data-safety review traced every write a job makes (AdaPot, fees, deposits, the PreAdaPot listeners, the reward replace, the stake/DRep snapshots, proposal status). It found each one an overwrite or a delete-then-insert keyed by epoch, so running a half-finished job again leaves no duplicate or double-counted rows.

Finding	Raised by	Change in 9ced88d
A failure that repeats (e.g. a db-sync mismatch) reran the whole job 4 times (the processor's own loop) every 5 min, forever	both review agents	The processor makes one attempt per call, and the manager does all retries. The backoff cap is 30 min.
InterruptedException from the processor's Thread.sleep was swallowed, so the manager kept retrying	all 3	That sleep is gone with the processor's loop. The job thread is interrupted on shutdown (@PreDestroy).
An Error (e.g. StackOverflowError) would still end the job thread	review 1	The retry loop catches Throwable.
A rollback during the backoff was noticed only when the wait ended (up to 5 min)	both review agents	The wait checks for a cancel every second.
Test gaps: N before N+1 through the real thread, Error, rollback while throwing and during the backoff, interrupt, the save itself failing	all 3	Added. The failing-save test now fails the save, not the lookup.
With balance pruning on (off by default), a long retry can rebuild the snapshot from pruned balances	Codex	Not changed: a restart reruns the job the same way. Listed in the PR under "Not changed here".
The partition cache marks an epoch as created before the snapshot transaction commits	review 2	Same.
A rollback between take() and activateJob(), or one whose deletes fail, isn't fully coordinated with the cancel	review 1	Same.

Tests: ./gradlew :aggregates:adapot:test: 58 tests, 0 failures. The job tests also passed 3 reruns. The 2.0.x backport #1354 has the same two commits: 47 tests, 0 failures.

[satran004]

Review round 2 · head 3c15ae1

A second Codex pass on 9ced88d found no production regression. It confirmed:

• The manager is the only production caller of processJob, so removing the processor's internal retries affects nothing else.
• error_message is still kept on failure and cleared on success.
• A cancel during the initial delay or the backoff is handled.
• The InterruptedException catch comes before Throwable.
• stop() is safe when the thread was never started.

It flagged one test as timing-dependent: the rollback-during-backoff test used a fixed 300 ms sleep instead of waiting for the job to start. In 3c15ae1:

• that test now waits for the first attempt;
• a new test checks that stop() ends the real job thread.

The 23 job tests passed 3 reruns on main, and pass on the 2.0.x backport #1354.

End-to-end check: a preprod instance built from test/recovery-combined-2.0.x with #1354 (ccbc427) is now syncing from genesis with the ledger-state profile. It will rerun fault scenarios L03 and L18, Postgres down 210 s during the reward calculation. Those are the scenarios that dropped the job before this fix. I'll post the result here.

[sonarqubecloud]

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[satran004]

End-to-end check on benchmark-vm: L03 and L18 both pass

Build: test/recovery-combined-2.0.x (bbac1a2) + this fix (backport #1354), jar cand2x4-bbf6b4664. Profiles ledger-state,rt, synced from genesis on preprod. Only the two fault scenarios that broke the job thread in the 2.0.x ledger-state run were injected:

• L03: Postgres down 210 s during the reward calculation (Shelley, epoch 4).
• L18: the same fault in Conway (epoch 163).

The JVM was never restarted: PID 2736717 runs through both scenarios. The RequiredRestartProcessor lines below restart the sync service inside that JVM, not the process.

	L03 (epoch 4)	L18 (epoch 163)
attempt 1	could not save the failure state → Retrying in 10 s	Reward calculation failed … (attempt 1). Retrying in 10 s
attempt 2	threw → Retrying in 20 s	threw → Retrying in 20 s
attempt 3	completed	threw → Retrying in 40 s
attempt 4	–	completed, Treasury and reserves value matched for epoch : 163
next epoch	epoch 5 starts only after 4 is COMPLETED; 5–9 matched	164–169 all matched
runner	recovered=True, ttr 6 s, no anomalies	recovered=True, ttr 34 s, no anomalies

L18 log excerpt (times are UTC+1):

18:47:11 AdaPotJobProcessor : Treasury and reserves value matched for epoch : 162
18:50:25 AdaPotJobManager   : Reward calculation failed for epoch : 163 (attempt 1). Retrying in 10 s
18:51:10 AdaPotJobManager   : Error processing reward calc job for epoch : 163 (attempt 2). Retrying in 20 s
18:52:05 AdaPotJobManager   : Error processing reward calc job for epoch : 163 (attempt 3). Retrying in 40 s
18:52:16 RequiredRestartProcessor : Sync restart completed for reason: HealthCheckFailed
18:52:51 AdaPotJobProcessor : Treasury and reserves value matched for epoch : 163
18:53:01 AdaPotJobProcessor : Treasury and reserves value matched for epoch : 164
...
18:56:15 AdaPotJobProcessor : Treasury and reserves value matched for epoch : 169

adapot_jobs after L18: epochs 4–163 COMPLETED. Epoch 164 was STARTED at that moment because it was the job running normally; it matched about 10 s later.

Without the fix, the same two scenarios left the job STARTED and stopped the job thread. The next epoch then failed Expected reserves until a restart.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MJrQUCWhJpLCQR5Asv7nDb