fix(engine): governor threads and drain-free spawn survive a failed spawn (#1122)

[xerj-org]

Advances #1122.

Why

The release profile is panic = "abort". A thread spawn fails in two ways, and both used to take the node down:

1. Builder::spawn returns Err on EAGAIN (thread or memory limit). std::thread::spawn and Scope::spawn turn that Err into a panic.
2. std panics inside the new thread when it cannot mmap the alternate signal stack under memory exhaustion. The caller never gets an Err. This is the xerj-drain-free abort in engine: node dies SIGABRT under memory exhaustion — std thread-spawn sigaltstack panic hits panic="abort" #1122.

Handling the Err fixes (1). The only way to avoid (2) is to not spawn while memory is exhausted. The governor's thread_spawn_safe() (rc.81) already gives that answer, but memtable.rs was the only caller that checked it.

What

xerj-common::thread_spawn is the shared helper the follow-up PRs use:

• try_spawn(name, f) never panics on a failed spawn. On failure the closure comes back in SpawnFailure::job (Builder::spawn drops it), so the caller can run it inline, retry, or drop it on purpose.
• A process-wide spawn gate (install_spawn_gate / spawn_allowed) refuses a spawn up front and hands the closure back the same way.
• spawn_with_retry(name, attempts, backoff, f) is for threads that must exist.
• The Spawn trait (StdSpawn in production) lets the tests inject a failing spawner, so the failure paths are tested without exhausting a real machine.

xerj-engine

• governor::init installs thread_spawn_safe as the gate. Storage, server and other crates then use the same RSS-watermark check without depending on the engine.

Call sites covered

Site	Before	After
engine.rs spawn_resource_sampler: xerj-mem-drain	let _ = Builder::spawn(..): a failure was dropped silently and the node ran without a breaker drain	3 attempts with 50 ms doubling back-off, then an ERROR log. MEMORY_DRAIN_TX is published only if the worker started
engine.rs: xerj-mem-sampler	same silent drop: no parent memory/disk breaker at all	retry, then ERROR ("breaker is degraded until restart")
engine.rs: xerj-memtable-sampler	same silent drop	retry, then ERROR
memtable.rs MemtableClaim::retire: xerj-drain-free	gate check, then let _ = spawn	try_spawn, which uses the same gate. On refusal or failure the returned job drops the dead generations inline, explicitly

Behaviour changes only on the spawn-failure path. With no governor installed (unit tests, library embedding), spawning is unrestricted, as before.

Tests

• xerj-common (6): a failed spawn returns the job; a closed gate never calls the spawner; a successful spawn runs the job once on the named thread; retry rides out a transient EAGAIN; retry gives up and hands the job back; the gate is re-sampled on every attempt.
• xerj-engine (2): a governor thread that cannot spawn reports false instead of aborting; a closed gate is respected.

cargo clippy -p xerj-common -p xerj-engine --all-targets -D warnings is clean, and cargo test -p xerj-common passes.

index::exact_scan_hydration_tests::exact_scan_is_bit_identical_to_the_clone_everything_scan fails locally on unmodified origin/main as well (different assertion each run). This PR does not touch it.

Not in this PR

Storage (merge.rs .expect, the WAL fsync worker using the gate), server, autoindex, and the engine's lazily built rayon pools (.build().expect(..)) are handled in follow-up PRs or listed on #1122.

🤖 Generated with Claude Code

[xerj-org]

Independent review: approve (after one follow-up fix, pushed)

Checked:

• try_spawn hands the closure back on both a gate refusal and a Builder::spawn Err. The Arc<Mutex<Option<F>>> slot is the right way to get it back, because the spawner drops the boxed wrapper.
• drain-free still drops the dead generations inline on any failure.
• MEMORY_DRAIN_TX is now published only when the drain worker actually started.
• The tests inject an EAGAIN spawner, so the failure paths are exercised without exhausting a real machine.

Defect fixed in 86f1934: spawn_governor_thread consulted the new process-wide gate, which governor::init installs as thread_spawn_safe. That gate refuses while the breaker is engaged or live RSS is at the watermark. It is the right gate for xerj-drain-free, which has an inline fallback, but the wrong one for the sampler and drain threads. Those threads are spawned once, at boot, and they are what samples RSS, engages the breaker and drains memory.

The failure mode: a node that boots above its watermark (for example after a large WAL replay under a low XERJ_MAX_PROCESS_MEMORY_MB) had all 3 attempts refused within about 150 ms. It then ran with no memory breaker, no 429 admission and no drain for the life of the process. Before this PR, those threads always started. They now ignore the soft gate, and a real spawn Err is still retried and then logged rather than aborting. The gated drain-free path is unchanged.

Verified:

• cargo test -p xerj-engine --lib: 759 passed.
• cargo test -p xerj-common thread_spawn: 6 passed.
• cargo clippy -p xerj-common -p xerj-engine --all-targets -D warnings: clean.
• ES-YAML on a private port with a throwaway data dir: 1400 passed, 0 failed, 4 skipped. The server log shows the sampler threads started.

"Advances #1122" is correct. Storage, server and the rayon pools are still open.

[cla-bot]

Thanks for this contribution — we'd like to merge it.

Before we can, we need a signed Contributor License Agreement from: @cursoragent

It is a one-time thing per contributor, not per pull request. Read CLA.md, then open a small pull request adding your GitHub username to .contributors — that pull request is your signature, since it is made from your own account.

Once it is merged, comment @cla-bot check here and this check will go green. If you believe you are already covered, comment @cla-bot check and it will re-verify.