fix(api,common): escalate low-confidence history votes to local decide tier (#1094)

[todie]

Summary

Closes #1094

Unfreezes the decision flywheel once the [decisions] index has BM25 support for common stop words by escalating split/low-confidence history votes (winning_share < escalate_below) to DecideTier::Local when the local tier is armed.

Changes

• [decisions] escalate_below (engine/crates/xerj-common/src/config.rs, engine/xerj.default.toml):
  • Added escalate_below: f64 (default 0.8, matching the gate's confidence bar; 0.0 disables confidence escalation) and validated 0.0..=1.0 & finiteness in Config::validate().
  • Updated the user-facing setting count invariants (132 → 133) across config.rs, lib.rs, xerj.default.toml, and product_experience.rs.
• Tier resolution & write-back (engine/crates/xerj-api/src/systemone_api.rs):
  • Extended HistorySupport::Supported(f64) to carry the winning vote share (0.0..=1.0).
  • Updated pure resolve_tier(support: &HistorySupport, escalate_below: f64, local_enabled: bool) -> Option<DecideTier> so HistorySupport::Supported(share) escalates to Some(DecideTier::Local) when local_enabled && *share < escalate_below, while preserving Some(DecideTier::History) when !local_enabled or *share >= escalate_below.
  • Wired HistorySupport::Supported(winning_share) and cfg.escalate_below into both POST /v1/systemone and POST /_decide, so escalated answers run through the local DeBERTa NLI head and cache their judgements into [decisions] index without writing back tier-1 history votes.

Verification

• cargo test --manifest-path engine/Cargo.toml -j 2 -p xerj-common --lib config
• cargo test --manifest-path engine/Cargo.toml -j 2 -p xerj-api --lib systemone_api
• cargo test --manifest-path engine/Cargo.toml -j 2 -p xerj-api --features decide-local --test decide_flywheel --test systemone_local_decide
• cargo test --manifest-path engine/Cargo.toml -j 2 -p xerj-engine --test product_experience

[xerj-org]

Independent review: approve (with fixes, carried in #1503)

Thank you @todie, this is a careful change. resolve_tier stays pure and table-tested, the escalation runs only when the local tier is armed, escalate_below = 0.0 is a clean off switch, the validation rejects NaN and out-of-range values, and the settings-count invariants were all updated. I checked the claims against the code: on both /_decide and /v1/systemone, a split vote below the floor reaches the local head and is written back, and a vote at or above the floor stays on history. Your new e2e test fails without the change and passes with it.

One real defect came up in review. The fix is in #1503, together with your two commits (authorship kept):

1. Repeat escalations appended duplicate cached answers. Once a split vote escalates, the same question asked again is often still split, because its own cached answer is only one neighbour among several. So it escalates again, and since the write-back used a random UUID, each repeat added another document voting for the local head's label. On the fixture, asking "refund money back check" twice left 4 docs in the index (2 seeded plus 2 identical cached answers). Repetition alone could therefore out-vote the history, human corrections included. The fix gives each write-back the deterministic id decide-<xxh3-128 of the text>, so re-deciding a text overwrites its one cached answer. The new test re_escalating_the_same_question_keeps_one_cached_answer fails on the original commits and passes with the fix.

Smaller items:

2. docs/DECISIONS.md said history always wins when it has support. It now documents the escalation exception, its off switch, and escalate_below in the config block. It also adds a caveat: the local head is near chance on choice labels it was not trained on (per DECIDE_MODEL.md), so operators with such indices should lower escalate_below or turn it off. That is a quality trade-off of the 0.8 default worth knowing about. It is not a blocker, because the local tier itself is opt-in.
3. "Closes decision flywheel: write-back freezes once the history index has BM25 support — 12 of 2,000 fill answers cached #1094" became "Refs decision flywheel: write-back freezes once the history index has BM25 support — 12 of 2,000 fill answers cached #1094". The change only affects nodes with the local tier armed, and the issue's Banking77 replay gate has not been re-measured.
4. Merging main: feat(server): support multi-instance execution with dynamic port and data-dir allocation #1487 added server.auto_port and also took the count from 132 to 133, and git merged the two identical "133" lines silently. The true total is now 134, and the carry branch updates every place it is quoted.

Note for later: CI does not build the decide-local feature, so the escalation e2e tests run only locally. I ran them on the merged tree, and they pass.

Gates on the merged tree: scoped tests for xerj-api (lib, plus decide_flywheel and systemone_local_decide with decide-local), xerj-common and xerj-engine product_experience all pass. fmt is clean, and clippy -D warnings is clean with and without decide-local. ES-YAML: 1399 passed, 0 failed, 5 skipped.

[xerj-org]

Thank you @todie! This landed via #1503 (merged as c4c2084), with your two commits carried and your authorship intact. On top of them we added: a deterministic write-back id, so a question that escalates more than once keeps a single cached answer instead of piling up votes for the local head; documentation of escalate_below in docs/DECISIONS.md; the settings count reconciled to 134 after #1487; and "Closes" changed to "Refs #1094", because the Banking77 replay gate still needs a re-measure. The full details are in the review comment above. Thanks again for a careful, well-tested change.