Skip to content

fix: bound jobstore copy connections per process - #25429

Merged
Git-on-my-level merged 1 commit into
mainfrom
fix/jobstore-process-pools
Nov 15, 2026
Merged

Git-on-my-level merged 1 commit into
mainfrom
fix/jobstore-process-pools

Conversation

@Git-on-my-level

@Git-on-my-level Git-on-my-level commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

What changed and why

Concurrent cold privacy purges created a new two-session pool per invocation, multiplying connections outside the namespace cache. The environment-selected jobstore factory now shares two normal sessions and one reserved erasure session across namespaces and cached/cold calls in each process. Normal saturation cannot consume deletion capacity; failures still raise into durable retry, and retargeting requires a process restart.

This is a per-process bound. Fleet admission, operating budgets and production rollout remain separate; no workflow, deployment, IAM, migration or production setting changes are included.

Product invariants affected

none

How it was verified

  • Ten focused unit files, each in a separate pytest process: 297 passed. The new counting-connector regression covers normal saturation, simultaneous cached/cold purges, connection/configuration/SQL/partial failures, cancellation and retry.
  • Disposable PostgreSQL16.16 plus loopback Firestore emulator: purge11, index8 and production-copy29 passed (48 total), with required backends and no skips. Both services cleaned up. All data synthetic; no production calls.
  • Image source-closure check passes all19 registered images. A direct connection-module import avoids traversing the package facade's unrelated exports. Repository make preflight at head 68120ee8846fd9644faae90a02012c9cb94c194c passes41 checks. No gate bypassed.

Tests

backend/tests/unit/test_jobstore_copy_connections.py drives the real pool and factory. backend/tests/integration/test_jobstore_purge_postgres.py now exercises erasure through a separate reserved connection, including mode-off cold calls. Existing index/cohort and current-source retirement suites pass.

UI evidence

UI-Evidence: none -- internal backend resource ownership only.

Failure class (fixes)

Failure-Class: new

Failure-class transition narrative

The violated contract is a shared resource ceiling multiplied by per-call pool construction. The new FC-per-call-pools-multiply-shared-capacity definition names the shared process-owned factory and concurrent counting-connector test as the canonical prevention. This patch provides the first source fix and guard; no previous merged PR is asserted as evidence.

View guided diff Turn on auto-fix

@Git-on-my-level
Git-on-my-level merged commit 97074e7 into main Nov 15, 2026
68 checks passed
@Git-on-my-level
Git-on-my-level deleted the fix/jobstore-process-pools branch November 15, 2026 21:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant