Performance Tuning

Benchmark data and sizing rationale for running agent-sandbox-controller under high claim rates.

Configuration is the canonical flag reference — every flag named here is defined there, and its High-Throughput & Scale Tuning section has a quick-start Deployment manifest. This guide adds what the reference deliberately leaves out: the benchmark evidence behind those recommendations, how to size each knob from your own traffic profile, and where each knob helps, is neutral, or can hurt.

All flags are on the agent-sandbox-controller binary. Defaults are safe for small deployments; everything below is opt-in. Flags not covered here (readiness grace period, unschedulable recheck interval, warm pool eviction, pprof) are in the flag reference.

Before tuning: confirm warm adoption applies

Most latency numbers below concern warm adoption — a SandboxClaim binding a pre-created sandbox from a SandboxWarmPool. Two claim fields disable warm adoption entirely: a claim that sets spec.env or spec.volumeClaimTemplates always cold-starts a fresh sandbox from the pool's template, even when the pool has ready capacity, because those customizations cannot be applied to an already-running warm pod. (spec.additionalPodMetadata does not have this effect — it is applied to the adopted sandbox in place.) If your claims set either field, pool sizing and refill shaping will not improve their latency.

Burst adoption vs. sustained throughput (and multi-pool sharding)

When planning capacity or comparing benchmark numbers, distinguish between two operating regimes:

Regime	What is on the critical path	Typical throughput ceiling
Burst warm adoption (draining an already-Ready warm pool)	Optimistic-lock `Sandbox` claim binding + `SandboxClaim` status update (no Pod creation or scheduling during the burst)	~300 claims/s per cluster at p90 ≤ 200 ms (e.g., 3,600 claims over a 12 s burst into a pre-warmed 3,700-replica pool)
Sustained warm adoption + concurrent refill (continuous claims where the pool must refill at the arrival rate)	Warm adoption plus `Sandbox` + `Pod` creation, `warmPoolExpectations` watch round-trips, `kube-scheduler` pod binding, and node container startup	~70–85 sandboxes/s per `SandboxWarmPool` (bounded by per-pool workqueue serialization, expectations-gate round-trips, and `kube-scheduler`'s default `--kube-api-qps=50`)

Why a single large `SandboxWarmPool` hits a ceiling

Per-pool workqueue serialization: SandboxWarmPoolReconciler serializes reconciles per pool key. Raising --sandbox-warm-pool-concurrent-workers only parallelizes across distinct SandboxWarmPool resources, not within a single pool.
Per-pool expectations gate: After issuing a batch of Sandbox creates, a pool's warmPoolExpectations tracker blocks further creates for that pool until the controller's informer cache observes the corresponding ADD watch events (or the 5-minute fallback timeout expires). If watch delivery lags for 10–30 s under heavy write load, a single pool freezes all replenishment for its entire replica count.
Unpaced refill bursts: With the default --sandbox-warm-pool-max-refill-rate=0, a large deficit triggers a batch of up to --sandbox-warm-pool-max-batch-size (default 300) parallel Sandbox creates at once. Because each Sandbox object is ~10 KB in etcd (inlined podTemplate + managedFields), firing hundreds of creates in a single reconcile spikes etcd MVCC growth, saturates the API server watch ring buffer, and competes with latency-critical claim adoptions.

Rule of thumb: shard large warm pools

For sustained claim rates above ~30 claims/s or warm pools larger than ~500–1,000 replicas:

Split capacity across multiple SandboxWarmPool resources (for example, 4 pools of 450 replicas instead of 1 pool of 1,800 replicas) and set --sandbox-warm-pool-concurrent-workers to match the number of active pools.
Set --sandbox-warm-pool-max-refill-rate (e.g., 50–100 per pool when watch delivery is healthy) so replenishment flows as a steady token-bucket stream rather than 300-sandbox bursts.
- Caveat under watch lag: The expectations gate admits one create batch per watch round-trip, and refillBucket caps each batch at max(1, rate) tokens (1 second of creates). Under an $L$-second watch lag, effective per-pool refill drops to $\approx \min(\text{rate}, \text{batchSize}) / L$ (for example, rate=100 with a 15 s lag yields only ~7 creates/s per pool, below the unpaced 300 / 15 = 20/s). When watch lag is present, the primary levers are pool count and --sandbox-warm-pool-max-batch-size, and the per-pool rate must be sized as $\text{rate} \ge \text{claims/s} \times L / \text{pools}$ rather than a flat 50–100.
Isolate watch streams with --separate-watch-connection=true and --api-connections=4 so mutating write traffic cannot starve the informer watch events that clear each pool's expectations gate.

Benchmark data

These numbers come from two sources:

PR-level benchmarks (kops, k8s 1.35.6, e2-standard-16 control plane): the authoritative warm-adoption numbers, run against a dedicated cluster under controlled conditions.

GKE live tests (k8s 1.36.2, 20× e2-standard-16 worker nodes, 75-claim burst from a 150-sandbox warm pool): two rounds of A/B comparison run against the same live cluster. Pod cold-start was ~42–50 s p50 on this cluster; warm-adoption latency (sub-second when the pool is healthy) is covered by the PR-level data. The relative impact of each flag combination is the signal.

Concurrency worker A/B — 75-claim burst

Worker count sweep, all other perf flags at recommended values (GKE, 75-claim burst from a 150-sandbox warm pool). Worker quadruples are sandbox/claim/warm-pool/template:

Config	p50	p90	p99	n/75
Baseline — no perf flags, 1000/1000/500/100 workers	47.3 s	85.3 s	93.8 s	75/75
Recommended workers (200/150/2/1) + all perf flags	49.8 s	89.5 s	98.2 s	75/75
Minimal workers (50/50/1/1) + all perf flags	55.6 s	95.6 s	104.5 s	75/75
High workers (1000/1000/500/100) + all perf flags	52.7 s	93.5 s	102.3 s	75/75
`--cache-label-selectors` only, baseline workers	49.5 s	89.8 s	98.5 s	75/75

What this shows:

Worker counts 200/150/2/1 are validated. The recommended values beat both high workers and minimal workers at this scale. High worker counts add unnecessary API server contention; too-few workers create a reconcile bottleneck.
50/50/1/1 has a measurable regression — p50 +11%, p90 +7% vs recommended values. There is a real floor below which under-provisioning hurts.
--cache-label-selectors is neutral on a small cluster. On clusters with

5000 total pods it reduces informer scope from O(cluster pods) to O(sandbox pods) — the benefit scales with how many non-sandbox pods the cache would otherwise hold.
Perf flags add ~5% cold-start overhead vs a no-flags baseline. This is expected: the flags reduce write contention during warm-adoption bursts, not cold pod scheduling latency.

GKE live A/B — complete results

Config	p50	p90	p99	n/75	Notes
A: Baseline (workers 1000/1000/500/100, batch 500)	50.4 s	88.2 s	96.8 s	75/75	cold-start floor
B: + `--api-connections=4 --separate-watch-connection`	51.2 s	92.9 s	102.0 s	75/75	neutral
C: + `--disable-claim-events --disable-claim-observability-annotations`	51.1 s	93.0 s	101.7 s	75/75	neutral
D: + `--sandbox-warm-pool-replenish-delay=15s`	41.5 s	74.6 s	82.2 s	75/75	neutral on its own
E: + `--sandbox-warm-pool-max-refill-rate=80`	42.2 s	75.3 s	82.7 s	75/75	−15% vs baseline
G: + `--sandbox-warm-pool-max-refill-rate=100`	41.6 s	74.6 s	82.0 s	75/75	same as rate=80
H: sustained profile below (all flags + rate=100, no delay)	41.4 s	74.2 s	82.6 s	75/75	✅ confirmed optimal
I: burst profile below (all flags + delay=20s + rate=100)	49.4 s	204.7 s	214.5 s	74/75	⚠️ 1 transient timeout → pool drain

What this shows:

--sandbox-warm-pool-max-refill-rate delivers a consistent ~15% reduction in p50/p90 by smoothing the pod-creation burst and reducing scheduler contention. Rate=80 and rate=100 perform identically, so the exact value matters less than enabling it.
--api-connections, --separate-watch-connection, and write-reduction flags are neutral on per-claim latency at this scale (75 concurrent claims). Their value appears at higher scale where the API server concurrency ceiling is hit.
--sandbox-warm-pool-replenish-delay is neutral in isolation (config D). However, when combined with other flags (config I), a single transient API timeout during claim creation (1/75 failed) allowed the pool to drain — because the delay prevented immediate refill — producing a catastrophic p90 of 205 s. Do not enable this flag unless you can guarantee zero claim creation failures under load.
Config H (the sustained profile below) is confirmed optimal across all 75/75 claims with no failures.

PR-level benchmark — warm-adoption latency

Under a 45/s Poisson arrival rate across 4 pools (113 replicas/ns) on kops:

Config	p50	p90	p99
Burst-tuned (`replenish-delay=20s`, no rate cap)	~57 ms first ~10 s, then ~1.4 s cold	2.28 s steady-state	—
Sustained-tuned (`replenish-delay=0`, `max-refill-rate=100`)	92 ms	182 ms	376 ms

The sustained config held p50 flat at 77–116 ms across all six 10-second windows with no pool drain. Per-pool refill ceiling is ~70–85 sandboxes/s (pod scheduling bounds at ~70/s at the kube-scheduler default --kube-api-qps=50).

PR-level benchmark — write-behind window

Two-configuration A/B (300-claim warm burst + 45/s × 60s sustained Poisson, 12-node kops cluster, e2-standard-16 control plane):

Config	burst p50/p90	sustained p50/p90	pod PATCHes ok	optimistic 409s
Baseline (`window=0`)	1465 / 3201 ms	320 / 680 ms	2,953	5,447
`--sandbox-write-behind-window=250ms`	1271 / 3158 ms	507 / 1027 ms	2,997	3,077

Key observations:

44% fewer 409 conflicts (5,447 → 3,077) — the most reliable benefit at scale.
Burst p50 −13% (1465 → 1271 ms) — write coalescing frees API seats for claim adoption.
Sustained p50 +58% (320 → 507 ms, p90 680 → 1027 ms) — a significant regression under continuous churn; see the write-behind section for when the trade-off is worth it.
+10% sandbox reconcile count (one extra deferred pass per coalesced write) at ~+2.5% total writes.
Pod patch bound is always ≤ min(window, 1s) — the safe-to-evict strip cannot lag the cluster autoscaler.

API connection tuning

Flag reference: API Transport and Connection Settings.

The kube-apiserver caps concurrent in-flight HTTP/2 streams per connection (100 by default, configurable server-side via --http2-max-streams-per-connection), so a single connection bounds the controller's effective request concurrency regardless of worker count. --api-connections=N shards non-watch traffic (writes, uncached reads, events, leader election) round-robin across N dedicated connections; --separate-watch-connection isolates the informer watch streams from write-burst congestion. Enable them together.

When they matter: live testing showed both are neutral on per-claim latency at 75 concurrent claims. The benefit appears when concurrent in-flight requests approach the per-connection stream limit (~100) — i.e. large worker counts with sustained bursts — where a shared connection also starves watch delivery and can wedge warm-pool refill on the expectations gate (see High-Throughput & Scale Tuning).

--kube-api-qps / --kube-api-burst: the default QPS of -1 means no client-side throttle — the server's APF policy controls admission instead, which is the preferred approach (see APF insulation below). If you must apply client-side throttling, set --kube-api-burst ≥ the total worker count (sandbox + claim + warm-pool + template); the controller logs a startup warning when it detects a positive QPS with a burst below the worker total.

API Priority and Fairness (APF) insulation

The most important server-side tuning is applying the APF insulation overlay at examples/apf-insulation/apf-insulation.yaml. This gives the controller dedicated APF concurrency seats and splits its traffic into three priority classes so refill bursts cannot queue out claim-adoption writes.

See docs/apf-insulation.md for the full explanation, seat-sizing math, and when to apply it.

Recommended: apply on any cluster where claim rates exceed ~50/s or where other workloads share the API server.

Concurrency workers

Workers control how many reconciles run in parallel within each controller. Raising workers increases throughput but also increases API server load.

Flag	When to raise	Practical ceiling
`--sandbox-concurrent-workers`	Many sandboxes in flux simultaneously	~500; bounded by node scheduling capacity
`--sandbox-claim-concurrent-workers`	High claim arrival rate	~200; bounded by API server write throughput
`--sandbox-warm-pool-concurrent-workers`	Multiple warm pools	Size to the number of active pools — reconciles for one pool are serialized, so extra workers only help across distinct pools
`--sandbox-template-concurrent-workers`	Many templates changing frequently	rarely > 2

Worker counts 200/150/2/1 are directly benchmarked — they outperform both high workers (1000/1000/500/100) and minimal workers (50/50/1/1) at 75 concurrent claims (see the worker A/B table above). The benchmark ran a single pool; per the sizing rule, --sandbox-warm-pool-concurrent-workers=1 is equivalent for one pool.

Warning: sandbox + claim + warm-pool + template workers totaling > 1000 triggers a startup warning. The practical ceiling is usually the API server's mutating inflight limit, not the worker count.

Warm pool refill shaping

When a burst of SandboxClaims adopts warm sandboxes, the warm-pool controller immediately tries to refill the deficit. Without shaping, this refill burst competes for API server write capacity with the very claim-adoption writes it is meant to support. Flag reference: Warm Pool Replenishment Shaping.

`--sandbox-warm-pool-max-batch-size` (default: 300)

Maximum sandboxes created per reconcile round. Creates advance one observed batch per watch round-trip (the expectations gate waits for a batch's ADD events before issuing the next), so a refill deficit of D sandboxes clears in about ceil(D / batchSize) round-trips — an initial pool fill is the D = replicas case. Raising it trades round-trips for burst size and is safe at any value under the gate. When a refill rate cap is set (below), the token bucket usually bounds each batch before the batch cap does.

`--sandbox-warm-pool-max-refill-rate` (default: 0 — unpaced)

Caps the refill rate in sandboxes/second per pool via a per-pool token bucket. Turns full-deficit burst creates into a smooth stream, reducing scheduler contention. When no token is available the controller requeues for exactly when the next token accrues; partial batches advance on the watch events of their own creates. The cap paces all pool creates, including initial fill and scale-ups — live verification of a 150-replica pool at rate=100 filled in two paced batches (100, then 50 a second later).

Live testing confirmed a consistent ~15% reduction in p50 and p90 latency (baseline 50.4 s → 41.4–42.2 s p50; 88.2 s → 74.2–75.3 s p90). Rate=80 and rate=100 produced statistically identical results, so the exact value above ~50/s is not critical.

This flag is safe to enable broadly — it helps even when claims fall through to cold start, and it has no failure mode.

Sizing: set per-pool rate ≥ your steady per-pool claim arrival rate to prevent pool drain. If you have 4 pools and 80 claims/s total, 20/s per pool is sufficient; 100/s leaves headroom for spikes. The per-pool ceiling on a standard control plane is ~70–85/s. (The quick-start manifest uses rate=25, sized for a ~20 claims/s workload by the same rule.)

`--sandbox-warm-pool-replenish-delay` (default: 0 — immediate)

Defers the start of refill after pool members are adopted. While claims are still arriving, the timer re-arms; it only fires once the pool is stable for the delay duration. Initial pool fill and scale-ups are never delayed by this flag (though the rate cap above, if set, still paces them).

In isolation, this flag is neutral — live testing of replenish-delay=15s alone produced the same latency as max-refill-rate=100 alone (p50=41.5 s, p90=74.6 s). The flag's benefit only appears when warm adoption is actually occurring and the pool is large enough to absorb the full burst.

⚠️ Fragility: when combined with other flags under load, even a single transient API timeout during claim creation is enough to trigger pool drain, because the delay prevents the immediate refill that would have replenished the pool before the straggler claim arrived. In live testing, 1/75 failed claim applies caused p90 to jump from 74 s to 205 s (config I). This failure mode does not exist with replenish-delay=0.

When to use: only enable this flag if warm adoption is confirmed (claim p50 in the sub-second range) and your pool is consistently larger than your maximum burst size with headroom for transient failures. Monitor claim p90 continuously after enabling — any jump is the signature of pool drain.

The two flags compose: replenish-delay defers the start of refill; max-refill-rate shapes its flow once started. In most cases max-refill-rate alone gives the same performance with none of the fragility.

Informer cache scoping

`--cache-label-selectors` (default: false)

Scopes the Pod and Service informer caches to objects carrying the sandbox tracking label (agents.x-k8s.io/sandbox-name-hash). The controller only ever creates Pods and Services it labeled itself, so on large or shared clusters this cuts informer list/watch volume, JSON decode CPU, and cache memory from O(cluster pods) to O(sandbox pods).

Recommended: enable on clusters with > ~5000 total pods or where the controller's pod cache is a meaningful fraction of memory. Benchmarked in isolation on a small clean cluster (worker A/B table above): neutral at small scale (p50 49.5 s vs 47.3 s baseline) as expected — the benefit scales with how many non-sandbox pods the cache would otherwise track.

Caveat: externally pre-provisioned resources that rely on the agents.x-k8s.io/adoptable=true adoption path must also carry the tracking label (value = the owning sandbox's name hash) to remain visible to the controller when this flag is enabled.

Write reduction

During a claim burst, every avoidable write competes for API server capacity with latency-critical adoption writes. Flag reference: API Write and Cache Optimization.

`--disable-sandbox-events` and `--disable-claim-events` (default: false)

Disable Kubernetes Event emission from the Sandbox and SandboxClaim controllers respectively. Events (SandboxPodCreated, SandboxReady, SandboxAdopted, etc.) are informational only; removing them eliminates roughly 1–2 API writes per sandbox and claim lifecycle transition.

Live testing showed no measurable per-claim latency improvement in isolation at 75 concurrent claims; the benefit appears during large warm-pool fills, when the events API level in APF is saturated, or at claim rates above ~200/s.

`--disable-claim-observability-annotations` (default: false)

Skips the dedicated API write that persists the SandboxClaim observability annotations (controller first-observed timestamp, trace context). The values are still stamped on the in-memory object, so startup-latency metrics and trace propagation within the current process keep working.

Treat this flag as removing one write per claim, not as a guarantee the annotations never appear on the object: when a later full-object update carries the claim (e.g. the adoption update on the warm path), the stamped values are persisted along with it. The write saving holds either way.

`--sandbox-write-behind-window` (default: 0 — disabled)

Coalesces the Sandbox controller's recoverable pod metadata patch via RequeueAfter deferral. The specific write deferred is the pod label/annotation reconciliation on the warm-pool adoption path (warm-pool label prune, cluster-autoscaler.kubernetes.io/safe-to-evict strip, propagated-keys tracking). When the window is open, the reconcile pass skips the patch and returns RequeueAfter: <remaining window>; the workqueue's per-key dedup coalesces repeated redeliveries into a single flush pass that recomputes the desired state from fresh informer state and issues one merge patch.

Properties:

Sandbox readiness is never gated on the deferred write — the claim Ready condition is set from in-memory state without waiting for the flush.
Pod patch is always flushed within min(window, 1s), bounding the safe-to-evict annotation lag well below the cluster autoscaler's 10 s scan interval.
Crash-safe: the deferral clock stores timestamps only, no mutation payload. A crash merely restarts a sub-second window on the next leader's first pass.

Trade-offs (benchmarked at 250ms):

	Baseline (0)	250ms
409 write conflicts	5,447	3,077 (−44%)
Burst p50/p90	1465/3201 ms	1271/3158 ms (−13% p50)
Sustained p50/p90	320/680 ms	507/1027 ms (+58% p50)
Sandbox reconciles	+0%	+10%

The sustained regression is significant: +58% p50 (320 → 507 ms) and +51% p90 (680 → 1027 ms) under continuous churn.

Recommended: enable --sandbox-write-behind-window=250ms only when 409 write conflicts are high (watch for them in controller logs) or the workload is burst-dominated, and your latency SLO tolerates ~500 ms sustained warm-adoption p50. Leave it at 0 when sustained warm-adoption latency is the primary constraint — which is why the sustained profile below omits it.

Troubleshooting at scale & managed control planes

Symptom-to-fix table

Symptom	Root cause	Fix
Warm pool stops refilling for 10 s–5 m while claims drain the pool (`warmPoolExpectations` gate stays closed)	Watch `ADD` events for newly created `Sandbox` objects are queued behind mutating write bursts on a shared HTTP/2 connection, or `kube-apiserver` is slow decoding ~10 KB `Sandbox` watch payloads	Enable `--separate-watch-connection=true`, `--api-connections=4`, and pace/shard across multiple `SandboxWarmPool`s; ensure `ConcurrentWatchObjectDecode` (KEP-6178; beta and default-off in Kubernetes 1.31–1.36, default-on in 1.37+) is enabled on the API server
`Sandbox` watch force-closed (`410 Gone` / relist) or etcd DB grows to multiple GBs between defrags	Unpaced 300-sandbox refill bursts writing ~10 KB per `Sandbox` (inlined `podTemplate` + `managedFields`) plus `Pod` specs saturate `kube-apiserver` watch delivery and bloat etcd MVCC revisions	Pace refills with `--sandbox-warm-pool-max-refill-rate`; enable `--disable-sandbox-events`, `--disable-claim-events`, and `--disable-claim-observability-annotations`; tune the managed control plane (below)
High claim adoption latency while warm sandboxes are Ready (`workqueue` wait or APF queuing)	Warm-pool refill `POST` requests share `workload-low` in APF and starve latency-critical claim adoption `PATCH`es	Apply `examples/apf-insulation/apf-insulation.yaml` (APF Insulation) and pace refills with `--sandbox-warm-pool-max-refill-rate`
High rate of HTTP `409 Conflict` errors in controller logs during bursts	Excessive worker concurrency colliding on `Sandbox` or `Pod` `resourceVersion` updates	Keep workers near the validated `200/150` profile (avoid 1000+ workers); optionally set `--sandbox-write-behind-window=250ms` for burst-dominated workloads

Managed Kubernetes control plane checklist

Each Sandbox custom resource stores an inlined PodTemplate plus metadata.managedFields (~10 KB per object in etcd). Although the controller strips managedFields and Pod.spec in its local informer cache, every Sandbox create, adoption, or status write still serializes the full object through etcd and the kube-apiserver watch cache. At 30+ sustained claims/s (~100+ Sandbox writes/s alongside Pod writes), the Kubernetes control plane itself can become the bottleneck.

When running on a managed control plane where you cannot edit kube-apiserver or etcd flags directly, check or request the following from your cloud provider:

Dedicated / higher control-plane tier & etcd disk IOPS: Rapid multi-GB etcd database growth between defragmentations or multi-second stalls across all API server replicas during etcd defrag indicates saturated etcd disk throughput/IOPS or synchronized defragmentation. Upgrade to a dedicated/high-scale control-plane tier and verify that etcd defragmentation runs rolling (one member at a time behind leader transfer).
ConcurrentWatchObjectDecode (KEP-6178): ConcurrentWatchObjectDecode lets kube-apiserver decode ~10 KB Sandbox watch payloads concurrently instead of falling 10–15 s behind on the controller's watch stream. It is beta but default-off in Kubernetes 1.31–1.36 and becomes default-on in 1.37+ — on 1.31–1.36 managed control planes, explicitly enable it (or ask your provider to enable the feature gate).
API server inflight seats & kube-scheduler QPS: Default Kubernetes API server limits (--max-requests-inflight=400, --max-mutating-requests-inflight=200) leave only ~77 concurrency seats for agent-sandbox-critical under the APF overlay. Scaling API server inflight limits (e.g., 1500–3000 read / 500–1000 mutating) and raising kube-scheduler --kube-api-qps above its default 50 removes the control-plane concurrency and pod-scheduling ceiling (see docs/apf-insulation.md).

Benchmark-validated configurations

These are the exact configurations validated in the GKE live A/B above. The quick-start manifest in configuration.md is a fine starting point for moderate sustained workloads (~20 claims/s, single pool); the profiles below show how the same sizing rules apply at benchmark scale, and what to change for burst-dominated traffic.

Sustained high-throughput profile ✅ (validated optimal)

Traffic pattern: continuous high arrival rate (e.g. 30–100 claims/s steady). Priority: keep the pool topped up at the arrival rate; warm latency must hold across the entire window.

This is config H from the live benchmark — 75/75 claims, p50=41.4 s, p90=74.2 s, no failures across all runs. It is the default recommendation for any high-throughput deployment.

args:
  - --extensions
  # API server connections (benefit when in-flight requests approach the
  # ~100-stream per-connection limit)
  - --api-connections=4
  - --separate-watch-connection=true
  # Concurrency — validated at 75 concurrent claims; outperforms 1000/1000/500/100.
  # Size warm-pool workers to the number of active pools (benchmark: single pool).
  - --sandbox-concurrent-workers=200
  - --sandbox-claim-concurrent-workers=150
  - --sandbox-warm-pool-concurrent-workers=2
  # Warm pool: smooth refill, no delay (size rate ≥ per-pool claim arrival rate)
  - --sandbox-warm-pool-replenish-delay=0
  - --sandbox-warm-pool-max-refill-rate=100
  # Cache (benefit on clusters with >5000 total pods)
  - --cache-label-selectors=true
  # Write reduction (benefit during large pool fills or at ≥200 claims/s)
  - --disable-sandbox-events=true
  - --disable-claim-events=true
  - --disable-claim-observability-annotations=true
  # NOT set: --sandbox-write-behind-window — costs +58% sustained p50; enable
  # only for burst-dominated traffic or high 409 rates (see Write reduction)

Also apply examples/apf-insulation/apf-insulation.yaml to the cluster.

Sizing --sandbox-warm-pool-max-refill-rate: set it ≥ your per-pool steady claim arrival rate. Rate=80 and rate=100 were statistically identical in testing; setting 100 provides headroom for spikes. The per-pool ceiling is ~70–85/s on a standard control plane.

Burst profile ⚠️ (use only with confirmed prerequisites)

Traffic pattern: many claims arrive together, then the cluster is quiet. Priority: serve the burst at warm latency; refill can wait.

Prerequisites before enabling replenish-delay:

Confirm warm adoption is working (claim p50 is sub-second, not tens of seconds).
Confirm your pool is consistently larger than your maximum burst size, with headroom for transient claim creation failures under API pressure.
Have monitoring on claim p90 in place — any jump is pool drain.

If you cannot confirm all three, use the sustained profile above instead. It delivers equivalent performance without the fragility.

args:
  - --extensions
  - --api-connections=4
  - --separate-watch-connection=true
  - --sandbox-concurrent-workers=200
  - --sandbox-claim-concurrent-workers=150
  - --sandbox-warm-pool-concurrent-workers=2
  - --sandbox-warm-pool-replenish-delay=20s   # ⚠️ see prerequisites above
  - --sandbox-warm-pool-max-refill-rate=100
  - --cache-label-selectors=true
  - --disable-sandbox-events=true
  - --disable-claim-events=true
  - --disable-claim-observability-annotations=true
  - --sandbox-write-behind-window=250ms   # burst-dominated traffic: −13% burst p50, −44% 409s

Also apply examples/apf-insulation/apf-insulation.yaml to the cluster.

Applying settings

For a complete tuned Deployment manifest (selector, service account, image), see the High-Throughput Deployment Example in the flag reference. The warm-pool and claim flags only take effect on a deployment running with --extensions.

Live cluster patch: use a strategic merge patch that targets the container by name and replaces the whole args list. Unlike appending individual args with a JSON patch, this is idempotent (re-running it cannot duplicate flags) and does not assume the controller is at container index 0:

kubectl patch deployment agent-sandbox-controller \
  -n agent-sandbox-system \
  --type=strategic \
  -p '{"spec":{"template":{"spec":{"containers":[{
    "name":"agent-sandbox-controller",
    "args":[
      "--leader-elect=true",
      "--extensions",
      "--api-connections=4",
      "--separate-watch-connection=true",
      "--sandbox-concurrent-workers=200",
      "--sandbox-claim-concurrent-workers=150",
      "--sandbox-warm-pool-concurrent-workers=2",
      "--sandbox-warm-pool-max-refill-rate=100",
      "--cache-label-selectors=true",
      "--disable-sandbox-events=true",
      "--disable-claim-events=true",
      "--disable-claim-observability-annotations=true"
    ]}]}}}}'

Because the patch replaces the full list, include every flag the deployment needs (including --leader-elect and --extensions shown above), not just the tuning flags. Check the current list first with:

kubectl get deployment agent-sandbox-controller -n agent-sandbox-system \
  -o jsonpath='{.spec.template.spec.containers[?(@.name=="agent-sandbox-controller")].args}'

Navigation Menu

Repository navigation

performance-tuning.md

performance-tuning.md

Performance Tuning

Before tuning: confirm warm adoption applies

Burst adoption vs. sustained throughput (and multi-pool sharding)

Why a single large `SandboxWarmPool` hits a ceiling

Rule of thumb: shard large warm pools

Benchmark data

Concurrency worker A/B — 75-claim burst

GKE live A/B — complete results

PR-level benchmark — warm-adoption latency

PR-level benchmark — write-behind window

API connection tuning

API Priority and Fairness (APF) insulation

Concurrency workers

Warm pool refill shaping

`--sandbox-warm-pool-max-batch-size` (default: 300)

`--sandbox-warm-pool-max-refill-rate` (default: 0 — unpaced)

`--sandbox-warm-pool-replenish-delay` (default: 0 — immediate)

Informer cache scoping

`--cache-label-selectors` (default: false)

Write reduction

`--disable-sandbox-events` and `--disable-claim-events` (default: false)

`--disable-claim-observability-annotations` (default: false)

`--sandbox-write-behind-window` (default: 0 — disabled)

Troubleshooting at scale & managed control planes

Symptom-to-fix table

Managed Kubernetes control plane checklist

Benchmark-validated configurations

Sustained high-throughput profile ✅ (validated optimal)

Burst profile ⚠️ (use only with confirmed prerequisites)

Applying settings

FilesExpand file tree

performance-tuning.md

Latest commit

History

performance-tuning.md

File metadata and controls

Performance Tuning

Before tuning: confirm warm adoption applies

Burst adoption vs. sustained throughput (and multi-pool sharding)

Why a single large SandboxWarmPool hits a ceiling

Rule of thumb: shard large warm pools

Benchmark data

Concurrency worker A/B — 75-claim burst

GKE live A/B — complete results

PR-level benchmark — warm-adoption latency

PR-level benchmark — write-behind window

API connection tuning

API Priority and Fairness (APF) insulation

Concurrency workers

Warm pool refill shaping

--sandbox-warm-pool-max-batch-size (default: 300)

--sandbox-warm-pool-max-refill-rate (default: 0 — unpaced)

--sandbox-warm-pool-replenish-delay (default: 0 — immediate)

Informer cache scoping

--cache-label-selectors (default: false)

Write reduction

--disable-sandbox-events and --disable-claim-events (default: false)

--disable-claim-observability-annotations (default: false)

--sandbox-write-behind-window (default: 0 — disabled)

Troubleshooting at scale & managed control planes

Symptom-to-fix table

Managed Kubernetes control plane checklist

Benchmark-validated configurations

Sustained high-throughput profile ✅ (validated optimal)

Burst profile ⚠️ (use only with confirmed prerequisites)

Applying settings

Why a single large `SandboxWarmPool` hits a ceiling

`--sandbox-warm-pool-max-batch-size` (default: 300)

`--sandbox-warm-pool-max-refill-rate` (default: 0 — unpaced)

`--sandbox-warm-pool-replenish-delay` (default: 0 — immediate)

`--cache-label-selectors` (default: false)

`--disable-sandbox-events` and `--disable-claim-events` (default: false)

`--disable-claim-observability-annotations` (default: false)

`--sandbox-write-behind-window` (default: 0 — disabled)