Model routing: live smoke workflow has failed 28 runs in a row (test asks gpt-5.4-mini for max effort; OpenAI key rejected) #9869

Problem

.github/workflows/test-model-routing.yml (the live model-routing smoke test) has failed 28 runs in a row since 30 Sep 23:10 UTC. It runs on pushes to main that touch its own paths, weekly on Mondays at 09:00 UTC, and on manual dispatch. It never runs on pull requests and nothing alerts on failure, so the streak went unnoticed.

The latest scheduled run, 38231085580 on 12 Oct (3082d0d718), has two independent failures:

• Exercise live routing, policy, and Copilot inference fails in the step "Run an effort-routed Copilot inference through AWF" with RoutingFailureExitError: Model routing failed (unsupported_reasoning_effort): Routed execution failed (exit code 78).
• Exercise live openai routing fails in the step "Run a routed openai inference through AWF" with InfrastructureReadinessError: Model routing failed (provider_unavailable): The selected provider model catalogue is unavailable (exit code 78). The agent never starts.
• Exercise live anthropic routing passes.

Neither failure is an AWF regression. The first is a bug in the test itself. The second is a problem with the OpenAI credential.

Corrected timeline

Run 36667996894 (057293fb8b) passed and 36668364575 (45507f00a2) failed. That pair is not the real boundary. Run 36668364575 failed once in the Copilot CLI step with model_routing_mismatch, and the next three runs passed.

• Last green: 36789662060 on 4efc3ef74a (30 Sep 23:10:17).
• First red: 36789691598 on 10b907bfec, pushed 20 seconds later. The only commit between the two is Model routing: make the selection advisory instead of pinning requests #9272, which made the routing selection advisory instead of pinning requests.

The 28 failures have three parts:

1. 30 Sep 23:10 to 1 Oct 23:33 (4 runs, from the Model routing: make the selection advisory instead of pinning requests #9272 merge to 4ab4b7d9e6). These failed in "Route between two Copilot candidates and allow a deviating choice". The step sends a deliberately deviating request and expects exit code 0. It got curl: (22) The requested URL returned error: 403, then Model routing failed (model_routing_mismatch): Routed execution was rejected, and exit code 78. The job was still pulling the released api-proxy:0.28.29 image, which pinned requests, so it could not exercise Model routing: make the selection advisory instead of pinning requests #9272's advisory behaviour.
   test: refresh model routing end-to-end smoke #9344 fixed this by building the API proxy from source in the workflow. This part is resolved, but the step has not run since, because an earlier step in the same job now fails first.
2. From 2 Oct 00:46 to now (24 runs). These start with 36947669193, the merge commit of test: refresh model routing end-to-end smoke #9344 (9bc799a9b9). Every run fails in "Run an effort-routed Copilot inference through AWF", including 37974846368, 38097576854 and 38231085580. This is root cause 1 below.
3. The OpenAI job (schedule and dispatch only). It passed on the 5 Oct schedule, 37290835565, with the selection openai/gpt-5.4, effort xhigh, endpoint /responses. It failed on the 12 Oct schedule. This is root cause 2 below.

Root cause 1: the test asks gpt-5.4-mini for an effort it does not support (test bug)

#9344 added explicit effort overrides to the effort step. After the routed inference, the step sends requests to the routed model on /responses with reasoning.effort set to none, then max.

The router selects github-copilot/gpt-5.4-mini. That model advertises the GPT-5.4 effort vocabulary (none, low, medium, high, xhigh). max belongs to the Anthropic vocabulary, and no GPT model in the Copilot catalogue advertises it.

Evidence from the 12 Oct run (job 114750447024):

• The routed selection is {"provider":"copilot","model":"github-copilot/gpt-5.4-mini","wire_model":"gpt-5.4-mini","effort":"xhigh","endpoint":"/responses"}.
• The routed inference prints ROUTED_COPILOT_EFFORT_INFERENCE=passed, and the none override prints COPILOT_EFFORT_INFERENCE=passed effort=none.
• The max request then fails with curl: (22) The requested URL returned error: 400, about 26 ms after the previous request. The proxy rejected it locally and never sent it upstream.
• AWF exits with [ERROR] Fatal error: RoutingFailureExitError: Model routing failed (unsupported_reasoning_effort): Routed execution failed and exit code 78.

Before #9520 ("Reject unsupported reasoning efforts before proxying", merged 7 Oct), the proxy forwarded the same max request upstream. Copilot rejected it with a 400 after about 460 ms, and the run failed with the code invalid_request_body instead (for example 37290835565 on 5 Oct). From 37551022668, the #9520 merge (a375a8f73f), the proxy rejects the request locally as unsupported_reasoning_effort. The cause is the same in both periods. Only the place of rejection and the error code changed.

Why this was not caught:

• The job only runs on main, so the test: refresh model routing end-to-end smoke #9344 pull request never ran it. It failed on its own merge commit.
• The unit tests in containers/api-proxy/routing-efforts.test.js use a synthetic model that does advertise max.
• The step uses curl --fail, which drops the response body. The proxy's "supported values" message never reaches the log.

Why one 400 fails the whole run:

• The API proxy treats a request that names the selected provider and model as "on the selected model", even when it overrides the effort.
• Any response with status 400 or above on the selected model is recorded as a terminal routing failure (runtime-failure.json).
• verifyRoutingCompletion turns that record into exit code 78, whatever the agent's own exit code is.

Root cause 2: OpenAI rejects the repository's OpenAI key (credential, not AWF)

In the 12 Oct OpenAI job (job 114750447394), AWF logs "Waiting for model routing selection..." at 09:44:59.319. Half a second later it fails with provider_unavailable: The selected provider model catalogue is unavailable. That message means an OpenAI key was configured, but the proxy's OpenAI model discovery returned no models. On 5 Oct the same wait took about 18 seconds and succeeded.

No AWF commit between 3469b7581e (green, 5 Oct) and 3082d0d718 (red, 12 Oct) touches OpenAI key validation or model discovery.

In the same window, smoke-codex (which uses CODEX_API_KEY, falling back to OPENAI_API_KEY) changed state:

• Its last green run was 37337456029 (5 Oct 16:00).
• Every run since 37397671730 (6 Oct 01:08) has failed.
• Its log shows OpenAI rejecting the key with unexpected status 401 Unauthorized: You do not have access to the organization tied to the API key ... auth error code: invalid_organization.

The OpenAI organization or project behind the repository's key most likely lost access on 5 Oct.

The routing workflow cannot confirm this directly, because it uploads no artifacts. The API proxy's key-validation and model-fetch logs, model-routing.jsonl, token-usage.jsonl and the generated AWF config stay on the runner and are lost. The log only says "API proxy logs preserved at /tmp/api-proxy-logs-…".

Does this affect real routed workflows (gh-aw)?

• Root cause 1 affects only this test. The router only offers efforts that the model catalogue advertises, taken from Copilot's capabilities.supports.reasoning_effort. A routed selection therefore never carries an unsupported effort. A gh-aw run that uses the routed effort, as the Copilot CLI does with --reasoning-effort, does not hit this.
• Root cause 2 affects any workflow that uses the same OpenAI key, including smoke-codex. It is not specific to model routing.
• A related design point, out of scope here. Effort-only deviations count as requests "on the selected model". A real agent that sends one rejected request on the routed model therefore ends the routed run with exit code 78, even if it recovers. Examples are a sub-agent default effort the model does not support, or any other recoverable 4xx. Model routing: make the selection advisory instead of pinning requests #9272 states that a deviating request is the agent's own choice and never ends the routed run, but effort deviations are not handled that way. If maintainers agree, this deserves a separate issue.

Proposed fix

1. Copilot effort step. Only send explicit effort overrides that the routed model advertises. Take them from the model metadata the API proxy already exposes on /reflect, or from the routed selection, not from a hard-coded list.
   • If a fixed list is kept, use values valid for the GPT-5.4 family (for example none and low), not max.
   • To keep the Reject unsupported reasoning efforts before proxying #9520 behaviour covered on purpose, add a separate negative check. It should assert that an unsupported effort gets a local unsupported_reasoning_effort 400, and that the routed run then ends with exit code 78 as designed.
2. OpenAI credential. A maintainer with secret access should check and, if needed, rotate OPENAI_API_KEY (and CODEX_API_KEY) so it belongs to an OpenAI organization that has model access again.
   • Add a cheap preflight to the native-provider job that lists OpenAI models with the key, outside AWF.
   • If the preflight fails, the step should report the HTTP status and OpenAI error code, not the generic "catalogue is unavailable" message.
3. Re-run. After step 1, dispatch the workflow on main and confirm the rest of the Copilot job passes. This includes the multi-candidate step from Model routing: make the selection advisory instead of pinning requests #9272, which has not run since 2 Oct, the audit assertions and the Copilot CLI step. Report any new failure there as a separate issue.

Noticing failures sooner

These are proposals for a follow-up pull request; see Scope.

• Run on pull requests. Run the Copilot job on same-repository pull requests that change routing code: containers/api-proxy/routing-*, containers/api-proxy/reasoning-effort-*, src/routing/**, this workflow, and tests/integration/model-routing.test.ts. The job needs no external secret: it uses github.token with copilot-requests: write. Skip it on fork pull requests. This would have caught both Model routing: make the selection advisory instead of pinning requests #9272 and test: refresh model routing end-to-end smoke #9344 before merge.
• Track failures in an issue. When a scheduled or dispatched run fails, a final job should open a single tracking issue, or comment on it if it already exists, with the run link. A red weekly run is then seen the same day, not 28 runs later.
• Upload logs on failure. Upload the API proxy log directory (including model-routing.jsonl and token-usage.jsonl), the AWF audit directory and the generated AWF config as short-retention artifacts. Check that none of them contain secrets.
• Show error bodies. Print the response body when a request in the workflow fails, so a 400 shows the proxy's reason.

Scope

Keep the pull request to this one fix: make the Copilot effort probe in .github/workflows/test-model-routing.yml send only efforts the routed model supports.

• Rotating the OpenAI secret is a maintainer action, not a code change.
• Propose the detection items above in a follow-up pull request unless a maintainer asks for them here.
• Do not fix other checks that also fail on main, such as smoke-codex or anything else unrelated to this workflow.
• Do not change how AWF decides that a routed run has failed (how effort deviations or 4xx responses lead to runtime-failure.json) in this pull request.
• If later steps of the job fail once the effort step passes, open a separate issue rather than widening this pull request.