feat(api-proxy): ordered fallback models on 5xx, timeout, or model_not_supported by Copilot · Pull Request #9356 · github/gh-aw-firewall

Copilot · 2026-10-02T02:38:01Z

engine.model accepts a single model, and retries re-run that same model. When the model fails with a 5xx, a timeout, or model_not_supported, the job ends with no output. This PR adds an optional ordered fallback chain to the API proxy: on a model-specific failure, the proxy rewrites the request's model and re-sends it.

apiProxy:
  fallbackModels:
    - gpt-5.4
    - claude-sonnet-4.6

Trigger policy (`model-fallback-chain.js`)

Falls back on:
- any 5xx (504 is reported as upstream_timeout)
- a connection error before any response arrives (upstream_connection_error)
- a 400/404 whose body names a model failure: model_not_supported, model_not_found, "not accessible via the … endpoint", Anthropic not_found_error with model:, Gemini models/X is not found (reported as model_not_supported)
Never falls back on: 401, 403, 429, or a generic 400 validation error. These reach the client unchanged.
Rewrite target: the body model field for OpenAI, Anthropic, and Copilot. Gemini carries the model in the URL, so the /models/<model>:<method> path segment is rewritten instead. A redundant <provider>/ prefix on an entry is stripped.
Order: entries are tried in list order, skipping models already attempted for this request. Once the chain is exhausted, the last upstream error is returned.
Guards: each candidate must pass the same checks as the original model (model policy, retired model, multiplier cap, budgets) via getCurrentGuardChecks. Without this, the fallback chain could bypass disallowedModels.

Request/response plumbing

upstream-http.js: works out the next candidate before dispatch. The response handler receives an onModelFallback callback only when a fallback actually exists. As a result, error bodies are buffered only when a switch is possible; with no chain configured, or once it is exhausted, 5xx responses still stream straight to the client.
Connection errors: a proxyReq error before any response arrives is treated as eligible.
upstream-response.js / upstream-retry.js: 5xx and 404 responses get a new buffered path. For 400, the fallback runs as a new step in handle400WithRetry, after the existing Copilot transient retry and alias-candidate retry. The shared tail that writes a buffered error response was extracted to sendBufferedUpstreamResponse.

Observability

Per switch: a model_fallback warn log with from_model, to_model, requested_model, attempt, reason, and status.
Token usage: in the record, model now holds the model that actually served the request. A new model_fallback object (requested_model, model, attempt, reason, status) is added to the record and to schemas/token-usage.schema.json, so gh-aw can report the served model in GH_AW_INFO_MODEL.

CLI / config

Wiring: apiProxy.fallbackModels flows through the config schema, config-file.ts, the mapper, build-config.ts, and the types, and reaches the proxy as AWF_FALLBACK_MODELS (JSON array).
Direct env use: when the variable is set by hand, a comma-separated list is also accepted.
Dockerfile: the new module is added to the api-proxy COPY list.

Not covered

WebSocket (Responses API) upgrades and internal routing-classifier requests.
The proxy sets no request timeout of its own, so "timeout" relies on Squid returning 504 or the connection dropping.

Tests

Per provider: server.model-fallback-chain.test.js covers OpenAI, Anthropic, and Copilot (5xx, model-specific error, full-chain walk, no fallback on 401/403/429 or a generic 400, connection error, no-op when unconfigured) plus Gemini path rewriting.
Unit: tests for the new module, the guard-permission filtering, the token record, the config schema, and the env mapping.

Docs: docs/awf-config-spec.md §12.7 and docs/api-proxy-sidecar.md.

Fixes [awf] api-proxy: support model fallback on 5xx/model_not_supported for engine.model #9353

Copilot

Copilot review overview

🟡 Changes recommended

Anthropic detection is order-dependent, and newly buffered error responses have no memory bound.

Review effort: Balanced
Findings: 1 · 1

Open (2)

Cap buffered upstream error bodies to prevent proxy memory exhaustion · New
Make Anthropic error matching independent of JSON property order · New

What changed in this PR

Adds ordered, guarded model fallback retries to the API proxy, addressing #9353.

Changes:

Adds fallback selection, request rewriting, failure detection, and guard enforcement.
Records fallback telemetry and the serving model.
Wires configuration, schemas, documentation, Docker packaging, and tests.

File	Description
`src/types/api-proxy-model-options.ts`	Defines fallback model options.
`src/services/api-proxy-env-config.ts`	Exports fallback configuration.
`src/services/api-proxy-env-config.test.ts`	Tests environment mapping.
`src/schema.test.ts`	Tests fallback schema validation.
`src/config-mapper.ts`	Maps file configuration.
`src/config-file.ts`	Types file configuration.
`src/config-file-mapping.test.ts`	Tests configuration mapping.
`src/commands/build-config.ts`	Builds runtime configuration.
`src/awf-config-schema.json`	Updates runtime schema.
`schemas/token-usage.schema.json`	Defines fallback telemetry.
`docs/awf-config.schema.json`	Updates canonical config schema.
`docs/awf-config-spec.md`	Documents fallback semantics.
`docs/api-proxy-sidecar.md`	Adds sidecar usage guidance.
`containers/api-proxy/upstream-token.js`	Passes fallback metadata.
`containers/api-proxy/upstream-retry.js`	Handles buffered fallback responses.
`containers/api-proxy/upstream-response.js`	Routes eligible responses through fallback.
`containers/api-proxy/upstream-http.test.js`	Tests guarded dispatch behavior.
`containers/api-proxy/upstream-http.js`	Dispatches rewritten fallback requests.
`containers/api-proxy/token-tracker-http.unit.test.js`	Tests fallback records.
`containers/api-proxy/token-tracker-http.js`	Persists fallback metadata.
`containers/api-proxy/server.model-fallback-chain.test.js`	Tests provider integrations.
`containers/api-proxy/proxy-request.js`	Applies existing guards to candidates.
`containers/api-proxy/model-fallback-chain.test.js`	Tests fallback utilities.
`containers/api-proxy/model-fallback-chain.js`	Implements fallback selection and rewriting.
`containers/api-proxy/Dockerfile`	Packages the new module.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

+    if (shouldBufferForFallback) {
+      const bufferedChunks = [];
+      proxyRes.on('data', (chunk) => {
+        responseBytes += chunk.length;
+        bufferedChunks.push(chunk);
+      });


+  /\b(?:unknown|invalid|unsupported) model\b/i,
+  /\bmodel\b[^"\n]{0,120}?\b(?:does not exist|not found|is not supported|not supported|is not available|not available|has been deprecated|is deprecated|is retired)\b/i,
+  /\bmodels\/[^\s"]+ is not found\b/i,
+  /"type"\s*:\s*"not_found_error"[^}]*"message"\s*:\s*"model:/i,


lpcox · 2026-10-02T03:24:23Z

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, apply the suggestion exactly.

Do not make changes beyond what is described in the linked review thread.

Copilot · 2026-10-02T03:31:56Z

@copilot Fix the code for all comments in this review thread.

When a review comment includes a suggested change, app...

Addressed both review findings in 932c61d: Anthropic matching now handles either JSON property order, and oversized 400/5xx error bodies are capped at 64 KiB before streaming through.

github-actions · 2026-10-02T03:38:00Z

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

github-actions · 2026-10-02T03:38:18Z

Documentation Preview

Documentation has been built for this PR.

Download preview artifact

To view locally:

Download the docs-preview-pr-9356 artifact from the workflow run
Unzip and open index.html in your browser

Built from commit 68de66f

github-actions · 2026-10-02T03:38:31Z

✅ Security Guard completed successfully!

Security review complete for PR #9356 (feat: ordered fallback models). All security-critical changes analyzed:

✅ Buffer overflow prevention: 64KB limit on fallback body inspection with streaming fallback for oversized responses
✅ Error pattern matching: Narrow regex patterns for model-specific errors only; generic errors surfaced to client
✅ Retry eligibility: 401/403/429 responses correctly excluded from model fallback
✅ Guard enforcement: Fallback models must pass same policy guards (model policy, retired models, caps, budgets)
✅ Routing classifier: Fallback logic disabled for routing classification requests
✅ Response handling: Headers and status codes properly managed; no injection vectors
✅ Loop prevention: Attempted models tracking prevents infinite retries

No security weakening detected. PR passes security review.

Generated by Security Guard for #9356

github-actions · 2026-10-02T03:38:31Z

✅ Build Test Suite completed successfully!

Generated by Build Test Suite for #9356

github-actions · 2026-10-02T03:38:31Z

❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

github-actions · 2026-10-02T03:38:32Z

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

🛡️ Egress verdict from Smoke Copilot Network Isolation

github-actions · 2026-10-02T03:38:32Z

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

🔑 BYOK report filed by Smoke Copilot BYOK

github-actions · 2026-10-02T03:38:32Z

❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed...

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

github-actions · 2026-10-02T03:38:36Z

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

github-actions · 2026-10-02T03:38:36Z

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

🔮 The oracle has spoken through Smoke Codex

github-actions · 2026-10-02T03:38:36Z

✅ Smoke Claude passed

Generated by Smoke Claude for #9356

github-actions · 2026-10-02T03:38:36Z

Smoke Cloud Hypervisor reports failed. Cloud Hypervisor + Copilot failed.

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor

github-actions · 2026-10-02T03:38:42Z

❌ Smoke Gemini reports failed. Facets need polishing...

💎 Faceted by Smoke Gemini

github-actions · 2026-10-02T03:38:43Z

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

github-actions · 2026-10-02T03:38:48Z

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

📡 OTel tracing validated by Smoke OTel Tracing

github-actions · 2026-10-02T03:39:39Z

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Copilot

github-actions · 2026-10-02T03:41:27Z

✅ Smoke Test: Copilot BYOK (Direct) Mode

Status: PASS

✅ GitHub MCP: Connected (2 recent merged PRs verified)
✅ GitHub.com: HTTP 200
✅ File Write/Read: Confirmed
✅ BYOK Inference: Active (api-proxy → api.githubcopilot.com)

Mode: Direct BYOK via COPILOT_PROVIDER_API_KEY

🔑 BYOK report filed by Smoke Copilot BYOK
_{Add label ready-for-aw to run again}

github-actions · 2026-10-02T03:41:30Z

EGRESS_RESULT allow=pass deny=pass
✅ Allowed domain (api.github.com): HTTP 200
✅ Blocked domain (example.com): blocked (curl TLS error)
Overall: PASS — cc @lpcox

🛡️ Egress verdict from Smoke Copilot Network Isolation
_{Add label ready-for-aw to run again}

github-actions · 2026-10-02T03:41:40Z

Smoke Test: Claude Engine Validation

API status: ✅ PASS
GH check: ✅ PASS
File status: ✅ PASS

Overall result: PASS

Generated by Smoke Claude for #9356 · claude · haiku45 · 24.2 AIC · ⊞ 6.2K · ◷
_{Add label ready-for-aw to run again}

github-actions · 2026-10-02T03:42:02Z

✅ Coverage Check Passed

Overall Coverage

Metric	Base	PR	Delta
Lines	92.77%	92.78%	📈 +0.01%
Statements	91.29%	91.30%	➡️ +0.01%
Functions	89.35%	89.35%	➡️ +0.00%
Branches	84.60%	84.61%	📈 +0.01%

📁 Per-file Coverage Changes (1 files)

File	Lines (Before → After)	Statements (Before → After)
`src/log-directory-setup.ts`	96.2% → 100.0% (+3.78%)	96.3% → 100.0% (+3.71%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

github-actions · 2026-10-02T03:42:16Z

OTEL smoke test results

✅ S1 Module loading: otel.js loads; exports startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled, etc.
✅ S2 Tests: 3 suites, 68 tests passed.
✅ S3 Env forwarding: trace/parent span IDs in env-passthrough.ts; OTLP vars + trace context in api-proxy-env-config.ts.
✅ S4 Token tracker: onUsage callback present in token-tracker-http.js.
✅ S5 Diagnostics: /tmp/gh-aw/otel.jsonl has 1 line; the api-proxy span export itself wasn't independently verified.

📡 OTel tracing validated by Smoke OTel Tracing
_{Add label ready-for-aw to run again}

github-actions · 2026-10-02T03:42:38Z

Smoke Copilot — PASS ✅

GitHub MCP (last merged/closed PR: "test: make cloud hypervisor tests portable"): ✅
github.com connectivity (HTTP 200): ✅
File write/read: ✅

Author: @Copilot · Assignees: @lpcox @Copilot

📰 BREAKING: Report filed by Smoke Copilot
_{Add label ready-for-aw to run again}

github-actions · 2026-10-02T03:43:01Z

Smoke services: ✅ Redis PONG · ✅ pg_isready · ✅ SELECT 1 → PASS

🔌 Service connectivity validated by Smoke Services
_{Add label ready-for-aw to run again}

github-actions · 2026-10-02T03:43:19Z

Smoke test results

GitHub PR reads: ❌
PR details: ❌
Playwright title check: ❌
File write/read: ❌
AWF build: ❌
Overall: FAIL

🔮 The oracle has spoken through Smoke Codex
_{Add label ready-for-aw to run again}

github-actions · 2026-10-02T03:43:28Z

Chroot Version Comparison

Runtime	Host Version	Chroot Version	Match?
Python	Python 3.12.14	Python 3.12.14	✅ YES
Node.js	v24.21.0	v22.23.2	❌ NO
Go	go1.22.12	go1.22.12	✅ YES

Result: Not all tests passed (Node.js version mismatch), so the smoke-chroot label was not added.

Tested by Smoke Chroot
_{Add label ready-for-aw to run again}

github-actions · 2026-10-02T03:43:56Z

🏗️ Build Test Suite Results

Ecosystem	Project	Build/Install	Tests	Status
Bun	elysia	✅	1/1 passed	✅ PASS
Bun	hono	✅	1/1 passed	✅ PASS
C++	fmt	✅	N/A	✅ PASS
C++	json	✅	N/A	✅ PASS
Deno	oak	N/A	1/1 passed	✅ PASS
Deno	std	N/A	1/1 passed	✅ PASS
.NET	hello-world	✅	N/A	✅ PASS
.NET	json-parse	✅	N/A	✅ PASS
Go	color	✅	1/1 passed	✅ PASS
Go	env	✅	1/1 passed	✅ PASS
Go	uuid	✅	1/1 passed	✅ PASS
Java	gson	✅	1/1 passed	✅ PASS (see note)
Java	caffeine	✅	1/1 passed	✅ PASS (see note)
Node.js	clsx	✅	all passed	✅ PASS
Node.js	execa	✅	all passed	✅ PASS
Node.js	p-limit	✅	all passed	✅ PASS
Rust	fd	✅	1/1 passed	✅ PASS
Rust	zoxide	✅	1/1 passed	✅ PASS

Overall: 8/8 ecosystems passed — PASS

Note: the first Java run failed with Could not create local repository at ~/.m2/repository. ~/.m2 is root-owned in the runner, so this is an environment issue, not a firewall one. Rerunning with -Dmaven.repo.local=<tmp dir> passed both projects.

Generated by Build Test Suite for #9356 · copilot · auto · 18.7 AIC · ⊞ 11.8K · ◷
_{Add label ready-for-aw to run again}

Initial plan

aefd923

Copilot AI assigned Copilot and lpcox Oct 2, 2026

Copilot AI linked an issue Oct 2, 2026 that may be closed by this pull request

[awf] api-proxy: support model fallback on 5xx/model_not_supported for engine.model #9353

Closed

Copilot started work on behalf of lpcox October 2, 2026 02:38 View session

feat(api-proxy): add ordered fallback models chain

0d32c3b

Copilot AI changed the title ~~[WIP] Add model fallback support for engine.model in api-proxy~~ feat(api-proxy): ordered fallback models on 5xx, timeout, or model_not_supported Oct 2, 2026

Copilot finished work on behalf of lpcox October 2, 2026 02:55

Copilot AI requested a review from lpcox October 2, 2026 02:55

lpcox marked this pull request as ready for review October 2, 2026 03:19

Copilot AI balanced review requested due to automatic review settings October 2, 2026 03:19

Copilot started reviewing on behalf of lpcox October 2, 2026 03:20 View session

Copilot AI reviewed Oct 2, 2026

View reviewed changes

Copilot started work on behalf of lpcox October 2, 2026 03:25 View session

fix(api-proxy): cap fallback error buffering

932c61d

Copilot finished work on behalf of lpcox October 2, 2026 03:32

lpcox added the ready-for-aw label Oct 2, 2026

lpcox enabled auto-merge (squash) October 2, 2026 03:38

lpcox had a problem deploying to aoai-model October 2, 2026 03:38 — with GitHub Actions Failure

github-actions Bot added smoke-copilot-byok smoke-copilot-network-isolation Copilot network-isolation egress smoke test labels Oct 2, 2026

github-actions Bot added the smoke-claude label Oct 2, 2026

lpcox deployed to aoai-model October 2, 2026 03:42 — with GitHub Actions Active

github-actions Bot added the smoke-copilot label Oct 2, 2026

github-actions Bot added the smoke-services label Oct 2, 2026

github-actions Bot added the build-test label Oct 2, 2026

lpcox merged commit b1da8ea into main Oct 2, 2026
133 of 137 checks passed

lpcox deleted the copilot/awf-api-proxy-support-model-fallback branch October 2, 2026 03:50

github-actions Bot mentioned this pull request Oct 2, 2026

[deps-release-notes] awf — upstream release action items githubnext/ado-aw#2252

Open

Conversation

Copilot AI commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Trigger policy (model-fallback-chain.js)

Request/response plumbing

Observability

CLI / config

Not covered

Tests

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Copilot review overview

🟡 Changes recommended

Uh oh!

lpcox commented Oct 2, 2026

Uh oh!

Copilot AI commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Documentation Preview

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Oct 2, 2026

✅ Smoke Test: Copilot BYOK (Direct) Mode

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

github-actions Bot commented Oct 2, 2026

Smoke Test: Claude Engine Validation

Uh oh!

github-actions Bot commented Oct 2, 2026

✅ Coverage Check Passed

Overall Coverage

Uh oh!

github-actions Bot commented Oct 2, 2026

Uh oh!

Copilot AI commented Oct 2, 2026 •

edited

Loading

Trigger policy (`model-fallback-chain.js`)

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading

github-actions Bot commented Oct 2, 2026 •

edited

Loading