Skip to content

Commit 2e6b695

Browse files
lpcoxCopilot
lpcox
and
Copilot
authored
ci: exercise a two-candidate routing decision in model-routing smoke (#9245)
* ci: exercise a two-candidate routing decision in model-routing smoke Allow claude-haiku-4.5 and gpt-5.4-mini together so the router must choose. The agent sends the choice reported by /reflect and gets a reply; a second run sends the non-selected candidate and asserts the 403 model_routing_mismatch rejection and AWF exit 78. Refs #9238 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9549be0-1ad6-4e3e-b641-b2f5f0485b1b * ci: temporarily allow model-routing smoke on branch Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9549be0-1ad6-4e3e-b641-b2f5f0485b1b * fix(cli): keep exit 78 when routing fails after cleanup A routing mismatch is detected after the workflow has already torn down containers and removed the work directory. The fatal handler then ran cleanup again; `docker compose down` in the removed directory threw an uncaught ENOENT and AWF exited 1 instead of the routing-failure code 78. Run cleanup at most once per invocation, and never let a fatal-path cleanup error replace the routing exit code. Refs #9238 Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9549be0-1ad6-4e3e-b641-b2f5f0485b1b * ci: restore main-only guard on model-routing smoke Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9549be0-1ad6-4e3e-b641-b2f5f0485b1b --------- Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: f9549be0-1ad6-4e3e-b641-b2f5f0485b1b
1 parent 1504d69 commit 2e6b695

3 files changed

Lines changed: 283 additions & 102 deletions

File tree

‎.github/workflows/test-model-routing.yml‎

Lines changed: 115 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -37,7 +37,7 @@ jobs:
3737
name: Exercise live routing and Copilot inference
3838
if: github.ref == 'refs/heads/main'
3939
runs-on: ubuntu-latest
40-
timeout-minutes: 20
40+
timeout-minutes: 30
4141
env:
4242
AGENT_IMAGE: ghcr.io/github/gh-aw-firewall/agent:0.28.29@sha256:edcf17ae63dd74366bc911a74678b9e264d66ac51c48ec156c80e2619892ebbb
4343
API_PROXY_IMAGE: ghcr.io/github/gh-aw-firewall/api-proxy:0.28.29@sha256:5cc683af8156b39c15bd2370615a85775a8b179bed9f49c490a3068d667dfa2b
@@ -204,6 +204,120 @@ jobs:
204204
sudo --preserve-env=COPILOT_GITHUB_TOKEN,GITHUB_WORKSPACE,HOME,PATH \
205205
node dist/cli.js --config "${ROUTING_DIR}/awf-config.yml" -- bash -euc "$ROUTED_COMMAND"
206206
207+
- name: Route between two Copilot candidates and enforce the choice
208+
env:
209+
COPILOT_GITHUB_TOKEN: ${{ github.token }}
210+
run: |
211+
set -euo pipefail
212+
ROUTING_DIR="${RUNNER_TEMP}/model-routing-multi"
213+
mkdir -m 700 -p "$ROUTING_DIR"
214+
cat > "${ROUTING_DIR}/conversation.json" <<'JSON'
215+
[{"role":"user","parts":[{"text":"Reply with exactly ROUTED_OK."}]}]
216+
JSON
217+
cat > "${ROUTING_DIR}/awf-config.yml" <<YAML
218+
experimental:
219+
modelRouting: true
220+
network:
221+
allowDomains:
222+
- api.githubcopilot.com
223+
apiProxy:
224+
enabled: true
225+
allowedModels:
226+
- github-copilot/claude-haiku-4.5
227+
- github-copilot/gpt-5.4-mini
228+
routing:
229+
objective:
230+
goal: cost
231+
mode: balanced
232+
task:
233+
conversationFile: ${ROUTING_DIR}/conversation.json
234+
container:
235+
images:
236+
agent: ${AGENT_IMAGE}
237+
apiProxy: ${API_PROXY_IMAGE}
238+
squid: ${SQUID_IMAGE}
239+
router: ${ROUTER_IMAGE}
240+
YAML
241+
242+
# ROUTING_TARGET=selected sends the routed choice; ROUTING_TARGET=other
243+
# sends the candidate the router did not pick, which must be rejected.
244+
ROUTED_COMMAND="$(cat <<'SCRIPT'
245+
set -euo pipefail
246+
test -n "${COPILOT_API_URL:-}" || { echo "COPILOT_API_URL is missing" >&2; exit 1; }
247+
reflect="$(curl --fail --silent --show-error --max-time 30 "${COPILOT_API_URL}/reflect")"
248+
plan="$(node -e '
249+
const assert = require("node:assert/strict");
250+
const routing = JSON.parse(process.argv[1]).routing;
251+
assert.equal(routing?.status, "selected", "/reflect did not report a routed selection: " + JSON.stringify(routing));
252+
const s = routing.selection;
253+
const candidates = {
254+
"github-copilot/claude-haiku-4.5": {
255+
path: "/chat/completions",
256+
body: { model: "claude-haiku-4.5", messages: [{ role: "user", content: "Reply with exactly ROUTED_OK." }], max_tokens: 16, stream: false },
257+
},
258+
"github-copilot/gpt-5.4-mini": {
259+
path: "/responses",
260+
body: { model: "gpt-5.4-mini", input: "Reply with exactly ROUTED_OK.", reasoning: { effort: s.effort ?? "low" }, max_output_tokens: 2048, stream: false },
261+
},
262+
};
263+
assert.ok(Object.hasOwn(candidates, s.model), "router selected a model outside the allow-list: " + s.model);
264+
assert.equal(s.endpoint, candidates[s.model].path, "unexpected endpoint for " + s.model);
265+
console.error("ROUTED_SELECTION=" + JSON.stringify(s));
266+
const model = process.env.ROUTING_TARGET === "other"
267+
? Object.keys(candidates).find(model => model !== s.model)
268+
: s.model;
269+
console.log(JSON.stringify({ model, ...candidates[model] }));
270+
' "$reflect")"
271+
path="$(node -e 'process.stdout.write(JSON.parse(process.argv[1]).path)' "$plan")"
272+
body="$(node -e 'process.stdout.write(JSON.stringify(JSON.parse(process.argv[1]).body))' "$plan")"
273+
target="$(node -e 'process.stdout.write(JSON.parse(process.argv[1]).model)' "$plan")"
274+
if [ "${ROUTING_TARGET}" = other ]; then
275+
rejected="$(mktemp)"
276+
status="$(curl --silent --show-error --max-time 60 -o "$rejected" -w '%{http_code}' \
277+
-H "Content-Type: application/json" --data "$body" "${COPILOT_API_URL}${path}")"
278+
node -e '
279+
const assert = require("node:assert/strict");
280+
assert.equal(process.argv[1], "403", "non-selected candidate was not rejected");
281+
const error = JSON.parse(require("node:fs").readFileSync(process.argv[3], "utf8")).error;
282+
assert.equal(error?.code, "model_routing_mismatch");
283+
console.log("ROUTED_ENFORCEMENT=rejected model=" + process.argv[2]);
284+
' "$status" "$target" "$rejected"
285+
rm -f "$rejected"
286+
exit 0
287+
fi
288+
response="$(curl --fail --silent --show-error --max-time 180 \
289+
-H "Content-Type: application/json" --data "$body" "${COPILOT_API_URL}${path}")"
290+
node -e '
291+
const assert = require("node:assert/strict");
292+
const result = JSON.parse(process.argv[1]);
293+
const text = result.choices?.[0]?.message?.content ?? (result.output || [])
294+
.filter(item => item.type === "message")
295+
.flatMap(item => item.content || [])
296+
.filter(part => part.type === "output_text")
297+
.map(part => part.text)
298+
.join("");
299+
assert.match(String(text), /ROUTED_OK/, "Copilot did not return the expected smoke response: " + JSON.stringify(result).slice(0, 500));
300+
console.log("ROUTED_MULTI_CANDIDATE_INFERENCE=passed model=" + process.argv[2]);
301+
' "$response" "$target"
302+
SCRIPT
303+
)"
304+
305+
echo "::group::Routed choice is admitted"
306+
sudo --preserve-env=COPILOT_GITHUB_TOKEN,GITHUB_WORKSPACE,HOME,PATH \
307+
node dist/cli.js --config "${ROUTING_DIR}/awf-config.yml" --env ROUTING_TARGET=selected -- bash -euc "$ROUTED_COMMAND"
308+
echo "::endgroup::"
309+
310+
echo "::group::Non-selected candidate is rejected"
311+
set +e
312+
sudo --preserve-env=COPILOT_GITHUB_TOKEN,GITHUB_WORKSPACE,HOME,PATH \
313+
node dist/cli.js --config "${ROUTING_DIR}/awf-config.yml" --env ROUTING_TARGET=other -- bash -euc "$ROUTED_COMMAND"
314+
status=$?
315+
set -e
316+
echo "::endgroup::"
317+
# A mismatched request is a terminal routing failure, reported as exit 78.
318+
test "$status" -eq 78 || { echo "expected AWF exit 78 after a routing mismatch, got ${status}" >&2; exit 1; }
319+
echo "ROUTED_MISMATCH_EXIT=78"
320+
207321
- name: Show Docker state on failure
208322
if: failure()
209323
run: |

‎src/commands/main-action.test.ts‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -878,6 +878,40 @@ describe('createMainAction', () => {
878878
expect(processExitSpy).toHaveBeenCalledWith(78);
879879
});
880880

881+
it('does not repeat cleanup and keeps exit 78 when routing fails after the workflow cleaned up', async () => {
882+
mockedCliWorkflow.runMainWorkflow.mockImplementation(
883+
async (_config, _deps, callbacks) => {
884+
callbacks.onContainersStarted?.();
885+
await callbacks.performCleanup();
886+
throw new RoutingFailureExitError('Model routing failed (model_routing_mismatch): Routed execution was rejected');
887+
}
888+
);
889+
mockedDockerManager.stopContainers.mockResolvedValue(undefined);
890+
mockedDockerManager.cleanup.mockResolvedValue(undefined);
891+
892+
const action = createMainAction(getOptionValueSource);
893+
await action(['echo hi'], {});
894+
895+
expect(mockedDockerManager.stopContainers).toHaveBeenCalledTimes(1);
896+
expect(processExitSpy).toHaveBeenCalledWith(78);
897+
});
898+
899+
it('keeps exit 78 when cleanup after a routing failure throws', async () => {
900+
mockedCliWorkflow.runMainWorkflow.mockImplementation(
901+
async (_config, _deps, callbacks) => {
902+
callbacks.onContainersStarted?.();
903+
throw new RoutingFailureExitError('Model routing selection timed out');
904+
}
905+
);
906+
mockedDockerManager.stopContainers.mockRejectedValueOnce(new Error('spawn docker ENOENT'));
907+
908+
const action = createMainAction(getOptionValueSource);
909+
await action(['echo hi'], {});
910+
911+
expect(mockedLogger.warn).toHaveBeenCalledWith('Cleanup after a fatal error failed.', expect.any(Error));
912+
expect(processExitSpy).toHaveBeenCalledWith(78);
913+
});
914+
881915
it('stops containers during cleanup when workflow fails after startup callbacks', async () => {
882916
mockedCliWorkflow.runMainWorkflow.mockImplementation(
883917
async (_config, _deps, callbacks) => {
@@ -1111,6 +1145,21 @@ describe('createMainAction', () => {
11111145
expect(mockedDockerManager.cleanup).toHaveBeenCalled();
11121146
});
11131147

1148+
it('buildCleanupFn tears down only once across repeated calls', async () => {
1149+
const performCleanup = testHelpers.buildCleanupFn(
1150+
MAIN_ACTION_STUB_CONFIG,
1151+
() => true,
1152+
() => true,
1153+
);
1154+
1155+
await Promise.all([performCleanup(), performCleanup('SIGTERM')]);
1156+
await performCleanup();
1157+
1158+
expect(mockedDockerManager.stopContainers).toHaveBeenCalledTimes(1);
1159+
expect(mockedHostIptables.cleanupHostIptables).toHaveBeenCalledTimes(1);
1160+
expect(mockedDockerManager.cleanup).toHaveBeenCalledTimes(1);
1161+
});
1162+
11141163
it('preserves audits after an enclave drain failure', async () => {
11151164
mockedEnclaveGateway.shutdownEnclaveGateway.mockRejectedValueOnce(
11161165
new Error('drain failed')

0 commit comments

Comments
 (0)