Skip to content

test(api-proxy): add end-to-end coverage for routed Copilot efforts none and max - #9325

Merged
lpcox merged 3 commits into
mainfrom
copilot/add-e2e-coverage-for-copilot-efforts
Oct 1, 2026
Merged

lpcox merged 3 commits into
mainfrom
copilot/add-e2e-coverage-for-copilot-efforts

Conversation

Copilot AI commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

gh-aw's Copilot harness rejected router-selected none and max efforts (github/gh-aw#64831). AWF already supports both values. This PR adds coverage across the routed request boundary so a later change can't drop or alter them without a test failing. It is test-only: the current implementation passes as-is.

What's added

containers/api-proxy/routing-efforts.test.js runs the real createRoutingController inside createProductionRoutingSession. Only the router, catalogue snapshot and executor are faked. Each case is followed through the whole chain:

catalogue → candidate pool → router pick → selection.json → getReflectState() → observeRequest telemetry

  • none / max (via describe.each)
    • Both are offered to the router as candidate choices, and the full effort set (none … max) is offered too.
    • The selected effort is kept unchanged in the selection, selection.json, the selection log record and /reflect (endpoint: /responses).
    • A request is recorded as as_selected only when model, endpoint and effort all match. Each of these is recorded as deviated with the specific mismatches listed:
      • a different effort, or no effort
      • a different model
      • /chat/completions, including when the effort is sent in the wrong shape for that endpoint
  • Omitted vs. explicit none
    • An omitted selection has no effort key; /reflect shows effort: null and /chat/completions.
    • reasoning_effort: 'none' against an omitted selection is a deviation, and so is an effortless request against a none selection.
  • Fail closed
    • The router returning any of the following gives routing_contract_error, failure.json, no selection.json, and a failed /reflect state:
      • ultra
      • null
      • a different effort on an offered id
      • no effort where one was offered, or none where it was omitted
    • Unsupported catalogue efforts (ultra, NONE) are filtered out. If none remain, the run fails with no_route.

Note for reviewers

Routing selection is advisory: a request that doesn't match is not rejected. Here "accepted only on matching requests" means the request is recorded as as_selected rather than deviated, which is what these tests assert. Actually rejecting mismatched requests would be a behaviour change and is out of scope.

The tests fail if max or none is removed from EFFORTS in routing-candidates.js or routing-contract.js. They also fail if none is mapped to the omitted-effort endpoint in routing-observation.js.

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Copilot AI changed the title [WIP] Add end-to-end coverage for routed Copilot efforts none and max test(api-proxy): add end-to-end coverage for routed Copilot efforts none and max Oct 1, 2026
Copilot AI requested a review from lpcox October 1, 2026 20:54
@lpcox
lpcox marked this pull request as ready for review October 1, 2026 20:55
Copilot AI balanced review requested due to automatic review settings October 1, 2026 20:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟢 Approval recommended

The focused test-only changes correctly cover the stated routing behavior and failure cases without altering production logic.

Review effort: Balanced
Findings: None

What changed in this PR

Adds end-to-end regression coverage ensuring routed Copilot efforts none and max remain supported across AWF’s API-proxy routing boundary.

Changes:

  • Verifies candidate selection, persistence, reflection, and request telemetry.
  • Tests omitted effort versus explicit none.
  • Confirms unsupported efforts fail closed.
File Description
containers/​api-proxy/​routing-efforts.test.js Adds comprehensive routed-effort regression tests.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

⚠️ Coverage Regression Detected

This PR decreases test coverage. Please add tests to maintain coverage levels.

Overall Coverage

Metric Base PR Delta
Lines 92.84% 92.83% 📉 -0.01%
Statements 91.34% 91.34% ➡️ +0.00%
Functions 89.37% 89.37% ➡️ +0.00%
Branches 84.55% 84.58% 📈 +0.03%
📁 Per-file Coverage Changes (9 files)
File Lines (Before → After) Statements (Before → After)
src/cloud-hypervisor/manager-start.ts 88.4% → 86.3% (-2.07%) 87.8% → 85.8% (-1.99%)
src/nvx/one-shot-adapter.ts 83.5% → 82.9% (-0.60%) 80.3% → 79.8% (-0.56%)
src/services/enclave-mcp-service.ts 97.9% → 97.8% (-0.07%) 96.0% → 95.9% (-0.13%)
src/enclave/github-gateway.ts 82.6% → 82.6% (+0.00%) 79.7% → 79.8% (+0.09%)
src/cloud-hypervisor/preflight.ts 95.9% → 95.9% (+0.02%) 96.0% → 96.0% (+0.03%)
src/microvm/network-plan.ts 93.3% → 93.6% (+0.24%) 92.7% → 93.0% (+0.25%)
src/microvm/network-reservation.ts 52.8% → 53.3% (+0.44%) 51.1% → 51.5% (+0.42%)
src/log-directory-setup.ts 96.2% → 100.0% (+3.78%) 96.3% → 100.0% (+3.71%)
src/cloud-hypervisor/cleanup-handle.ts 88.3% → 93.6% (+5.32%) 85.0% → 90.0% (+5.00%)
✨ New Files (1 files)
  • src/cloud-hypervisor/enclave-network.ts: 81.0% lines

Coverage comparison generated by scripts/ci/compare-coverage.ts

…ries

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Documentation Preview

Documentation has been built for this PR.

Download preview artifact

To view locally:

  1. Download the docs-preview-pr-9325 artifact from the workflow run
  2. Unzip and open index.html in your browser

Built from commit bb46e46

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed...

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

🛡️ Egress verdict from Smoke Copilot Network Isolation

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Copilot

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

✅ Smoke Claude passed

Generated by Smoke Claude for #9325

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

✅ Build Test Suite completed successfully!

Generated by Build Test Suite for #9325

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

❌ Smoke Gemini reports failed. Facets need polishing...

💎 Faceted by Smoke Gemini

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Smoke Cloud Hypervisor reports failed. Cloud Hypervisor + Copilot failed.

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

📡 OTel tracing validated by Smoke OTel Tracing

@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

🌑 The shadows whisper... Smoke Codex failed. The oracle requires further meditation...

🔮 The oracle has spoken through Smoke Codex

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

  • API status: ✅ PASS
  • GitHub check: ✅ PASS
  • File status: ✅ PASS

Overall result: PASS

Generated by Smoke Claude for #9325 · claude · haiku45 · 49.8 AIC · ⊞ 6.2K · ◷
Add label ready-for-aw to run again

@github-actions github-actions Bot added smoke-claude smoke-copilot-network-isolation Copilot network-isolation egress smoke test labels Oct 1, 2026
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

EGRESS_RESULT allow=pass deny=pass

  • ✅ Allowed domain (api.github.com): HTTP 200
  • ✅ Blocked domain (example.com): blocked (TLS cert error, no response)
  • Overall: PASS

cc @lpcox

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct) Mode – PASS ✅

Test Results:

  • ✅ MCP connectivity: GitHub API reachable
  • ✅ GitHub.com connectivity: HTTP 200
  • ✅ File write/read: Test file verified
  • ✅ BYOK inference: Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY via api-proxy → api.githubcopilot.com)

Status: All tests passed. Direct BYOK mode operational.

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Chroot Version Comparison

Runtime Host Version Chroot Version Match?
Python Python 3.12.14 Python 3.12.14 ✅
Node.js v24.21.0 v2.98.0 ❌
Go go1.22.12 go1.22.12 ✅

Result: not all tests passed (Node.js mismatch), so the smoke-chroot label was not added.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot: PASS ✅

  • GitHub MCP: ✅ (latest merged PR: "Filter API proxy /reflect models by policy")
  • github.com HTTP 200: ✅
  • File write/read: ✅
    cc @Copilot @lpcox

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

OTEL smoke test

  • ✅ S1 Module loading: otel.js loads and exports its functions
  • ✅ S2 Tests: 3 suites, 68 tests passed (otel)
  • ✅ S3 Env forwarding: trace context in env-passthrough.ts; OTEL vars in api-proxy-env-config.ts
  • ✅ S4 Token tracker: onUsage hook exists in token-tracker-http.js
  • ✅ S5 Diagnostics: otel.jsonl has 1 record (local re-check; I didn't see the post-step output)

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

@lpcox
lpcox deployed to aoai-model October 1, 2026 22:33 — with GitHub Actions Active
@lpcox
lpcox deployed to aoai-model October 1, 2026 22:33 — with GitHub Actions Active
@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia ❌ not run ❌ FAIL
Bun hono ❌ not run ❌ FAIL
C++ fmt ✅ N/A ✅ PASS
C++ json ✅ N/A ✅ PASS
Deno oak ❌ not run ❌ FAIL
Deno std ❌ not run ❌ FAIL
.NET hello-world ✅ N/A (run OK) ✅ PASS
.NET json-parse ✅ N/A (run OK) ✅ PASS
Go color ✅ 1/1 passed ✅ PASS
Go env ✅ 1/1 passed ✅ PASS
Go uuid ✅ 1/1 passed ✅ PASS
Java gson ❌ not run ❌ FAIL
Java caffeine ❌ not run ❌ FAIL
Node.js clsx ✅ all passed ✅ PASS
Node.js execa ✅ all passed ✅ PASS
Node.js p-limit ✅ all passed ✅ PASS
Rust fd ✅ 1/1 passed ✅ PASS
Rust zoxide ✅ 1/1 passed ✅ PASS

Overall: 5/8 ecosystems passed — FAIL

Failures:

  • Bun / Deno: all repos cloned, but the install script download (curl bun.sh/install) was denied by the sandbox ("Permission denied and could not request permission from user"). I did not work around it. Neither bun nor deno was on PATH, so no tests ran.
  • Java: Maven failed with Could not create local repository at /home/runner/.m2/repository (LocalRepositoryNotAccessibleException). ~/.m2/settings.xml was written as instructed. The ~/.m2 directory is likely not writable in this sandbox. Compile and tests did not run.

Generated by Build Test Suite for #9325 · copilot · auto · 24.2 AIC · ⊞ 11.8K · ◷
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Oct 1, 2026

Copy link
Copy Markdown
Contributor

Smoke services: ✅ Redis PONG · ✅ pg_isready accepting connections · ✅ SELECT 1 = 1 → PASS

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@lpcox
lpcox enabled auto-merge (squash) October 1, 2026 22:37
@lpcox
lpcox merged commit 0c7d7f3 into main Oct 1, 2026
127 of 134 checks passed
@lpcox
lpcox deleted the copilot/add-e2e-coverage-for-copilot-efforts branch October 1, 2026 22:41

This branch was successfully deployed

1 active deployment
aoai-model — 57751c1e Deployed Oct 1, 2026 by lpcox via conclusion #1846
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add end-to-end coverage for routed Copilot efforts none and max

3 participants