Skip to content

Report unsupported token accounting for Cursor native RunSSE - #9429

Merged
lpcox merged 3 commits into
mainfrom
copilot/api-proxy-fix-token-usage-accounting
Oct 4, 2026
Merged

lpcox merged 3 commits into
mainfrom
copilot/api-proxy-fix-token-usage-accounting

Conversation

Copilot AI commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Cursor inference succeeds through the custom OpenAI upstream, but native RunSSE responses without recognized usage leave telemetry empty. Authoritative native usage and executed-model metadata remain unverified; missing counts must not imply zero cost or an emitter failure.

  • Explicit accounting state

    • Emit TRACK_END result=unsupported_accounting and a structured warning for 2xx RunSSE streams without recognized usage.
    • Include route, protocol, status, and reason so consumers can distinguish inference from BidiAppend traffic sharing the request ID.
  • Accounting boundaries

    • Document incomplete effective-token and AI-credit budgets for unsupported responses.
    • Preserve recognized usage parsing, inference routing, and authentication. Do not emit fabricated zero counts or byte-derived estimates.
  • Regression coverage

    • Cover opaque and binary-framed streams, shared-ID BidiAppend traffic, missing usage, error responses, and existing OpenAI/Anthropic/Gemini accounting.

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix token usage accounting in Cursor RunSSE inference Report unsupported token accounting for Cursor native RunSSE Oct 4, 2026
Copilot AI requested a review from lpcox October 4, 2026 04:04
@lpcox
lpcox marked this pull request as ready for review October 4, 2026 15:44
Copilot AI balanced review requested due to automatic review settings October 4, 2026 15:44

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

Successful RunSSE responses with unsupported compression bypass the documented terminal accounting state.

Review effort: Balanced
Findings: 1 Medium severity

Open (1)
What changed in this PR

Adds explicit unsupported-accounting diagnostics for Cursor native RunSSE responses in the API proxy, without estimating token usage.

Changes:

  • Reports unsupported accounting when native streams lack recognized usage.
  • Adds regression coverage for native streams and existing provider accounting.
  • Documents telemetry interpretation and budget limitations.
File Description
docs/​api-proxy-sidecar.md Documents unsupported accounting and incomplete budgets.
containers/​api-proxy/​token-tracker.cursor.test.js Tests native diagnostics and recognized usage preservation.
containers/​api-proxy/​token-tracker-http.js Adds the accounting state and structured warning.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +396 to +397
if (streaming && typeof reqPath === 'string'
&& reqPath.split('?')[0] === '/agent.v1.AgentService/RunSSE') {
@lpcox

lpcox commented Oct 4, 2026

Copy link
Copy Markdown
Collaborator

@copilot address review feedback

Co-authored-by: lpcox <15877973+lpcox@users.noreply.github.com>

Copilot AI commented Oct 4, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address review feedback

Addressed in c560ca5: successful RunSSE responses skipped for unsupported encoding now emit unsupported_accounting with an encoding-specific reason. The regression test verifies that no token record is written.

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Documentation Preview

Documentation has been built for this PR.

Download preview artifact

To view locally:

  1. Download the docs-preview-pr-9429 artifact from the workflow run
  2. Unzip and open index.html in your browser

Built from commit 602a2d9

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

✅ Coverage Check Passed

Overall Coverage

Metric Base PR Delta
Lines 92.91% 92.91% ➡️ +0.00%
Statements 91.38% 91.38% ➡️ +0.00%
Functions 89.56% 89.56% ➡️ +0.00%
Branches 85.03% 85.03% ➡️ +0.00%
📁 Per-file Coverage Changes (2 files)
File Lines (Before → After) Statements (Before → After)
src/nvx/one-shot-adapter.ts 83.5% → 82.9% (-0.60%) 80.3% → 79.8% (-0.56%)
src/log-directory-setup.ts 96.8% → 100.0% (+3.18%) 96.9% → 100.0% (+3.13%)

Coverage comparison generated by scripts/ci/compare-coverage.ts

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed...

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

🛡️ Smoke Copilot Network Isolation confirmed the egress allowlist is enforced. ✅

🛡️ Egress verdict from Smoke Copilot Network Isolation

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

✅ Smoke Claude passed

Generated by Smoke Claude for #9429

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Chroot tests passed! Smoke Chroot - All security and functionality tests succeeded.

Tested by Smoke Chroot

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

✅ Security Guard completed successfully!

Security review complete — PR #9429 adds Cursor native RunSSE token accounting reporting with no security weakening. Changes are audit/logging only; no firewall, ACL, capability, or validation changes. No security vulnerabilities detected.

Generated by Security Guard for #9429

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

❌ Smoke Gemini reports failed. Facets need polishing...

💎 Faceted by Smoke Gemini

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

🌑 The shadows whisper... Smoke Codex failed. The oracle requires further meditation...

🔮 The oracle has spoken through Smoke Codex

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

Smoke Cloud Hypervisor completed. Cloud Hypervisor + Copilot passed.

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

📰 VERDICT: Smoke Copilot has concluded. All systems operational. This is a developing story. 🎤

📰 BREAKING: Report filed by Smoke Copilot

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

✅ Build Test Suite completed successfully!

Generated by Build Test Suite for #9429

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

📡 Smoke OTel Tracing completed. All tracing scenarios validated. ✅

📡 OTel tracing validated by Smoke OTel Tracing

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

❌ Smoke Copilot BYOK AOAI (api-key) reports failed. AOAI BYOK (api-key) mode investigation needed...

🔑 BYOK (AOAI api-key) report filed by Smoke Copilot BYOK AOAI (api-key)

@github-actions

github-actions Bot commented Oct 4, 2026 •

Copy link
Copy Markdown
Contributor

✅ Smoke Copilot BYOK completed. Copilot BYOK mode operational. 🔓

🔑 BYOK report filed by Smoke Copilot BYOK

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

  • API status: ✅ PASS
  • gh check: ✅ PASS
  • File status: ✅ PASS

Overall result: PASS

Generated by Smoke Claude for #9429 · claude · haiku45 · 49.8 AIC · ⊞ 6.2K · ◷
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct Mode) ✅ PASS

  1. ✅ GitHub MCP: PRs fetched (merged Upgrade gh-aw prerelease workflows #9435, fix(ci): pin tsx for docs audit enforcement #9430)
  2. ✅ GitHub.com HTTP: 200
  3. ✅ File Write/Read: /tmp/gh-aw/agent/smoke-test-copilot-byok.txt
  4. ✅ BYOK Inference: Running in direct BYOK mode (COPILOT_PROVIDER_API_KEY) via api-proxy → api.githubcopilot.com

Status: All tests passed. Direct BYOK inference path validated.

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Smoke Test: Cloud Hypervisor + Copilot

  1. PASS — list_pull_requests MCP call succeeded (github/gh-aw-firewall).
  2. BLOCKED — curl https://github.com could not run: sandbox denied curl execution ("Permission denied and could not request permission from user") before any network attempt, unrelated to AWF domain rules.
  3. PASS — Wrote and read back /tmp/gh-aw/agent/smoke-cloud-hypervisor-37216048847.txt.
  4. BLOCKED — curl (example.com/redacted) likewise blocked by sandbox permission gate, not testable here.

Note: checks 2/4 could not verify AWF allow/deny behavior because the bash sandbox itself denies curl network calls outright (no exit code, immediate permission error), independent of the firewall. No bypass attempted.

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Smoke Copilot: PASS

  • ✅ GitHub MCP (last merged PR: "Upgrade gh-aw prerelease workflows")
  • ✅ github.com HTTP 200
  • ✅ File write/read
    Author: @Copilot · Assignees: @lpcox @Copilot

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Smoke OTEL tracing:

  • ✅ 1 Module loading: otel.js loads and exports its functions
  • ✅ 2 Tests: 68/68 passed (3 suites)
  • ✅ 3 Env forwarding: trace context in env-passthrough.ts, OTEL vars in api-proxy-env-config.ts
  • ✅ 4 Token tracker: onUsage hook present in token-tracker-http.js
  • ⚠️ 5 Diagnostics: /tmp/gh-aw/otel.jsonl has 1 line. I didn't inspect it for exported spans.

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

@github-actions github-actions Bot added the smoke-copilot-network-isolation Copilot network-isolation egress smoke test label Oct 4, 2026
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

EGRESS_RESULT allow=pass deny=pass

  • ✅ Allowed domain (api.github.com): allowed=200
  • ✅ Blocked domain (example.com): blocked
  • Overall: PASS
  • PR author: @lpcox

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Services smoke test: PASS

  • Redis PING: ✅
  • pg_isready: ✅
  • psql SELECT 1: ✅

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@lpcox
lpcox deployed to aoai-model October 4, 2026 16:19 — with GitHub Actions Active
@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

Chroot Version Comparison

Runtime Host Version Chroot Version Match?
Python Python 3.12.14 Python 3.12.14 ✅
Node.js v24.21.0 v2.98.0 ❌
Go go1.22.12 go1.22.12 ✅

Not all tests passed: Node.js differs between host and chroot, so smoke-chroot was not added. The chroot "Node" version v2.98.0 looks unusual, so the node binary resolved inside the chroot may not be the host's Node.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

github-actions Bot commented Oct 4, 2026

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia ❌ not run ❌ FAIL
Bun hono ❌ not run ❌ FAIL
C++ fmt ✅ N/A ✅ PASS
C++ json ✅ N/A ✅ PASS
Deno oak ❌ not run ❌ FAIL
Deno std ❌ not run ❌ FAIL
.NET hello-world ✅ N/A ✅ PASS
.NET json-parse ✅ N/A ✅ PASS
Go color ✅ 1/1 pkg ok ✅ PASS
Go env ✅ 1/1 pkg ok ✅ PASS
Go uuid ✅ 1/1 pkg ok ✅ PASS
Java gson ✅ 1/1 passed ✅ PASS*
Java caffeine ✅ 1/1 passed ✅ PASS*
Node.js clsx ✅ passed ✅ PASS
Node.js execa ✅ passed ✅ PASS
Node.js p-limit ✅ passed ✅ PASS
Rust fd ✅ 1/1 passed ✅ PASS
Rust zoxide ✅ 1/1 passed ✅ PASS

Overall: 6/8 ecosystems passed — FAIL

Failures

  • Bun and Deno: the curl | bash / curl | sh installer commands were denied by the sandbox ("Permission denied and could not request permission from user"). bun and deno are not on PATH, so the tests were not run.
  • *Java: with the default ~/.m2 repository, Maven failed with LocalRepositoryNotAccessibleException. Both projects passed after I pointed Maven at a writable local repo (-Dmaven.repo.local). This is an environment issue, not a project failure.

All 8 repositories cloned successfully. The build-test label was not added because of the failures.

Generated by Build Test Suite for #9429 · copilot · auto · 25 AIC · ⊞ 11.8K · ◷
Add label ready-for-aw to run again

@lpcox
lpcox merged commit c652378 into main Oct 4, 2026
130 of 135 checks passed
@lpcox
lpcox deleted the copilot/api-proxy-fix-token-usage-accounting branch October 4, 2026 16:41

This branch was successfully deployed

1 active deployment
aoai-model — c560ca5f Deployed Oct 4, 2026 by lpcox via conclusion #1871
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[api-proxy] Cursor RunSSE inference succeeds but token usage remains unaccounted

3 participants