Skip to content

Pin Gemini CLI auth type so api-proxy runs stop failing with exit 41 - #8916

Merged
lpcox merged 5 commits into
mainfrom
copilot/awf-gemini-cli-auth-fix
Sep 23, 2026
Merged

lpcox merged 5 commits into
mainfrom
copilot/awf-gemini-cli-auth-fix

Conversation

Copilot AI commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

Gemini CLI runs through --enable-api-proxy abort at startup with Invalid auth method selected. (exit 41). This is an upstream regression, not a proxy routing or placeholder-key problem.

Root cause

Since gemini-cli 0.44.0, getAuthTypeFromEnv() maps any non-empty GOOGLE_GEMINI_BASE_URL to AuthType.GATEWAY, evaluated before the GEMINI_API_KEY check. validateAuthMethod() has no gateway branch, so it falls through to return 'Invalid auth method selected.', which validateNonInteractiveAuth turns into FatalAuthenticationError (exit 41). Upstream: google-gemini/gemini-cli#27550 (open, two closed fix PRs).

AWF must set GOOGLE_GEMINI_BASE_URL to route the CLI at the sidecar, so every proxied Gemini run hits this. Confirmed against the smoke-gemini artifact from run 35862843304 (Invalid auth method selected. → exitCode=41) and by diffing contentGenerator.ts / auth.ts between v0.43.0 and v0.55.1.

Notably, the placeholder key format is irrelevant — gemini-cli performs no key-format validation anywhere, so adjusting gemini-api-key-placeholder-for-credential-isolation (as the issue proposed) would not have helped.

Changes

  • src/services/credentials/gemini-cli-settings.ts (new) — settings content, path helper, and the enable/skip predicates shared by the writer and the env builder.
  • src/chroot-home-setup.ts — writes $HOME/.awf/gemini-cli-system-settings.json into the empty chroot home volume, owned by the agent uid/gid. The host's real ~/.gemini is untouched.
  • src/services/credentials/gemini-credential-env.ts — injects GEMINI_CLI_SYSTEM_SETTINGS_PATH under the same gate (api-proxy enabled + Gemini routed + not Vertex/GCA), so the env var can never point at a file AWF didn't write.
  • Docs — troubleshooting entry and env-var row in docs/api-proxy-sidecar.md; Gemini CLI compatibility matrix in docs/environment.md.

The file AWF writes:

{
  "security": {
    "auth": {
      "selectedType": "gemini-api-key"
    }
  }
}

The CLI resolves settings.merged.security.auth.selectedType || getAuthTypeFromEnv(), and the system settings scope has the highest merge precedence, so API-key auth wins over the gateway inference while GOOGLE_GEMINI_BASE_URL is still honoured for request routing. The file sets only security.auth.selectedType, so gh-aw's workspace mcpServers block deep-merges untouched. A missing file is a silent no-op in the CLI, so non-compose backends degrade to current behaviour rather than erroring.

The pin is skipped when GOOGLE_GENAI_USE_VERTEXAI=true or GOOGLE_GENAI_USE_GCA=true — those auth types resolve before the gateway branch, and pinning API-key auth would break them.

Not covered

Issue item 4 (a live engine: gemini integration test through the sidecar) is not added — smoke-gemini.lock.yml already exercises this path end-to-end and will exercise the fix once a release image ships. Since the fix is host-side only (no containers/agent changes), it does not depend on a new agent image.

Copilot AI changed the title [WIP] Fix Gemini CLI API key rejection issue Pin Gemini CLI auth type so api-proxy runs stop failing with exit 41 Sep 23, 2026
@lpcox
lpcox marked this pull request as ready for review September 23, 2026 15:58
Copilot AI balanced review requested due to automatic review settings September 23, 2026 15:58
Copilot AI requested a review from lpcox September 23, 2026 16:02

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot review overview

🟡 Changes recommended

The unsafe file write and incorrect backend/environment handling must be fixed before approval.

Get a fresh assessment by requesting another Copilot review.

Review effort: Balanced
Findings: 1 High severity · 2 Medium severity · 1 Low severity

Open (4)
What changed in this PR

Pins Gemini CLI API-key authentication for API-proxy runs to prevent exit 41.

Changes:

  • Adds and injects AWF-owned Gemini system settings.
  • Adds authentication-mode regression tests.
  • Updates compatibility and troubleshooting documentation.
File Review
src/​workdir-setup.test.ts Tests settings-file creation and exclusions.
src/​services/​credentials/​gemini-credential-env.ts Moderate: Settings are unavailable on sbx/Cloud Hypervisor backends, and auth flags are resolved from the wrong environment source.
src/​services/​credentials/​gemini-credential-env.test.ts Tests environment injection and gating.
src/​services/​credentials/​gemini-cli-settings.ts Moderate: Vertex/GCA predicates do not use the agent’s effective configured environment.
src/​chroot-home-setup.ts Critical: The root-side write follows symlinks, permitting redirection and ownership changes to arbitrary host files.
docs/​environment.md Nit: Compatibility claims do not account for unsupported external guest backends.
docs/​api-proxy-sidecar.md Nit: Existing exit-41 guidance contradicts the new diagnosis.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread src/chroot-home-setup.ts Outdated
Comment on lines +58 to +60
fs.writeFileSync(settingsPath, buildGeminiSystemSettingsContent(), { mode: 0o644 });
fs.chownSync(settingsPath, uid, gid);
fs.chmodSync(settingsPath, 0o644);
Comment on lines +52 to +53
export function shouldPinGeminiAuthType(env: NodeJS.ProcessEnv = process.env): boolean {
return env.GOOGLE_GENAI_USE_VERTEXAI !== 'true' && env.GOOGLE_GENAI_USE_GCA !== 'true';
Comment on lines +40 to +41
extraEnv: isGeminiProxyRoutingEnabled(config) && shouldPinGeminiAuthType()
? { GEMINI_CLI_SYSTEM_SETTINGS_PATH: getGeminiSystemSettingsPath(getRealUserHome()) }
Comment thread docs/api-proxy-sidecar.md Outdated
@@ -677,6 +680,28 @@ When `GEMINI_API_KEY` is provided to the AWF runner, `GOOGLE_GEMINI_BASE_URL`, `

> **Note:** Exit code 41 ("no auth method") should no longer occur since the placeholder key satisfies the CLI's pre-flight check. If you see exit 41, verify `GEMINI_API_KEY` is exported in the AWF runner environment.
@lpcox

lpcox commented Sep 23, 2026

Copy link
Copy Markdown
Collaborator

@copilot address review feedback

Copilot AI commented Sep 23, 2026

Copy link
Copy Markdown
Contributor Author

@copilot address review feedback

Addressed the review feedback in bd8c6da: hardened the settings write, resolved auth flags from the effective agent environment, restricted the pin to Compose-backed runtimes, and updated tests/docs.

@github-actions

Copy link
Copy Markdown
Contributor

✅ Copilot review passed with no inline comments.

@copilot Add the ready-for-aw label to this PR to trigger agentic CI smoke tests.

@github-actions

Copy link
Copy Markdown
Contributor

Documentation Preview

Documentation has been built for this PR.

Download preview artifact

To view locally:

  1. Download the docs-preview-pr-8916 artifact from the workflow run
  2. Unzip and open index.html in your browser

Built from commit 33fffe8

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

🔌 Smoke Services — All services reachable! ✅

🔌 Service connectivity validated by Smoke Services

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

✅ Smoke Claude passed

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by Smoke Claude for #8916

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

✅ Smoke Gemini completed. All facets verified. 💎

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • github.com
  • play.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"
    - "play.googleapis.com"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

✨ The prophecy is fulfilled... Smoke Codex has completed its mystical journey. The stars align. 🌟

Warning

Firewall blocked 12 domains

The following domains were blocked by the firewall during workflow execution:

  • ab.chatgpt.com
  • accounts.google.com
  • api.github.com
  • clients2.google.com
  • collector.github.com
  • contentautofill.googleapis.com
  • github.com
  • github.githubassets.com
  • msfeed25.pkgs.visualstudio.com
  • update.googleapis.com
  • www.google.com
  • www.gstatic.com

[!TIP]
api.github.com is blocked because GitHub API access uses the built-in GitHub tools by default. Instead of adding api.github.com to network.allowed, use tools.github.mode: gh-proxy for direct pre-authenticated GitHub CLI access without requiring network access to api.github.com:

tools:
  github:
    mode: gh-proxy

See GitHub Tools for more information on gh-proxy mode.

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "ab.chatgpt.com"
    - "accounts.google.com"
    - "api.github.com"
    - "clients2.google.com"
    - "collector.github.com"
    - "contentautofill.googleapis.com"
    - "github.com"
    - "github.githubassets.com"
    - "msfeed25.pkgs.visualstudio.com"
    - "update.googleapis.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

✅ Build Test Suite completed successfully!

Warning

Firewall blocked 8 domains

The following domains were blocked by the firewall during workflow execution:

  • api.nuget.org
  • bun.sh
  • dc.services.visualstudio.com
  • deno.land
  • dl.deno.land
  • github.com
  • releaseassets.githubusercontent.com
  • repo.maven.apache.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.nuget.org"
    - "bun.sh"
    - "dc.services.visualstudio.com"
    - "deno.land"
    - "dl.deno.land"
    - "github.com"
    - "releaseassets.githubusercontent.com"
    - "repo.maven.apache.org"

See Network Configuration for more information.

Generated by Build Test Suite for #8916

@github-actions

Copy link
Copy Markdown
Contributor

🚀 Security Guard has started processing this pull request

@github-actions

github-actions Bot commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

❌ Smoke Copilot BYOK AOAI (Entra) reports failed. AOAI BYOK (Entra) mode investigation needed...

🪪 BYOK (AOAI Entra) report filed by Smoke Copilot BYOK AOAI (Entra)

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Claude Engine Validation

Check Result
API status ✅ PASS
gh check ✅ PASS
File status ✅ PASS

Overall result: PASS

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • api.anthropic.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.anthropic.com"

See Network Configuration for more information.

Generated by Smoke Claude for #8916 · claude · haiku45 · 55.8 AIC · ⊞ 4.7K · ◷
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Cloud Hypervisor + Copilot

  1. list_pull_requests (merged, limit 1): PASS
  2. curl https://github.com → 200: PASS
  3. Write/read temp file: PASS
  4. curl (example.com/redacted) blocked (000): PASS

All checks PASS.

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • example.com
  • github.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "example.com"
    - "github.com"

See Network Configuration for more information.

Cloud Hypervisor + Copilot smoke test by Smoke Cloud Hypervisor
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot BYOK (Direct Mode) — PASS ✅

  1. GitHub MCP: ✅ (PR feat: add NVX Phase 3f promotion evidence #8919 & chore: update model-to-API mapping (2026-09-23) #8915 verified)
  2. HTTP connectivity: ✅ (HTTP 200)
  3. File write/read: ✅ (/tmp/gh-aw/agent/smoke-test-copilot-byok.txt)
  4. BYOK inference: ✅ (Direct mode via api-proxy → api.githubcopilot.com)

Mode: Direct BYOK (COPILOT_PROVIDER_API_KEY) with api-proxy sidecar credential injection

cc/ @lpcox @Copilot

🔑 BYOK report filed by Smoke Copilot BYOK
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

@lpcox Network isolation smoke test results:

EGRESS_RESULT allow=pass deny=pass

✅ Allowed domain (github.com) reachable: allowed=200
✅ Non-allowed domain (example.com) blocked (TLS/proxy denial)

Overall status: PASS

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • api.github.com
  • example.com

[!TIP]
api.github.com is blocked because GitHub API access uses the built-in GitHub tools by default. Instead of adding api.github.com to network.allowed, use tools.github.mode: gh-proxy for direct pre-authenticated GitHub CLI access without requiring network access to api.github.com:

tools:
  github:
    mode: gh-proxy

See GitHub Tools for more information on gh-proxy mode.

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.github.com"
    - "example.com"

See Network Configuration for more information.

🛡️ Egress verdict from Smoke Copilot Network Isolation
Add label ready-for-aw to run again

@github-actions github-actions Bot added smoke-copilot-byok smoke-copilot-network-isolation Copilot network-isolation egress smoke test labels Sep 23, 2026
@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Copilot Engine — cc @lpcox

Overall: PASS

📰 BREAKING: Report filed by Smoke Copilot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Services Connectivity

  • Redis PING: ✅ PONG
  • PostgreSQL pg_isready: ✅ accepting connections
  • PostgreSQL SELECT 1: ✅ returned 1

Overall: PASS

🔌 Service connectivity validated by Smoke Services
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

🏗️ Build Test Suite Results

Ecosystem Project Build/Install Tests Status
Bun elysia ✅ 1/1 passed ✅ PASS
Bun hono ✅ 1/1 passed ✅ PASS
C++ fmt ✅ N/A ✅ PASS
C++ json ✅ N/A ✅ PASS
Deno oak N/A 1/1 passed ✅ PASS
Deno std N/A 1/1 passed ✅ PASS
.NET hello-world ✅ N/A (ran, output correct) ✅ PASS
.NET json-parse ✅ N/A (ran, output correct) ✅ PASS
Go color ✅ ok ✅ PASS
Go env ✅ ok ✅ PASS
Go uuid ✅ ok ✅ PASS
Java gson ✅ 1/1 passed ✅ PASS
Java caffeine ✅ 1/1 passed ✅ PASS
Node.js clsx ✅ passed ✅ PASS
Node.js execa ✅ passed ✅ PASS
Node.js p-limit ✅ passed ✅ PASS
Rust fd ✅ 1/1 passed ✅ PASS
Rust zoxide ✅ 1/1 passed ✅ PASS

Overall: 8/8 ecosystems passed — ✅ PASS

Notes:

  • Initial Deno clone failed to appear in the first batch (repo cloned to a stale path); re-cloned and re-ran — both projects passed.
  • Java builds initially failed with Could not create local repository at /home/runner/.m2/repository (root-owned .m2 dir on this runner, unrelated to the firewall). Worked around by pointing Maven's localRepository to a writable path via -s settings; the Squid HTTP/HTTPS proxy configuration itself worked correctly (dependencies downloaded fine through squid-proxy:3128) once the local repo path was writable.

All 18 test projects across all 8 ecosystems built and passed successfully through the firewall proxy.

Warning

Firewall blocked 8 domains

The following domains were blocked by the firewall during workflow execution:

  • api.nuget.org
  • bun.sh
  • dc.services.visualstudio.com
  • deno.land
  • dl.deno.land
  • github.com
  • releaseassets.githubusercontent.com
  • repo.maven.apache.org

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "api.nuget.org"
    - "bun.sh"
    - "dc.services.visualstudio.com"
    - "deno.land"
    - "dl.deno.land"
    - "github.com"
    - "releaseassets.githubusercontent.com"
    - "repo.maven.apache.org"

See Network Configuration for more information.

Generated by Build Test Suite for #8916 · copilot · auto · 48.4 AIC · ⊞ 11.8K · ◷
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Chroot Version Comparison Results

Runtime Host Version Chroot Version Match?
Python Python 3.12.14 Python 3.12.14 ✅ YES
Node.js v24.21.0 v22.23.2 ❌ NO
Go go1.22.12 go1.22.12 ✅ YES

Overall: ❌ FAILED — Node.js version mismatch between host and chroot environment. The smoke-chroot label was not added.

Tested by Smoke Chroot
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

feat: add NVX Phase 3f promotion evidence
chore: update model-to-API mapping (2026-09-23)
Merged PR review: ✅
PR detail lookup: ✅
Playwright title check: ✅
File write/read: ✅
AWF build: ✅
Overall: PASS

Warning

Firewall blocked 12 domains

The following domains were blocked by the firewall during workflow execution:

  • ab.chatgpt.com
  • accounts.google.com
  • api.github.com
  • clients2.google.com
  • collector.github.com
  • contentautofill.googleapis.com
  • github.com
  • github.githubassets.com
  • msfeed25.pkgs.visualstudio.com
  • update.googleapis.com
  • www.google.com
  • www.gstatic.com

[!TIP]
api.github.com is blocked because GitHub API access uses the built-in GitHub tools by default. Instead of adding api.github.com to network.allowed, use tools.github.mode: gh-proxy for direct pre-authenticated GitHub CLI access without requiring network access to api.github.com:

tools:
  github:
    mode: gh-proxy

See GitHub Tools for more information on gh-proxy mode.

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "ab.chatgpt.com"
    - "accounts.google.com"
    - "api.github.com"
    - "clients2.google.com"
    - "collector.github.com"
    - "contentautofill.googleapis.com"
    - "github.com"
    - "github.githubassets.com"
    - "msfeed25.pkgs.visualstudio.com"
    - "update.googleapis.com"
    - "www.google.com"
    - "www.gstatic.com"

See Network Configuration for more information.

🔮 The oracle has spoken through Smoke Codex
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: Gemini Engine Validation

Warning

Firewall blocked 2 domains

The following domains were blocked by the firewall during workflow execution:

  • github.com
  • play.googleapis.com

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "github.com"
    - "play.googleapis.com"

See Network Configuration for more information.

💎 Faceted by Smoke Gemini
Add label ready-for-aw to run again

@github-actions

Copy link
Copy Markdown
Contributor

Smoke Test: API Proxy OpenTelemetry Tracing — Results

# Scenario Result
1 Module Loading ✅ otel.js loaded; isEnabled() → true; exports startRequestSpan, setTokenAttributes, setBudgetAttributes, endSpan, endSpanError, shutdown, isEnabled, _provider, ...
2 Test Suite ✅ 68/68 tests passed (3 suites: otel.test.js, otel-fanout.test.js, split-module tests)
3 Env Var Forwarding ✅ env-passthrough.ts forwards GITHUB_AW_OTEL_TRACE_ID/GITHUB_AW_OTEL_PARENT_SPAN_ID; api-proxy-env-config.ts forwards GH_AW_OTLP_ENDPOINTS, OTEL_EXPORTER_OTLP_ENDPOINT, trace ID, and parent span ID
4 Token Tracker Integration ✅ token-tracker-http.js exposes onUsage callback as the OTEL hook point
5 OTEL Diagnostics ✅ otel.jsonl present with 1 exported span (gh-aw.agent.setup, status OK). No api-proxy request spans present — expected, since this workflow validates code/config paths rather than making live LLM proxy calls

Overall: ✅ All 5 scenarios pass. No unexpected failures detected.

Warning

Firewall blocked 1 domain

The following domain was blocked by the firewall during workflow execution:

  • o205451.ingest.us.sentry.io

To allow these domains, add them to the network.allowed list in your workflow frontmatter:

network:
  allowed:
    - defaults
    - "o205451.ingest.us.sentry.io"

See Network Configuration for more information.

📡 OTel tracing validated by Smoke OTel Tracing
Add label ready-for-aw to run again

This branch was successfully deployed

1 active deployment
aoai-model — bd8c6dad Deployed Sep 23, 2026 by lpcox via conclusion #1747
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[awf] api-proxy: Gemini CLI 0.55.1 rejects placeholder API key with "Invalid auth method selected" (exit 41)

3 participants