Testing Claude Code and Codex through the agentic CLI

This guide is the shareable checklist for exercising a coding harness (Claude Code or Codex) against Agentic API, both with a local gateway started by the CLI and against a gateway already running on Kubernetes. It was written while verifying issue #190 / PR #197 and records the exact commands and expected output.

Prerequisites

• A running OpenAI-compatible upstream. The examples use vLLM serving Qwen on http://127.0.0.1:8000:

  vllm serve Qwen/Qwen3.8-27B-FP8 \
    --served-model-name Qwen/Qwen3.8-27B-FP8 ...
  

  Check what is served with:

  curl -s http://127.0.0.1:8000/v1/models | python3 -c 'import sys,json; print([m["id"] for m in json.load(sys.stdin)["data"]])'
  

• The harness binary on PATH: claude --version or codex --version. Override discovery with AGENTIC_CLAUDE_BIN or AGENTIC_CODEX_BIN.

• Both Agentic API binaries built from this repository:

  cargo build -p agentic-server --bins
  

CI pins Claude Code 2.1.245 and Codex 0.149.1 and runs both real CLIs through the attach commands against recorded Qwen/vLLM streams. The Claude job verifies a gateway-owned web-search round trip; the Codex job verifies a completed Responses answer and a real PNG attachment through both agentic harness codex and agentic run codex. The image checks use the committed Qwen2.5-VL response recording, compare the upstream image bytes with the attached PNG, and verify that the same attachment is absent with an explicit text-only catalog. They exercise the actual pinned CLI and gateway, without contacting a live model or claiming fresh vision inference. Run the same checks locally with bash scripts/claude-code-smoke.sh and bash scripts/codex-smoke.sh after building both binaries with cargo build -p agentic-server --bins.

Claude Code with a self-hosted model

Do not start Claude Code directly with a self-hosted model ID such as zai-org/GLM-5.2-FP8. Claude Code validates the selected model against its own catalog before it sends a request, so it reports an unrecognized_model error for IDs it does not recognize. Updating Claude Code alone does not make an arbitrary served model a catalog entry.

Use the attach command for an existing Agentic API gateway:

agentic harness claude \
  --gateway-url http://127.0.0.1:9000 \
  --model zai-org/GLM-5.2-FP8 \
  -- -p "summarize the files in this directory"

For a locally started gateway, use agentic run claude --upstream <vLLM URL> --model <served model> instead. Both commands generate a private settings file that maps Claude Code's canonical claude-sonnet-4-5-20250929 identifier to the selected served model, then launch Claude with the canonical identifier. They also isolate Claude state from the user's normal configuration and remove cloud-provider routing variables.

For a lightweight wrapper around an already-running gateway, use scripts/agentic-claude.sh. It reads GET /v1/models to select or validate AGENTIC_MODEL, records the server-advertised context limit when available, and writes the same canonical-model override. It requires curl and jq. By default, each invocation creates an isolated temporary Claude configuration and removes it when Claude exits; set AGENTIC_CLAUDE_CONFIG_DIR to preserve configuration and session state between invocations:

AGENTIC_GATEWAY_URL=http://127.0.0.1:9000 \
AGENTIC_MODEL=zai-org/GLM-5.2-FP8 \
bash scripts/agentic-claude.sh -p "summarize the files in this directory"

Set AGENTIC_GATEWAY_API_KEY when the gateway requires authentication; the wrapper uses it for both model discovery and Claude Code's gateway requests.

CLI behavior worth knowing

1. Validate the configuration

./target/debug/agentic validate --upstream http://127.0.0.1:8000 --harness claude

Expected: Agentic API configuration looks valid. This checks the gateway port is free, the database URL is usable, the harness binary resolves, and the upstream URL is well formed.

2. Non-interactive smoke test

Claude Code:

./target/debug/agentic run claude --upstream http://127.0.0.1:8000 --model Qwen/Qwen3.8-27B-FP8 -- -p "Reply with exactly one word: pong"

Codex:

./target/debug/agentic run codex --upstream http://127.0.0.1:8000 -- exec "Reply with exactly one word: pong"

Expected lifecycle output, then the harness answer and a clean exit:

Starting Claude via http://127.0.0.1:3000
... agentic_server::server: LLM ready: http://127.0.0.1:8000
... agentic_server::server: gateway listening on 127.0.0.1:3000
Claude Code config: /tmp/agentic-api-session-... (gateway: http://127.0.0.1:3000, model: Qwen/Qwen3.8-27B-FP8)
pong

Claude Code's header may still display the canonical Sonnet label. Requests are routed to the Qwen model by the generated modelOverrides entry; the display label is not evidence that Anthropic billing or an enterprise provider is in use.

3. Tool-call round trip

Tool calls are where the parallel-tool-call handling from #190 is exercised, so run at least one. Start a normal session and approve the permission prompt when the harness asks to run the command:

./target/debug/agentic run claude --upstream http://127.0.0.1:8000 --model Qwen/Qwen3.8-27B-FP8
> Run 'ls crates' with the Bash tool and list the directory names.

Expected: the harness runs the command and answers with agentic-praxis, agentic-server, agentic-server-core. The gateway log must not contain invalid tool config.

Permission prompts are the default. Only for an unattended run in an externally isolated environment (CI, a throwaway container) add --yolo, which forwards the harness's native bypass flag:

./target/debug/agentic run claude --upstream http://127.0.0.1:8000 --model Qwen/Qwen3.8-27B-FP8 --yolo \
  -- -p "Run 'ls crates' with the Bash tool and list the directory names."

4. Interactive session

./target/debug/agentic run claude --upstream http://127.0.0.1:8000 --model Qwen/Qwen3.8-27B-FP8

Inside the session, useful checks are a plain question (no tools), a file read, a multi-step edit, and /model (should show the discovered or passed model). Ctrl-C stops the harness and the gateway together; confirm nothing is left behind with pgrep -fa agentic-server.

5. Reproduce and verify #190 directly

The bug was a gateway-side rejection of parallel_tool_calls: true whenever a built-in tool was declared. Codex always sends true, so every Codex session with a built-in tool failed:

curl -s -w '\nHTTP %{http_code}\n' -H 'content-type: application/json' http://127.0.0.1:3000/v1/responses -d '{
  "model": "Qwen/Qwen3.8-27B-FP8",
  "input": "Reply with the single word: pong",
  "max_output_tokens": 64,
  "parallel_tool_calls": true,
  "tools": [{"type": "web_search_preview"}]
}'

Also run the mixed shape (a function tool plus a built-in such as code_interpreter) with parallel_tool_calls: true; it must return HTTP 200 as well. Unit coverage lives in crates/agentic-server-core/src/types/request_response.rs (to_upstream_request_*parallel_tool_calls*).

6. Against a Kubernetes deployment

Use the attach-only agentic harness command to test a cluster-hosted gateway without starting another local gateway. The kind-based development cluster below is the one from Deploy agentic-api on Kubernetes.

Roll out a new image

docker build -t agentic-api:kind .
kind load docker-image agentic-api:kind --name agentic-api
kubectl --namespace agentic-api rollout restart deploy/agentic-api
kubectl --namespace agentic-api rollout status deploy/agentic-api --timeout=180s

The Kubernetes guide deploys into the agentic-api namespace, so every kubectl command below passes --namespace agentic-api; without it kubectl reports deployments.apps "agentic-api" not found.

Port-forward and run the same checks

kubectl --namespace agentic-api port-forward svc/agentic-api 9000:9000 &

# #190 repro (expect HTTP 200)
curl -s -w '\nHTTP %{http_code}\n' -H 'content-type: application/json' http://127.0.0.1:9000/v1/responses -d '{
  "model": "Qwen/Qwen3.8-27B-FP8", "input": "Reply with the single word: pong", "max_output_tokens": 64,
  "parallel_tool_calls": true, "tools": [{"type": "web_search_preview"}]
}'

# Claude Code through the cluster gateway (interactive)
agentic harness claude \
  --gateway-url http://127.0.0.1:9000 \
  --model Qwen/Qwen3.8-27B-FP8

# The same path as a one-shot smoke test
agentic harness claude \
  --gateway-url http://127.0.0.1:9000 \
  --model Qwen/Qwen3.8-27B-FP8 \
  -- -p "Reply with exactly one word: pong"

The Claude command waits for /health and /ready without following redirects, creates owner-only temporary settings, removes inherited cloud-provider routing variables, and deletes the temporary settings when Claude exits. Claude's session history persists under $AGENTIC_API_HOME/harnesses/claude, separate from the user's normal Claude Code configuration. It advertises the compact Bash,Edit,Read,WebSearch tool set. There is intentionally no --web-search switch: whether Claude Code's WebSearch function is gateway-owned is a deployment policy controlled by MESSAGES_GATEWAY_TOOL_ALIASES=WebSearch=web_search.

For a raw-command diagnosis, the critical pieces are a persistent isolated CLAUDE_CONFIG_DIR, a temporary settings.json whose modelOverrides maps the full canonical ID claude-sonnet-4-5-20250929 to the served model, removal of inherited CLAUDE_CODE_USE_VERTEX/CLAUDE_CODE_USE_BEDROCK/CLAUDE_CODE_USE_FOUNDRY, and passing the canonical ID to claude --model. Short aliases such as claude-sonnet-4-5 are not sufficient for this override.

CLAUDE_SETTINGS_HOME=$(mktemp -d)
CLAUDE_STATE_HOME="${AGENTIC_API_HOME:-$HOME/.agentic-api}/harnesses/claude"
mkdir -p -m 700 "$CLAUDE_STATE_HOME"
trap 'rm -rf -- "$CLAUDE_SETTINGS_HOME"' EXIT
jq --null-input --arg model 'Qwen/Qwen3.8-27B-FP8' \
  '{modelOverrides: {"claude-sonnet-4-5-20250929": $model}}' \
  >"$CLAUDE_SETTINGS_HOME/settings.json"

env -u CLAUDE_CODE_USE_VERTEX \
  -u CLAUDE_CODE_USE_BEDROCK \
  -u CLAUDE_CODE_USE_FOUNDRY \
  -u CLAUDE_CODE_USE_ANTHROPIC_AWS \
  -u CLAUDE_CODE_USE_MANTLE \
  -u CLAUDE_CODE_PROVIDER_MANAGED_BY_HOST \
  -u ANTHROPIC_VERTEX_PROJECT_ID \
  -u CLOUD_ML_REGION \
  CLAUDE_CONFIG_DIR="$CLAUDE_STATE_HOME" \
  ANTHROPIC_BASE_URL=http://127.0.0.1:9000 \
  ANTHROPIC_MODEL=Qwen/Qwen3.8-27B-FP8 \
  ANTHROPIC_SMALL_FAST_MODEL=Qwen/Qwen3.8-27B-FP8 \
  ANTHROPIC_DEFAULT_OPUS_MODEL=Qwen/Qwen3.8-27B-FP8 \
  ANTHROPIC_DEFAULT_SONNET_MODEL=Qwen/Qwen3.8-27B-FP8 \
  ANTHROPIC_DEFAULT_HAIKU_MODEL=Qwen/Qwen3.8-27B-FP8 \
  ANTHROPIC_API_KEY=agentic-api-local \
  ANTHROPIC_AUTH_TOKEN=agentic-api-local \
  CLAUDE_CODE_MAX_CONTEXT_TOKENS=32768 \
  CLAUDE_CODE_MAX_OUTPUT_TOKENS=2048 \
  MAX_THINKING_TOKENS=0 \
  CLAUDE_CODE_EFFORT_LEVEL=medium \
  claude --model claude-sonnet-4-5-20250929 \
    --tools Bash,Edit,Read,WebSearch --setting-sources user --effort medium \
    --settings "$CLAUDE_SETTINGS_HOME/settings.json"

Codex uses the same attach-only workflow. The CLI generates an isolated CODEX_HOME, Responses provider configuration, and model catalog, then removes them when Codex exits:

# Interactive Codex session through the cluster gateway
agentic harness codex \
  --gateway-url http://127.0.0.1:9000 \
  --model Qwen/Qwen3.8-27B-FP8

# One-shot smoke test
agentic harness codex \
  --gateway-url http://127.0.0.1:9000 \
  --model Qwen/Qwen3.8-27B-FP8 \
  -- exec --skip-git-repo-check "Reply with exactly one word: pong"

Codex uses the gateway's WebSocket transport (supports_websockets = true), so this also verifies /v1/responses over WebSocket through the port-forward. On Linux hosts where Codex's sandbox (codex-linux-sandbox/bwrap) cannot start, shell tool calls fail inside Codex itself; that is unrelated to the gateway and --dangerously-bypass-approvals-and-sandbox (or the CLI's --yolo) confirms the gateway side. Codex declares web search structurally through the Responses API, so it does not require MESSAGES_GATEWAY_TOOL_ALIASES.

Replace agentic-api-local with a real key when the deployment enforces inbound authentication. For agentic harness, pass it explicitly with --api-key or set AGENTIC_GATEWAY_API_KEY; ambient OPENAI_API_KEY and ANTHROPIC_CUSTOM_HEADERS are deliberately ignored because they commonly carry unrelated provider credentials. Finish by confirming the gateway logged no errors during the run:

kubectl --namespace agentic-api logs deploy/agentic-api --all-pods --since=10m | grep -c ' ERROR '   # expect 0

Match the log level rather than the word error: Codex closes its WebSocket without a closing handshake when exec finishes, which the gateway logs as a WARN ... WebSocket protocol error: Connection reset without closing handshake line per run. That line is expected; anything at ERROR level is not.

Troubleshooting