Skip to content

protocol-translation.md

Latest commit

 

History

History
273 lines (192 loc) · 34.1 KB

protocol-translation.md

File metadata and controls

273 lines (192 loc) · 34.1 KB
Raw

The Compatibility Surface — Why Gateways Break Claude Code (and How to Tell Before It Bites You)

Last updated 2026-07-29 · Part of Awesome AI Gateway — the only AI-gateway list with a reproducible cost benchmark and a security-honest scorecard. ⭐ Star it.

Languages: English · 简体中文

📊 Key numbers · The three major LLM wire protocols share zero field names in their usage objects (prompt_tokens vs input_tokens vs promptTokenCount — verified against the official references below, 2026-07-29), so every cross-format request is a live, lossy translation. On the hardest path — an Anthropic-format client like Claude Code routed to an OpenAI-format upstream — the independent measurement is LiteLLM 3/3 · Bifrost 3/3 · Portkey OSS: path not offered (xformat.json, neutral CI, 2026-07-10). The phrase "claude code" appears in 465 LiteLLM issues and 138 new-api issues (repo:<owner>/<repo> "claude code" is:issue, 2026-07-29 — without is:issue the LiteLLM figure is ~2,000 because pull requests are counted too). And one silent failure has a price tag: a stripped cache_control breakpoint re-bills cached input at 10× — cache reads cost 0.1× base input on Anthropic's official price sheet.

Every gateway on the list advertises some version of "OpenAI-compatible, works with Claude, supports Gemini." What that sentence hides is where the compatibility lives. A gateway that accepts Anthropic's Messages format from your agent and forwards it to an Anthropic upstream is doing passthrough — cheap and hard to get wrong. A gateway that accepts Messages format and serves it from an OpenAI-format upstream (or vice versa) is doing structural translation of every request and every streamed byte of every response. That translation layer is the single largest source of "the gateway broke my agent" bug reports, and this chapter maps it: what the three protocols actually disagree on, the five ways translation fails (each anchored to a real, verified GitHub issue), what independent measurement shows, and how to test your own gateway in ten minutes.

1. The concept in 60 seconds

There are three wire-protocol families that matter in 2026:

Family Endpoint Spoken natively by Reference (verified 2026-07-29)
OpenAI Chat Completions POST /v1/chat/completions OpenAI + the entire "OpenAI-compatible" ecosystem developers.openai.com
Anthropic Messages POST /v1/messages Anthropic; the format Claude Code emits platform.claude.com
Gemini generateContent POST …:generateContent / :streamGenerateContent Google Gemini ai.google.dev

A gateway that "speaks all three" sits in the middle doing live translation: request body down to the upstream's schema, then the response — including the SSE stream, event by event — back up to the client's schema. Translation must preserve tool-call identity across turns, re-shape streaming envelopes on the fly, and re-derive token accounting between schemas that don't share a single field name. Passthrough fidelity is a solved problem; translation fidelity is where agents die, because coding agents exercise exactly the corners the schemas disagree on: parallel tool calls, long system prompts, incremental tool-argument streaming, and cache breakpoints.

The same tool call, three ways

Here is one model response — "call get_weather for San Francisco" — as each protocol actually puts it on the wire (shapes from the official references above; the Gemini function shapes also per Google's function-calling guide, retrieved 2026-07-29):

// OpenAI chat.completions — arguments is a JSON-encoded STRING; finish_reason: "tool_calls"
{ "role": "assistant",
  "tool_calls": [{
    "id": "call_abc123",
    "type": "function",
    "function": { "name": "get_weather",
                  "arguments": "{\"location\": \"San Francisco, CA\"}" } }] }
// Anthropic messages — input is a parsed OBJECT; stop_reason: "tool_use"
{ "role": "assistant",
  "content": [{
    "type": "tool_use",
    "id": "toolu_01T1x1fJ34qAmk2tNTrN7Up6",
    "name": "get_weather",
    "input": { "location": "San Francisco, CA" } }] }
// Gemini generateContent — a functionCall PART inside a "model" content
{ "role": "model",
  "parts": [{
    "functionCall": { "name": "get_weather",
                      "args": { "location": "San Francisco, CA" } } }] }

Three different container shapes, two different argument encodings (string vs object), and three different pairing mechanisms for the tool result coming back: OpenAI pairs by tool_call_id in a dedicated tool-role message, Anthropic by tool_use_id in a tool_result block inside a user message, and Gemini's documented functionResponse shape pairs by function name, not id. A translating gateway carries ids like toolu_01N9FRKhMkWtQ77NLCKGy4An across schemas that never minted them — the verbatim payload in Portkey-AI/gateway#980 shows exactly such Anthropic-minted toolu_… ids riding inside OpenAI-shaped tool_calls messages at the moment the pairing broke.

2. The field-by-field mismatch

Everything below is taken from the current official references — OpenAI Chat Completions (+ streaming events), Anthropic Messages (+ streaming, prompt caching), and Gemini generateContent — all retrieved 2026-07-29.

Concern OpenAI chat.completions Anthropic messages Gemini generateContent What the translator must not fumble
System prompt A message: role: "system" (or "developer") inside messages[]; multiple allowed Top-level system parameter (string or text-block array); messages[] alternates user/assistant Top-level systemInstruction object with parts Hoist N system messages into 1 top-level field — without dropping any (see failure mode 4)
Roles developer · system · user · assistant · tool user · assistant user · model tool-role messages have no direct Anthropic/Gemini equivalent — they must become content blocks/parts inside a user turn
Tool call (model → you) assistant message with tool_calls[]: {id, type: "function", function: {name, arguments}} — arguments is a JSON-encoded string tool_use content block: {type, id, name, input} — input is a parsed object functionCall part inside a model content String↔object conversion of the arguments, both directions; a classic bug is relaying input as a raw string
Tool result (you → model) Separate message: role: "tool", tool_call_id tool_result content block inside a user message: {tool_use_id, content, is_error} functionResponse part Id pairing: every tool_use.id must match a tool_result.tool_use_id — Anthropic 400s if any is orphaned (see failure mode 1)
Stream envelope Unnamed SSE data: lines, each a chat.completion.chunk with choices[].delta; terminates with data: [DONE] Named SSE events: message_start → content_block_start/content_block_delta/content_block_stop (per block, with index) → message_delta → message_stop, plus ping/error SSE via ?alt=sse; each chunk a full GenerateContentResponse JSON The shapes are structurally alien: flat delta stream ↔ indexed block state machine. The translator must synthesize events that never existed upstream
Streaming text delta.content string fragments content_block_delta with delta.type: "text_delta" candidates[].content.parts[].text Losing granularity here = buffered "fake" streaming (see failure mode 2)
Streaming tool args delta.tool_calls[] fragments keyed by index; function.arguments accretes as string pieces input_json_delta with partial_json string pieces; final tool_use.input must be a parsed object (no equivalent incremental-JSON contract documented) Accumulate partial JSON, then parse — mismatched block index bookkeeping breaks clients mid-stream
Finish/stop finish_reason: stop · length · tool_calls · content_filter · function_call stop_reason: end_turn · max_tokens · stop_sequence · tool_use · pause_turn · refusal · model_context_window_exceeded; arrives in message_delta finishReason (own enum) tool_calls↔tool_use must map exactly, or the agent doesn't know it's supposed to run a tool
Thinking / reasoning Reasoning is exposed only as a count: completion_tokens_details.reasoning_tokens First-class thinking content blocks, streamed via thinking_delta + a signature_delta (integrity signature) before content_block_stop thoughtsTokenCount in usageMetadata Anthropic thinking blocks (and their signatures) simply have nowhere to go in an OpenAI-format hop — round-tripping loses them
Usage naming usage: prompt_tokens · completion_tokens · total_tokens; stream: only the final chunk carries usage, and only if stream_options: {"include_usage": true} usage: input_tokens · output_tokens (+ cache fields); stream: message_start carries initial usage, message_delta carries cumulative totals usageMetadata: promptTokenCount · candidatesTokenCount · totalTokenCount Zero shared names. If the translator forgets include_usage downstream or drops message_delta.usage upstream, billing can't be reconciled
Cache accounting prompt_tokens includes cached tokens; prompt_tokens_details.cached_tokens is a subset breakdown input_tokens excludes cache traffic: total input = input_tokens + cache_read_input_tokens + cache_creation_input_tokens cachedContentTokenCount in usageMetadata The inclusion-semantics flip is a built-in double-count trap (see failure mode 3)
Cache breakpoints None — server-side automatic caching Explicit cache_control: {"type": "ephemeral"} (optional ttl: "5m"/"1h") on content blocks, ≤4 per request Separate cached-content mechanism cache_control exists only in Anthropic's schema — a naive translator strips it, silently (see failure mode 5)

The same two-word reply, streamed, both ways

The row that is hardest to picture from a table is the stream envelope, so here it is concretely. An OpenAI-format upstream streams "Hello!" as unnamed data: chunks (shape per the streaming-events reference):

data: {"object":"chat.completion.chunk","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}
data: {"object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}
data: {"object":"chat.completion.chunk","choices":[{"index":0,"delta":{"content":"!"},"finish_reason":null}]}
data: {"object":"chat.completion.chunk","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
data: [DONE]

An Anthropic-format client expects the same two words as a named-event state machine (this sequence is the official docs' own example, abbreviated):

event: message_start
data: {"type":"message_start","message":{"id":"msg_…","role":"assistant","content":[],"usage":{"input_tokens":25,"output_tokens":1},…}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Hello"}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"!"}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn","stop_sequence":null},"usage":{"output_tokens":15}}

event: message_stop
data: {"type":"message_stop"}

A translating gateway must manufacture the right-hand side from the left in real time: invent message_start before the first upstream chunk arrives, open and close numbered content blocks it was never told about, convert finish_reason: "stop" into a message_delta carrying stop_reason: "end_turn" plus usage the upstream only sends on its final chunk (and only if the gateway remembered to request stream_options: {"include_usage": true} downstream). Add a tool call mid-stream and it must interleave a second block with input_json_delta fragments at the correct index. Get any of the bookkeeping wrong and the client's stream parser throws — which is precisely the Content block not found error in failure mode 2.

Two rows in the mismatch table do the most damage in practice. The usage-semantics flip (OpenAI counts cached tokens inside prompt_tokens; Anthropic counts them outside input_tokens) means a translator that maps fields 1:1 either double-counts or under-counts cached input — that's not a bug in any one gateway, it's a trap in the schema pair. And cache_control having no OpenAI equivalent means any "normalize to OpenAI format internally" architecture destroys it by default unless someone writes explicit preservation code — per provider adapter, forever.

3. The five failure modes

Each failure mode below is anchored to a real, public GitHub issue, verified to exist and to say what we cite it for (via GitHub API, 2026-07-29). Dates are issue-creation dates; closed issues are fixed in current releases — they're cited as evidence of the class, which recurs.

3.1 Tool-call rewriting or dropping

The translator loses tool-call identity or structure across the format boundary. The canonical shape: a client sends parallel tool calls and their results back through the gateway, and the id pairing breaks in translation — the upstream rejects the conversation with 400 … the following tool_use ids were not found in tool_result blocks (Portkey-AI/gateway#980, 2025-03-09, closed — OpenAI-format client with multiple tool_calls relayed to Anthropic). The subtler variant is dropping or string-ifying arguments: the cross-format probe explicitly checks that tool_use.input arrives as a parsed object, "the classic mistranslation" being a raw JSON string. For an agent, either variant is fatal: it can't execute the tool or can't continue the conversation.

3.2 Fake or reshaped streaming

Translating a flat OpenAI delta stream into Anthropic's indexed block state machine (or vice versa) in real time is genuinely hard, and two distinct failures come out of it. Reshaped-broken: the synthesized event stream violates the client's state expectations — Claude Code driving an OpenAI model through LiteLLM's /v1/messages logged Error streaming, falling back to non-streaming mode: Content block not found on every tool call (BerriAI/litellm#13373, 2025-08-07, closed). Fake streaming: the gateway buffers the whole upstream response and emits it as one blob — you lose time-to-first-token and any ability to see queue latency. The measured signature is a stream that arrives as 0–1 delta events instead of many: exactly what fidelity.json recorded for Portkey OSS v1.15.2 on a clean CI runner ("only 0 chunk(s) — collapsed/buffered", 2026-07-10; non-streaming worked, hosted product untested).

3.3 Usage misreporting and inflation

Because the three usage schemas disagree on both names and semantics, translated usage is where billing quietly goes wrong. Two verified instances on opposite sides of the same trap: LiteLLM's cost calculation once charged cache_creation_input_tokens twice — "once as prompt tokens and then again as cache creation tokens" — reporting $0.091311 against an Anthropic-console-verified $0.05439, ~1.7× the real cost (BerriAI/litellm#9812, 2025-04-08, closed). And new-api's /v1/messages → OpenAI-compatible-upstream conversion returned input_tokens that still included cached tokens — inflated usage on the Anthropic-format side, precisely the inclusion-semantics flip from the table above (QuantumNous/new-api#4395, 2026-04-22, open, zh). If you pay per token through a gateway, this failure mode is invisible until you reconcile against the provider's own console.

3.4 Context and system-prompt truncation

Translation between "system prompts are messages, plural" (OpenAI) and "system prompt is one top-level field" (Anthropic) invites silent content loss. Verified instance: Portkey's Anthropic adapter overwrote the system parameter on each iteration, so when a client sent multiple system messages only the last one was forwarded — every earlier instruction silently vanished (Portkey-AI/gateway#457, 2024-07-11, closed). The mirror image is mangling rather than truncation: new-api's message re-serialization produced empty text content blocks that Anthropic rejects (400 … text content blocks must be non-empty), making Claude Code entirely unusable through the gateway against Anthropic's own models (QuantumNous/new-api#1854, 2025-09-20, closed, zh). Truncation is the nastier of the two: a 400 you notice; a dropped system prompt just makes your agent quietly worse.

3.5 cache_control stripping — the silent 10× bill

cache_control breakpoints exist only in Anthropic's schema, so any internal normalization step that doesn't explicitly carry them drops them — and nothing fails. Requests succeed, responses look identical; the only symptom is cache_read_input_tokens: 0 and a bigger bill. Verified instances: Portkey stripping cache_control en route to Vertex AI Anthropic models, with the reporter's own before/after usage showing zero cache tokens (Portkey-AI/gateway#1579, 2026-03-25, open), and LiteLLM's SDK→proxy path silently no-opping cache-control injection (BerriAI/litellm#30319, 2026-06-12, closed). The class keeps recurring per adapter — e.g. BerriAI/litellm#34797 (2026-07-27, open) reports the same stripping in a newer provider path. The arithmetic: Anthropic bills cache reads at 0.1× base input (pricing, retrieved 2026-07-29), so input that should have been cache-read now bills at full price — 10× on those tokens. For a Claude Code session, where the system prompt and tool definitions dominate input and repeat on every request, "those tokens" is most of your bill (production telemetry cited on the list has system prompts at 69% of input tokens — Datadog, 2026-04).

4. Why the measured results look the way they do

The llm-gateway-bench project measures exactly this chapter's subject, black-box, against a spec-correct mock upstream on a neutral CI runner (no API keys, no vendor involvement). Two datasets, both from 2026-07-10:

Same-format passthrough (fidelity.json) — OpenAI client → gateway → OpenAI-format upstream. Does a spec-correct response survive the relay?

Gateway (version) tool_calls streaming stream usage Score
LiteLLM 1.91.1 ✅ intact ✅ 7 chunks, content intact ✅ total_tokens=14 relayed 3/3
Bifrost (docker 95caedb1c368) ✅ intact ✅ 5 chunks, content intact ✅ relayed 3/3
Portkey OSS 1.15.2 ✅ intact ❌ 0 chunks — collapsed/buffered ❌ no usage in stream 1/3

Cross-format translation (xformat.json) — Anthropic-format client (the Claude Code path) → gateway /v1/messages → OpenAI-format upstream. The hardest path, and the one this chapter is about:

Gateway (version) tool_use streaming stream usage Score
LiteLLM 1.91.1 ✅ name + parsed input ✅ 3 text_delta events, intact ✅ output_tokens in message_delta 3/3
Bifrost (docker 95caedb1c368) ✅ ✅ 4 text_delta events ✅ 3/3
Portkey OSS 1.15.2 — — — not offered — its /v1/messages is Anthropic-provider-only in the header-config self-host; targeting an OpenAI upstream returns 500 "messages is not supported by openai"

Three readings, grounded in the raw data:

  1. LiteLLM and Bifrost genuinely translate, and translate cleanly — on this probe's three checks, both synthesize a well-formed Anthropic event stream (message_start → content_block_delta … — visible verbatim in the recorded stream_snippets), deliver tool_use.input as a parsed object, and land output_tokens in the final message_delta. That's all five failure-mode surfaces from §3 exercised at their choke points, passing.
  2. Portkey OSS's cross-format "failure" is honest scoping, not breakage — the data records unsupported: true, not a lying 3/3 or a crash. Its /v1/messages endpoint translates only toward Anthropic-family providers in this deployment mode. If your agent speaks Anthropic format and your upstream speaks OpenAI format, this gateway (in OSS header-config mode) is not on the menu — its 1/3 on the same-format streaming probe is the separate, real finding.
  3. The path under your feet changes between versions — pin them. The probe's own header documents (measured 2026-07-09) that LiteLLM ≤1.57.x served /v1/messages by translating to the upstream's Chat Completions endpoint, while ≥~1.9x rewrote the path to the OpenAI Responses API (/v1/responses, input/max_output_tokens) — and the new transformer raises KeyError('created_at') if the upstream answers with a chat.completion body. Same gateway, same config, different minor version: a chat-completions-only upstream went from working to a Python traceback. The bench handles this by mocking both endpoints and flagging the mismatch inconclusive rather than 0/3; your production stack won't be so forgiving.

5. How to verify your gateway in 10 minutes

Don't trust this chapter — or any vendor README. The whole point of the failure catalog is that every signature is cheap to check:

  1. Pin and record versions first — gateway version/image digest and agent version. §4's KeyError('created_at') story is what "it worked last month" looks like across a minor bump.
  2. Run the black-box probes (no API keys needed): 
    git clone https://github.com/cuihuan/llm-gateway-bench && cd llm-gateway-bench

# 1. Sanity-check the probe itself — runs standalone, nothing to configure:
node probe/fidelity.mjs --gateway self-test            # expect: 3/3

# 2. Same-format passthrough through YOUR gateway (tool_calls / streaming / usage).
#    Configure the gateway's openai provider to point at http://127.0.0.1:9020/v1 first:
node probe/fidelity.mjs --gateway mine \
  --gateway-url http://localhost:4000/v1/chat/completions

# 3. The Claude Code path (Anthropic client → OpenAI upstream), mock on port 9010:
node probe/xformat.mjs --gateway mine \
  --messages-url http://localhost:4000/v1/messages
    Both probes stand up a spec-correct mock upstream locally and score the same 3 checks as §4 — no provider API key is needed, but steps 2 and 3 do need your gateway running with the mock configured as its openai provider. --help prints every flag.
  3. Run your real agent through it — a trivial Claude Code task in a scratch repo (claude -p "list the files here and read one") exercises parallel tools, streaming, system prompt, and caching in one shot. Run it twice back-to-back (the second run is the cache check).
  4. Reconcile one request's usage against the provider's own console — the only way to catch failure mode 3.

Then read the output against the five signatures:

Signature you see Failure mode Anchor issue
400 … tool_use ids were not found in tool_result blocks; tools fire but the next turn 400s Tool-call rewriting/dropping Portkey#980
Error: Streaming fallback triggered / long pause then the whole answer at once / probe reports "collapsed/buffered" Fake or reshaped streaming litellm#13373
Gateway-billed tokens ≠ provider console; input_tokens moves with cache activity Usage misreporting litellm#9812, new-api#4395
Agent "forgets" standing instructions; 400 … text content blocks must be non-empty Context/system-prompt truncation or mangling Portkey#457, new-api#1854
cache_read_input_tokens: 0 on the second identical request cache_control stripping (silent 10×) Portkey#1579, litellm#30319

Ten minutes. The alternative is finding out from your invoice.

6. Per-gateway implementation notes

Only gateways whose behavior we can source — from their own docs or from the measured data — get a row with claims; everything else on the list is honestly not measured. Vendor docs describe intent; the measured column is what actually happened on a neutral runner (2026-07-10).

Gateway Anthropic-format inbound (/v1/messages) Cross-format translation Source (retrieved 2026-07-29) Measured (xformat · fidelity)
LiteLLM ✅ documented ✅ to "all LiteLLM supported providers" (openai, bedrock, vertex, gemini, azure…) docs.litellm.ai — /v1/messages 3/3 · 3/3 (v1.91.1) — note the §4 transport change; pin your version
Bifrost ✅ drop-in Anthropic SDK endpoint (/anthropic) ✅ unified across 23+ providers behind an OpenAI-compatible core README · Anthropic SDK integration docs 3/3 · 3/3 (docker 95caedb1c368)
Portkey OSS Endpoint exists, but Anthropic-provider-only in header-config self-host ❌ not offered on the Anthropic→OpenAI path in that mode ("messages is not supported by openai") measured behavior, xformat.json not offered · 1/3 (v1.15.2; hosted product untested)
new-api ✅ "Native Claude Format" documented ✅ converts /v1/messages ↔ OpenAI-compatible upstreams (usage-conversion caveat: #4395) docs.newapi.pro — Native Claude Format not measured
one-api ❌ — inbound is OpenAI format only ("access all models via the standard OpenAI API format") Outbound only: rewrites request/response bodies toward non-OpenAI downstream channels (incl. Claude) README (zh; incl. architecture diagram) not measured
Everything else on the list — — — not measured — treat all compatibility claims as vendor claims until probed

7. What this means for choosing

If your agent speaks Anthropic format and your upstream doesn't (or might not, once routing kicks in), the translation layer is the product — shortlist from gateways with measured cross-format fidelity in Self-hosted open source, currently LiteLLM and Bifrost at 3/3, and pin the exact version you validated. Before any gateway touches production traffic, spend the ten minutes in §5, because four of the five failure modes are silent and one of them costs 10× on your biggest token bucket — the caching economics are laid out in Prompt caching through a gateway — the money question. And keep reconciling usage against the provider console on a schedule, per How to choose safely — a gateway that was faithful at version N is one refactor away from failure mode 3 at version N+1.

Appendix — every source this chapter relies on

Primary protocol references (all retrieved and field-checked 2026-07-29):

GitHub issues (each verified to exist and to say what it's cited for, via GitHub API, 2026-07-29):

Issue Title (abridged) Created State Cited in
Portkey-AI/gateway#980 400 anthropic error: tool_use ids not found in tool_result blocks 2025-03-09 closed §3.1
BerriAI/litellm#13373 Claude Code with an OpenAI model throws "Streaming fallback triggered" 2025-08-07 closed §3.2
BerriAI/litellm#9812 Anthropic cost calculations incorrect with prompt caching 2025-04-08 closed §3.3
QuantumNous/new-api#4395 /v1/messages → OpenAI-compatible upstream usage conversion (zh) 2026-04-22 open §3.3
Portkey-AI/gateway#457 Anthropic only uses last system message 2024-07-11 closed §3.4
QuantumNous/new-api#1854 Claude Code via new-api → "text content blocks must be non-empty" (zh) 2025-09-20 closed §3.4
Portkey-AI/gateway#1579 cache_control stripped when routing to Vertex AI Anthropic 2026-03-25 open §3.5
BerriAI/litellm#30319 Prompt caching silently stripped through proxy path 2026-06-12 closed §3.5
BerriAI/litellm#34797 cache_control stripped in SAP provider path 2026-07-27 open §3.5

Measured data & vendor docs:

Found this useful? ⭐ Star the list — it's how the next engineer choosing a gateway finds it. Corrections & additions welcome via PR or issue — every claim above is dated and linked so you can re-check it; if a linked issue is fixed or a probe result changes, that's a PR we want.