Testing Conversation Preservation Across LLM APIs: A Rule-Based Oracle for Detecting Silent Information Loss
Organizations: Independent Researcher · School of Engineering and Built Environment Griffith University
Abstract
LLM applications increasingly rely on translators and compatibility layers to move conversations across APIs. Yet a translated request can be accepted by the target API while silently losing information about instruction roles, tool interactions, error states, or generation controls. We introduce isloss, a deterministic oracle for testing conversation-information preservation across LLM API translations. It maps source and translated requests and replies to a shared intermediate representation, ConvIR, and applies explicit rules to distinguish legitimate representation changes from information loss. Unlike schema validation or reference-based differential testing, isloss requires neither a reference translation nor additional model calls. It also distinguishes avoidable loss from degradation caused by target API limitations. We evaluate isloss using specification-based fault injection, human review, and experiments on translators, gateways, and client libraries. On held-out tests, it detects 95/125 planted faults (76.0%) with 2/100 false alarms (2.0%). Across a synthetic request suite spanning six configurations of five translator codebases, 127/685 completed translations (18.5%) pass static target checks while silently changing prompt, role, or tool information in ways the target API could have preserved. In translated replies, 203/558 streams (36.4%) cause exceptions or silent corruption in at least one tested official OpenAI SDK. Targeted experiments across six models show that translation-induced changes can alter model behaviour when tasks depend on the affected information. These results establish conversation preservation as a distinct correctness requirement for reliable LLM interoperability, agent execution, and model evaluation.
Figures & tables
| Term | Meaning |
|---|---|
| Translation test (cell) | One request template ( shape ) sent through one translator and source-to-target route ( path ). |
| Silent change | The translator returns output without warning about the change. |
| Avoidable change | The target API has a documented native field that could preserve the altered information. |
| Static validity | A request passes our offline checks of API schema and documented constraints. |
| Live acceptance | A vendor endpoint accepts the request with HTTP 200; this is tested separately from static validity. |
| Primary finding | A T1 finding under the reporting policy in Section 3.2 that is silent, avoidable, affects prompt, role or tool fields, and occurs in a request passing static target checks. One finding should meet all criteria. |
| Stage | Criterion | Retained cells |
|---|---|---|
| 1. Compare representations | At least one potential information-changing difference | 489 |
| 2. Filter acceptable changes | Exclude four known classes of permitted change | 485 |
| 3. Apply reporting policy | Retain cells with at least one T1 finding | 467/685 (68.18 %) |
| 4. Check static target validity | Retain cells whose translated requests pass the static target contract | 333/685 (48.61 %) |
| 5. Select primary findings | Require an avoidable, silent change to prompt, role, or tool information | 127/685 (18.54 %) |
| Check | Material | Faults found | Allowed changes flagged |
|---|---|---|---|
| First held-out batch (faults) | 36 faults, 34 activated | 28/34 = 82.4% (66.5–91.7%) | — |
| First held-out batch (controls) | 28 allowed changes, 28 activated | — | 1/28 |
| Primary held-out evaluation | 125 activated faults, 100 allowed changes | 95/125 = 76.0% (67.8–82.6%) | 2/100 = 2.0% (0.6–7.0%) |
| Extended held-out evaluation | 220 activated faults, 170 allowed changes | 160/220 = 72.7% | 5/170 = 2.9% |
| New public issues | 16 within scope, 14 encodable | 10/14 = 71.4% (45.4–88.3%) | — |
| Seed recall (requests) | 2,129 single-fault wires, 335 controls | 1,595/2,129 = 74.9% (73.0–76.7%) | 0/335 |
| What we measure | Value | Unit | Tested on | Sec. | |
| Translated requests | |||||
| Primary rate, full synthetic suite (Stratum 1) | 127/685 (18.5 %) | cell | 685 | 6 configs, 14 paths | § 6.1 |
| Same rate, shapes claude-sonnet-4-6 accepts live (Stratum 2a) | 34/182 (18.7 %) | cell | 182 | 6 configs, 14 paths | § 6.1 |
| Same rate, shapes claude-opus-4-8 accepts live (Stratum 2b) | 23/112 (20.5 %) | cell | 112 | 6 configs, 14 paths | § 6.1 |
| Range of the primary rate under other rules | 12.9–19.6 % (validity rules); 16.5–25.7 % (counting conventions) | cell | 308–685 | 4–6 configs | § 6.1 |
| Reviewed candidates | 54 candidates (31 confirmed defects) across 18 mechanisms | (translator, rule) pair | 54 | 5 codebases | § 6.1 |
| OpenAI | Gemini | Anthropic | |||||
|---|---|---|---|---|---|---|---|
| Translator | T1/judged | Static | T1/judged | Static | T1/judged | Static | Cand. (pre) |
| LiteLLM | 37/51 | 0 | – | – | – | – | 4 |
| claude-code-proxy | 21/22 | 7 | – | – | – | – | 7 |
| LLM-Rosetta | 42/51 | 22 | 42/51 | 42 | 36/51 | 36 | 10 |
| @musistudio/llms | 44/51 | 17 | 51/51 | 51 | 0/51 | 0 | 20 |
| AI SDK v5 | 39/51 | 21 | 40/51 | 40 | 18/51 | 18 | 7 |
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
| Request & session features ( a – i ) | Reply & CI ( j – n ) | |||||||||||||||
| Code base | a | b | c | d | e | f | g | h | i | j | k | l | m | n | Y(+Y*)/14 | Test style |
| LiteLLM | 12/14 | literal asserts + VCR | ||||||||||||||
| Vercel AI SDK | 5/14 | inline snapshots | ||||||||||||||
| @musistudio/llms | 0/14 | none | ||||||||||||||
| claude-code-proxy | 0/14 | print-only smoke | ||||||||||||||
| LLM-Rosetta | 10/14 | IR round-trip + literals | ||||||||||||||
| Method (venue) | Object under test | Oracle | LLM | Det. | State | BB | Reported outcome |
|---|---|---|---|---|---|---|---|
| isloss (this paper) | captured requests, replies and SSE | documented preservation rules; stream equivalence | N a | Y | Y | N | see § 6 |
| Endpoint probes (this study) | hosted compatibility endpoint | model replies vs. native controls | Y | N | Y | Y | see § 6.5 |
| LogicHunter (ISSTA’26) | object API (agent frameworks) | LLM agent judge; crash baseline | Y | N | N | N | 40 found / 30 conf. / 26 fixed; precision 91% |
| GateScope (IMC’26) | hosted gateway (network path) | differential vs. vendor; statistical | Y | N | Y | Y | 10 gateways; substitution, memory loss, billing drift |
| HTTP Garden (preprint’24) | HTTP parser; gateway transforms | differential | N | Y | N | part. | 100 bugs, 68 fixed |
| Netalyzr / Web proxies (IMC’10 / PAM’14) | network path (middleboxes) | known-answer probes | N | Y | N | Y | 130k sessions; 14% of 646k IPs behind proxies |
| Construct | OpenAI Chat Completions | Anthropic Messages | Gemini generateContent |
|---|---|---|---|
| System prompt | system / developer message, accepted at any position | top-level system ; a mid-conversation {role:system} turn is documented for recent models while the API reference (our copy of 1 October 2026) stated there is no system role † | top-level systemInstruction ; role:system inside contents is rejected (400) |
| Roles | system, developer, user, assistant, tool | user, assistant (typed content blocks) | user, model (typed parts) |
| Tool call | assistant.tool_calls[] : id , function.name , function.arguments (JSON string) | tool_use block: id , name , input (object) | functionCall part: name , args (object), optional id |
| Tool result | role:tool message with tool_call_id ; no error flag ‡ | tool_result block: tool_use_id , content , is_error | functionResponse part: name , response object; an "error" key is the documented convention |
| Reasoning | no reasoning content in messages; reasoning_effort knob only ‡ | thinking block with signature , replayed verbatim | thought parts with thoughtSignature ; Gemini 3 rejects a missing signature (400) † |
| Forced tool choice | tool_choice : none / auto / required / named function | tool_choice.type : auto / any / tool / none | functionCallingConfig.mode : AUTO / ANY / NONE / VALIDATED, allowedFunctionNames |
| Rule | Dimension | What changes | Vendor source (artifact copy, location) |
|---|---|---|---|
| L01 | Representation | role name remap (tool result carried in user/model turn) | anthropic:messages (L48-50) |
| L02 | Representation | merge consecutive same-author messages into one multi-block message, blocks intact | anthropic:messages (L48) |
| L03 | Representation | consistent renaming of call ids across call and result | anthropic:agents-and-tools_tool-use_handle-tool-calls (L65) |
| L04 | Representation | split one text block into adjacent text blocks of the same author, concatenation identical | anthropic:messages (L95) |
| L05 | Representation | developer text hoisted to a dedicated system slot when it is the leading developer text | anthropic:messages (L91) |
| L06 | Degradation | unsupported control dropped with documented degradation; recorded as a loss, not as semantic equivalence | anthropic:messages (L3058) |
| # | Construct | Gap kind | Documented by | Live probe |
|---|---|---|---|---|
| 1 | system message mid-conversation | contradiction | A†(n/a: G) | Gemini: resolved |
| 2 | system between tool_use and tool_result | unspecified | A (n/a: G) | not probed |
| 5 | empty tool result content | unspecified | A | Gemini: accepts |
| 6 | assistant text after tool_use in the same turn | unspecified | none (n/a: O) | Gemini: accepts |
| 7 | tool error flag | mapping unspecified | A,G (n/a: O) | Gemini: resolved |
| 8 | cross-vendor replay of thinking / redacted thinking | field unspecified | A,G (n/a: O) | Gemini: resolved |
| Check | Predicate | Material | Result |
| Final isloss measurements and comparisons | |||
| Seed recall (request) | pre-specified | 2,129 single-fault payloads; 335 allowed-change controls | isloss T1: 1,595/2,129 = 74.9% (Wilson 73.0–76.7%); intended rule 1,383/2,129 = 65.0%; 0/335 false alarms |
| Historical bugs, sample A (request) | pre-specified | 29 sample-A issues; 12 replayed | isloss T1: 9/12 (Wilson 47–91%). A separate post-hoc diagnostic detects 10/12; it is not the reported final-rule estimate |
| Historical bugs, sample A (stream) | pre-specified | 4 replayed; 6 not run; 7 out of scope | isloss T1: 4/4; one upstream fix found incomplete |
| Historical bugs, sample B–C | before the run | 56 further issues: 18 replayed (12 request, 6 stream), 17 without a harness, 20 out of scope, 1 not reproduced | isloss T1: 6/11 requests and 3/6 streams. The 11-request scoring subset differs from the 12 replayed requests; the inverse case is reported separately in the records |
| Program mutants (request) | pre-specified | LiteLLM 1.104.0 adapter, OpenAI target; 300 mutants, 135 activated | isloss T1 kills 113/135 = 83.7% of activated mutants |
| Judge | Setting | Fault recall (95% CI) | Allowed-change FPR (95% CI) | [95% CI] | Cost/1k calls |
|---|---|---|---|---|---|
| Held-out Sets 9–14, all activated items ( : , ) | |||||
| isloss (T1) | Ref.-free | 95/125 (76.0% [67.8, 82.6]) | 2/100 (2.0% [0.6, 7.0]) | 0.740 [0.658, 0.816] | $0 |
| Normalised diff D3 | Ref. given (same wire) | 124/125 (99.2% [95.6, 99.9]) | 15/100 (15.0% [9.3, 23.3]) | 0.842 [0.768, 0.910] | $0 |
| gemini-3.5-flash-lite | Ref. given (same wire) | 108/125 (86.4% [79.3, 91.3]) | 6/100 (6.0% [2.8, 12.5]) | 0.804 [0.724, 0.876] | $0.116 † |
| gemini-3.8-flash | Ref. given (same wire) | 120/125 (96.0% [91.0, 98.3]) | 12/100 (12.0% [7.0, 19.8]) | 0.840 [0.762, 0.910] | $0.116 † |
| claude-sonnet-4-6 | Ref. given (same wire) | 122/125 (97.6% [93.2, 99.2]) | 25/100 (25.0% [17.5, 34.3]) | 0.726 [0.636, 0.812] | $3.74 |
| Subject | Position | Language | Directions tested | Version / commit | Scale (2026-10-01) |
|---|---|---|---|---|---|
| LiteLLM | library; also a proxy | Python | A O; O/G A; 2nd hop | 1.103.1 / 1.103.2 | 89.4M/30 d; 60.0k |
| Vercel AI SDK v5 | in-process library | TypeScript | A O/G/A; O/G A | 5.0.271 | ai 32.7M/wk; 27.1k |
| Vercel AI SDK v7 | in-process library | TypeScript | A O/G/A; O/G A | 7.0.126 | (same package) |
| @musistudio/llms | local proxy (engine of claude-code-router 1.x) | TypeScript | A O/G/A; O/G A | 1.0.53 | 308 ; router 37.5k |
| @the-next-ai/ai-gateway | local proxy (engine of claude-code-router 2.0); not pooled | TypeScript | A O/G/A; 2nd hop | 1.0.21 | router 37.5k |
| claude-code-proxy | local proxy | Python | A O; O A | git 7ea4177 | 2.8k |
| Headline claim & value | Unit & | Corpus | Subjects | Method and supporting evidence | Sec. |
|---|---|---|---|---|---|
| 1. Primary (Stratum 1) : (18.5%) | Cell; ( ) | 51 of 357 shapes ( S-strict-R9 ) | 6 configs (5 codebases), 14p | isloss primary criteria; static T1; finding-aligned mechanism coverage | § 6.1 |
| 2. Live-source (Stratum 2a) : (18.7%) | Cell; ( ) | 13 of 51 shapes HTTP 200 (Sonnet) | 6 configs, 14p | isloss primary criteria; source HTTP 200 ( claude-sonnet-4-6 ) | § 6.1 |
| 3. Opus (Stratum 2b) : (20.5%) | Cell; ( ) | 8 of 51 shapes HTTP 200 (Opus) | 6 configs, 14p | isloss primary criteria; source HTTP 200 ( claude-opus-4-8 ) | § 6.1 |
| 4. Live target : Gem. (78.4%); Anth. (37.7%) | Target cell ( Gem., Anth.) | S-strict-R9 wire ( OpenAI cells checked offline) | 8 live paths (4 Gem., 4 Anth.) | Static checks vs. live acceptance; Gemini and Anthropic refusals | § 5.3 |
| 5. Candidates & mechs : 54 ( A/ B/ C), 18 mechs | (translator,rule) ( ); mechs | S-strict-R9 ( cells) | 5 codebases (v5/v7 merged) | isloss and 4 raters; among non-disputed cases | § 6.1 |
| 6. Sensitivity : 12.9%–19.6% (valid.); 16.5%–25.7% (conv.) | Cell; selected main-suite variants ( ) | Stratum-1 subsets & reclassifications | 4–6 configs (8–14p) | isloss under the stated variant ( to validity; to counting) | § 6.1 |
| Mechanism (pre-review) | Cand. | T1 cells | Static | M/U/Borderline | Code | Upstream |
|---|---|---|---|---|---|---|
| redacted thinking block dropped | 9 | 261 | 210 | 1/8/0 | O | 3 rep.: 3 merged |
| tool-result error flag dropped | 9 | 154 | 92 | 3/6/0 | E+O | 8 rep.: 5 merged, 1 ack., 2 open |
| metadata.user_id dropped | 6 | 92 | 71 | 4/2/0 | O | – |
| top_k dropped | 6 | 86 | 55 | 1/5/0 | O | – |
| assistant text after a call moved before it | 3 | 54 | 54 | 3/0/0 | E | – |
| max_tokens dropped | 1 | 51 | 51 | 1/0/0 | O | – |
| Var. | Specification | ( ) | T1 | Static T1 | Static + Av. (ledger) | Cell-level criteria | Primary (one finding) |
|---|---|---|---|---|---|---|---|
| 0 | Baseline S-strict-R9 (14 paths) | 685 (51) | 467 (68.18 %) | 333 (48.61 %) | 206 (30.07 %) | 146 (21.31 %) | 127 (18.54 %) |
| a1 | Drop redacted_thinking_dropped (O) findings | 685 (51) | 340 (49.64 %) | 234 (34.16 %) | 179 (26.13 %) | 146 (21.31 %) | 127 (18.54 %) |
| a2 | Drop 29 shapes with thinking=redacted | 308 (22) | 165 (53.57 %) | 101 (32.79 %) | 80 (25.97 %) | 69 (22.40 %) | 60 (19.48 %) |
| b | Drop 17 temperature + top_p shapes | 457 (34) | 297 (64.99 %) | 206 (45.08 %) | 125 (27.35 %) | 93 (20.35 %) | 84 (18.38 %) |
| c1 | Drop AI SDK v5 (11 paths, 5 codebases) | 532 (51) | 370 (69.55 %) | 254 (47.74 %) | 163 (30.64 %) | 107 (20.11 %) | 88 (16.54 %) |
| c2 | Drop all AI SDK paths (8 paths, 4 projects) | 379 (51) | 273 (72.03 %) | 175 (46.17 %) | 120 (31.66 %) | 68 (17.94 %) | 49 (12.93 %) |
| Stratum | Acceptance or coverage criterion | ( ) | T1 | Static T1 | Static + Av. | Primary (one finding) | Cell-level criteria |
| 1 | Static schema/doc contract S-strict-R9 (14 paths) | 685 (51) | 467 (68.18 %) | 333 (48.61 %) | 206 (30.07 %) | 127 (18.54 %) | 146 (21.31 %) † |
| 1 (11p) | Static S-strict-R9 , AI SDK v5 deduplicated (11 paths) | 532 (51) | 370 (69.55 %) | 254 (47.74 %) | 163 (30.64 %) | 88 (16.54 %) | 107 (20.11 %) † |
| 2a | Source live HTTP 200 on claude-sonnet-4-6 (14 paths) | 182 (13) | 80 (43.96 %) | 51 (28.02 %) | 42 (23.08 %) | 34 (18.68 %) | 39 (21.43 %) |
| 2a (11p) | Source live HTTP 200 on sonnet-4-6 , dedup. (11 paths) | 143 (13) | 65 (45.45 %) | 38 (26.57 %) | 31 (21.68 %) | 23 (16.08 %) | 28 (19.58 %) |
| 2b | Source live HTTP 200 on claude-opus-4-8 (14 paths) | 112 (8) | 58 (51.79 %) | 39 (34.82 %) | 30 (26.79 %) | 23 (20.54 %) | 28 (25.00 %) |
| 2b (11p) | Source live HTTP 200 on opus-4-8 , dedup. (11 paths) | 88 (8) | 47 (53.41 %) | 29 (32.95 %) | 22 (25.00 %) | 15 (17.05 %) | 20 (22.73 %) |
| Track | Model / mode | Mutation (translator behaviour) | Ref outcome | Mut outcome | Pre-specified prediction |
|---|---|---|---|---|---|
| MA-1 | claude-sonnet-4-6 , manual thinking (budget 1,024) | thinking blocks stripped from the assistant turn before tool_result | 10/10 HTTP 200; output_tokens_details 10/10 | 10/10 HTTP 200, ; thinking 0/10; output_tokens_details 0/10 | Ref HTTP 200 ; Mut HTTP 400 : not met |
| MA-2 | claude-sonnet-4-6 , adaptive | same | 10/10 200; output_tokens_details 10/10 | 10/10 200; thinking 0/10; output_tokens_details 10/10 | Mut 200 9/10: met |
| MA-2 | claude-haiku-5-5 , adaptive | same | 10/10 200 | 10/10 200; thinking 1/10 | Same criterion: met |
| MB-1 | claude-sonnet-4-6 | top_p 0.05 dropped | distinct strings per 5 samples: mean 3.8 ( 38/10) | 4.8 ( 48/10); [0.5, 1.5], | , : met |
| MB-2 | claude-haiku-5-5 | top_p 0.05 dropped | 10/10 HTTP 400 “top_p is deprecated for this model.” | 10/10 HTTP 200 | Ref HTTP 400 ; Mut HTTP 200 : met |
| MC3-1 | claude-sonnet-4-6 | mid-turn [harness] <tool> is DISABLED block (inlined in the last user turn) dropped | 10/30 refuse (no tool_use ); 20/30 call the tool, 9 of them stating that an in-chat instruction cannot disable a tool | 30/30 call the tool; [0.10, 0.63], | P-C3-1 ( , ): not met |
| Mechanism | Model | Intact (%; Wilson CI) | Translated (%; Wilson CI) | |
| Prompt, role, and tool information | ||||
| tool-result error flag dropped | gemini-3.5-flash-lite | 0.7 | 30/30 (100.0 %; 88.65–100.00) | 0/30 (0.0 %; 0.00–11.35) |
| (154 census cells) | gemini-3.8-flash | 0.7 | 30/30 (100.0 %; 88.65–100.00) | 0/30 (0.0 %; 0.00–11.35) |
| Qwen2.5-7B-Instruct | 0.7 | 25/60 (41.67 %; 30.06–54.27) | 0/60 (0.0 %; 0.00–6.02) | |
| gemini-3.5-flash-lite | 0.0 | 10/10 (100.0 %; 72.25–100.00) | 0/10 (0.0 %; 0.00–27.75) | |
| gemini-3.8-flash | 0.0 | 10/10 (100.0 %; 72.25–100.00) | 10/10 (100.0 %; 72.25–100.00) | |
| Arm | Input / control | Format % [CI] | Exact tag % [CI] | Last integer % [CI] | MAX | Mean tokens | Format [CI] | Exact tag [CI] | Last integer [CI] |
| A0 | Intact ref. ( max_tokens: 1024 , temp: 0.0 ) | 98 [93.0, 99.4] | 90 [82.6, 94.5] | 97 [91.5, 99.0] | 2 | 254.6 | — | — | — |
| A1 | musistudio->gemini (D02 + S08) | 100 [96.3, 100.0] | 91 [83.8, 95.2] | 99 [94.6, 99.8] | 0 | 291.8 | [ ] | [ ] | [ ] |
| A2 | aisdk7->gemini : driver-omission control; translator-fault claim withdrawn | 100 [96.3, 100.0] | 91 [83.8, 95.2] | 98 [93.0, 99.4] | 0 | 250.8 | [ ] | [ ] | [ ] |
| A4 | A0 repeated (nondeterminism floor) | 99 [94.6, 99.8] | 90 [82.6, 94.5] | 97 [91.5, 99.0] | 1 | 250.5 | [ ] | [ ] | [ ] |
| A5 | rosetta->gemini (clean wire) | 100 [96.3, 100.0] | 92 [85.0, 95.9] | 100 [96.3, 100.0] | 0 | 249.9 | [ ] | [ ] | [ ] |
| A6 | A0 minus temperature , maxOutputTokens | 100 [96.3, 100.0] | 91 [83.8, 95.2] | 98 [93.0, 99.4] | 0 | 263.4 | [ ] | [ ] | [ ] |
| openai-python 2.54.0 | openai-node 5.23.2 | Two-SDK union ( ) | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Family | Exc. | Corrupt | Clean/U | Exc. | Corrupt | Clean/U | Fail (%) | Any exc. | Both corr. | Exc./corr. split | |
| Bifrost | 80 | 2 | 8 | 59/11 | 11 | 6 | 52/11 | 17 (21.25 %) | 11 | 6 | 2 / 2 / 7 / 6 |
| new-api | 80 | 15 | 1 | 53/11 | 6 | 3 | 60/11 | 20 (25.00 %) | 20 | 0 | 1 / 4 / 15 / 0 |
| CLIProxyAPI | 78 | 4 | 13 | 50/11 | 9 | 6 | 52/11 | 17 (21.79 %) | 11 | 6 | 2 / 7 / 2 / 6 |
| Portkey | 80 | 2 | 21 | 47/10 | 5 | 19 | 46/10 | 24 (30.00 %) | 5 | 19 | 2 / 2 / 1 / 19 |
| one-api | 80 | 14 | 40 | 21/5 | 41 | 17 | 17/5 | 58 (72.50 %) | 55 | 3 | 0 / 51 / 4 / 3 |
| LiteLLM | LLM-Rosetta | CLIProxyAPI | Bifrost | AI SDK | ||||||
| Property (applicable) | c | R | c | R | c | R | c | R | c | R |
| Top-level system prompt kept (239) | 239 | 239 | 239 | 239 | 239 | 239 | 239 | 0 ‡ | 79/79 | 79/79 ‡ |
| Mid-turn system: role and place kept (236) | 0 | 236 | 160 | 236 | 0 | 0 | 236 | 236 | 160/160 B | 160/160 B |
| Mid-turn system: moved behind the next tool result (76) | 76 | 0 | 76 | 0 | 76 | 76 | 0 | 0 | rej. | rej. |
| Thinking text kept (121) | 121 | 121 | 121 | 0 | 0 | 0 | 121 | 0 | 0/40 | 0/40 |
| Signature (121), redacted data (116), is_error trace (146) | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 | 0 |
| Project | Issues | Comments | Window | State in the thread |
|---|---|---|---|---|
| LiteLLM | Oct. 2026 | 3 competing fix PRs on 1 issue; | ||
| fix PR open (verified by us on the PR head) on 1; others no reply | ||||
| Vercel AI SDK | Oct. 2026 | fixed, released in ai 7.0.132; re-run by us | ||
| LLM-Rosetta | Oct. 2026 | 3 issues fixed on master; re-run by us | ||
| @musistudio/llms | – | Oct. 2026 | no reply | |
| claude-code-router | – | Oct. 2026 | no reply |
| Pattern (rules) | Portkey | Kong | new-api | one-api | Bifrost | Envoy | |
| audited commit | 669825c | 8927af6 | 56758ed | 8df4a26 | 080a6bb | 97f5609 | |
| P1 | Forged Gemini thought signature (V10) | C | R | – | C | ||
| P2 | Inconsistent mid-conversation system handling (V01/V02/V05) | C | ? + | C | C | ||
| P3 | Tool-result error status dropped or mis-mapped (V03) | – | – | C | – | C | |
| P4 | Gemini tool-call/result identifiers dropped (V06) | C | C | ? | |||
| P5 | Finish/stop reason mapping (RV01/RV06) | ? | ? |
| Second hop | Target | Judged (refused) | No end-to-end T1 | Carry | Inject | Accum. | Cancel | Two-hop-only T1 pure mixed total (%) | End-to-end T1 (%) |
| S_all_357 (24 chains; includes original-driver AI SDK captures) | |||||||||
| LiteLLM | Anthropic | 1,585 (557) | 0 | 0 | 70 | 969 | 30 | 469+47 = 516 (32.56%) | 1,585 (100.00%) |
| LiteLLM | Gemini | 1,514 (628) | 16 | 308 | 70 | 607 | 67 | 406+40 = 446 (29.46%) | 1,498 (98.94%) |
| LLM-Rosetta | Anthropic | 1,585 (557) | 18 | 250 | 60 | 693 | 18 | 544+2 = 546 (34.45%) | 1,567 (98.86%) |
| LLM-Rosetta | Gemini | 1,585 (557) | 16 | 346 | 73 | 669 | 0 | 481+0 = 481 (30.35%) | 1,569 (98.99%) |
| all | 6,269 (2,299) | 50 | 904 | 273 | 2,938 | 115 | 1,900+89 = 1,989 (31.73%) | 6,219 (99.20%) | |
| Evidence | Protocol and scope | Interpretation and location |
|---|---|---|
| Preservation rules | The main checking framework has 27 rules, grouped as T1 , T2 and T3 . The primary rules were fixed before the main held-out sets were authored. | Rule mappings: Table 10 ; request results: Section 6 ; replies: Appendix B.4 . |
| Primary held-out evaluation | LLM agents designed Sets 9–14 from the specification without access to checker code. Report activated faults and allowed-change controls separately. | Independent detection and false-alarm estimates, not development-set performance (Table 12 ; Appendices A.1 and A.3 ). |
| Development and repair analyses | Earlier rule calibration (in-sample Sets 1–8) and exploratory post-freeze repairs (evaluated on Sets 17–20; script-generated Sets 15–16 voided) use separate batches. A set used to devise a repair cannot independently evaluate that repair; only subsequently authored blind sets can do so. | Historical comparisons remain separate in Appendix A.3 and the replication archive. They are not additional isloss results. |
| Census and human review | Use the specified capture points and corrected drivers; distinguish completed translations, individual findings, root-cause candidates and mechanisms. | Request census, behavioural probes and deployments have different units (Sections 6.1 and 6.5 ; Table 15 ). |
| Corrections and sensitivity checks | Reconcile parser, harness and review errors separately from alternative validity or counting policies. | Table 30 identifies corrected preliminary counts and sensitivity bounds; Table 17 reports alternative choices. |
| Track / experiment | Held | Scope | Missed predictions (pre-specified threshold observed) |
|---|---|---|---|
| RQ1 mechanism concentration | 1/4 | 51 shapes; 14 paths | “after removing the top-3 most pervasive mechanisms, the T1 share of judged cells stays 60 % (from 85.6 %) and the share of 4,998 stays 45 % (from 64.0 %)” judged 32.8 % CI [28.5, 37.2]; of 714 31.5 % CI [27.3, 35.7]; “the top-3 by pervasiveness are exactly { is_error_dropped , control_max_tokens_dropped , redacted_thinking_dropped } (any order)” redacted_thinking_dropped, is_error_dropped, control_user_id_dropped; “binary real-request estimate averaged over the 14 paths: 50 % with the max_tokens mechanisms, 50 % without them” with 33.2 %, without 32.1 %. |
| RQ1 K-class candidate count | 2/3 | 56 pre-review candidates | “Among the 56 path mechanism candidates, V accounts for candidates and K accounts for candidates” V = 27 , K = 14 . |
| RQ1 upstream report coverage | 2/3 | 56 pre-review candidates | “ 40 of 56 candidates have a U-id (we reported them), and 10 of those are fixed today” 15 reported, 8 fixed. |
| RQ1 replies (reverse, forward) & SDKs | 9/12 | 1,115 cells / 558 streams / 61 forward reply shapes | “ >= 4/7 families ( T1 S12 STOP : Gemini finishReason: "STOP" + functionCall translated to OpenAI finish_reason == "stop" while tool_calls are present)” 3/7 families (rosetta, oneapi, portkey); “ >= 2/5 HTTP proxy families ( T1 S13 STREAM : Missing terminal event / [DONE] or broken stream ledger)” 0/5 HTTP proxy families fire S13 under default context (no non_stream_body); “B4c — Forward non-stream ranking preservation: Under isloss ( side="response" ), aisdk5 and litellm have strictly higher non-stream T1 -clean cell rates ( 0 T1 findings) than rosetta , musistudio , and fuergaosi233 across spec-compliant backend response shapes” nonstream T1-clean rates: aisdk=80.3%, aisdk5=80.3%, fuergaosi233=85.2%, litellm=91.8%, musistudio=49.2%, rosetta=62.3%. |
| RQ1 live vendor acceptance (C1) | 2/5, 1 n/e | 548 calls | “ claude-sonnet-4-6 accepts all 34 strict shapes that do not set temperature and top_p together (34/34)” 13/34; “ Gemini agreement (valid 200, invalid 4xx) 90%” 160/204; “Restated: The 19 midsys-only shapes (which all carry role: system inside messages[] ) are rejected by both models ( 18/19 each)” 19/19 & 16/19 (sonnet 19/19 ( 18 met), opus 16/19 ( 18 not met)). |
| RQ2 consequence | 13/16 | 4 models | “ M3: is_error_dropped : Mean scenario-level difference , with on scenarios and ” mean_diff=0.4167, pos=5/12, p_perm=0.062500 ; “On Meta-Llama-3.1-8B-Instruct , at least 4 / 5 mechanisms achieve with pp” 3/5 mechanisms; “New mechanism N2 ( function_response_id_dropped , 37 silent cells) produces pp and on both Llama-3.1-8B and Qwen2.5-7B ” Llama-3.1-8B: +0.00 pp (p_Holm=1.0); Qwen2.5-7B: +0.00 pp (p_Holm=1.0). |
| Track / experiment | Held | Scope | Missed predictions (pre-specified threshold observed) |
|---|---|---|---|
| RQ3 LiteLLM 50-release timeline | 3/11, 1 n/e | 50 rel. | 8 of 11 missed (1 n/e): 50-release timeline 8/9 missed (1 n/e), isloss recount 0/2 missed. |
| RQ3 five-library two-exits | 161/187 | 5 libs | 26 of 187 missed: LiteLLM Responses 4/22 and Chat side-by-side 1/18, LLM-Rosetta 1/32, CLIProxyAPI 3/28, Bifrost 1/22 and #8209 check 0/5, AI SDK run A 0/23, run B ( allowSystemInMessages ) 12/14, version-equality 1/1, and GA follow-up 3/22. |
| RQ3 same-vendor controls | 764/804, 2 n/a | 5 gw/libs | 40 of 804 missed (2 n/a): Anthropic request round-trip 3/192 and follow-up A 1/86, Anthropic response 6/110 (1 n/a) and follow-ups B 0/49, C 5/88 (1 n/a), D 1/58, E 3/79, OpenAI response 13/92, and OpenAI request 8/50. |
| RQ3 corroboration | 1/2 | 80 items | “3-class Cohen’s : Unweighted 3-class Cohen’s ( exact vs related vs unrelated / none ) across the 80 audited items is >= 0.60 ” ( 57/80 = 71.25% ; binary exact vs not-exact ). |
| RQ4 chains / gateways / agents | 21/39, 1 n/e | 4 gw + CLI | 18 of 39 missed (1 n/e): CLI coding-agent round-trip 0/4, forward SSE stream census 3/4, Responses ingress census 0/4, self-hosted gateway census 14/26 (1 n/e), and 2-hop chains 1/1. |
| RQ4 fallback & accounting | 414/500 | Bifrost/LiteLLM | 86 of 500 missed: LiteLLM main two-exit re-run 0/3; 42 Bifrost/LiteLLM routing, fallback, key, budget, and accounting addenda 86/497 missed (37 single-tree addenda 78/459 missed; 5 multi-tree accounting addenda 8/38 missed: CE 0/8, CF 2/10, CG 2/5, CH 0/8, CI 4/7). |
| Item | Audit finding or correction | Effect on results | Location |
|---|---|---|---|
| Motivating-example placement | The initial Case A placed system directly before user : this placement was absent from the 10 recorded occurrences in 36 captured requests and failed static beta rules. The main example instead uses Case A ′ placements before assistant (3/10 occurrences) and at messages[-1] (7/10); all 22 paraphrased requests pass static checks. Case A remains an auxiliary experiment ( , temperature 0, including token-limit stops; LiteLLM contrast within the experiment and over 64 contrasts). | Primary rate unchanged: none of the 51 main-suite shapes has a mid-conversation system message. Case A ′ reports for two Gemini models and Qwen2.5-7B; on gemini-3.5-flash-lite , deletion reduces refusals from 11/11 to 0/11 in both placements. | § 2.1 ; App. A.1 |
| Claude instruction-loss probe | Design C3 has 240 calls across the instruction-loss ( MC3-1 ) and token-limit ( MC3-2 ) experiments. The instruction-loss experiment comprises 120 calls: 10 scenarios 3 repeats 2 arms 2 models. Its notice is the first text block in the last user turn, retained in Ref and dropped in Mut ( max_tokens =1,000; no role:system , beta header or sampling controls). Haiku refusals are 30/30 vs. 0/30 ( , , prediction met); Sonnet refusals are 10/30 vs. 0/30 ( , CI , , prediction not met; 9/20 Ref tool calls mention user-turn placement). | Both models are reported alongside the separate Gemini and Qwen system-role placement experiments. Request-census counts are unchanged. | § 2.1 ; § 6.2 ; Table 19 |
| Static checks and live rejection | Case A rejection and Case A ′ validity under the mid-conversation-system-2026-04-07 beta rules are static-validator results, not observations of a live beta-header experiment. The live Sonnet C1 run omitted that header and rejected 19/19 role:system shapes. Separately, among 29 shapes carrying redacted_thinking , live rejections comprise 19 thinking, 9 sampling-combination and 1 empty error-result rejection; the associated LLM-Rosetta comparison has 19/38 flips. This 29-shape subset differs from the full source-acceptance rejection count. | Wording correction only; no headline count changes. | App. A.1 ; App. A.2 |
| Voided Sets 15–16 | These sets were produced by a fuzzer script rather than an independently tasked LLM agent without checker-code access, and were excluded before scoring the held-out summary. Diagnostic counts are retained for transparency: Set 15 has 48 fault mutants, including 4 uncaught script exceptions (40/48 = 83.3 % if crashes count as detections; 36/48 = 75.0 % otherwise); Set 16 has 36 allowed changes, with 1/36 false alarms (2.8 %). | Both sets are excluded. Primary Sets 9–14 give 95/125 recall and 2/100 false alarms; pooling non-voided Sets 9–14 and 17–20 gives 160/220 (72.7 %) and 5/170 (2.9 %). | § 4 ; App. A.3 |
| Human-review reconciliation | Removed LLM-Rosetta to Gemini, list-valued tool-result re-encoding; 7 statically accepted cells, the only consensus Grade B candidate classified Borderline. Merged duplicate cards AI SDK v7/v5 to Anthropic, moved assistant text; 18 statically accepted cells each, Grade A. The avoidable class has 28 cards: 23 A, 1 B and 4 C; confirmation is 23/24 (95.8 %) among agreed cases or 23/28 (82.1 %) overall. The earlier tally had incorrectly subtracted that Borderline candidate. | Catalogue: 56 to 54 candidates (31 A, 11 B, 12 C; 49 to 48 on 11 paths), and 19 to 18 mechanisms. Primary counts stay 127/685 and 88/532. Removing the accepted finding gives audit-adjusted T1 cells 467 to 464, statically accepted T1 cells 333 to 330, and the alternative cell-level conjunction 146 to 140/685 (20.44 %); these are not changes to the primary finding-level rate. | § 4 ; § 6.1 ; App. C |
| Stream-reassembly audit | A nine-line correction to _parse_openai_chat_stream assembles delta.reasoning_content as the non-stream parser does. We rechecked 41 affected reply streams. | All 714 request attempts (685 judged translations) and Sets 9–14 are unchanged. Preliminary reverse-reply T1 : 402 to 391/1,115 (35.1 %); gateway-stream T1 : 61 to 57/198 (28.8 %); affected agent sessions: 12 to 9/12. Main results use the corrected counts. | § 4.1 ; § 6.3 ; § 6.5 ; App. C |
LLM Reliability
Large language model reliability studies whether outputs remain correct, calibrated, robust, and controllable across prompts, contexts, and deployment conditions.
See all 866 papers