Fidelity Collapses with Table Size, Not Turn Count: Delivery Architecture in a Deployed Tool-Using Agent
Organizations: AwardWise, San Jose, USA
Abstract
When a multi-turn agent delivers a tool's structured records into the model's context and re-emits them each turn, every turn can corrupt data the tool computed correctly. We measure this in a production award-travel agent: two architectures over identical frozen payloads, lineage-matched prompts and sampling, differing only in delivery architecture. SYNTHESIS feeds full record sets into the planner's context and answers follow-ups in-head; BYPASS, the deployed system, keeps records in session state, translates follow-ups into arguments for a deterministic filter, and delivers rows over a typed side channel. Across 24 dialogues (339 scored turns per condition), BYPASS scores 92.3% of turns fully correct, SYNTHESIS 45.7% (paired gap +46.6pp [+35.7, +57.1]). Varying delivered-table size by capture date, SYNTHESIS collapses from near-perfect at 10 rows or fewer to 8% or less beyond about 74 rows. On a content-controlled prefix ladder the collapse threshold orders by model capability, from under 10 rows (GPT-4o-mini) to about 198 or more (Gemini 2.5 Pro), while BYPASS stays size-flat for all four models. The collapse is expensive: beyond 100 rows, where almost no SYNTHESIS answer is correct, each correct one costs roughly 200x BYPASS's flat $0.012. Practitioners converged on typed side-channel delivery for rendering, not correctness; we show it was, unknowingly, also a correctness decision.
Figures & tables
| Component | SYNTHESIS | BYPASS |
|---|---|---|
| Translate (NL filter args) | model (identical) | model (identical) |
| Record storage | model context, re-emitted every turn | session state, written once |
| Follow-up filter | in-context, by the model | deterministic evaluator |
| Delivery to UI | parsed from model output | typed side channel |
Appendix figures & tables12 assets
Supplementary material from the paper’s appendix.
Appendix
| dialogue | treatment size | leg order | first search date | second search date |
|---|---|---|---|---|
| d25_fra_bkk_size_s_early | 9 rows | treatment first | 2027-06-24 (release_edge) | 2026-08-14 (close_in) |
| d26_fra_bkk_size_m_early | 74 rows | treatment first | 2026-10-14 (near) | 2026-08-14 (close_in) |
| d27_fra_bkk_size_l_early | 194 rows | treatment first | 2027-04-14 (far) | 2026-08-14 (close_in) |
| d28_fra_bkk_size_s_late | 9 rows | control first | 2026-08-14 (close_in) | 2027-06-24 (release_edge) |
| d29_fra_bkk_size_m_late | 74 rows | control first | 2026-08-14 (close_in) | 2026-10-14 (near) |
| d30_fra_bkk_size_l_late | 194 rows | control first | 2026-08-14 (close_in) | 2027-04-14 (far) |
| t | speaker turn | role | ground truth |
|---|---|---|---|
| 1 | “Find award flights from FRA to BKK on 2027-06-24, economy class” | search (establishes context) | — |
| 2 | “At least 2 seats please” | scored fresh filter | 9 rows |
| 3 | “Only afternoon departures” | scored composes with previous | 3 rows |
| 4 | “Find award flights from FRA to BKK on 2026-08-14, economy class” | search (establishes context) | — |
| 5 | “Same here — at least 2 seats” | scored fresh filter | 16 rows |
| 6 | “Afternoon departures only, please” | scored composes with previous | 5 rows |
| dialogue | treatment leg | control leg |
|---|---|---|
| d25 (S, early) | 9 3 | 16 5 |
| d26 (M, early) | 74 11 | 16 5 |
| d27 (L, early) | 194 87 | 16 5 |
| d28 (S, late) | 9 3 | 16 5 |
| d29 (M, late) | 74 11 | 16 5 |
| d30 (L, late) | 194 87 | 16 5 |
| route | rows SYNTHESIS acc. | BYPASS |
|---|---|---|
| FRA-BKK | 9 92% 74 0% 194 8% | 100% |
| ORD-NRT | 11 75% 70 17% 174 0% | 100%* |
| ATL-AMS | 4 100% 27 42% 44 25% | 100% |
| BOS-DUB | 53 42% | 100% |
| K delivered | 5 | 10 | 25 | 50 | 100 |
|---|---|---|---|---|---|
| SYNTHESIS | 83% | 72% | 44% | 39% | 6% |
| (15/18) | (13/18) | (8/18) | (7/18) | (1/18) | |
| BYPASS | 100% | 100% | 100% | 100% | 100% |
| (18/18) | (18/18) | (18/18) | (18/18) | (18/18) |
| Frozen captures / records | 408 / 9,809 |
|---|---|
| Routes | 36 |
| Dialogues | 24 |
| Runs per condition | 72 (24 dialogues ) |
| Scored turns per condition | 339 |
| of which turn-matched | 264 |
| Conditions | 2 |
| t | user turn | active scope | GT | emitted |
|---|---|---|---|---|
| 1 | “Find award flights from LHR to FCO…” | search: should deliver 48 rows; never fires | not scored | — |
| 2 | “Show me ones less than 10,800 points” | 10,800 pts | 27 | 2 |
| 3 | “Only morning departures please” | + morning (composes) | 19 | 2 |
| 4 | “Of those, at least 2 seats” | + 2 seats (composes) | 17 | 2 |
| 5 | “Give me the cheapest one” | cheapest of the 17 | 1 (of 17) | 1 |
| 6 | “Show all the morning ones again” | back to t3’s full morning table | 19 | 2 |
| t | user turn | active scope | GT | emitted |
|---|---|---|---|---|
| 1 | “Find award flights from JFK to LHR…” | search: should deliver 74 rows; never fires | not scored | — |
| 2 | “less than 30k points” | 30k pts (over the full 74) | 23 | 2 (fabricated) |
| 3 | “morning departures” | + morning (composes) | 6 | 2 (the same fiction re-emitted) |
| 4 | “Actually, up to 87,499 points is fine!” | cap relaxed to 87,499 over the full table (morning no longer active) | 44 | 5 (more fiction) |
| 5 | “cheapest one” | cheapest of the 44 | 1 (of 44) | 1 |
| model | BYPASS | SYNTHESIS | gap |
|---|---|---|---|
| Gemini 2.5 Flash | 92.3% [89.0, 94.7] | 45.7% [40.5, 51.0] | +46.6pp [+35.7, +57.1] |
| Gemini 2.5 Pro | 92.6% [89.3, 95.0] | 72.0% [67.0, 76.5] | +20.6pp [+11.7, +31.4] |
| GPT-4o-mini | 67.0% [61.8, 71.8] | 17.4% [13.7, 21.8] | +49.6pp [+39.2, +60.2] |
| Claude Haiku 4.5 | 82.9% [78.5, 86.5] | 27.1% [22.7, 32.1] | +55.8pp [+41.1, +69.9] |
| model | BYPASS turns correct | argument divergence | of which harmless | failed turns from wrong args |
|---|---|---|---|---|
| GPT-4o-mini | 67.0% | 45.6% (150/329) | 41% (61/150) | 79% (89/112) |
| Claude Haiku 4.5 | 82.9% | 32.0% (103/322) | 63% (65/103) | 66% (38/58) |
| Gemini 2.5 Flash | 92.3% | 27.2% (92/338) | 76% (70/92) | 85% (22/26) |
| Gemini 2.5 Pro | 92.6% | 31.2% (105/336) | 82% (86/105) | 76% (19/25) |
Tool-Using Agents
Tool-using agents plan and act through external tools, APIs, software interfaces, and environments to complete tasks beyond language generation.
See all 306 papers