cs.CLOct 14, 2026
SaveSame Context, Different Agents: A Behavioral Study of Semantic Layers, Multi-Agent Orchestration and Memory Compaction in Data Agents
Organizations: University of Southern California · Oracle AI · University of Pennsylvania
Abstract
Answering questions over heterogeneous data means combining evidence from files, databases, documents and the web. Data agents automate this, but their success depends on context. Three context-engineering techniques (CETs) help by shaping it: semantic layers describe the data, orchestration splits work among agents, and compaction condenses interaction history. Existing studies validate each CET alone, leaving open how agent behavior shapes its effect. We present a comprehensive analysis of the three CETs, alone and combined, with seven large language models (LLMs) on 1,514 long-horizon tasks from five benchmarks, covering behavior, accuracy and cost. We find that data agents without CETs form three behavior groups: investigators, premature answerers that answer without inspecting data, and early stoppers that give up. Each CET delivers its known benefit, from more investigation to smaller prompts, but only the behavior group it matches turns that benefit into accuracy, such as premature answerers under orchestration. For investigators, the same CETs can hurt: the semantic layer stops them from looking, orchestration makes them overconfident, costing 5 percentage points, and compaction makes them repeat steps. Combining CETs yields smaller gains and losses than their sum. We derive guidance for choosing CETs by agent behavior, task demands and accuracy-cost trade-offs.
Figures & tables
Figure 1: Same context, different agents (illustration). Given the same question and data, an investigator checks every source and answers from the data, a premature answerer answers without looking, and an early stopper gives up partway.
Figure 2: Data agents and the context-engineering techniques that help them. Data agents answer questions over heterogeneous data, and context-engineering techniques (CETs) help by shaping the context the model works from: a semantic layer describes the data, orchestration splits the work and compaction condenses the history. What we do not yet know is how each changes what different LLMs do, alone and combined.
Figure 3: How the baseline agent behaves, by behavior group. Each axis is the share of a group’s baseline runs that show a behavior, from 0 at the center to 100% at the outer ring (mean of the group’s models). Investigators look and persist, premature answerers often answer without acting, and early stoppers act but give up and repeat themselves.
| Krama | DataC. | FDA rep. | FDA ch. | HDR | |
|---|---|---|---|---|---|
| Opus 5.5 | 82 | 71 | 60 | 70 | 61 |
| GPT-6 Astra | 81 | 72 | 59 | 67 | 69 |
| Sonnet 5.5 | 79 | 66 | 46 | 56 | 61 |
| GPT-5.5 | 22 | 63 | 10 | 3 | 9 |
| GPT-5.4 | 37 | 28 | 28 | 4 | 8 |
| Gemini 2.5 Flash | 50 | 22 | 24 | 8 | 19 |
Table 1: Baseline agent accuracy in % on each benchmark’s own metric, with rows by behavior group (investigators, premature answerers, early stoppers).
Figure 4: Accuracy-cost trade-offs depend on the behavior group. Each arrow goes from the baseline agent (square) to the agent with one CET: accuracy change in points and dollars per task as a multiple of the baseline’s. Orchestration gives the largest gain for premature answerers and the semantic layer for early stoppers; for investigators every CET lowers accuracy or leaves it unchanged, and every CET raises the cost.
Figure 5: Task demands modify CET effects, and process gains can accompany accuracy losses. Rows show each CET’s effect on accuracy (overall and within-benchmark contrasts between tasks with and without each demand), process score (the mean of three benchmarks’ step scores), and dollars per task. Blue indicates improvement and red degradation; white marks effects below the display thresholds of one accuracy point, three process-score points, and 1.1 times the cost. Figure 16 reports every demand, each benchmark’s process measure, and which changes are significant.
Figure 6: How each CET changes what agents do. For each group, a line outside the dashed ring means the CET makes a behavior more common and a line inside it less common; behaviors are grouped into engagement, reliability and commitment. Orchestration raises engagement, the semantic layer raises reliability at the cost of engagement, and compaction lowers reliability; Figure 17 gives every measure.
Figure 7: The semantic layer cuts give-ups, long loops and calls, but raises the bill and makes investigators stop looking. It helps most the agents that give up, halving their give-ups and loops at little extra cost. Investigators instead stop using tools on most runs whose tool use changes, and their accuracy there falls from 68% to 31%.
Figure 8: Compaction limits prompt growth but increases cost and repetition. “Loses a value” measures incorrect final answers after the correct value appeared in a tool output on DataClaw.
| baseline | +C | +S | +S+C | +M | +M+C | +M+S | +M+S+C | |
|---|---|---|---|---|---|---|---|---|
| Opus 5.5 | 68.8 | 65.2 | 65.6 | 65.3 | 61.9 | 64.0 | 61.8 | 63.1 |
| GPT-6 Astra | 69.6 | 65.1 | 69.6 | 65.7 | 61.0 | 56.4 | 61.0 | 56.4 |
| Sonnet 5.5 | 61.3 | 57.7 | 61.2 | 61.8 | 60.0 | 58.4 | 57.0 | 56.0 |
| GPT-5.5 | 21.4 | 20.8 | 23.7 | 23.9 | 58.1 | 55.6 | 57.8 | 56.4 |
| GPT-5.4 | 21.0 | 27.0 | 29.2 | 31.3 | 37.2 | 36.4 | 40.1 | 39.3 |
| Gemini 2.5 Flash | 24.6 | 22.3 | 27.0 | 26.0 | 24.7 | 20.9 | 29.2 | 27.7 |
Table 2: Accuracy of every combination of CETs (%, mean of the five benchmarks; M: orchestration, S: semantic layer, C: compaction). The best combination depends on the group: no CET for investigators, orchestration, mostly with the layer, for the others.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Baseline behavior profile of every model. Same axes as Figure 3 , one radar per model, grouped by behavior group.
Figure 10: Each benchmark demands something different. Share of each benchmark’s tasks that need each capability, with the same labels for every benchmark.
Figure 11: Full demand profiles. Share of each benchmark’s tasks at each of the 88 levels of the 19 demand dimensions.
Figure 12: Reasoning removes GPT-5.4’s habit of answering before looking, but not GPT-5.5’s. Share of baseline runs that answer without any tool (orange) and accuracy (blue) at each reasoning effort, on KramaBench and HDR. GPT-5.4 does not reason at its default; GPT-5.5 does.
Figure 13: Behavior types by model. Share of each model’s baseline runs in eight behavior types. Investigators spend almost all runs in three productive types; GPT-5.5 answers without looking in half of its runs; GLM spends a third of its runs in trial and error and repetitive loops.
Figure 14: Most wrong values after looking come from the wrong slice of data or from external evidence, not from arithmetic. Share of each group’s wrong values in twelve causes grouped into six families (color: family; shade: cause), from 790 annotated baseline wrong values weighted to the wrong values they represent.
Figure 15: Each CET changes size on a few demands, and only for some groups. For each CET (rows) and behavior group (columns), the demands on which the effect against the baseline agent departs from the group’s effect on all tasks (dashed line) by at least 2 points, with a 95% task-bootstrap interval that excludes it; at most the three largest departures in each direction (“top of ” when more qualify). Blue: helps more than on all tasks; red: helps less or hurts more. Values are points of accuracy, groups are the mean of their models, and each cell has its own scale. : at least 74% of the level’s tasks come from DataClaw.
Figure 16: Every demand and every benchmark’s process measure. Same as Figure 5 , with three more demands and each benchmark’s process measure in place of their mean. HDR has no step score, so its row is the share of runs that time out (fewer is better).
Figure 17: Orchestration raises engagement, the layer raises reliability at the expense of engagement, and compaction lowers reliability. Change in each behavior, in points of runs, from the agent without the CET to the agent with it, for each behavior group; blue is more of the property, red less, and bold an interval that excludes no change. Outlined cells are the main shifts; other cells are muted. The failure mix covers wrong data slice, wrong conclusion from web or document evidence, code that never ran and arithmetic, with purple for the size of the shift.
Figure 18: Behavioral fingerprint by model. Same as Figure 17 , one row per model.
Figure 19: Forcing a single GPT-5.5 agent to act, or to spend the orchestrated agent’s calls, closes only 7-37% of orchestration’s gain. Accuracy against dollars per task for GPT-5.5: the baseline agent, the single agent forced to act before answering (A), the single agent forced to spend the orchestrated agent’s number of calls on each task (B) and the orchestrated agent; 95% intervals over tasks.
Figure 20: Orchestration moves premature answerers’ runs into looking. Behavior types of runs on the same tasks without (left) and with orchestration (right), for each behavior group; bold ribbons are the main moves.
Figure 21: Orchestration helps most where the baseline agent is weakest. Accuracy change of orchestration against the baseline agent’s accuracy on the benchmark, one point per model and benchmark (Spearman ; against the baseline share of answers without a tool, ).
Figure 22: The layer moves every group toward quick lookups: it replaces failure for early stoppers and work for investigators. Behavior types of runs without and with the layer, by group.
Figure 23: Each part’s content reaches the agent’s actions where it is meant to. Without (open) and with (filled) the layer: runs whose first action opens the source the layer lists first (KramaBench and DataClaw, both architectures); runs with a failed read (KramaBench, single agent); runs that use a join key from the layer on KramaBench tasks that need no join (open) or need one (filled). An element counts as used only when it appears in an executed action.
Figure 24: A small author suffices, and the layer makes models alike in how they work more than in accuracy. Left: accuracy gain of a GPT-5.4 single agent over its no-layer run for layers written by models of increasing size on DataClaw (28% accuracy without a layer), with 95% intervals paired by task. Right: spread across the seven models (standard deviation, points) of each measure without (open) and with (filled) the layer; reaching a gold source is measured on single-agent KramaBench and DataClaw runs, the other measures over all runs.
Figure 25: Compaction changes little for investigators and pushes the weaker groups into long repetitive runs. Behavior types of runs without and with compaction, by group.
Figure 26: The best configuration depends on the model. Accuracy of the eight configurations (mean of five benchmarks), one line per model; rings mark each model’s best configuration (a gain under one point counts as the baseline agent).
Figure 27: CETs that repair the same habit add up to less than their parts. Each CET as a vector from the baseline agent (square): accuracy change in points (mean of five benchmarks) and dollars per task as a multiple of the baseline agent’s (log scale). For each combination (columns) and group (rows), the parts are chained head to tail in color, their sum is the open circle, and the black arrow is the measured combination; the dotted segment is the interaction. Groups are the mean of their models (geometric mean for dollars).
Figure 28: The CETs change accuracy where some agent already succeeds, not where the gold says a task is long. Effect of each CET (averaged over the settings of the other two) by task difficulty, for each group: (A) how many of the seven baseline agents solve the task; (B) the share of the other groups’ baseline agents that solve it, which excludes the group’s own runs; (C) terciles of the gold number of dependent steps within each benchmark; (D) the benchmarks’ own easy, medium and hard labels; (E) terciles of the other groups’ baseline run length in model calls and (F) in output tokens, within each benchmark. Filled markers have a 95% task-bootstrap interval that excludes 0.
Figure 29: Dollars per task by model and configuration. Pooled over the five benchmarks at list prices (right column: input, cached input and output dollars per million tokens); darker cells cost more.
| Score | Arch. | Astro. | Biomed. | Env. | Legal | Wildf. | Pipeline | Calls | In (k) | Out (k) | Time (s) | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Opus 5.5 | baseline | 82.4 | 50.0 | 75.0 | 83.3 | 83.3 | 90.0 | 93.1 | 35.8 | 3.8 | 47.8 | 1.9 | 104 |
| +S | 0.9 | +0.0 | 2.1 | 3.0 | 8.8 | +1.7 | +3.8 | +3.7 | 1.1 | +73.1 | 0.5 | 19 | |
| +M | +0.5 | +0.0 | +6.2 | +8.1 | 1.2 | 1.7 | 1.2 | +5.7 | +2.0 | +39.1 | +4.7 | +42 | |
| +C | +0.4 | +0.0 | +2.1 | 5.4 | +1.2 | +0.0 | +2.0 | +2.1 | +1.2 | 13.3 | +1.5 | +31 | |
| GPT-6 Astra | baseline | 81.0 | 50.0 | 75.0 | 88.9 | 88.3 | 80.0 | 93.1 | 45.7 | 3.5 | 35.4 | 1.5 | 49 |
| +S | 1.4 | +0.0 | 6.2 | 2.2 | 4.1 | 0.8 | +2.8 | +0.1 | 1.5 | +57.4 | 0.6 | +9 |
Table 3: KramaBench in its own metrics. End-to-end score overall and in each of the six domains, pipeline design (share of the human-annotated key functionalities an LLM judge finds in the executed code, judged by GPT-5.4 with the benchmark’s prompt), LLM calls, input and output tokens (thousands) and runtime per task. Each model’s first row is the baseline agent; the next three rows give the change made by the layer (+S), orchestration (+M) and compaction (+C), each averaged over the settings of the other two. Rates are in points.
| Acc | Easy | Med. | Hard | EE | GPR | TPE | Calls (C) | Calls (W) | In (k) | Out (k) | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Opus 5.5 | baseline | 70.8 | 91.6 | 67.8 | 46.0 | 57.9 | 40.9 | 88.7 | 6.8 | 8.0 | 68.3 | 3.1 |
| +S | 2.3 | +0.2 | 2.1 | 7.1 | +29.7 | 5.5 | +9.5 | 5.3 | 6.2 | +381 | 4.3 | |
| +M | +1.5 | +0.6 | +1.8 | +2.0 | 16.0 | +12.1 | 7.7 | +7.9 | +11.2 | +321 | +12.7 | |
| +C | 0.3 | +0.2 | 0.8 | +0.5 | 3.8 | 0.2 | +0.0 | +6.6 | +8.5 | +44.9 | +12.1 | |
| GPT-6 Astra | baseline | 71.9 | 90.8 | 68.5 | 52.0 | 65.7 | 44.3 | 93.7 | 5.8 | 7.2 | 79.2 | 1.6 |
| +S | 1.2 | 0.2 | 0.6 | 5.5 | +27.0 | +0.1 | +7.0 | 9.5 | 35.2 | +222 | 35.2 |
Table 4: DataClaw in its own metrics. Judged accuracy overall and by difficulty; execution efficiency (EE, gold steps over agent steps, correct tasks only); goal progress rate (GPR, share of gold milestones reached) and temporal progress efficiency (TPE, ), both on incorrect tasks only, judged by GPT-5.4; LLM calls on correct (C) and wrong (W) tasks; input and output tokens (thousands) per task. Each model’s first row is the baseline agent; the next three rows give the change made by the layer (+S), orchestration (+M) and compaction (+C), each averaged over the settings of the other two. Rates are in points.
| RS | SQL | Ext. | Logic | Compl. | R-1 | R-L | TR | TF1 | TOS | SR | Calls | In (k) | Out (k) | Time (s) | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Opus 5.5 | baseline | 59.8 | 32.7 | 76.3 | 85.7 | 45.6 | 36.5 | 14.2 | 57.3 | 70.6 | 35.5 | 97.9 | 4.7 | 102 | 4.0 | 133 |
| +S | 1.3 | 0.1 | 6.4 | 1.6 | 0.2 | +0.3 | +0.1 | 4.6 | 3.7 | 2.5 | +0.4 | 2.8 | +90.8 | 3.9 | +44 | |
| +M | 8.1 | 4.9 | 8.0 | 4.7 | 7.9 | 1.4 | 0.7 | +25.6 | +19.7 | +55.3 | 1.2 | +13.9 | +210 | +28.6 | +364 | |
| +C | +1.5 | 2.5 | 0.2 | +0.3 | +1.6 | 0.4 | 0.2 | +0.1 | +0.2 | +0.3 | 1.4 | +4.9 | 53.5 | +9.6 | +135 | |
| GPT-6 Astra | baseline | 59.4 | 25.2 | 83.7 | 86.7 | 42.0 | 33.4 | 12.6 | 72.0 | 82.6 | 36.4 | 96.2 | 9.7 | 241 | 7.7 | 545 |
| +S | +0.8 | +3.6 | 2.4 | +0.5 | +1.7 | +0.4 | +0.1 | +2.0 | +1.4 | +1.5 | 1.0 | +2.3 | +425 | +3.0 | +1605 |
Table 5: FDABench reports in their own metrics. Rubric score (RS) and its four keys (SQL accuracy, external integration, logical reasoning, completeness), ROUGE-1 and ROUGE-L, tool recall (TR) and tool F1 (TF1) at the level of tool names, tool orchestration score (TOS, the released DAG evaluator), tool success rate (SR, runs that attempted a tool), external LLM calls, input and output tokens (thousands) and latency per task; 218 gold-labelled report tasks. Each model’s first row is the baseline agent; the next three rows give the change made by the layer (+S), orchestration (+M) and compaction (+C), each averaged over the settings of the other two. Rates are in points.
| EX | Easy | Med. | Hard | Prec. | Rec. | TF1 | TOS | SR | Calls | In (k) | Out (k) | Time (s) | ||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Opus 5.5 | baseline | 70.0 | 65.2 | 66.7 | 73.8 | 93.3 | 88.3 | 75.7 | 32.2 | 99.7 | 6.3 | 129 | 2.6 | 150 |
| +S | +0.0 | +1.1 | 0.2 | 0.2 | 0.2 | +0.5 | 0.5 | 1.1 | +0.1 | 0.4 | +100 | +0.2 | +164 | |
| +M | 10.5 | 9.8 | 9.9 | 11.1 | 2.5 | 6.5 | +15.6 | +22.4 | 0.4 | +8.0 | +47.4 | +11.7 | +216 | |
| +C | 3.1 | +4.3 | 5.0 | 3.9 | 1.4 | 1.6 | +0.5 | +0.1 | 0.4 | +3.2 | 41.9 | +6.3 | +90 | |
| GPT-6 Astra | baseline | 66.6 | 69.6 | 64.0 | 67.5 | 90.0 | 86.7 | 78.9 | 33.5 | 98.7 | 9.9 | 280 | 2.7 | 601 |
| +S | +2.3 | 0.5 | +3.7 | +2.0 | 0.2 | +1.3 | +0.1 | 1.5 | +0.1 | +4.3 | +290 | +5.7 | +1762 |
Table 6: FDABench multiple choice in its own metrics. Exact match of the option set (EX) overall and by difficulty, precision and recall of the chosen options, tool F1, tool orchestration score (the released evaluator’s fallback, as choice tasks have no released DAG), tool success rate (runs that attempted a tool), LLM calls, input and output tokens (thousands) and latency per task; 320 multiple-choice tasks. Each model’s first row is the baseline agent; the next three rows give the change made by the layer (+S), orchestration (+M) and compaction (+C), each averaged over the settings of the other two. Rates are in points.
| Succ. | S2SQL | SQL2S | Par. | LLM calls | Tool calls | Tool SR | In (k) | Out (k) | Timeout | ||
|---|---|---|---|---|---|---|---|---|---|---|---|
| Opus 5.5 | baseline | 61.1 | 50.0 | 54.7 | 77.3 | 7.0 | 5.8 | 99.4 | 264 | 4.9 | 0.0 |
| +S | 0.8 | 0.4 | 2.1 | +1.3 | 1.6 | 1.6 | +0.4 | +197 | 1.0 | +0.2 | |
| +M | 1.1 | +5.6 | 2.6 | 1.7 | +15.1 | +9.4 | 0.5 | +94.6 | +18.1 | +0.2 | |
| +C | +0.9 | +3.9 | +0.1 | +0.8 | +6.5 | +0.2 | 0.8 | 22.6 | +6.5 | +0.3 | |
| GPT-6 Astra | baseline | 68.9 | 63.8 | 61.1 | 84.9 | 32.7 | 32.6 | 99.7 | 2127 | 13.1 | 4.0 |
| +S | +0.1 | 1.3 | 0.6 | +1.9 | 1.8 | 0.5 | 0.0 | +767 | 3.4 | 0.2 |
Table 7: HDR in its own metrics. Success rate overall and by task type (search then SQL, SQL then search, parallel), LLM calls, tool calls and their success rate (runs that attempted a tool), input and output tokens (thousands) per task, and the share of runs that exceed the benchmark’s limits of 300 tool-calling turns or 3,600 seconds. Each model’s first row is the baseline agent; the next three rows give the change made by the layer (+S), orchestration (+M) and compaction (+C), each averaged over the settings of the other two. Rates are in points.