Answering questions over heterogeneous data means combining evidence from files, databases, documents and the web. Data agents automate this, but their success depends on context. Three context-engineering techniques (CETs) help by shaping it: semantic layers describe the data, orchestration splits work among agents, and compaction condenses interaction history. Existing studies validate each CET alone, leaving open how agent behavior shapes its effect. We present a comprehensive analysis of the three CETs, alone and combined, with seven large language models (LLMs) on 1,514 long-horizon tasks from five benchmarks, covering behavior, accuracy and cost. We find that data agents without CETs form three behavior groups: investigators, premature answerers that answer without inspecting data, and early stoppers that give up. Each CET delivers its known benefit, from more investigation to smaller prompts, but only the behavior group it matches turns that benefit into accuracy, such as premature answerers under orchestration. For investigators, the same CETs can hurt: the semantic layer stops them from looking, orchestration makes them overconfident, costing 5 percentage points, and compaction makes them repeat steps. Combining CETs yields smaller gains and losses than their sum. We derive guidance for choosing CETs by agent behavior, task demands and accuracy-cost trade-offs.
Figures & tables
Figure 1: Same context, different agents (illustration). Given the same question and data, an investigator checks every source and answers from the data, a premature answerer answers without looking, and an early stopper gives up partway.
Figure 2: Data agents and the context-engineering techniques that help them. Data agents answer questions over heterogeneous data, and context-engineering techniques (CETs) help by shaping the context the model works from: a semantic layer describes the data, orchestration splits the work and compaction condenses the history. What we do not yet know is how each changes what different LLMs do, alone and combined.
Figure 3: How the baseline agent behaves, by behavior group. Each axis is the share of a group’s baseline runs that show a behavior, from 0 at the center to 100% at the outer ring (mean of the group’s models). Investigators look and persist, premature answerers often answer without acting, and early stoppers act but give up and repeat themselves.
Krama
DataC.
FDA rep.
FDA ch.
HDR
Opus 5.5
82
71
60
70
61
GPT-6 Astra
81
72
59
67
69
Sonnet 5.5
79
66
46
56
61
GPT-5.5
22
63
10
3
9
GPT-5.4
37
28
28
4
8
Gemini 2.5 Flash
50
22
24
8
19
Table 1: Baseline agent accuracy in % on each benchmark’s own metric, with rows by behavior group (investigators, premature answerers, early stoppers).
Figure 4: Accuracy-cost trade-offs depend on the behavior group. Each arrow goes from the baseline agent (square) to the agent with one CET: accuracy change in points and dollars per task as a multiple of the baseline’s. Orchestration gives the largest gain for premature answerers and the semantic layer for early stoppers; for investigators every CET lowers accuracy or leaves it unchanged, and every CET raises the cost.
Figure 5: Task demands modify CET effects, and process gains can accompany accuracy losses. Rows show each CET’s effect on accuracy (overall and within-benchmark contrasts between tasks with and without each demand), process score (the mean of three benchmarks’ step scores), and dollars per task. Blue indicates improvement and red degradation; white marks effects below the display thresholds of one accuracy point, three process-score points, and 1.1 times the cost. Figure 16 reports every demand, each benchmark’s process measure, and which changes are significant.
Figure 6: How each CET changes what agents do. For each group, a line outside the dashed ring means the CET makes a behavior more common and a line inside it less common; behaviors are grouped into engagement, reliability and commitment. Orchestration raises engagement, the semantic layer raises reliability at the cost of engagement, and compaction lowers reliability; Figure 17 gives every measure.
Figure 7: The semantic layer cuts give-ups, long loops and calls, but raises the bill and makes investigators stop looking. It helps most the agents that give up, halving their give-ups and loops at little extra cost. Investigators instead stop using tools on most runs whose tool use changes, and their accuracy there falls from 68% to 31%.
Figure 8: Compaction limits prompt growth but increases cost and repetition. “Loses a value” measures incorrect final answers after the correct value appeared in a tool output on DataClaw.
baseline
+C
+S
+S+C
+M
+M+C
+M+S
+M+S+C
Opus 5.5
68.8
65.2
65.6
65.3
61.9
64.0
61.8
63.1
GPT-6 Astra
69.6
65.1
69.6
65.7
61.0
56.4
61.0
56.4
Sonnet 5.5
61.3
57.7
61.2
61.8
60.0
58.4
57.0
56.0
GPT-5.5
21.4
20.8
23.7
23.9
58.1
55.6
57.8
56.4
GPT-5.4
21.0
27.0
29.2
31.3
37.2
36.4
40.1
39.3
Gemini 2.5 Flash
24.6
22.3
27.0
26.0
24.7
20.9
29.2
27.7
Table 2: Accuracy of every combination of CETs (%, mean of the five benchmarks; M: orchestration, S: semantic layer, C: compaction). The best combination depends on the group: no CET for investigators, orchestration, mostly with the layer, for the others.
Appendix figures & tables26 assets
Supplementary material from the paper’s appendix.
Appendix
Figure 9: Baseline behavior profile of every model. Same axes as Figure 3 , one radar per model, grouped by behavior group.
Figure 10: Each benchmark demands something different. Share of each benchmark’s tasks that need each capability, with the same labels for every benchmark.
Figure 11: Full demand profiles. Share of each benchmark’s tasks at each of the 88 levels of the 19 demand dimensions.
Figure 12: Reasoning removes GPT-5.4’s habit of answering before looking, but not GPT-5.5’s. Share of baseline runs that answer without any tool (orange) and accuracy (blue) at each reasoning effort, on KramaBench and HDR. GPT-5.4 does not reason at its default; GPT-5.5 does.
Figure 13: Behavior types by model. Share of each model’s baseline runs in eight behavior types. Investigators spend almost all runs in three productive types; GPT-5.5 answers without looking in half of its runs; GLM spends a third of its runs in trial and error and repetitive loops.
Figure 14: Most wrong values after looking come from the wrong slice of data or from external evidence, not from arithmetic. Share of each group’s wrong values in twelve causes grouped into six families (color: family; shade: cause), from 790 annotated baseline wrong values weighted to the wrong values they represent.
Figure 15: Each CET changes size on a few demands, and only for some groups. For each CET (rows) and behavior group (columns), the demands on which the effect against the baseline agent departs from the group’s effect on all tasks (dashed line) by at least 2 points, with a 95% task-bootstrap interval that excludes it; at most the three largest departures in each direction (“top k of n ” when more qualify). Blue: helps more than on all tasks; red: helps less or hurts more. Values are points of accuracy, groups are the mean of their models, and each cell has its own scale. † : at least 74% of the level’s tasks come from DataClaw.
Figure 16: Every demand and every benchmark’s process measure. Same as Figure 5 , with three more demands and each benchmark’s process measure in place of their mean. HDR has no step score, so its row is the share of runs that time out (fewer is better).
Figure 17: Orchestration raises engagement, the layer raises reliability at the expense of engagement, and compaction lowers reliability. Change in each behavior, in points of runs, from the agent without the CET to the agent with it, for each behavior group; blue is more of the property, red less, and bold an interval that excludes no change. Outlined cells are the main shifts; other cells are muted. The failure mix covers wrong data slice, wrong conclusion from web or document evidence, code that never ran and arithmetic, with purple for the size of the shift.
Figure 18: Behavioral fingerprint by model. Same as Figure 17 , one row per model.
Figure 19: Forcing a single GPT-5.5 agent to act, or to spend the orchestrated agent’s calls, closes only 7-37% of orchestration’s gain. Accuracy against dollars per task for GPT-5.5: the baseline agent, the single agent forced to act before answering (A), the single agent forced to spend the orchestrated agent’s number of calls on each task (B) and the orchestrated agent; 95% intervals over tasks.
Figure 20: Orchestration moves premature answerers’ runs into looking. Behavior types of runs on the same tasks without (left) and with orchestration (right), for each behavior group; bold ribbons are the main moves.
Figure 21: Orchestration helps most where the baseline agent is weakest. Accuracy change of orchestration against the baseline agent’s accuracy on the benchmark, one point per model and benchmark (Spearman −.65 ; against the baseline share of answers without a tool, +.60 ).
Figure 22: The layer moves every group toward quick lookups: it replaces failure for early stoppers and work for investigators. Behavior types of runs without and with the layer, by group.
Figure 23: Each part’s content reaches the agent’s actions where it is meant to. Without (open) and with (filled) the layer: runs whose first action opens the source the layer lists first (KramaBench and DataClaw, both architectures); runs with a failed read (KramaBench, single agent); runs that use a join key from the layer on KramaBench tasks that need no join (open) or need one (filled). An element counts as used only when it appears in an executed action.
Figure 24: A small author suffices, and the layer makes models alike in how they work more than in accuracy. Left: accuracy gain of a GPT-5.4 single agent over its no-layer run for layers written by models of increasing size on DataClaw (28% accuracy without a layer), with 95% intervals paired by task. Right: spread across the seven models (standard deviation, points) of each measure without (open) and with (filled) the layer; reaching a gold source is measured on single-agent KramaBench and DataClaw runs, the other measures over all runs.
Figure 25: Compaction changes little for investigators and pushes the weaker groups into long repetitive runs. Behavior types of runs without and with compaction, by group.
Figure 26: The best configuration depends on the model. Accuracy of the eight configurations (mean of five benchmarks), one line per model; rings mark each model’s best configuration (a gain under one point counts as the baseline agent).
Figure 27: CETs that repair the same habit add up to less than their parts. Each CET as a vector from the baseline agent (square): accuracy change in points (mean of five benchmarks) and dollars per task as a multiple of the baseline agent’s (log scale). For each combination (columns) and group (rows), the parts are chained head to tail in color, their sum is the open circle, and the black arrow is the measured combination; the dotted segment is the interaction. Groups are the mean of their models (geometric mean for dollars).
Figure 28: The CETs change accuracy where some agent already succeeds, not where the gold says a task is long. Effect of each CET (averaged over the settings of the other two) by task difficulty, for each group: (A) how many of the seven baseline agents solve the task; (B) the share of the other groups’ baseline agents that solve it, which excludes the group’s own runs; (C) terciles of the gold number of dependent steps within each benchmark; (D) the benchmarks’ own easy, medium and hard labels; (E) terciles of the other groups’ baseline run length in model calls and (F) in output tokens, within each benchmark. Filled markers have a 95% task-bootstrap interval that excludes 0.
Figure 29: Dollars per task by model and configuration. Pooled over the five benchmarks at list prices (right column: input, cached input and output dollars per million tokens); darker cells cost more.
Score
Arch.
Astro.
Biomed.
Env.
Legal
Wildf.
Pipeline
Calls
In (k)
Out (k)
Time (s)
Opus 5.5
baseline
82.4
50.0
75.0
83.3
83.3
90.0
93.1
35.8
3.8
47.8
1.9
104
+S
− 0.9
+0.0
− 2.1
− 3.0
− 8.8
+1.7
+3.8
+3.7
− 1.1
+73.1
− 0.5
− 19
+M
+0.5
+0.0
+6.2
+8.1
− 1.2
− 1.7
− 1.2
+5.7
+2.0
+39.1
+4.7
+42
+C
+0.4
+0.0
+2.1
− 5.4
+1.2
+0.0
+2.0
+2.1
+1.2
− 13.3
+1.5
+31
GPT-6 Astra
baseline
81.0
50.0
75.0
88.9
88.3
80.0
93.1
45.7
3.5
35.4
1.5
49
+S
− 1.4
+0.0
− 6.2
− 2.2
− 4.1
− 0.8
+2.8
+0.1
− 1.5
+57.4
− 0.6
+9
Appendix
Table 3: KramaBench in its own metrics. End-to-end score overall and in each of the six domains, pipeline design (share of the human-annotated key functionalities an LLM judge finds in the executed code, judged by GPT-5.4 with the benchmark’s prompt), LLM calls, input and output tokens (thousands) and runtime per task. Each model’s first row is the baseline agent; the next three rows give the change made by the layer (+S), orchestration (+M) and compaction (+C), each averaged over the settings of the other two. Rates are in points.
Acc
Easy
Med.
Hard
EE
GPR
TPE
Calls (C)
Calls (W)
In (k)
Out (k)
Opus 5.5
baseline
70.8
91.6
67.8
46.0
57.9
40.9
88.7
6.8
8.0
68.3
3.1
+S
− 2.3
+0.2
− 2.1
− 7.1
+29.7
− 5.5
+9.5
− 5.3
− 6.2
+381
− 4.3
+M
+1.5
+0.6
+1.8
+2.0
− 16.0
+12.1
− 7.7
+7.9
+11.2
+321
+12.7
+C
− 0.3
+0.2
− 0.8
+0.5
− 3.8
− 0.2
+0.0
+6.6
+8.5
+44.9
+12.1
GPT-6 Astra
baseline
71.9
90.8
68.5
52.0
65.7
44.3
93.7
5.8
7.2
79.2
1.6
+S
− 1.2
− 0.2
− 0.6
− 5.5
+27.0
+0.1
+7.0
− 9.5
− 35.2
+222
− 35.2
Appendix
Table 4: DataClaw in its own metrics. Judged accuracy overall and by difficulty; execution efficiency (EE, gold steps over agent steps, correct tasks only); goal progress rate (GPR, share of gold milestones reached) and temporal progress efficiency (TPE, γ=0.9 ), both on incorrect tasks only, judged by GPT-5.4; LLM calls on correct (C) and wrong (W) tasks; input and output tokens (thousands) per task. Each model’s first row is the baseline agent; the next three rows give the change made by the layer (+S), orchestration (+M) and compaction (+C), each averaged over the settings of the other two. Rates are in points.
RS
SQL
Ext.
Logic
Compl.
R-1
R-L
TR
TF1
TOS
SR
Calls
In (k)
Out (k)
Time (s)
Opus 5.5
baseline
59.8
32.7
76.3
85.7
45.6
36.5
14.2
57.3
70.6
35.5
97.9
4.7
102
4.0
133
+S
− 1.3
− 0.1
− 6.4
− 1.6
− 0.2
+0.3
+0.1
− 4.6
− 3.7
− 2.5
+0.4
− 2.8
+90.8
− 3.9
+44
+M
− 8.1
− 4.9
− 8.0
− 4.7
− 7.9
− 1.4
− 0.7
+25.6
+19.7
+55.3
− 1.2
+13.9
+210
+28.6
+364
+C
+1.5
− 2.5
− 0.2
+0.3
+1.6
− 0.4
− 0.2
+0.1
+0.2
+0.3
− 1.4
+4.9
− 53.5
+9.6
+135
GPT-6 Astra
baseline
59.4
25.2
83.7
86.7
42.0
33.4
12.6
72.0
82.6
36.4
96.2
9.7
241
7.7
545
+S
+0.8
+3.6
− 2.4
+0.5
+1.7
+0.4
+0.1
+2.0
+1.4
+1.5
− 1.0
+2.3
+425
+3.0
+1605
Appendix
Table 5: FDABench reports in their own metrics. Rubric score (RS) and its four keys (SQL accuracy, external integration, logical reasoning, completeness), ROUGE-1 and ROUGE-L, tool recall (TR) and tool F1 (TF1) at the level of tool names, tool orchestration score (TOS, the released DAG evaluator), tool success rate (SR, runs that attempted a tool), external LLM calls, input and output tokens (thousands) and latency per task; 218 gold-labelled report tasks. Each model’s first row is the baseline agent; the next three rows give the change made by the layer (+S), orchestration (+M) and compaction (+C), each averaged over the settings of the other two. Rates are in points.
EX
Easy
Med.
Hard
Prec.
Rec.
TF1
TOS
SR
Calls
In (k)
Out (k)
Time (s)
Opus 5.5
baseline
70.0
65.2
66.7
73.8
93.3
88.3
75.7
32.2
99.7
6.3
129
2.6
150
+S
+0.0
+1.1
− 0.2
− 0.2
− 0.2
+0.5
− 0.5
− 1.1
+0.1
− 0.4
+100
+0.2
+164
+M
− 10.5
− 9.8
− 9.9
− 11.1
− 2.5
− 6.5
+15.6
+22.4
− 0.4
+8.0
+47.4
+11.7
+216
+C
− 3.1
+4.3
− 5.0
− 3.9
− 1.4
− 1.6
+0.5
+0.1
− 0.4
+3.2
− 41.9
+6.3
+90
GPT-6 Astra
baseline
66.6
69.6
64.0
67.5
90.0
86.7
78.9
33.5
98.7
9.9
280
2.7
601
+S
+2.3
− 0.5
+3.7
+2.0
− 0.2
+1.3
+0.1
− 1.5
+0.1
+4.3
+290
+5.7
+1762
Appendix
Table 6: FDABench multiple choice in its own metrics. Exact match of the option set (EX) overall and by difficulty, precision and recall of the chosen options, tool F1, tool orchestration score (the released evaluator’s fallback, as choice tasks have no released DAG), tool success rate (runs that attempted a tool), LLM calls, input and output tokens (thousands) and latency per task; 320 multiple-choice tasks. Each model’s first row is the baseline agent; the next three rows give the change made by the layer (+S), orchestration (+M) and compaction (+C), each averaged over the settings of the other two. Rates are in points.
Succ.
S2SQL
SQL2S
Par.
LLM calls
Tool calls
Tool SR
In (k)
Out (k)
Timeout
Opus 5.5
baseline
61.1
50.0
54.7
77.3
7.0
5.8
99.4
264
4.9
0.0
+S
− 0.8
− 0.4
− 2.1
+1.3
− 1.6
− 1.6
+0.4
+197
− 1.0
+0.2
+M
− 1.1
+5.6
− 2.6
− 1.7
+15.1
+9.4
− 0.5
+94.6
+18.1
+0.2
+C
+0.9
+3.9
+0.1
+0.8
+6.5
+0.2
− 0.8
− 22.6
+6.5
+0.3
GPT-6 Astra
baseline
68.9
63.8
61.1
84.9
32.7
32.6
99.7
2127
13.1
4.0
+S
+0.1
− 1.3
− 0.6
+1.9
− 1.8
− 0.5
− 0.0
+767
− 3.4
− 0.2
Appendix
Table 7: HDR in its own metrics. Success rate overall and by task type (search then SQL, SQL then search, parallel), LLM calls, tool calls and their success rate (runs that attempted a tool), input and output tokens (thousands) per task, and the share of runs that exceed the benchmark’s limits of 300 tool-calling turns or 3,600 seconds. Each model’s first row is the baseline agent; the next three rows give the change made by the layer (+S), orchestration (+M) and compaction (+C), each averaged over the settings of the other two. Rates are in points.
Large language model (LLM) agents assist users with everyday tasks that can be completed in many reasonable ways. Even when their answers are useful, how agents carry out these tasks may not match users' preferences and needs. For example, agents differ in whether they ask clarifying questions or search the web. We introduce HABIT, a taxonomy of 23 behavioral axes in five categories, which three authors and three LLMs derive bottom-up from 408 agent trajectories across 17 domains. On held-out tasks, HABIT distinguishes models more clearly than existing taxonomies of human values and agent actions while supporting comparably consistent annotation. Building on HABIT, we construct AgentHABIT, a benchmark that profiles each agent's behavioral tendencies from its trajectories on 86 everyday tasks. Profiling 18 models with AgentHABIT reveals a range of distinctive tendencies. For example, most GPT and Claude models state their assumptions and offer alternatives when requirements conflict, whereas Qwen and Google's models more often leave assumptions or changes to requirements unstated. These profiles remain recognizable even when built from entirely different sets of tasks, indicating that they reflect general tendencies rather than task-specific behavior. Prompting agents to adopt specific behaviors shifts some axes readily but barely changes others, while fine-tuning on another model's trajectories changes only part of a model's profile and leaves much of it intact. Overall, HABIT and AgentHABIT provide a systematic framework for characterizing how agents carry out everyday tasks beyond task success, offering insights to guide the development of agents whose behavior better fits users' needs.
Woojung Song, Hoyeol Yang, Jeonghoon Shim +4
Seoul National University, Seoul, Republic of Korea
Structured-data agents in enterprise settings must reason over complex data environments whose relevant evidence is distributed across schemas, relationships, policies, and recurring business roles. Modern agentic systems often address this burden through reusable markdown-style memory or skill files that preserve previously discovered information for later queries, reducing the need to rediscover the same structure repeatedly. This is useful, but it obscures a natural division of labor: agents are well suited to semantic reasoning, while learned systems are well suited to predicting and organizing recurring structure. We introduce latent equivalence learning to bridge this gap. The framework separates persistent task-relevant identities from their dataset-relative realizations. In our realization, supporting and opposing evidence shape support-realized Gaussian prototypes that learn how those identities are expressed in a particular data environment, while soft-membership profiles retain distinctions lost under a hard assignment. A separate learned query-prototype system represents recurring evidential requirements and maps them through a learned compatibility function into the same persistent identity structure. This identity-factorized, query-conditioned routing materializes the relevant dataset-specific evidence for downstream reasoning, allowing the agent to operate over an already organized evidential state rather than reconstructing cross-schema structure at every query. On the Data Agent Benchmark, spanning 54 queries across 12 heterogeneous datasets, our full implementation achieves 94.67% dataset-macro stratified Pass@1 over five complete trials and 258/270 successful raw query attempts, compared with 55.51% for the benchmark's Claude Opus 4.6 reference agent, ranking first among 40 leaderboard entries at submission.
Across many domains (e.g., IoT, observability, telecommunications, cybersecurity), there is an emerging adoption of conversational data analysis agents that enable users to "talk to your data" to extract insights. Such data analysis agents operate on timeseries data models; e.g., measurements from sensors or events monitoring user clicks and actions in product analytics. We evaluate 6 popular data analysis agents (both open-source and proprietary) on domain-specific data and query types, and find that they fail on stateful and incident-specific queries. We observe two key expressivity gaps in existing evals: domain-customized datasets and domain-specific query types. To enable practitioners in such domains to generate customized and expressive evals for such timeseries data agents, we present AgentFuel. AgentFuel helps domain experts quickly create customized evals to perform end-to-end functional tests. We show that AgentFuel's benchmarks expose key directions for improvement in existing data agent frameworks. We also present anecdotal evidence that using AgentFuel can improve agent performance (e.g., with GEPA). AgentFuel benchmarks are available at https://huggingface.co/datasets/RockfishData/TimeSeriesAgentEvals.