Context layers, curated documentation that an analytics agent fetches at query time, produce large accuracy gains on text-to-SQL benchmarks. A with/without comparison cannot say which part of the layer does the work: the semantic content, the retrieval scaffolding that delivers it, or the pre-computed views that usually accompany it. We report a four-arm ablation on DABStep on four models that separates the three. The instrument is a data contract: a YAML artifact that carries a domain's semantics and the rules an agent's tools enforce. One arm empties every field of prose in the frozen contract while holding the tool surface, retrieval instruction, table allow-list and operation rules byte-for-byte fixed. Compiling the contract's own SQL expressions into views gives the ceiling a pre-computed layer would reach: gold on all 176 tasks it covers. Content dominates. It raises hard-task accuracy from 13.9% to 55.1%, 22.6% to 56.6%, 22.9% to 68.4% and 37.0% to 77.4%, beating the same knowledge pasted into the prompt on every model. Scaffolding without content is worth 0 to 5 points on two flash models and 14 to 15 on two frontier models. Against the compiled ceiling the contract arm's shortfall is a failure to derive, and it falls from 39 points to 5 with model capability. The contract beats the prompt because the rule it needs is one lookup away rather than buried in a long prompt: its SQL carries the fee semantics up to 98% of the time against the prompt arm's 4%, and at the lowest cost per correct answer on three of four models. For practitioners: semantics first, scaffolding second, pre-computed macros only where an agent demonstrably fails to derive. Ungoverned arms submitted 166 mutating statements; governed arms none. The gain is confined to the contract's domain. On one model the benchmark's own withheld golds grade the contract arm at 51.9% against 18.8% for the prompt baseline.
Figures & tables
Views / macros
Declarative contract
Compiled
ahead of time, by a human
per query, by the model
Covers
anticipated metrics
any question in the domain
Cost
one macro per metric, maintained
one description per domain
Fails when
nobody anticipated the question
the model misapplies the rule
Table 1: Two ways to deliver the same semantics. The compiled contract of Section 5.3 is the left column built from the right column’s content, which is what lets this paper measure the difference between them.
no scaffolding
scaffolding
no content
schema
hollow
content
manual
contract
Table 2: The four arms as a two-factor design (artifact names: schema_only , contract_hollow , manual_prompt , contract ). hollow differs from contract only in that its prose fields are empty. manual carries the same knowledge as contract in its original form, pasted into the prompt.
Model
Pinned id
Route
T
Tasks
GLM
glm-5.3-flash
OpenRouter, fp8
0
401
DeepSeek
deepseek-v4-flash
OpenRouter, fp8
0
279
Sonnet 5
Claude Sonnet 5
Bedrock via gateway
—
401
GPT-5.6
gpt-5.6-sol
OpenRouter
—
401
Table 3: The four models, in the order used throughout: bare-schema hard accuracy. This is not the order they were run in; Sonnet 5 ran last. Each is pinned to one provider endpoint with fallbacks disabled. T is temperature; a dash means the route does not accept the parameter. DeepSeek’s task count is its complete-case set, defined below.
Arm
Overall
Hard
Easy
Cost
GLM ( glm-5.3-flash , 401 tasks)
schema
22.7%
13.9%
65.2%
$0.76
hollow
26.4%
19.3%
60.9%
$0.85
manual
31.7%
22.9%
73.9%
$0.96
contract
57.9%
55.1%
71.0%
$0.67
DeepSeek ( deepseek-v4-flash , 279 tasks)
Table 4: Accuracy and cost, models in order of bare-schema capability. Hard splits are n=332 , 226, 332 and 332; easy are 69, 53, 69 and 69. contract is the most accurate arm on all four models, the cheapest on the two flash models, and within about 3% of the cheapest on Sonnet 5. Arms are in ladder order, so the two content-free arms are adjacent.
Comparison
GLM
DeepSeek
Sonnet 5
GPT-5.6
schema vs contract
3×10−31 14/ 155
2×10−15 14/ 93
1×10−35 11/ 162
5×10−32 12/ 152
manual vs contract
2×10−17 29/ 134
0.0067 30/ 56
5×10−27 22/ 160
2×10−17 12/ 96
hollow vs contract
9×10−24 23/ 149
8×10−17 13/ 96
2×10−17 25/ 126
2×10−17 18/ 111
schema vs manual
7×10−6 14/50
2×10−12 5/58
0.066 15/28
2×10−8 22/78
manual vs hollow
0.033 55/34
4×10−13 63/6
1×10−4 27/ 64
0.44 58/49
schema vs hollow
0.058 20/35
0.61 19/15
9×10−9 14/ 64
4×10−7 20/ 67
Table 5: Every pairwise paired McNemar over all tasks: p -value, then discordant pairs won by the left arm / by the right arm. The last row is the scaffolding step of the decomposition, on which the flash and frontier models split two and two. Sonnet 5’s fourth row is its own result: the pasted manual is not distinguishable from a bare schema.
Figure 2: The two steps of the decomposition on hard tasks, four models in order of bare-schema capability. Adding the nine governed tools, the retrieval instruction, the table allow-list and the operation rules with the prose emptied ( schema to hollow ) moves accuracy by +5.4 , +0.0 , +15.1 and +14.2 points; restoring the prose ( hollow to contract ) moves it by a further +35.8 , +34.0 , +30.4 and +26.2 . Bars are Wilson 95% intervals.
GLM
DeepSeek
Sonnet 5
GPT-5.6
Hard tasks
macro
derived
macro
derived
macro
derived
macro
derived
schema
2.8%
25.0%
8.5%
34.7%
6.8%
37.8%
31.8%
41.2%
hollow
10.2%
25.7%
10.2%
31.7%
36.4%
37.2%
46.6%
54.1%
manual
14.8%
32.4%
38.1%
44.6%
8.5%
38.5%
49.4%
48.6%
contract
60.8%
45.9%
62.7%
46.5%
73.9%
60.1%
94.9%
56.1%
compiled contract
100%
—
100%
—
100%
—
100%
—
Table 6: Hard tasks split by whether the compiled contract answers the task outright ( macro , n=176 ; 118 on DeepSeek’s complete-case set) or requires reasoning on top of it ( derived , n=148 ; 101). The gap between contract and the compiled contract on macro is derivation failure on information the agent provably had.
Wrote all six clauses
GLM
DeepSeek
Sonnet 5
GPT-5.6
schema
7%
4%
3%
8%
hollow
14%
4%
4%
8%
manual
3%
9%
4%
4%
contract
39%
65%
55%
98%
Table 7: Share of the 97 tasks requiring all six fee clauses (69 on DeepSeek) on which the agent’s submitted SQL expresses all six. manual holds the same knowledge as prose in its system prompt. contract against manual is p=2×10−10 , 3×10−12 , 9×10−16 and 2×10−47 (Fisher exact).
schema
hollow
manual
contract
GLM
Turns / task
12.1
8.4
9.8
7.1
Tool calls
5,134
4,110
4,282
3,355
Reasoning tokens
428k
525k
400k
154k
Cost / correct
$0.0084
$0.0080
$0.0076
$0.0029
Forced answers
41
0
18
0
Table 8: Work and cost per arm. The contract arm uses the fewest turns and the least reasoning on every model and never exhausts its output budget; it is the cheapest arm per correct answer on three of the four. DeepSeek’s reasoning row is per task rather than a total, because its complete-case set has fewer rows than the other runs.
GPT-5.6
Fresh input
Cached input
Output
Total
manual
$3.55
$3.85
$7.02
$14.42
contract
$8.87
$3.78
$8.67
$21.32
Table 9: GPT-5.6’s bill for the two content-bearing arms, decomposed from the recorded token counts at the pinned endpoint’s prices: 2.00permillionfreshinputtokens,0.20 cached, $10.00 output (reasoning included).
schema
hollow
manual
contract
GLM
27 (8)
0
92 (13)
0
DeepSeek
24 (2)
0
1 (1)
0
Sonnet 5
0
0
22 (2)
0
GPT-5.6
0
0
0
0
Total
51 (10)
0
115 (16)
0
Table 10: Mutating statements submitted, by arm (tasks in parentheses). Fisher exact on traces containing at least one: p=8×10−7 for GLM alone, p=3×10−8 pooled over the three models where any arm attempted a write. DeepSeek and Sonnet 5 alone are underpowered ( p=0.25 and 0.50 ).
Appendix figures & tables3 assets
Supplementary material from the paper’s appendix.
Appendix
GLM
DeepSeek
Sonnet 5
GPT-5.6
Arm
exp.
corr.
exp.
corr.
exp.
corr.
exp.
corr.
contract
15.8
50.1
17.2
51.0
10.6
64.2
10.5
72.7
hollow
26.6
16.3
21.6
19.8
14.3
34.8
14.7
46.9
manual
19.7
20.3
23.7
37.1
40.5
18.2
13.8
46.3
schema
37.0
10.9
31.4
18.5
36.8
18.0
10.6
34.8
Appendix
Table 11: Exposure to scorer leniency: the share of each arm’s correct hard-split answers that required more than exact match against the reconstructed gold (exp.), and the accuracy that follows from correcting it (corr.). An upper bound.
Model
Bucket
schema
hollow
manual
contract
GLM
fee (294)
13.9%
18.7%
24.5%
59.2%
GLM
non-fee (38)
5
9
4
9
DeepSeek
fee (196)
23.0%
22.4%
44.9%
61.2%
DeepSeek
non-fee (30)
6
7
9
8
Sonnet 5
fee (294)
22.4%
40.1%
24.1%
73.5%
Sonnet 5
non-fee (38)
10
8
8
11
Appendix
Table 12: Hard-task accuracy split by whether the question concerns the domain the contract describes. Non-fee cells are counts, not rates.
Figure 3: Hard accuracy against base-model capability, where capability is the bare-schema arm’s own accuracy. The margin over manual is +32.2 , +13.7 , +44.6 and +27.1 : not monotone.
LLM-based agents for text-to-SQL often struggle with latency-performance trade-off, where performance improvements come at the cost of latency or vice versa. We reformulate text-to-SQL generation within the lens of software test coverage where the original query is prepared with a suite of test cases with simpler, atomic SQLs that are executed in parallel and together ensure semantic coverage of the original query. After iterating on test case coverage, the final SQL is generated only when enough information is gathered, leveraging the explored test case SQLs to ground the final generation. We validated our framework on a state-of-the-art benchmark for text-to-SQL, Spider 2.0, achieving a new state-of-the-art with 70.2% execution accuracy.
Real-world text-to-SQL is often under-specified until user phrases are grounded in how the database stores values. Prior work attempts to address this by requiring a semantic layer to specify groundings in advance, but such specifications are often incomplete, especially in expert domains where domain-specific conventions are under-documented. As this leaves multiple grounding hypotheses open for the same SQL part, we introduce GATE (Grouding After Test from Execution), which bootstraps missing groundings from execution feedback. GATE keeps grounding hypotheses open while executing the already grounded parts to obtain observations. Then, only the hypothesis supported by that observation is grounded and stored as a memory entry, recording what was tested and how the open part should be written in SQL. These entries accumulate into execution-grounded memory, allowing later steps to reuse supported groundings. Across real-world and controlled benchmarks, GATE consistently improves over strong baselines, demonstrating that execution can serve not only as validation but also as a bootstrapping mechanism for reusable memory in text-to-SQL.
LLM-based Text-to-SQL progress is reported across heterogeneous benchmarks, backbones, and inference protocols, making cross-system comparison fragile. We reframe the field as a leaderboard aggregation: we collect the metrics authors themselves report and organize them along an inference-autonomy axis spanning constrained, in-context, iterative, agentic, and reasoning-internalized generation, with traceable provenance for every cell. To anchor the aggregation empirically, we run a focused case study on Spider, comparing 8B open-source backbones with and without chain-of-thought (CoT) supervision against few-shot DeepSeekV3 and GLM-4 baselines. Four patterns emerge: Spider gains transfer unevenly to BIRD and Spider2.0; autonomy buys robustness at non-trivial cost; reasoning internalization sits between answer-only decoding and externally orchestrated agents; and CoT gains concentrate on Hard and Extra-Hard queries. We release a Python harness mirroring the autonomy axis so that future methods can be added directly to the leaderboard.