Measure Learning at Steady State: A BIRD-SQL Formula 1 Case Study

Manoj Bajaj

cs.LGSep 11, 2026

Measure Learning at Steady State: A BIRD-SQL Formula 1 Case Study

Authors: Manoj Bajaj

Abstract

Continual Learning Bench scores learning as short-horizon gain versus a reset baseline and finds naive full-context ICL strongest among the memories it tested. We treat ICL as one learning system and score it on a longer shared-world schedule. Steady-state learning is the gap versus baseline on a pre-set late window (last 40 of 174 BIRD-SQL formula-1 questions). We split the score into exploration efficiency (SQL probes), task reward (hits), and delivery cost (API dollars and context size). On gpt-5.6-luna, late probes fall from 4.6-5.6 to 0.95 while hits rise only modestly and ICL context grows to about 95k tokens with cost roughly doubling. Short-horizon gain understates the late probe saving and misses the cost inversion, so we find that unbounded ICL is a poor candidate for the learning mechanism.

Figures & tables

Run	Hits	Hit rate	Mean queries	Cost (USD)
Luna baseline r1	65	37.4%	4.78	4.55
Luna baseline r2	67	38.5%	4.40	4.51
Luna baseline r3	65	37.4%	4.43	4.47
Luna ICL	74	42.5%	1.07	9.92
Terra baseline	63	36.2%	3.76	3.63
Terra ICL	87	50.0%	1.17	11.10

Table 1: Whole run (

n=174

Run	Hits / 40	Mean queries	Window cost (USD)
Luna baseline r1	11	5.58	1.16
Luna baseline r2	11	4.60	1.09
Luna baseline r3	10	5.25	1.18
Luna ICL	13	0.95	3.56
Terra baseline	9	3.85	0.91
Terra ICL	19	1.08	—

Table 2: Steady-state window (questions 135–174).

Figure 1: Luna r1. Each point is the mean of ten consecutive questions (last plotted block is q161–170). ICL stays near one probe. Baseline does not.

Figure 2: Terra. Same block means. ICL again sits near one probe after the first block.

Figure 3: The measure we want: late-window mean probes. ICL is near 1 on both models. Baseline stays in the 4–6 (Luna) or $\sim$ 4 (Terra) range.

Window	Base hits	ICL hits	Base $q$	ICL $q$	Probe cut
q1–40 (CL-Bench length)	19	22	3.95	1.23	69%
q135–174 (steady state)	11	13	5.58	0.95	83%

Table 3: Luna r1, first 40 vs. last 40.

Figure 4: ICL context (input + cache-read). Luna ends at $\sim$ 95k tokens, Terra at $\sim$ 103k. Baseline is a new prompt each time.

Figure 5: Delivery cost. ICL starts cheaper, then the curve rises with context. Baseline stays roughly flat. This inversion is invisible on a 10- or 40-question schedule.

Run	Hits	Mean $q$	Late $q$	Cost (USD)	End ctx
Luna ICL	74	1.07	0.95	9.92	$\sim$ 95k
Luna compact 25k	71	1.14	1.18	12.73	$\sim$ 26k
Terra ICL	87	1.17	1.08	11.10	$\sim$ 103k
Terra compact 25k	81	1.16	1.25	14.21	$\sim$ 29k

Table 4: Compact versus ICL (full 174).

Continual Learning for LLMs

Continual learning for large language models studies how models absorb new knowledge and skills while retaining previously learned capabilities.

See all 233 papers

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL
Aug 28, 2026Jiayan Lin, Yujia Liu, Zijin Hong +6Execution-Guided Text-to-SQL Text-to-SQL
Jinan University Guangzhou, China · The Hong Kong Polytechnic University Kowloon, Hong Kong · City University of Macau Taipa, Macau +2
A Theoretical Interpretation of In-Context Learning via Probabilistic Modeling
Jun 27, 2026Zhenyu Liu, Huaze Tang, Shao-Lun HuangIn-Context Learning Exponential Family
Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, China
CA-SQL: Complexity-Aware Inference Time Reasoning for Text-to-SQL via Exploration and Compute Budget Allocation
May 8, 2026James Petullo, Nianwen XueInference-Time Search Inference-Time Scaling
Brandeis University / Waltham, MA

Aug 28, 2026J/K move · Enter open · S save

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

Jiayan Lin, Yujia Liu, Zijin Hong, Zheng Yuan+5

Jinan University Guangzhou, China · The Hong Kong Polytechnic University Kowloon, Hong Kong · City University of Macau Taipa, Macau+2

Execution-Guided Text-to-SQL Text-to-SQL In-Context Learning Cost-Aware Inference

Open paper AskPDF arXiv

Recent advances in in-context learning (ICL) text-to-SQL have substantially improved execution accuracy on public benchmarks by assembling increasingly elaborate pipelines around the base generator, yet existing studies typically report aggregate end-to-end accuracy, without quantifying the marginal accuracy-cost contribution of individual design choices. Consequently, providing a unified, paradigm-level cost-accuracy quantification remains a critical challenge for understanding and configuring modern text-to-SQL. To address this, we instantiate 17 paradigm-level configurations across five recurring modules of the ICL text-to-SQL pipeline under a single controlled implementation, and attribute each paradigm's marginal contribution and incurred cost across all four backbones spanning diverse capability levels and reasoning styles. Our analysis reveals that execution-feedback refinement is the only paradigm whose benefit holds universally at consistently low cost, while most other modules help only under backbone-dependent conditions. Token accounting shows that input demand is more closely tied to pipeline structure, whereas output demand is more sensitive to backbone generation behavior. Cross-module analysis further shows that stacking improves accuracy on most backbones, although how the gains compose varies with backbone capability. We also find that a fixed budget is often better spent engineering a more elaborate pipeline over a mid-tier backbone than upgrading to a frontier model with a lean pipeline. These findings distill into an actionable, cost-aware tiered guideline that transfers to five additional backbones without per-paradigm search.

Abstract

Continual Learning for LLMs

Explore similar work

Are These Modules Worth Their Cost? A Paradigm-Level Accuracy-Cost Analysis of In-context Learning Text-to-SQL

A Theoretical Interpretation of In-Context Learning via Probabilistic Modeling

CA-SQL: Complexity-Aware Inference Time Reasoning for Text-to-SQL via Exploration and Compute Budget Allocation