Where Should Learning Live? Regret Analysis for Text, Parameter, and Joint Adaptation

Jungtaek Kim; Jaden Park; Jongwon Jeong; Emmanouil-Vasileios Vlatakis-Gkaragkounis; Kangwook Lee

stat.MLNov 6, 2026

Where Should Learning Live? Regret Analysis for Text, Parameter, and Joint Adaptation

Authors: Jungtaek Kim, Jaden Park, Jongwon Jeong, Emmanouil-Vasileios Vlatakis-Gkaragkounis, Kangwook Lee

Organizations: University of Arkansas · University of Wisconsin–Madison · KRAFTON · Ludo Robotics

Abstract

Language-model systems can select prompts and parameter adapters repeatedly during deployment, using feedback to guide later choices. Earlier choices may change the context or environment inherited by later configurations. We study selection among preconstructed candidates and compare cumulative loss with the best fixed prompt--parameter pair used from the beginning. This excess loss separates into three terms: selection regret within the available family, the gap from restricting that family, and the benefit or harm of the actual deployment history. Selection is evaluated using the history each candidate would have produced if used throughout, which separates candidate quality from the effects of switching. When loss functions are fixed independently of selector randomness, past choices affect loss for only a bounded number of rounds, and every candidate's own-history loss is revealed, limiting switches yields sublinear expected regret within the selected family. We also quantify the costs and additional assumptions of independent probing and deployment-only feedback. Controlled experiments show that fewer switches can either lower or raise deployed loss, depending on whether inherited histories are harmful or helpful. With Qwen3.5-9B on Spider, text-only selection has lower within-family regret than joint selection, yet produces more failures because its candidate family excludes stronger configurations.

Figures & tables

Panel of Figure 1 — Figure 1: Text-only and parameter-only families within the joint family.

Panel of Figure 4 — Figure 4: Independent-minus-lazy loss contrasts at $T=4{,}096$ , with $\lambda\in\{-0.2,0,0.2\}$ . Reversing the mechanism reverses the selector ordering while expected diagonal selection remains unchanged.

Panel of Figure 7 — Figure 7: Diagnostics with prompt–adapter interactions at adapter learning rate $10^{-4}$ and SParC state effects, followed by restricted-feedback excess failures. Negative interactions indicate improvement beyond additive effects. Full and Probe denote full-diagonal and one-probe lazy EW, $b$ the deployment block length, and Once draw-once.

Appendix figures & tables13 assets

Supplementary material from the paper’s appendix.

Appendix

Panel of Figure 8 — Figure 8: Sensitivity of exact expected regret to interaction strength and coordinatewise sweep count at $T=100{,}000$ , with a shared logarithmic scale. The dotted line in Panel 8(a) marks the onset of a strict coordinatewise local minimum.

Figure 9: Decomposition of cumulative excess loss on fixed-advantage tables at

T=1{,}024

and

\lambda=0.2

. Black marks show observed totals, including the diagonal sampling residual.

Panel of Figure 10 — Figure 10: Horizon dependence and the draw-once control under harmful state with $\lambda=0.2$ .

Panel of Figure 11 — Figure 11: Approximate-minus-exact loss components on changing-quality tables under a common offset in Panel 11(a) and reference-opposing bias for lazy EW in Panel 11(b) . The offset panel fixes $\lambda=0.2$ , and the panels use different vertical scales.

Panel of Figure 12 — Figure 12: Feedback accuracy and excess deployed loss as the sampling budget grows. Dotted lines mark exact feedback, and the gray curve is the capped Hoeffding envelope. The loss panels use different vertical scales.

Samples $\boldsymbol{n}$	Observations/table	Max. error	Excess loss $\pm$ SE
1	16,384	0.660	$147.835\pm 4.906$
16	262,144	0.479	$150.527\pm 1.305$
64	1,048,576	0.252	$149.555\pm 2.290$
256	4,194,304	0.132	$148.311\pm 1.913$

Appendix

Table 1: Monte Carlo feedback for lazy EW on fixed-advantage tables at

T=4{,}096

(m_{Z},m_{W})=(4,0)

, and

\lambda=0.2

. Max. error is the table mean of the maximum candidate–round error, and observations count represented Bernoulli samples.

Feedback and Selection	$\boldsymbol{\lambda=-0.2}$	$\boldsymbol{\lambda=0}$	$\boldsymbol{\lambda=0.2}$
Full diagonal, lazy	$51.574\pm 0.126$	$56.707\pm 0.000$	$61.840\pm 0.126$
One probe, lazy	$58.574\pm 0.495$	$62.983\pm 0.503$	$67.392\pm 0.534$
Deployment only, $b=1$	$-138.617\pm 0.232$	$52.135\pm 0.565$	$232.257\pm 2.160$
Deployment only, $b=16$	$41.820\pm 0.436$	$78.855\pm 0.459$	$115.524\pm 0.565$
Draw once	$87.241\pm 0.000$	$87.241\pm 0.000$	$87.241\pm 0.000$

Appendix

Table 2: Controlled fixed-advantage excess loss at

T=1{,}024

and

m_{Z}=m_{W}=4

, using 64 selector repetitions in each of four fixed environments. Draw-once expectations are exact for each table.

Prompt	Base	$\mathbf{5\times 10^{-5}}$	$\mathbf{10^{-4}}$	$\mathbf{2\times 10^{-4}}$
Direct $z_{0}$	66.05	$69.21\pm 0.28$	$69.34\pm 0.45$	$69.50\pm 1.01$
Verification $z_{1}$	65.28	$68.73\pm 0.45$	$69.25\pm 0.19$	$69.54\pm 0.88$
Demonstrations $z_{2}$	63.73	$69.86\pm 0.70$	$70.15\pm 0.87$	$69.21\pm 0.71$
Schema linking $z_{3}$	65.09	$69.54\pm 0.51$	$69.60\pm 0.35$	$70.08\pm 1.10$
Clause planning $z_{4}$	65.86	$68.99\pm 0.09$	$69.60\pm 0.21$	$70.02\pm 0.81$
Demonstrations + verification $z_{5}$	64.31	$70.08\pm 0.78$	$70.25\pm 0.86$	$69.34\pm 0.71$

Appendix

Table 3: Spider candidate test-suite accuracy (%). Columns identify the base model and adapter learning rates.

Family	EW	Follow-the-leader	Uniform	Development-fixed
Text	$359.903\pm 0.000$	$358.517\pm 0.000$	$361.333\pm 0.000$	$351.000\pm 0.000$
Parameter	$319.354\pm 5.750$	$316.222\pm 8.814$	$325.417\pm 3.495$	$313.667\pm 9.404$
Union	$325.435\pm 5.962$	$319.139\pm 8.585$	$346.519\pm 1.553$	$313.667\pm 9.404$
Joint	$315.024\pm 6.332$	$310.990\pm 9.123$	$326.306\pm 3.513$	$309.333\pm 7.311$

Appendix

Table 4: Spider expected cumulative failures by family and selector. Uniform also gives the draw-once expectation.

$\mathbf{K}$	Text	Parameter	Union	Joint
2	$56.437\pm 7.265$	$26.627\pm 5.059$	$45.258\pm 6.154$	$31.479\pm 4.488$
3	$58.223\pm 7.265$	$21.693\pm 4.498$	$39.184\pm 5.430$	$23.931\pm 3.665$
4	$59.195\pm 7.265$	$19.687\pm 4.162$	$34.503\pm 4.988$	$20.851\pm 3.293$

Appendix

Table 5: Spider excess failures after averaging all base-anchored subsets of each size within each seed, relative to the full joint-family comparator.

Prompt	$\mathbf{5\times 10^{-5}}$	$\mathbf{10^{-4}}$	$\mathbf{2\times 10^{-4}}$
Verification $z_{1}$	$-0.29\pm 0.17$	$-0.68\pm 0.30$	$-0.81\pm 0.17$
Demonstrations $z_{2}$	$-2.97\pm 0.47$	$-3.13\pm 0.59$	$-2.03\pm 0.31$
Schema linking $z_{3}$	$-1.29\pm 0.23$	$-1.23\pm 0.23$	$-1.55\pm 0.15$
Clause planning $z_{4}$	$0.03\pm 0.23$	$-0.45\pm 0.31$	$-0.71\pm 0.21$
Demonstrations + verification $z_{5}$	$-2.61\pm 0.53$	$-2.64\pm 0.49$	$-1.58\pm 0.31$

Appendix

Table 6: Spider prompt–adapter failure interactions in percentage points. Negative values indicate improvement beyond additive effects.

Coupling	Selection Regret	State Effect	Deployed Loss
Independent	26.108	$-1.979\pm 1.678$	$489.795\pm 1.678$
Lazy	26.108	$0.396\pm 0.175$	$492.170\pm 0.175$

Appendix

Table 7: Recursive SParC accounting over 1,203 turns. Both couplings share exact selection regret, and restriction is zero.

Feedback and Selection	Spider		Cleared SParC
Feedback and Selection	Mean $\pm$ SE	MCSE	Mean $\pm$ SE	MCSE
Full diagonal, lazy	$15.358\pm 1.893$	0.000	$17.674\pm 0.443$	0.000
One probe, lazy	$24.165\pm 4.312$	0.130	$34.304\pm 1.979$	0.241
Deployment only, $b=1$	$24.340\pm 4.536$	0.300	$34.182\pm 2.254$	0.336
Deployment only, $b=2$	$25.232\pm 4.541$	0.327	$36.094\pm 2.696$	0.342
Deployment only, $b=4$	$25.431\pm 4.694$	0.366	$38.145\pm 2.836$	0.368

Appendix

Table 8: SQL restricted-feedback excess failures above each bank’s best fixed joint candidate, with separate conditional MCSEs. SParC uses cleared history and the zero-influence rate.

Inference-Time Adaptation of Language Models

See all 43 papers

Selection Is Not Learning: Choosing Model Futures
Oct 15, 2026Haoyi Yang, Kshitij Mishra, Nils Lukas +1Language Model Self-Improvement Self-Training for Language Models
Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE
Can Language Models Learn When to Abstain? Policy Learning and Calibrated Answer Selection
Oct 9, 2026Juan Terven, Diana-Margarita Córdova-Esparza, Ruben Gomez +3Selective QA LLM Abstention
CICATA Qro., Instituto Politécnico Nacional, México · Facultad de Informática, Universidad Autónoma de Querétaro, México
PARALLEL: A Prefrontal-Aligned Reinforcement inspired Approach for Language-Model Learning under Explicit Limits
Jul 31, 2026Namkyung Yoon, Sanghong Kim, Hwangnam KimContinual Learning for LLMs LLM Fine-Tuning
School of Electrical Engineering, Korea University Seoul 02841, Republic of Korea

Oct 15, 2026J/K move · Enter open · S save

Selection Is Not Learning: Choosing Model Futures

Haoyi Yang, Kshitij Mishra, Nils Lukas, Salem Lahlou

Mohamed bin Zayed University of Artificial Intelligence, Abu Dhabi, UAE

Language Model Self-Improvement Self-Training for Language Models Continual Learning for LLMs LLM Reasoning

Open paper AskPDF arXiv

Self-evolving language models train on tasks they generate, so every round must decide which updated model the next round inherits. Current loops such as R-Zero, Agent0, and Skill Self-Play make this decision through the task, admitting tasks that look learnable and inheriting every resulting update. We identify a selection-to-update gap in these loops, where a task that looks learnable need not produce a useful update. Most updates on verified tasks fail to make correct answers more likely, yet these loops inherit them all. HeirCommit freezes the updated model and uses fresh evaluation questions to decide whether to inherit it or retain the current model. EvoHeir separates parameter inheritance from experience inheritance, using current and archived experience to repair failures and preserve successes before final confirmation. Our analysis characterizes selection-induced optimism and decomposes regret into selection, absorption, and confirmation errors. HeirCommit cuts the harmful share of accepted updates by 68-83% relative to random inheritance at the same acceptance rate. Across four backbones, EvoHeir achieves the best mathematics and general-reasoning averages, raising the math average by 5.3% relative to the strongest baseline, Agent0. The framework lets useful experience persist even when the model that produced it is not inherited.

Abstract

Inference-Time Adaptation of Language Models

Explore similar work

Selection Is Not Learning: Choosing Model Futures

Can Language Models Learn When to Abstain? Policy Learning and Calibrated Answer Selection

PARALLEL: A Prefrontal-Aligned Reinforcement inspired Approach for Language-Model Learning under Explicit Limits