Certified Predictive Value-of-Advice Gating for Cost-Aware Language-Model Guidance in Reinforcement Learning

Ibne Farabi Shihab; Md Najmus Swaqeeb; Abu Sa-Adat Mohamed Moon-Im Al Ahsan

cs.LGAug 26, 2026

Certified Predictive Value-of-Advice Gating for Cost-Aware Language-Model Guidance in Reinforcement Learning

Authors: Ibne Farabi Shihab, Md Najmus Swaqeeb, Abu Sa-Adat Mohamed Moon-Im Al Ahsan

Organizations: Department of Computer Science, Iowa State University · Standard Chartered, Bangladesh · Department of Computer Science and Engineering, BRAC University, Bangladesh

Abstract

Language-model advice can accelerate reinforcement learning, but calls are costly and returned actions may be stale or wrong. We formulate advice acquisition as a response-contingent metareasoning problem: before querying, the controller predicts possible parsed responses, evaluates the decision and declared continuation that would follow each response, and queries only when a lower confidence bound on predictive value exceeds the priced cost. Execution is governed separately by an action-specific certificate. Under explicit assumptions, certified advice is near-optimal, a wrapped learner inherits fallback regret only under intervention stability, and conservative allocation loses at most the declared query-value estimation error relative to a myopic oracle. On BabyAI, a proxy-calibrated controller with Qwen2.5-1.5B and 7B advisors improves GoToObj return over no querying by 0.029 +/- 0.016 and 0.030 +/- 0.015 across 20 seeds while reducing calls by more than 97% relative to always-query. GoToLocal is a null result. Exactly matched-call tests show an advantage over random placement only for the 1.5B advisor and no advantage over an equal-budget early schedule. Mondrian calibration improves decision-relevant empirical coverage from 0.47 to 0.85, still below the 0.90 target, while the formally covered radius is vacuous. The demonstrated benefit is therefore robust sparse advice volume on a useful task, not a proven per-state placement advantage.

Figures & tables

Study	Question answered
Real advisor	Does selective advice improve return and reduce calls?
Matched calls	Is gain due to placement or only volume/schedule?
Calibration	Does the deployed radius cover relevant errors?
Price sweep	Do actual calls respond to the declared query cost?
Corruption	Does the certificate reject harmful advice?
Shift/cache	Does invalidation help with budget reserved after shift?

Table 1: Primary evidence chain and its role.

Task family	Role	Split/tasks	Horizon	Advisor	Seeds
BabyAI	Core	GoToObj, GoToLocal	64	Qwen2.5-1.5B and 7B	20
ALFWorld	Extension	pick-and-place, look-at-object	50	Simulated ( $\eta=0.85$ )	5
Crafter	Extension	procedural seed split	200	Simulated ( $\eta=0.8$ )	5
Controlled chain/grid	Diagnostic	exact or clonable	task-specific	known/corrupted	5 to 20

Table 2: Benchmark roles. Only BabyAI uses real LLM advisors for the headline result; the other environments are controlled diagnostics.

Estimator	Query-value error	Posterior log score	Surrogate loss	Return	Calls
No query	$0.300$	$-0.693$	$0.090$	$0.457\pm 0.046$	0
Marginal factorization	$0.300$	$-0.693$	$0.090$	$0.457\pm 0.046$	0
Candidate-only value	$0.000$	$-0.693$	$0.000$	$0.566\pm 0.042$	12
Response-conditioned value	$\mathbf{0.000}$	$\mathbf{-0.602}$	$\mathbf{0.000}$	$\mathbf{0.620\pm 0.040}$	12

Table 3: Known-likelihood informative-advisor diagnostic. Query-value error is against the exact Bayesian pre-posterior one-step value.

Fallback/history	Fallback gap	Width sum	Bound holds	Log-log slope	Return
Optimistic, standalone	$883.0\pm 3.7$	$1507.1\pm 3.2$	20/20	$0.724\pm 0.009$	$72.2$
Optimistic, wrapped	$942.4\pm 4.8$	$2201.9\pm 36.9$	20/20	$0.975\pm 0.032$	$72.5$
PSRL, standalone	$554.3\pm 16.3$	n/a	conditional	$0.474\pm 0.025$	$73.4$
PSRL, wrapped	$492.8\pm 17.5$	n/a	conditional	$0.691\pm 0.048$	$82.8$

Table 4: Intervention-stability audit on the controlled chain.

		Qwen-1.5B		Qwen-7B
Task	Method	Return	Calls	Return	Calls
GoToObj	Never query	$0.093\pm 0.014$	0	$0.093\pm 0.014$	0
	Always query	$0.056\pm 0.009$	3408	$0.057\pm 0.009$	3403
	ASK/uncertainty	$0.087\pm 0.014$	557	$0.086\pm 0.014$	557
	Ours, $\beta=0$	$0.117\pm 0.022$	139	$0.119\pm 0.021$	144
	Ours, $\beta_{\rm conf}$	$\mathbf{0.122\pm 0.019}$	75	$\mathbf{0.123\pm 0.021}$	82

Table 5: Faithful controller with real Qwen advisors (

20

seeds,

60

episodes). Return is mean

\pm 95\%

CI; calls are totals across episodes. Bold paired differences from never-query exclude zero.

Task (advisor)	Baseline	Return (calls)	$\Delta$ vs ours
GoToObj (1.5B)	Ours, $\beta_{\rm conf}$	$0.116$ ( $87.2$ )	n/a
	random exact	$0.097$ ( $87.2$ )	$\mathbf{+0.019\pm 0.018}$
	early schedule	$0.106$ ( $86.8$ )	$+0.010\pm 0.018$
GoToObj (7B)	Ours, $\beta_{\rm conf}$	$0.108$ ( $100.5$ )	n/a
	random exact	$0.106$ ( $100.5$ )	$+0.003\pm 0.017$
	early schedule	$0.100$ ( $99.5$ )	$+0.009\pm 0.017$

Table 6: Exactly-call-matched placement comparison (

20

seeds).

\Delta

is paired

\beta_{\rm conf}

minus the baseline; bold excludes zero before multiplicity correction.

Quantity	GoToObj	GoToLocal
Median residual	$6.8\!\times\!10^{-4}$	$8.2\!\times\!10^{-4}$
Median $\beta_{\rm conf}$	$2.2\!\times\!10^{-3}$	$3.0\!\times\!10^{-3}$
Median $\beta_{\rm form}$	$2.14$	$2.14$
Held-out marginal coverage	$0.97$	$0.97$
Decision-relevant marginal	$0.49$	$0.45$
Decision-relevant Mondrian	$0.86$	$0.83$

Table 7: BabyAI query-value calibration audit.

Diagnostic	Supported conclusion
Formal radius	MC term dominates; controller abstains
Faithful price sweep	realized calls decrease monotonically to zero
Five-task volume	robust return without per-task tuning; no optimal-volume tracking
Leave-one-task-out	competitive mean utility, but loses on low-headroom tasks
Certificate sweep	acceptance and return trade-off under corruption
Shift pilot	inconclusive because post-shift budget is nearly exhausted

Table 8: Claim boundary from deployment diagnostics.

Appendix figures & tables33 assets

Supplementary material from the paper’s appendix.

Appendix

Task family	Role	Train/eval split	Horizon	Advisor(s)	Parser	Seeds
MiniGrid/BabyAI	Core	GoToObj / GoToLocal (random layout)	$H{=}64$	Qwen2.5-1.5B & 7B	constrained action (7 admissible)	20
TextWorld/ALFWorld	Core	pick_and_place / look_at_obj (task split)	$H{=}50$	simulated ( $\eta{=}0.85$ )	admissible command	5
Crafter	Extension	procedural world seeds (train/eval split)	$H{=}200$	simulated ( $\eta{=}0.8$ )	skill/action (17 admissible)	5

Appendix

Table 9: Real-advisor benchmark. The table records exact versions, split construction, training budget, advisor prompts, evaluation role, and independent seeds.

Estimator	Query-value error	Posterior log score	Surrogate loss	Return	Calls
No query	$0.300$	$-0.693$	$0.090$	$0.457\pm 0.046$	0
Marginal factorization	$0.300$	$-0.693$	$0.090$	$0.457\pm 0.046$	0
Candidate-only value	$0.000$	$-0.693$	$0.000$	$0.566\pm 0.042$	12
Response-conditioned value	$\mathbf{0.000}$	$\mathbf{-0.602}$	$\mathbf{0.000}$	$\mathbf{0.620\pm 0.040}$	12

Appendix

Table 10: Known-likelihood informative-advisor experiment. Query-value error is computed against the exact Bayesian pre-posterior one-step response value.

Fallback and history	Fallback gap	Width sum $W_{T}$	Gap $\leq W_{T}$ ?	Growth slope	Final return
Optimistic, standalone	$883.0\pm 3.7$	$1507.1\pm 3.2$	yes (20/20)	$0.724\pm 0.009$	$72.2$
Optimistic, wrapped	$942.4\pm 4.8$	$2201.9\pm 36.9$	yes (20/20)	$0.975\pm 0.032$	$72.5$
PSRL, standalone	$554.3\pm 16.3$	—	conditional	$0.474\pm 0.025$	$73.4$
PSRL, wrapped	$492.8\pm 17.5$	—	conditional	$0.691\pm 0.048$	$82.8$

Appendix

Table 11: Intervention-stability audit on the tabular chain. Slope is fitted to cumulative fallback gap versus

T

on log–log axes with a seed-level confidence interval.

Task	Method	Return / success	Calls	Tokens	Latency	Cost-adjusted return
BabyAI-GoToObj	Never query	$0.188\pm 0.038$	0	0	0	$0.188$
	ASK / uncertainty gate	$0.071\pm 0.028$	3362	93/call	2033s	$0.003$
	VOE/Q	$0.071\pm 0.028$	3362	93/call	2032s	$0.003$
	Ours (surrogate)	$0.196\pm 0.025$	686	93/call	414s	$0.182$
BabyAI-GoToLocal	Never query	$0.137\pm 0.035$	0	0	0	$0.137$
	ASK / uncertainty gate	$0.062\pm 0.018$	3391	105/call	2125s	$-0.006$

Appendix

Table 12: Real language-model advice with the deployable epistemic-gap surrogate gate (

10

seeds). Report mean and 95% CI over independent seeds. Cost-adjusted return uses measured token or serving cost. Boldface marks the surrogate’s demonstrated win—the order-of-magnitude call reduction at matched return relative to the naive always-on gates (ASK/VOE)—not a return advantage over never-query, which these CIs do not support (on GoToObj the return difference is

0.196

vs.

0.188

with overlapping intervals and a cost-adjusted return of

0.182<0.188

). The faithful Algorithm 2 controller (Tables 13 , 14 ) is evaluated separately with paired tests.

Task	Method	Return	vs never (paired)	Calls	Cost-adj.	Cert. accept
GoToObj	Never query	$0.093\pm 0.014$	—	$0$	$0.093$	—
	Always query	$0.056\pm 0.009$	$-0.037$ (sig)	$3408$	$-0.228$	—
	ASK / uncertainty	$0.087\pm 0.014$	$-0.007\pm 0.008$	$557$	$0.040$	$0.08$
	Ours, $\beta{=}0$	$0.117\pm 0.022$	$\mathbf{+0.024\pm 0.018}$	$139$	$0.105$	$0.79$
	Ours, $\beta_{\rm conf}$	$\mathbf{0.122\pm 0.019}$	$\mathbf{+0.029\pm 0.016}$	$\mathbf{75}$	$\mathbf{0.116}$	$0.78$
	Ours, $\beta_{\rm form}$	$0.093\pm 0.014$	$+0.000\pm 0.000$	$0$	$0.093$	—

Appendix

Table 13: Faithful Algorithm 2 with the real Qwen2.5-1.5B advisor on both BabyAI tasks (

20

seeds,

60

episodes, shared warm start; run_voi_babyai_alg1.py ). Return is mean

\pm 95\%

CI; “vs never” is the paired per-seed difference against never-query (bold when its CI excludes zero); calls are summed over

60

episodes and averaged over seeds; cost-adjusted return subtracts

\kappa\!\cdot\!(\text{calls}/\text{episode})

\kappa{=}0.005

. The calibrated controller

\beta_{\rm conf}

significantly beats never-query on GoToObj at

\gtrsim 97\%

fewer calls than always-query; the uncalibrated

\beta{=}0

gate and (at matched budget) an early-episode schedule also significantly beat never-query there (Table 14 ), so the gain is not unique to

\beta_{\rm conf}

. Always-query hurts on both tasks; on GoToLocal no strategy beats the strong never-query baseline (reported as a null).

			Qwen-1.5B		Qwen-7B
Task	Baseline	Isolates	Return	$\Delta$ vs ours	Return	$\Delta$ vs ours
GoToObj	Ours, $\beta_{\rm conf}$	—	$0.122{\pm}0.019$	—	$0.123{\pm}0.021$	—
	Random matched	placement	$0.098{\pm}0.018$	$\mathbf{+0.024{\pm}0.014}$	$0.101{\pm}0.018$	$\mathbf{+0.022{\pm}0.022}$
	Schedule matched	placement	$0.117{\pm}0.017$	$+0.005{\pm}0.016$	$0.120{\pm}0.020$	$+0.003{\pm}0.023$
	Ours, cert-off	certificate	$0.110{\pm}0.019$	$+0.012{\pm}0.015$	$0.108{\pm}0.022$	$+0.016{\pm}0.020$
	Ours, global-pred	predictor	$0.110{\pm}0.022$	$+0.013{\pm}0.017$	$0.112{\pm}0.022$	$+0.011{\pm}0.018$

Appendix

Table 14: Placement and component isolation for the calibrated controller, against both a

1.5

B and a

7

B Qwen advisor (

20

seeds,

60

episodes; run_voi_babyai_alg1.py ). Each baseline is run at

\beta_{\rm conf}

’s realized call budget (matched gates) or at fixed

\beta_{\rm conf}

placement (component ablations). Realized calls confirm the match is close but not exact: at

1.5

\beta_{\rm conf}

issues

74.8

59.1

calls (GoToObj/GoToLocal) versus

75.0

59.2

for the schedule gate and

68.5

52.5

for the random gate, which falls

\sim 9

–

11\%

short because episodes that terminate early cap its remaining draws (the schedule front-loads and matches exactly). “

\Delta

vs ours” is the unadjusted paired per-seed return of

\beta_{\rm conf}

minus that baseline; bold marks an unadjusted CI excluding zero. There are

5

baselines

\times\,2

tasks

\times\,2

advisor sizes

=20

such comparisons; both advisor sizes use the corrected online select-

k

-of-remaining budget matching for the random gate. The controller beats the random -placement gate on GoToObj at both sizes (

+0.024\pm 0.014

1.5

+0.022\pm 0.022

7

B), but only the

1.5

B case survives Holm correction over the

20

-comparison family (

1

20

survives;

2

20

clear zero unadjusted). Against the equal-budget scheduled gate the controller is statistically indistinguishable on both tasks and sizes, and the certificate-off and predictor ablations are within noise everywhere. Because the scheduled gate itself matches the controller, the end-to-end gain over never-query (Table 13 ) is attributable to spending a calibrated call volume on the task where advice helps, not to a per-state placement advantage over an equal-budget schedule.

Task (advisor)	Baseline	Return (calls)	$\Delta$ vs ours
GoToObj (1.5B)	Ours, $\beta_{\rm conf}$	$0.116$ ( $87.2$ )	—
	random_exact	$0.097$ ( $87.2$ )	$\mathbf{+0.019{\pm}0.018}$
	schedule matched	$0.106$ ( $86.8$ )	$+0.010{\pm}0.018$
GoToObj (7B)	Ours, $\beta_{\rm conf}$	$0.108$ ( $100.5$ )	—
	random_exact	$0.106$ ( $100.5$ )	$+0.003{\pm}0.017$
	schedule matched	$0.100$ ( $99.5$ )	$+0.009{\pm}0.017$

Appendix

Table 15: Exactly-call-matched placement comparison ( R2Cal15B / R2Cal7B ;

20

seeds, fresh run, so absolute returns differ slightly from Table 14 ). random_exact carries the per-episode call deficit forward so its cumulative calls equal

\beta_{\rm conf}

’s exactly; the schedule gate front-loads the same budget. “

\Delta

” is paired

\beta_{\rm conf}

minus the baseline (bold =

95\%

CI excludes zero, unadjusted ; a Holm correction over this

8

-comparison family leaves only the GoToObj-

1.5

B random cell surviving). At exactly matched calls the controller beats random placement only at

1.5

B on GoToObj and is indistinguishable from the equal-budget schedule everywhere.

Task	Method	Return	vs never (paired)	Calls	Cost-adj.	Cert. accept
GoToObj	Never query	$0.093\pm 0.014$	—	$0$	$0.093$	—
	Always query	$0.057\pm 0.009$	$-0.036$ (sig)	$3403$	$-0.226$	$1.00$
	ASK / uncertainty	$0.086\pm 0.014$	$-0.007\pm 0.008$	$557$	$0.040$	$0.08$
	Ours, $\beta{=}0$	$0.119\pm 0.021$	$\mathbf{+0.026\pm 0.016}$	$144$	$0.107$	$0.61$
	Ours, $\beta_{\rm conf}$	$\mathbf{0.123\pm 0.021}$	$\mathbf{+0.030\pm 0.015}$	$\mathbf{82}$	$\mathbf{0.116}$	$0.71$
	Ours, $\beta_{\rm form}$	$0.093\pm 0.014$	$+0.000\pm 0.000$	$0$	$0.093$	—

Appendix

Table 16: Full faithful-controller panel with the stronger Qwen2.5-7B advisor (

20

seeds,

60

episodes; run_voi_babyai_alg1.py ), the

7

B counterpart of Table 13 . Columns as there; “vs never” is the unadjusted paired per-seed difference (bold when its CI excludes zero). The GoToObj gain of

\beta_{\rm conf}

(and of

\beta{=}0

) replicates at

7

\beta_{\rm form}

again abstains at every state; GoToLocal is again a null.

Task	v25	v50	v75	v100	v150	best fixed	$\beta_{\rm conf}$ (calls)	never
Qwen2.5-1.5B advisor
GoToObj	$0.111$	$0.114$	$0.105$	$0.099$	$0.101$	v50 $0.114$	$0.122$ ( $75$ )	$0.093$
GoToLocal	$0.105$	$0.112$	$0.110$	$0.118$	$0.121$	v150 $0.121$	$0.116$ ( $59$ )	$0.123$
GoToObjDoor	$0.097$	$0.093$	$0.109$	$0.100$	$0.101$	v75 $0.109$	$0.104$ ( $122$ )	$0.104$
Open	$0.015$	$0.016$	$0.013$	$0.014$	$0.011$	v50 $0.016$	$0.016$ ( $157$ )	$0.012$
Pickup	$0.006$	$0.006$	$0.009$	$0.007$	$0.008$	v75 $0.009$	$0.013$ ( $224$ )	$0.011$

Appendix

Table 17: Robustness to untuned query volume across five BabyAI tasks at both advisor sizes ( run_voi_babyai_alg1.py , VOLUME_STUDY ;

20

seeds,

60

episodes, clean advisor). Columns v25 – v150 are fixed front-loaded call schedules of that size (mean return); “best fixed” names the top-scoring fixed schedule per task selected in hindsight .

\beta_{\rm conf}

reports the controller’s return and its realized, untuned call count. The controller’s return lands within the best fixed schedule’s

95\%

CI on all five tasks at both sizes, though its realized volume (

59

–

224

calls at

1.5

65

–

218

7

B) is not an estimate of the best fixed volume and frequently diverges from it—so this is robustness to an untuned budget, not volume tracking. The GoToObj gain over never-query replicates at

7

B (

0.123

0.093

); on the two lowest-signal tasks (Open, Pickup) the

7

B controller sits a few thousandths below best-fixed and never-query, within overlapping CIs (see text). The fixed-volume return surface is flat relative to the per-seed CI.

Quantity	GoToObj	GoToLocal
Mean estimator residual (median over seeds)	$6.8\times 10^{-4}$	$8.2\times 10^{-4}$
Median $\beta_{\rm conf}$	$2.2\times 10^{-3}$	$3.0\times 10^{-3}$
$\beta_{\rm conf}$ IQR	$[1.0,3.9]\times 10^{-3}$	$[0.3,6.6]\times 10^{-3}$
Median $\beta_{\rm form}$ (MC-dominated)	$2.14$	$2.14$
Held-out coverage ( $\alpha{=}0.1$ ; mean / min over seeds)	$0.97/0.83$	$0.97/0.71$
Conditional coverage on decision-relevant states ( $\widehat{\Delta}_{t}{>}0$ )	$0.59$ mean ( $[0.00,1.00]$ )

Appendix

Table 18: BabyAI

\beta_{t}

calibration audit ( beta_calibration ;

20

seeds,

n_{\rm cal}{=}48

each,

\alpha{=}0.1

r_{\rm pair}{=}3

\beta_{\rm conf}

is the split-conformal radius on the pure estimator residual;

\beta_{\rm form}

adds the range-based MC term of Eq. ( 19 ). Held-out coverage is measured by fitting the conformal radius on the first half of each seed’s calibration residuals and testing on the disjoint second half; it is approximate under chronological splits (see text). We report the mean and worst-seed (min) value.

Coverage on $\widehat{\Delta}_{t}>0$ stratum	GoToObj	GoToLocal	overall
Marginal $\beta_{\rm conf}$ radius	$0.49$	$0.45$	$0.47$
Mondrian per-stratum radius	$0.86$	$0.83$	$0.85$
(marginal coverage, all opportunities)	$0.83$	$0.79$	$0.81$

Appendix

Table 19: Group-conditional (Mondrian) stratification of the query-value radius ( R2Cal15B ;

20

seeds,

n_{\rm cal}{=}200

\alpha{=}0.1

). The marginal split-conformal radius undercovers badly on the decision-relevant

\widehat{\Delta}_{t}>0

stratum (

0.47

pooled); per-stratum stratification raises pooled decision-relevant coverage to

0.85

—an improvement, but still below the

0.90

target and not uniform across subgroups (worst

0.30

–

0.43

). Return is unchanged (paired

\beta_{\rm conf}-

Mondrian within noise). Coverage is held-out (fit on the first calibration half, tested on the disjoint second) and advisor-independent (see text).

wrong-advice acceptance (cert on; cert-off ${=}1.00$ where queries occur; over $10$ seeds)
	corruption rate
$\epsilon$	$0.0$	$0.25$	$0.5$	$0.75$	$1.0$
$0.05$	—	$0.00^{\dagger}$	$0.00^{\dagger}$	$0.00^{\dagger}$	$0.00^{\dagger}$
$0.10$	—	$0.50$	$0.00$	$0.17$	$0.42$
$0.25$	—	$0.44$	$0.47$	$0.47$	$0.44$
$0.50$	—	$0.66$	$0.72$	$0.69$	$0.68$

Appendix

Table 20: Certificate safety–return frontier on BabyAI-GoToObj with the real Qwen-

1.5

B advisor ( sweep_alg1_faithful.py MODE=eps ;

10

seeds, recalibrated per

\epsilon

; the cert-off arm shares the cert-on query opportunities so only the execution branch differs). “wrong-acc” is the fraction of corrupted advice executed with the certificate on conditional on reaching a queried corrupted response ; cert-off executes such responses at rate

1.00

wherever queries occur (at

\epsilon{=}0.05

neither arm reaches that branch, as the shared placement has zero calls). It is a proportion over the

10

-seed pool with no per-cell denominator logged, so small values (e.g.

0.00

–

0.17

) are noisy. “

\Delta

ret” is the paired cert-on minus cert-off return with its

95\%

CI half-width; bold marks a CI excluding zero unadjusted (three cells, all negative:

\epsilon{=}0.5

/corrupt

{=}0.75

and

\epsilon{=}1.0

/corrupt

{=}0.5,0.75

; a Holm correction over the

20

-comparison family leaves none). Tightening

\epsilon

reduces corrupted-advice acceptance; no tolerance shows a statistically supported return gain. The

\epsilon{=}0.05

row is a degenerate zero-query regime (see text and

\dagger

Task	metric	$\kappa{=}0$	$0.002$	$0.005$	$0.01$	$0.02$	$0.05$
GoToObj	calls/ep	$4.16$	$1.73$	$1.45$	$0.56$	$0.03$	$0.00$
(never $.093$ )	ret ( $\pm$ CI)	$.122{\pm}.027$	$.110{\pm}.021$	$.116{\pm}.024$	$.104{\pm}.017$	$.101{\pm}.015$	$.093{\pm}.014$
	util $U$	$.122$	$.107$	$.109$	$.098$	$.101$	$.093$
GoToLocal	calls/ep	$4.02$	$1.91$	$1.00$	$0.19$	$0.003$	$0.00$
(never $.124$ )	ret ( $\pm$ CI)	$.125{\pm}.020$	$.115{\pm}.024$	$.103{\pm}.020$	$.115{\pm}.025$	$.122{\pm}.024$	$.124{\pm}.024$
	util $U$	$.125$	$.111$	$.098$	$.113$	$.122$	$.124$

Appendix

Table 21: Faithful Algorithm 2 controller cost sweep ( sweep_alg1_faithful.py , MODE=kappa ; real Qwen-

1.5

20

seeds,

60

episodes). Each

\kappa

is a full rerun of the gate (a complete deployment trajectory, not offline replay);

\kappa

enters only the gate rule so calibration is shared per seed. “ret” is mean return

\pm 95\%

CI over seeds; “util” is

U=R-\kappa\cdot(\text{calls/ep})

. Calls/episode fall monotonically toward zero as the price rises. Return equals never-query exactly only when calls reach exactly zero, which occurs at

\kappa{=}0.05

on all three tasks (GoToObj

.093

, GoToLocal

.124

, GoToObjDoor

.104

, each matching its never-query value); at

\kappa{=}0.02

on GoToLocal a residual

0.003

calls/ep remain, so its return (

0.122

) differs slightly from never-query (

0.124

)—a rounding artifact, not exact abstention. The behavioral claim we make is only that the price knob monotonically reduces calls. On GoToObj the cost-adjusted utility is nominally above never-query (

0.093

) for

\kappa\leq 0.02

(e.g.

0.101

–

0.122

), but paired utility intervals were not retained for this sweep, so we do not claim a statistically established utility gain from it; GoToLocal and GoToObjDoor show no consistent utility benefit over never-query.

Task	Method	$\kappa$	Return	Calls/ep	Net utility	Pareto efficient?
BabyAI-GoToObj	ASK / uncertainty	0.01	$0.046\pm 0.029$	$57.4$	$-0.528$	no
	Ours	0.01	$0.189\pm 0.029$	$25.3$	$-0.064$	no
	ASK / uncertainty	0.02	$0.046\pm 0.029$	$57.4$	$-1.102$	no
	Ours	0.02	$0.208\pm 0.018$	$11.5$	$-0.022$	no
	ASK / uncertainty	0.05	$0.046\pm 0.029$	$57.4$	$-2.825$	no
	Ours	0.05	$\mathbf{0.227\pm 0.049}$	$\mathbf{0.5}$	$\mathbf{0.204}$	yes

Appendix

Table 22: Behavioral cost sensitivity. Every cell comes from an independent run in which

\kappa

changes the query decision. Report mean and 95% CI.

Panel of Figure 1 — Figure 1: Cost–return frontier on BabyAI-GoToObj for the surrogate VoI gate (Table 23 ), which is the deployable single-rollout approximation; each point is a separately trained gate at a different price $\kappa$ , and the online frontier is not assumed monotone. The full Algorithm 2 controller is evaluated separately in Table 13 .

Advisor condition	Certificate	Return	Calls	Accept rate	Unsafe accept	Correct reject	Prevented loss
Natural (real LLM)	off	$0.196\pm 0.025$	686	1.00	n/l	—	—
Natural (real LLM)	on	$0.196\pm 0.025$	686	0.884	n/l	0.116	$\approx 0$
25% corruption (real LLM)	off	$0.214\pm 0.034$	387	1.00	n/l	—	—
25% corruption (real LLM)	on	$0.214\pm 0.034$	387	0.801	n/l	0.199	$\approx 0$
Correlated error (simulated)	off	$0.361\pm 0.041$	400	1.00	0.884	—	—
Correlated error (simulated)	on	$0.368\pm 0.044$	400	0.272	0.234	0.650	$+0.007$

Appendix

Table 24: Reliability and safety. The first four rows use the real Qwen2.5 advisor on BabyAI-GoToObj; unsafe-accept is marked “n/l” (not logged) because that run recorded only aggregate return, calls, and certificate-acceptance, not a per-decision ground-truth wrong-action flag. The last two rows use the simulated advisor on the tabular chain, where the known optimal action makes unsafe-accept observable. The certificate-on and certificate-off returns are nearly identical because the certificate rejects advice that the greedy fallback would have matched anyway (it removes accepted-wrong-advice risk without changing the realized action distribution in these low-headroom tasks). Prevented loss compares otherwise identical runs with and without the certificate.

Variant	Query-value ECE	Return	Calls
Rich-BLL VOI (ours)	$\mathbf{0.262\pm 0.085}$	$0.850$	$51.6$
Deep ensemble	$0.698\pm 0.075$	$0.850$	$65.4$
Novelty proxy	$0.675\pm 0.136$	$0.850$	$90.0$
Variance $\times 0.5$ (overconfident)	—	$0.186\pm 0.813$	$20.8$
Variance $\times 1.0$ (calibrated)	—	$0.850$	$51.6$
Variance $\times 2.0$ (underconfident)	—	$0.850$	$51.6$

Appendix

Table 25: Uncertainty-estimator calibration on the controlled gridworld ( EXTRA3_calibration , budget 90,

\eta=1.0

, 5 seeds). ECE is expected calibration error of the state-value uncertainty; lower is better. The Rich-BLL estimator the controller uses is far better calibrated (ECE

0.26

) than the ensemble (

0.70

) or novelty (

0.67

) proxies, and its VOI gate reaches the same return at fewer calls. This table reports ECE and call counts only; the split-conformal

\beta_{t}

marginal/trajectory coverage is a separate audit (Table 26 ) and is not claimed here. The variance-scaling rows show graceful degradation: an overconfident model (

\times 0.5

) under-queries (20.8 calls) and loses return.

Variant	$n_{\rm cal}$	$J$	$R$	Marginal cov.	Trajectory cov.	Median $\beta$	Abstain rate	Surrogate loss
No radius, $\beta=0$	200	5	20	0.518	0.000	0	0.000	0.001
Constant conformal radius	200	5	20	0.900	0.255	0.057	0.131	0.005
Scaled conformal radius	200	5	20	0.886	0.160	0.000	0.115	0.005
Bonferroni trajectory radius	200	5	20	0.997	0.960	0.138	0.241	0.016

Appendix

Table 26: End-to-end calibration of

\beta_{t}

(

n_{\rm cal}{=}200

so the Bonferroni level

\alpha/T_{\rm eval}

admits a finite split-conformal quantile; paired continuations use common random numbers). Trajectory coverage is the fraction of evaluation trajectories on which all audited opportunities satisfy the conservative proxy check

|\widehat{\Delta}_{t}-\widetilde{\Delta}_{t}|+\rho_{t}^{\rm MC}\leq\beta_{t}

. The

\beta{=}0

row still attains

0.52

marginal coverage because, under common random numbers, the paired residual is exactly zero whenever the advised action equals the greedy action (the two branches then share the identical trajectory); it nonetheless misses the

0.90

target, which is the point of the row. The scaled-radius median

\beta

0.000

because more than half of the per-opportunity local

\sigma_{t}

are zero (again the advice-equals-greedy case), so the median scaled radius

q\,\sigma_{t}

is zero even though its mean is positive and its marginal coverage reaches

0.89

Predictor	Predictor error	Resolutions	Queries used	Rank
Uniform	$0.397$	$5.55$	12	5
Global frequency (cached)	$0.385$	$5.55$	12	4
Per-context online (state)	$0.265$	$5.55$	12	3
Feature-generalizing online	$0.202$	$8.82$	12	2
Replay oracle, diagnostic	$0.000$	$10.22$	12	1

Appendix

Table 27: Response-predictor ablation on the controlled tabular chain with state-varying advisor reliability (the setting that makes predictor quality separable from allocation quality without confounding from a real LLM; verify_predictor_ablation ). Predictor error is the mean absolute error of the predicted response distribution; resolutions is the number of state–action pairs correctly resolved under a hard query budget of 12; 60 seeds. The retrospective oracle is diagnostic and never selects an online query. Regime shown: high/low reliability

0.9/0.1

Variant	Predictor	Values $d_{t}(y)$	Certificate	$\beta_{t}$	Return	Calls	Unsafe accept	Allocation loss
VOE/Q-style value gate	no	no	no	0	$34.63\pm 1.07$	36.8	0.200	1.91
Marginal predictor, raw action	yes	no	no	0	$34.42\pm 1.07$	32.3	0.201	2.23
Response-contingent gate	yes	yes	no	0	$35.52\pm 1.21$	32.3	0.093	2.24
Response-contingent + certificate	yes	yes	yes	0	$35.54\pm 1.19$	32.4	0.088	2.18
Full conservative controller	yes	yes	yes	calibrated	$34.95\pm 1.20$	16.9	0.056	2.98

Appendix

Table 28: Main component ablation at matched environment steps and advisor. Allocation loss uses the myopic surrogate in Eq. ( 29 ); unsafe accept is measured against simulator-optimal actions.

Cache policy	Pre-shift return	Final return	Recovery episodes	Post-shift calls
No cache	$-0.81$	$0.194\pm 0.72$	$92.4$	$1.6$
Time-to-live	$-0.81$	$-0.122\pm 0.74$	$108.0$	$1.2$
Epistemic (certificate)	$-0.81$	$-0.81$	$150.0$	$1.0$

Appendix

Table 29: Distribution shift on the controlled gridworld ( EXTRA4_shift_cache : goal relocated at 50% of training, 5 seeds), reported as a budget-limited pilot : post-shift calls are only

1.0

–

1.6

, so the budget was effectively spent before the shift and the cache policies cannot make materially different post-shift acquisition decisions. Recovery is the number of episodes to regain the prespecified fraction of pre-shift performance. Even so the finding is negative : epistemic (certificate-based) cache invalidation uses the fewest post-shift calls (

1.0

vs.

1.6

) but recovers slower , not faster, than no cache—the epistemic signal does not by itself detect the goal move. The reserved-budget protocol needed to separate the policies cleanly is not yet run.

$\eta$	0.0	0.25	0.5	0.75	1.0
Queries, measured	60.0	38.6	24.2	15.9	12.0
Strict comparator	96.0	72.0	48.0	24.0	0.0

Appendix

Table 30: Controlled chain query counts. The second row is a strict comparator, not a proved bound: it omits discovery calls and therefore incorrectly reaches zero at

\eta=1

Method	Return	Calls	Cost-adjusted
Never-query	0.182	0	0.182
Fixed-period	0.186	90	-0.714
Random-gated	-0.48	60	-1.08
Novelty-gated	0.850	90	-0.050
Ensemble-gated	0.850	65.4	0.196
Always-query	-0.48	90	-1.38

Appendix

Table 31: Pilot GridWorld means at the operating points used by the calibration and repricing diagnostics. Cost-adjusted return is

R-0.01N_{\rm calls}

. These five-seed estimates are not used for significance claims.

Advisor property	Rate
Local response-rule adherence	1.000
Optimal-action accuracy	n/a (no oracle)
Obstacle-conflict rate	n/a (not instrumented)
Parser failure	0.000
Certificate acceptance	0.884

Appendix

Table 32: Audit of the deterministic

\eta=1

pilot advisor. Reliability denotes agreement with the local response rule; optimal-action accuracy is evaluated against dynamic-programming actions in the simulator.

Reliability	Return	Calls
$\eta=1.0$	0.85	51.6
$\eta=0.75$	0.52	61.4
$\eta=0.5$	0.19	64.4
$\eta=0.25$	-0.15	77.6
$\eta=0.0$	0.18	87.0

Appendix

Table 33: Pilot reliability sweep without the execution certificate.

Reliability	Return	Calls	Unsafe accept
$\eta=1.0$	0.850	51.6	0.000
$\eta=0.75$	0.850	62.8	0.046
$\eta=0.5$	0.186	73.0	0.150
$\eta=0.25$	0.514	78.2	0.163
$\eta=0.0$	0.182	75.8	0.275

Appendix

Table 34: Pilot reliability sweep with the execution certificate, budget 90 and five seeds. Unsafe accept is the fraction of wrong advice that is executed.

Estimator / variant	ECE	Calls	Return
Rich-BLL gate	0.262	51.6	0.850
Ensemble	0.698	65.4	0.850
Novelty	0.675	90.0	0.850
Variance $\times 0.5$ , overconfident	—	20.8	0.186
Variance $\times 1$ , nominal	—	51.6	0.850
Variance $\times 2$ , underconfident	—	51.6	0.850

Appendix

Table 35: Pilot gate calibration and variance scaling. Labels follow statistical convention: smaller variance is more overconfident, and larger variance is more underconfident.

Method	0.001	0.005	0.010	0.020
Fixed-period	+0.096	-0.264	-0.714	-1.614
Novelty gate	+0.760	+0.400	-0.050	-0.950
Ensemble gate	+0.785	+0.523	+0.196	-0.458
Oracle response model	+0.798	+0.592	+0.334	-0.182
Safe Rich-BLL gate	+0.798	+0.592	+0.334	-0.182

Appendix

Table 36: Pilot post hoc repricing of fixed trajectories. This table does not show behavioral adaptation to query price.

Quantity	$9\times 9$	$14\times 14$
Never-query return	0.18	0.75
Gate return	0.85	0.75
Gate calls	51.6	90.4
Gate cost-adjusted return	0.33	-0.15
Always-query cost-adjusted return	-1.38	-0.45
Return lift over never-query	0.67	0.00

Appendix

Table 37: Pilot scale transfer. The larger task is a negative result for net utility, not evidence of successful query allocation.

Cache strategy	Pre-shift	Final return	Recovery ep.	Post-shift calls
No cache	-0.810	+0.194	92.4	1.6
Time-to-live	-0.810	-0.122	108.0	1.2
Epistemic invalidation	-0.810	-0.810	150.0	1.0

Appendix

Table 38: Pilot goal-relocation result with an exhausted post-shift budget.

Predictor	Resolutions / budget	Predictor error
Oracle	10.2	0.00
Feature-generalizing online	8.8	0.20
Per-state online	5.5	0.26
Cached global frequency	5.5	0.39
Uniform	5.5	0.40

Appendix

Table 39: Constructed chain predictor diagnostic with

\eta_{\rm hi}=0.9

\eta_{\rm lo}=0.1

, 24 states, and budget 12.

Regime	Acquisition	Execution
$c_{t}\to 0$	query positive lower-bound value	certified action only
$c_{t}\to\infty$	no fresh query	fallback or certified cache
Reliable advisor	sparse discovery and refresh	often accepted
Correlated wrong advisor	predictor should suppress repeats	rejected if uncertified

Appendix

Table 40: Acquisition and execution are distinct at important limits.

Execution rule	Return	Calls	Advice acceptance	Wrong-advice acceptance	Prevented loss
Variance-only cache rule	$0.50\pm 0.16$	37.3	1.000	1.000	—
Action-gap certificate	$0.73\pm 0.01$	38.7	0.772	0.180	$-$ 42%
Certificate without conservative query LCB	$0.74\pm 0.01$	32.5	0.752	0.041	$-$ 48%
Full controller	$0.74\pm 0.01$	20.4	0.740	0.255	$-$ 46%

Appendix

Table 41: Variance-only execution versus the action-gap certificate at matched query opportunities.

Explore similar work

Sep 29, 2026cs.AI

AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation

A small trainable advisor can steer a frozen language-model executor using natural-language advice. In addition to learning from task rewards, the advisor can use feedback from completed interactions to improve its advice. However, a plausible correction need not change execution, yet learning from such corrections can still affect the advisor's future decisions in other contexts. In a shared-parameter model, we prove that such corrections can limit learning if their targets favor useful advice less strongly than those of other corrections. Keeping them less often than the rest improves the model's eventual performance compared to learning from every correction. Motivated by this, our method, Advisor Self-Distillation (AdviSD), pairs outcome-based reinforcement learning with self-distillation from a feedback-conditioned copy of the advisor selectively. Reflection proposes corrections, and the advisor scores the same recorded executor response with and without its issued advice, using the magnitude of the difference to select decisions for supervision. This approach does not require executor likelihoods or additional executor rollouts. Experiments with Qwen3-8B advisors for Gemini and Claude show that AdviSD outperforms advisor-GRPO by 4.2-6.4 percentage points on BFCL-v3 and by 3.9-5.1 score points on EnvScaler. The trained advisors generalize to out-of-domain tasks and transfer across different executor versions and model families. AdviSD also beats matched-count random selection, supporting the value of its selection rule.

Rishabh Agrawal, Hejie Cui, Shasha Li +2

Google · University of Southern California

Language Model Steering LLM Agent Self-Improvement Self-Distillation RL for LLM Agents

Ask PDF

Sep 30, 2026cs.AI

OpenJev-RLCD: A Working RLCD Implementation

Decision models such as Jev answer questions with probabilities, which are only useful if they are calibrated. Open-source reproductions rely on supervised fine-tuning plus temperature scaling, while reinforcement learning from verifiable rewards (RLVR) makes reasoning models overconfident. We present a working implementation of reinforcement learning for calibrated decisions (RLCD) for reasoning models: the model samples a rationale, and we score the answer distribution it commits to afterwards with a strictly proper scoring rule. A variance identity shows that scoring the mixture of several samples rewards disagreeing rationales, and that RLVR is exactly this mixture objective without its diversity term. Optimized naively, the per-rationale objective either switches reasoning off or is drowned out by policy-gradient noise, which leads to a two-stage recipe: calibrate, then reinforce. With Qwen3-1.7B on two reasoning tasks (3 seeds, paired tests), RLCD matches or beats SFT, RFT/STaR and GRPO (each temperature-scaled) in accuracy and beats all of them in selective prediction; on GSM8K answer verification a single query decides \gvTwoCovFive% of the items at

\le

5% error, versus \gvGrpoCovFive% for GRPO. When uncertainty comes from annotator disagreement, RLCD provably cannot beat cross-entropy. Code and results: https://github.com/ZimmyGao/openjev-rlcd.

Zhimin Gao, Pichao Wang

RL for Language Models RL for Language Model Reasoning Language Model Calibration

Ask PDF

May 27, 2026cs.LG

Knowing When to Ask: Segment-Level Credit Assignment for LLM Tool Use

Humans know when to reach for help e.g.

347 \times 28

warrants a calculator while

2+2

does not. Language models do not. Prompt-based approaches can instruct a model when to invoke tools, but this scaffolding does not teach it to recognize the boundary of its own knowledge. RL approaches that assign a single outcome reward to the whole trajectory fare no better: trajectory-level credit cannot isolate which tool call in a successful episode actually helped, nor penalize unnecessary calls. We propose \textbf{CARL} (\textbf{C}ompetence-\textbf{A}ware \textbf{R}einforcement \textbf{L}earning), which trains a critic on the model's own rollouts to learn where parametric knowledge suffices and where it needs external help. By decomposing each rollout at natural tool-use boundaries (e.g., code fence delimiters and context block transitions), CARL assigns independent credit to each segment from a single binary outcome, without external judges or step-level annotations. As a result, erroneous tool calls, incorrect extractions, and unnecessary calls each receive appropriately signed advantages. The trained critic captures the model's domain competence: it separates parametrically solvable from tool-dependent questions with AUC 0.93 at 7B. On five benchmarks spanning arithmetic, multi-hop factual QA, and numerical reasoning over financial tables, CARL improves exact-match accuracy by 6.7 points at 7B and 9.7 points at 3B over the best RL baseline, with the largest gain (+8.3 EM at 7B, +9.0 EM at 3B) on Musique. The model issues 53% fewer tool calls on parametrically answerable questions while remaining

{\sim}10

EM points more accurate on them. Gains are largest at small scale: the 3B improvement is

1.4\times

the 7B improvement, suggesting that knowing when to ask disproportionately benefits models with smaller parametric memory.

Abhijit Kumar, Zoey Wu, Mohit Suley

Microsoft AI Redmond, WA

Tool-Augmented Language Model Agents Learning to Defer LLM Tool Use Credit Assignment Tool Selection Reward Design for RL+1

Ask PDF