When Should Enterprise Policy Decisions Be Learned, and When Should They Be Reasoned?

Hyojung Han; Ryangkyung Kang; Youngeun Kim; Seung-Hun Jeon

cs.AIOct 13, 2026

When Should Enterprise Policy Decisions Be Learned, and When Should They Be Reasoned?

Authors: Hyojung Han, Ryangkyung Kang, Youngeun Kim, Seung-Hun Jeon

Organizations: ThakiCloud · Korea University

Abstract

An enterprise agent that answers under a written policy must choose an action before it says anything: answer, ask for a missing fact, ask for confirmation, call one or several tools, escalate, or refuse. Holding one 27B Korean enterprise model fixed, we vary whether the decision is a generated JSON object or a single constrained token, and whether the model may reason first. On 2800 in-distribution items we did not detect an accuracy difference between the two deployed paths (93.46% vs 94.14%), while they differ about tenfold in restart stability (3.00% of actions flip vs 0.29%); they also differ in reasoning, so this is an operational comparison. On a sealed benchmark of 300 items whose labels come from a hidden executable policy, reasoning is worth +17.3 points [+12.3, +23.0] on exact action. With reasoning off, the format difference was 0.0 points [-4.3, +4.0], a failure to detect an advantage for either interface, not equivalence; with reasoning on it is unresolved. Against a token-matched random control, boundary-targeted supervision moves the two covered boundaries by +17.5 and +20.0 points and leaves a held-out boundary unchanged. In a 3-seed training x reasoning factorial, the sealed-rendering interaction is -4.9 [-8.1, -1.7], consistent with partial substitution. All supervision was Korean. On independently rendered English versions, the model is 0.82 points lower [-2.68, +0.93] with reasoning on, the same order as re-rendering in Korean; the reasoning-off gain from targeted supervision is not stable in either language, so we do not claim transfer. The original text stays private as the confirmatory set. An open-model re-rendering made after these results reproduced the reasoning effect and the English verdict; the interaction kept its sign but did not replicate under item-clustered intervals. We release the model and that re-rendering with gold labels.

Figures & tables

Operational path	Clean	Full	Restarts	Pairs	Flip mean	Flip range
Reasoning ${+}$ JSON action	96.86%	93.46%	5	10	3.00%	2.54–3.36%
No reasoning, single token	96.75%	94.14%	4	6	0.29%	0.18–0.50%

Table 1: Two operational paths on the in-distribution split. Clean is the 2414-item strict split, Full the 2800-item split. Flip is the fraction of items whose chosen action differs between two restarts of the same checkpoint on the same hardware, as the mean and range over restart pairs; pairs share restarts and are not independent, so no pooled interval is given. The two paths differ in reasoning as well as in output format, so these rows compare deployments, not output interfaces.

Figure 1: One scenario of the public open re-rendering, end to end. (a) Hidden factors of item b3-coun-00083-cf ; the executor checks its rules in a fixed priority order and the first that holds, call_tool , is the gold action, with a proof naming the rule and the factors it read. Below: three rendered Korean lines, each followed by the independently rendered English line. (b) Its counterfactual twin: one generating condition is flipped, the dependent factors follow, the executor fires answer , and exactly one rendered line differs. Long lines wrap with an indent.

Metric	Reasoning	JSON object	Single token
Exact action	off	70.7%	70.7%
	on	85.3%	88.0%
Both-correct (CF pairs)	off	46.7%	36.7%
	on	76.7%	80.0%

Table 2: Reasoning crossed with output format on the sealed benchmark (

n=300

; 30 counterfactual pairs). Each cell is a single run of the same checkpoint. Reasoning moves both metrics by a wide margin. With reasoning off, we did not detect an output-format effect on exact action on this sample (a failure to detect a difference, not an equivalence result); with reasoning on, the two arms do not share a reasoning prompt, so the format contrast in that row is unresolved. On the counterfactual endpoint the format effect is not identically zero, and the one secondary endpoint where it is clearly non-zero favours the single token on this rendering only (see text and Section 10 ).

Figure 2: Exact-action accuracy on the sealed benchmark. The vertical gap between the two lines is test-time reasoning; the horizontal slope of either line is the output format. The reasoning-off horizontal contrast isolates the output format cleanly; the reasoning-on horizontal contrast retains the prompt confound described in the text, because those two arms do not share a reasoning prompt. Holding format fixed, reasoning is worth

{+}17.3\,

[{+}12.3,{+}23.0]

. Holding reasoning off, changing the format is worth

0.0\,

[-4.3,{+}4.0]

; holding it on,

{+}2.7\,

[-0.7,{+}6.0]

. Format effects are written as single token minus JSON. Intervals are paired item-resampling bootstraps over the 300 items.

Figure 3: The same two-by-two scored on counterfactual pairs, where an item counts only if the model gets both members right. The effect of reasoning is larger here (

{+}43.3\,

[{+}23.3,{+}63.3]

) because getting both members right requires responding to the one latent variable that changed rather than to the surface of either item. Unlike exact action, the format effect here is not identically zero (

-10.0\,

pp with reasoning off, interval

[-20.0,-0.0]

), so we report it as inconclusive. This is a different quantity from Figure 2 and is therefore plotted on its own axis.

	$n$	Random	Targeted	$\Delta$	95% CI
Pre-registered primary endpoints
Exact action	300	74.4%	79.2%	${+}4.8$	lower $={+}2.00$ pp
Both-correct (CF pairs)	30	46.7%	55.6%	${+}8.9$	lower $={-}1.11$ pp
Any-tool recall	118	60.2%	70.1%	${+}9.9$	lower $={+}4.24$ pp
Per boundary (reporting only)
ANSWER / CALL_TOOL	42	64.3%	81.7%	${+}17.5$	$[+6.3,+29.4]$

Table 3: Boundary-targeted supervision against a token-budget-matched random control, 3 seeds per arm, evaluated on the sealed benchmark. Note that

n

differs by endpoint: 300 items overall, 30 counterfactual pairs, 118 items where a tool call is required (this endpoint’s ledger counts call_multiple_tools as tool-requiring; the two-by-two ledger of Section 5 does not, and reports 89). Primary endpoints were fixed before the benchmark was scored; per-boundary rows are reporting only. Every interval, including the pre-registered lower bounds, is the item-clustered bootstrap used throughout the paper: items (or counterfactual pairs) are resampled with replacement, the 3 seeds of an item move together, the statistic is the seed-averaged paired difference (

B=10{,}000

, seed 12345, 95% percentile). The registered gate reads only the lower end of that interval, so the primary rows print the lower bound alone. The intervals carry evaluation sampling uncertainty for these 3 trained seed pairs, not retraining uncertainty, which we report separately as per-seed deltas in the text. The same exact-action contrast appears as

T-C0

in Table 9 with the same interval. The held-out row is a boundary that neither arm’s added data covers. Pooled boundary groups is item accuracy over the union of the four per-boundary rows above it (each item counted once, held-out boundary included;

n=128

), averaged over seeds. It is not the “boundary layers” column of Table 9 , which is defined differently.

Figure 4: Boundary-targeted supervision minus the token-budget-matched random control, per boundary, with 95% item-clustered bootstrap intervals (the method of Table 3 ) over the 3 trained seed pairs. The two covered boundaries with headroom clear zero; the held-out boundary (hollow diamond) does not. Missing information barely moves: its control arm already sits at 80.0 % on 30 items, leaving little headroom. This is consistent with limited headroom rather than a failure of the supervision, but the resolution here does not establish that on its own, and we do not interpret the

{+}2.2\,

pp estimate.

Figure 5: Alternative configurations of data and test-time compute, scored on the sealed benchmark. These are competing options, not a cumulative progression: the pointer head and the random-data checkpoint are different artefacts, not one built on the other. The first four spend a single forward pass; the last spends test-time reasoning. The three trained rows — the pointer head (3 seeds, 67.0–80.3 %) and the two supervision arms (3 seeds each) — are seed means and carry a bar spanning their seed minimum and maximum; the other two are single runs and carry no bar, because they have no spread to show rather than a small one. The trained bars overlap, which is the point of Section 6 ’s pre-registered reading: only any-tool recall cleared its bar. Two harnesses share this axis: the first and last bars are the vLLM constrained-token probe of Section 5 on the base checkpoint, the three trained bars the pointer-head harness of Section 6 .

Configuration	Exact action
No thinking, constrained single token	70.7%
No thinking, LoRA pointer head	74.6%
Trained on random in-domain data	74.4%
Trained on boundary-targeted data	79.2%
Always thinking, constrained single token	88.0%

Table 4: The same configurations as numbers. Rows above the rule are single-pass; the row below spends test-time reasoning. The pointer-head row is a 3-seed mean (sd 6.8 pp) and the two supervision rows are 3-seed means; the constrained-token and always-thinking rows are single runs. The rows come from two harnesses: the first and last rows from the vLLM constrained-token probe of Section 5 on the base checkpoint, the middle three from the pointer-head harness of Section 6 . They are competing configurations measured by different decision mechanisms, not differences of one another.

Added supervision	No reasoning	Reasoning
Random, budget-matched	72.1%	88.7%
Boundary-targeted	75.2%	86.9%

Table 5: Training

\times

reasoning, all four cells measured on one harness, 3 seeds per arm, 300 sealed items. Lower panel: the pre-registered estimands. “Stable” requires both that the 3 seed-paired differences share a sign and that the item-resampling interval excludes zero;

E_{3}

’s interval excludes zero but its seed signs do not agree, so the rule calls it indeterminate rather than letting us claim it. These cells use the vLLM LoRA serving harness with its constrained-token probe, a different harness from the pointer head of Table 3 ; the reasoning-off cells here are therefore not the numbers of Table 3 and should not be compared with them.

Figure 6: The same four cells. The vertical gap within a line is what reasoning buys; the gap between the lines is what the supervision buys. The lines are not parallel, and they cross: that non-parallelism is the interaction, and its sign says the two levers substitute.

Effect (pp)	All 300 items	Shortcut-clean 240
Reasoning effect (single token)	${+}17.3$	${+}20.8$
Format effect, reasoning off	$0.0$	${-}0.8$
Format effect, reasoning on	${+}2.7$	${+}4.2$
Targeted ${-}$ random, exact action	${+}4.8$	${+}5.4$
Targeted ${-}$ random, CALL_TOOL-only recall	${+}15.7$	${+}17.8$
Targeted ${-}$ random, any-tool recall	${+}9.9$	${+}11.7$

Table 6: Every main effect recomputed on the 5 layers whose lexical shortcut probes stayed under their ceiling. Recomputed from stored predictions; no new runs, and not pre-registered. Positive favours reasoning on, the single token, or targeted supervision respectively.

Frame	Document	No reasoning	Reasoning
Korean	Korean	77.51%	89.16%
Korean	English	76.34%	88.34%
English	Korean	76.81%	89.63%
English	English	77.86%	88.34%
Korean	Korean, re-rendered	76.22%	88.69%

Table 7: Zero-shot transfer to English. All cells use the 286 items valid in every rendering; this subset is 1.5–2.2 pp easier than the full benchmark, so absolute accuracies are not comparable with other tables. Top: exact action under each combination of instruction-frame and policy-document language, and a second independent Korean rendering. Bottom: paired contrasts with item-bootstrap 95% intervals, means over 3 training seeds of the boundary-targeted arm (first two rows) or the targeted and budget-matched arms (last two). No English data was used in training.

Finding (pp)	Original (sealed)	Open re-rendering	Grade
Reasoning, single-token column	${+}17.3$ $[12.3,23.0]$	${+}21.8$ $[16.1,27.8]$	Replicated
Format, reasoning off	${+}0.0$ $[-4.3,4.0]$	${-}4.8$ $[-10.1,0.4]$	Null replicated
Format, reasoning off, call_tool -only recall ( $n=89$ )	${+}12.4$ $[4.5,20.2]$	${+}2.6$ $[-5.3,10.5]$	Did not replicate
Format, reasoning on	${+}2.7$ $[-0.7,6.0]$	${+}2.4$ $[0.0,4.8]$	Null replicated
Targeted ${-}$ random, exact action	${+}4.8$ (lo $+2.0$ )	${+}7.7$ (lo $+4.7$ )	Replicated
Targeted ${-}$ random, CF both-correct	${+}8.9$ (lo $-1.1$ )	${+}13.9$ (lo $+2.8$ )	Directionally consistent

Table 8: Replication on the open re-rendering. Original : the sealed, private rendering on which every primary analysis rests (

n=300

; 286 for the English rows). Open : the public re-rendering of the same latent scenarios and gold labels with open-weight models, sealed on 248 common items and generated after the original results were known. Paired contrasts in pp with item-bootstrap 95% intervals (lower bound only for the pre-registered endpoints of Section 6 ); format contrasts are single token minus JSON. Grades use the post-hoc robustness taxonomy of Section 10 ; they are descriptive, assigned by code from the ledger, and are not pre-registered endpoints. R4 and C1 rows use the pre-registered stability rule.

Added supervision	Exact action	Boundary layers ( $n=120$ )	CALL_TOOL-only
Random, budget-matched (C0)	74.4%	73.6%	59.2%
Random, label-matched (C1)	73.4%	73.1%	56.2%
Boundary-targeted (T)	79.2%	79.4%	74.9%
$T-C1$ exact action	${+}5.8$	$[2.7,8.9]$	indeterminate
$T-C1$ boundary layers	${+}6.4$	$[0.6,12.5]$	indeterminate
$T-C0$ exact action	${+}4.8$	$[2.0,7.7]$	stable

Table 9: The label-matched control (post-hoc, not pre-registered). Upper panel: arm means over 3 seeds on the sealed benchmark. C0 is the same checkpoints as the random arm of Table 3 , so its exact-action value matches that table. Boundary layers is item accuracy over all

n=120

items of the 3 boundary-constructed layers (answer versus tool, confirm versus refuse, missing information), whatever their gold action, averaged over seeds; it excludes the policy-versus-principle layer. It is therefore not the pooled boundary groups of Table 3 (

n=128

), which select items by layer and gold-action pair and include the held-out boundary in both of its layers. Lower panel: contrasts, their item-resampling intervals, and the stability verdict. C1 matches the treatment arm’s action-count histogram exactly — the builder asserts equality and refuses otherwise — and withholds the contrastive pairing. The separate token budget lands within 0.26 % overall, with two actions short of their per-label token target because the factory pool contains no records long enough; the counts are exact, the token totals are not. The last column of the upper panel is call_tool -only recall over the

n=89

items whose gold action is call_tool , not the any-tool recall of Table 3 (

n=118

	call_tool
Arm	P	R	F1	Pred. rate	Macro-F1
Gold (n=300)				29.7
Random, budget-matched (C0)	91.9	59.2	72.0	19.1	72.8
Random, label-matched (C1)	91.2	56.2	69.2	18.3	72.9
Boundary-targeted (T)	89.9	74.9	81.3	24.8	76.1
$T-C0$ , mean (pp)	${-}2.0$	${+}15.7$	${+}9.3$	${+}5.7$	${+}3.3$

Table 10: Per-class view of the C0, C1 and T arms of Table 9 (post-hoc, recomputed from stored pointer-head predictions; 3 seeds per arm, 300 items, 89 gold call_tool ). Precision, recall, F1 and predicted rate are for call_tool only; macro-F1 averages the seven actions. Seed-mean rows are in %; contrast rows give the seed-mean difference in pp and the sign of each seed-paired difference.

Figure 7: Per-class recall (left) and the marginal predicted-action distribution against the gold distribution (right) for C0, C1 and T, seed means over 3 seeds on the 300 sealed items. The right panel is a marginal, not a confusion matrix; the full matrix is Figure 8 . T’s call_tool recall and predicted call_tool share rise together; its call_tool precision does not (Table 10 ). The two panels are separate figures and assign colours to arms independently; read each against its own legend.

Figure 8: Full

7{\times}7

confusion matrices for C0, C1 and T on the 300 sealed items (post-hoc, stored pointer-head predictions). Rows are the gold action with its item count, columns the predicted action. Each cell is the seed-mean count over 3 seeds divided by the row’s gold count, in %; the grey level encodes the same number, which is printed so the figure reads without colour. Rows sum to 100 up to rounding. Per-action precision and recall are in Table 11 .

		C0		C1		T
Gold action	$n$	P	R	P	R	P	R
ANSWER	52	91.9	94.2	93.3	84.0	91.2	98.7
ASK_CLARIFICATION	16	65.3	100.0	62.4	100.0	66.5	93.8
ASK_CONFIRMATION	24	66.2	94.4	67.5	94.4	78.4	91.7
CALL_MULTIPLE_TOOLS	29	40.3	63.2	47.4	59.8	58.0	55.2
CALL_TOOL	89	91.9	59.2	91.2	56.2	89.9	74.9

Table 11: Per-action precision (P) and recall (R) for all seven actions, C0, C1 and T, in %, each the mean of the 3 per-seed values (post-hoc, recomputed from stored pointer-head predictions on the 300 sealed items;

n

is the gold count). Precision is taken as 0 for a seed that never predicts the action. Macro-F1 averages the seven per-action F1 scores. The call_tool row is the one summarised in Table 10 .

Appendix figures & tables7 assets

Supplementary material from the paper’s appendix.

Appendix

Result	Checkpoint / interface	$n$	Runs / seeds	Denominator	Uncertainty	Harness
Tab. 1 Reasoning+JSON vs no-reasoning single token, in-distribution	Qwen3.8-27B-Human-KO-Enterprise-v0.1 (no adapter) JSON action object (deployed path) vs constrained single token (letter code)	clean 2414; full 2800	json restarts 5; json pairs 10; constrained restarts 4; constrained pairs 6; accuracy reps used 2,3,4,5 no training seeds; restarts of the same serving checkpoint (temperature 0)	accuracy: strict clean split 2414 / full 2800 items; flip: same two splits, items whose action differs between two restarts	none pooled; mean/min/max over restart pairs (pairs share restarts, not independent)	Section 3 harness (pace/r0, vLLM v0.28.0, H200 x1); NOT the Section 6 pointer-head harness
Tab. 2 , Fig. 2 , Fig. 3 Reasoning x output format 2x2 on sealed Blind-A	v0.1 base (no adapter) { reasoning on/off } x { JSON object, single token }	items 300; cf pairs 30; invariance pairs 15	1 run per cell single run per cell (no seeds); temperature 0	exact action: all 300 items; both-correct: 30 CF pairs (pair is the unit); call_tool_recall (secondary): 89 CALL_TOOL-only items	paired item bootstrap, n_boot=2000, fixed seed 20261001; pair-level resampling for pair metrics	Section 5 serving harness pace/race/race_2x2.py (vLLM v0.28.0, B200 x1, concurrency 16)
Tab. 3 , Fig. 4 Boundary-targeted (T) vs token-budget-matched random (C) supervision	v0.1 + LoRA r=16 + pointer head, 3 seeds per arm trained pointer head (argmax over 7 options), no generation, no reasoning	items 300; cf pairs 30; any tool required 118; boundary pooled 128; invariance pairs 15	3 seeds x 2 arms 1,2,3 (T_i vs C_i seed-paired)	Recall endpoint here = ANY-tool-required items n=118 (CALL_TOOL + CALL_MULTIPLE_TOOLS); the 2x2 ledger CALL_TOOL-only n=89	item-clustered bootstrap (item_clustered_bootstrap_v1): units with replacement; all seeds of a unit stay together; B=10000, seed 12345; stable = unchanged registered definition: per-seed signs agree (len({d¿0})==1) AND CI excludes 0	Section 6 pointer-head harness (kev eval_kev.py, LocalPredictor + head.pt); trained T-s1/C-s1/T-s3/C-s3 on H200 NVL, T-s2/C-s2 on B200
Tab. 4 , Fig. 5 Configuration ladder (competing options)	mixed: v0.1 base (rows 1,5), R1b pointer head (row 2), R3 C and T adapters (rows 3,4) row1 vLLM constrained token; row2-4 pointer head; row5 vLLM reasoning-on single token	items 300	row1 1; row2 3; row3-4 3; row5 1 rows 2-4 seed means (3 each); rows 1,5 single run	exact action over 300	no intervals in the table; bars = seed min/max in figure	MIXED: rows 1,5 = Section 5 vLLM probe (B200); rows 2,3,4 = Section 6 pointer-head harness
Tab. 5 , Fig. 6 Training x reasoning, 4 cells, one harness	v0.1 + LoRA r=16 (R3 adapters, pointer head unused), 3 seeds per arm, key-fixed for vLLM vLLM LoRA serving; no-think = enable_thinking=false 1-token; think = on_1tok two-stage	items 300	12 cells (6 adapters x 2 reasoning states) [1, 2, 3]	exact action over 300; recalls secondary (CALL_TOOL-only 89 / any-tool 118)	item-clustered bootstrap (item_clustered_bootstrap_v1): units with replacement; all seeds of a unit stay together; B=10000, seed 12345; stable = unchanged registered definition: per-seed signs agree (len({d¿0})==1) AND CI excludes 0	Section 7 serving harness (one vLLM v0.28.0 server, –enable-lora, H200 x1 per r6en ledger reproduction_check.note; spec pace/r4/job/pace-r4-serve-h200.json)
Tab. 6 Denominator and shortcut sensitivity (recompute only)	as in Tables 2,3 as in Tables 2,3	all 300; shortcut clean 240; call tool only 89; any tool 118	reuses Table 2/3 predictions as source	CALL_TOOL-only n=89 vs any-tool n=118; clean layers = 5 of 7 (dialect_noisy, missing_information dropped)	point estimates only (P2 ledger has no CI)	mixed: 2x2 effects = Section 5 serving; T-C = Section 6 pointer head

Appendix

Table 20

Figure 9: Verbatim Korean prompt contract (a) and single-token suffix (b), rendered from the code that built the requests. Lines too long for the page are wrapped with an indent; no other change is made.

Action	Code	Token id	Tokens in action name
ANSWER	A	32	2
ASK_CLARIFICATION	B	33	4
ASK_CONFIRMATION	C	34	3
CALL_TOOL	D	35	2
CALL_MULTIPLE_TOOLS	E	36	5
ESCALATE	F	37	3

Appendix

Table 13: Action, letter code and token id in the Qwen3.8 tokenizer, with the number of tokens in each action name. The same mapping was used on every restart.

Item	Value
Base checkpoint	Qwen3.8-27B, revision 1d4bf0f2
Adapter	LoRA rank 16, alpha 32, dropout 0.05, bias none
LoRA targets	down_proj, gate_proj, in_proj_a, in_proj_b, in_proj_qkv, in_proj_z, k_proj, o_proj, out_proj, q_proj, up_proj, v_proj
Pointer head	hidden dim 256, trained from scratch
Trainable parameters	119,349,760
Learning rate	LoRA 5e-5, head 1e-3

Appendix

Table 14: Training configuration of the three supervision arms (boundary-targeted, budget-matched random, and the post-hoc label-matched arm of Section 13 ), read from each arm’s job specification, its token-budget record, the adapter configuration and the per-run training metrics. The build asserts that all nine job specifications share these training arguments and base revision and that the adapters match.

Flag	Value
Engine image	vllm/vllm-openai:v0.28.0
dtype	bfloat16
max-model-len	8192
max-num-seqs	256
gpu-memory-utilization	0.9
kv-cache-dtype	fp8

Appendix

Table 15: vLLM engine configuration, identical across the in-distribution and sealed-benchmark harnesses (checked against both readbacks).

Setting	Path	p50 (s)	p95 (s)	Median gen. tokens
In-distribution, $c=1$	Reasoning + JSON (cold prefix)	8.89	27.43	538
	No reasoning, single token (cold prefix)	0.085	0.168	1
In-distribution, $c=8$	Reasoning + JSON (cold prefix)	10.68	27.71	568
	No reasoning, single token (cold prefix)	0.107	0.173	1
Sealed benchmark, $c=16$	Reasoning on, JSON	12.55	68.27	972
	Reasoning on, single token (both stages)	4.19	41.25	255

Appendix

Table 16: Absolute latency and median generated tokens (reasoning included). In distribution: 40-item sample, 4 restarts. Sealed benchmark: all 300 items, one run per path; the reasoning-on JSON latency comes from a rerun, while its accuracy in Table 2 comes from the original run.

Gate	Criterion	Value	Pass
G1 positive control, planted correct	both readouts $\geq 90$ % follow the planted action	93.0 / 97.7	yes
G1 positive control, planted wrong	both readouts $\geq 90$ % follow the planted action	10.3 / 50.0	no
G2 no collapse (real)	both readouts $\geq 40$ %	85.7 / 86.0	yes
G3 trace sensitivity	real ${-}$ shuffled $\geq 3$ pp in either readout	67.7 / 66.3 pp	yes
G4 determinism	re-scored argmax agreement $\geq 95$ %	100.0	yes
G5 truncation	truncated traces $\leq 5$ %	3.3	yes

Appendix

Table 17: Gates of the matched-state diagnostic (generated from the measurement ledger). Paired values are single token / JSON, in % of 300 items; for the planted conditions the value is the share of items whose readout equals the planted action. G4 re-scores a subset of the real items. G1 fails on the planted-wrong condition, so the run is void.

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 242 papers

Aug 21, 2026

Why2Speak: Faithful Reasoning for Abstaining Action Policies

Many agentic systems must repeatedly choose between acting and abstaining, making faithful reasoning important for oversight: an explanation is useful only if it reflects the computation that produced the action. We study this problem through intervention timing in multi-party conversation, where an assistant must decide whether to speak or remain silent. This setting exposes class imbalance, asymmetric action costs, and the possibility that exposing reasoning changes the policy being audited. Using Qwen3-8B, decoded with or without chain-of-thought reasoning, we compare direct decision policies, reasoning policies, supervised fine-tuning, and reinforcement learning. We find a capability-auditability tradeoff: the strongest direct policy achieves higher quality but exposes no reasoning to inspect, while the reasoning policy provides a trace at the cost of lower performance, particularly recall of true intervention opportunities. Supervised fine-tuning either suppresses reasoning or preserves it without improving decision quality, while reinforcement learning also fails to improve the reasoning policy. We identify one mechanism underlying this failure: group relative objectives provide no learning signal on confidently wrong prompts when sampled rollouts all select the same action. Controlled activation probes and behavioral ablations show that standard faithfulness methods can overstate evidence that exposed reasoning reflects the underlying decision process. Probability-based metrics saturate under confident decisions, probes are vulnerable to class imbalance and textual leakage, and reasoning ablations can confound reasoning content with changes in inference mode. Together, these results show that exposing reasoning can change an agent's action policy rather than simply make it observable. We provide controls for evaluating reasoning-based oversight of agents that can act or abstain.

Shreya Mendi, Brinnae Bent

Pratt School of Engineering, Duke University

Faithfulness of Language Model Explanations AI Agent Auditing Selective Prediction

Ask PDF

Apr 1, 2026

Therefore I am. I Think

We consider the question: when a large language reasoning model makes a choice, did it think first and then decide to, or decide first and then think? In this paper, we present evidence that detectable, early-encoded decisions shape chain-of-thought in reasoning models. Specifically, we show that a simple linear probe successfully decodes tool-calling decisions from pre-generation activations with very high confidence, and in some cases, even before a single reasoning token is produced. Activation steering supports this causally: perturbing the decision direction leads to inflated deliberation, and flips behavior in many examples (between 7 - 79% depending on model and benchmark). We also show through behavioral analysis that, when steering changes the decision, the chain-of-thought process often rationalizes the flip rather than resisting it. Together, these results suggest that reasoning models can encode action choices before they begin to deliberate in text.

Esakkivel Esakkiraja, Sai Rajeswar, Denis Akhiyarov +1

Khoury College of Computer Sciences Northeastern University · Mila, ServiceNow Research · ServiceNow

LLM Decision-Making Implicit Reasoning in Language Models LLM Interpretability

Ask PDF

Sep 2, 2026

MasterControl Seventeen Every Time

We study a governed approach to enterprise analytics: a language model interprets the question, while deterministic policy selects and runs a pre-approved analytical program that returns both results and evidence. We show that this restriction can remain expressive within a defined analytical class, using relational operations plus aggregation, comparison, windows, ranking, and similarity. Fixed meaning, policy, data, and execution rules also make results replayable. Across 440 runs, three 8B models generated SQL and selected tools at runtime, while Qwen3-8B interpreted intent only and policy executed the approved program. None of 330 runtime-planning episodes matched the full answer-and-evidence contract across all test datasets; the policy-executed analyzer matched 110 of 110. This is a configuration-specific result, not evidence that runtime agents cannot succeed under other designs.

MasterControl AI Lab

Enterprise Analytics LLM Agent Evaluation Text-to-SQL Runtime Enforcement for AI Agents AI Agent Governance LLM Agent Reliability+1

Ask PDF

Abstract

LLM Decision-Making

Explore similar work

Why2Speak: Faithful Reasoning for Abstaining Action Policies

Therefore I am. I Think

MasterControl Seventeen Every Time