Not All Information Is Worth Asking For: Decision-Informed Information Seeking with LLMs

Seunghan Lee; Jun Seo; Junhyeok Kang; Jaehoon Lee; Sangjun Han; Sungdong Yoo; Minjae Kim; Tae Yoon Lim

cs.AIOct 23, 2026

Not All Information Is Worth Asking For: Decision-Informed Information Seeking with LLMs

Authors: Seunghan Lee, Jun Seo, Junhyeok Kang, Jaehoon Lee, Sangjun Han, Sungdong Yoo, Minjae Kim, Tae Yoon Lim,

Organizations: LG AI Research

Abstract

Large language models (LLMs) are increasingly used to support decisions under incomplete information, where deciding what to look up and when to stop matters as much as the final answer. However, LLMs fall into an information trap: They pay for checks, i.e., costly pieces of evidence, that cannot change their decision or are not worth their cost, and they keep asking when the evidence is already sufficient. To this end, we propose Decision-Informed Value-of-information Acquisition (DIVA), a simple yet effective procedure for LLM information seeking based on the decision value of evidence. DIVA 1) lets the LLM specify a small decision model, i.e., how likely a bad outcome is and how each check would shift this probability, through short numeric questions and 2) values each candidate check by how much its imagined outcomes improve the decision net of its cost, stopping when no check is worth its cost. Its variant, DIVA-U, further accounts for the LLM's uncertain estimates of how strongly each check shifts this probability. DIVA is training-free and applies to any LLM. Across four domains and 20 LLMs, DIVA-U has lower regret, i.e., value lost relative to the best possible decision, than every LLM baseline on real-world data, mainly by stopping once evidence suffices, and DIVA cuts the regret of the best LLM baseline by 75% in synthetic scenarios where every check's reliability is given. Code: github.com/seunghan96/diva.

Figures & tables

	[1] Which information to acquire?		[2] When to stop?	Training-free
	Decision value	Elicited model	Stopping rule	Training-free
Information seeking with LLMs
FinRiskAtlas ( Zhong et al., 2026 )	–	–	Ask-or-proceed	✓
QuestBench ( Li et al., 2025a )	–	–	–	✓
MediQ ( Li et al., 2024 )	–	–	Abstention	✓
Information gain

Table 1: Comparison with the closest work. Unlike prior work, DIVA (1) values each check by its decision value under (2) an LLM-elicited model , i.e., a decision model whose probabilities are obtained by asking the LLM short numeric questions, (3) stops by a value-based rule , and (4) works without training with any LLM.

Figure 1: Desiderata of information seeking. A good decision maker acquires only checks that can change its decision net of their cost ( what ) and stops once no check is worth its cost ( when ). Prompted LLMs and information gain fail both, whereas DIVA meets both.

Figure 2: LLMs ask for information, not value. Rank correlation between 1) how likely each LLM is to pick a check and 2) the check’s exact information gain or net decision value . Each dot is one LLM.

Figure 3: Overall framework of DIVA. In an earnings case, information gain asks for the gross margin, whose outcomes cannot move the belief

\hat{b}

across the decision threshold

b^{\dagger}

, so the cost is paid without changing the decision. In contrast, DIVA imagines every outcome of every check and values each by how much it can change the decision net of its cost. It asks for the forward guidance, whose cut would flip buy to pass , and then stops.

Action	$Y{=}0$ (Benign)	$Y{=}1$ (Adverse)
act	$+g$	$-l$
pass	$0$	$0$

Table 2: Payoff table. Utility

U(a,Y)

of each action under each state. Acting is optimal when the belief is below

b^{\dagger}=g/(g+l)

. Table 10 lists all notation.

Figure 4: LLMs do not imagine evidence coherently. (a) How much each LLM says the belief would change after an imagined check outcome vs. the exact Bayesian change, binned and pooled over 20 LLMs. (b) How often the top check by imagined decision value is the best one when acquiring is optimal.

Method (Avg. across 20 LLMs)	Knowledge-required environments								Controlled environments
	Credit		Clinical		SEC filings (2025–2026)				Finance	Credit	Clinical	Legal
	Taiwan	German	CDC	Heart	Ret-1d	Ret-5d	Vol-5d	Vol-20d	Finance	Credit	Clinical	Legal
Prompting
Direct	6.47	5.15	5.23	6.30	1.26	1.21	1.24	1.08	4.69	4.35	6.22	5.80
CoT ( Wei et al., 2022 )	7.50	6.53	6.56	6.87	1.19	1.25	1.28	1.10	4.71	4.20	5.78	5.96
SC ( Wang et al., 2023 )	7.52	6.67	6.82	6.81	1.17	1.26	1.27	1.03	4.42	4.17	5.84	5.64

Table 3: DP regret of the first decision, averaged over 20 LLMs. DIVA-U equals DIVA in the controlled environments. Wins: DIVA-U beats every LLM baseline. Ret / Vol-

k

k

-day return / volatility.

Figure 5: Effectiveness across 20 LLMs. Regret of the best LLM baseline of each LLM (orange) and DIVA (blue, DIVA-U on knowledge-required environments).

Prompting
Method (Avg. across 20 LLMs)	Knowledge- required	Controlled
Direct	18.98	28.46
CoT ( Wei et al., 2022 )	20.64	28.00
SC ( Wang et al., 2023 )	22.87	29.84
ReAct ( Yao et al., 2023 )	19.99	23.98
Decision-aware prompting

Table 4: Episode regret of full acquire-then-decide episodes, averaged over environments and 20 LLMs.

Variant	Regret ( $\times$ 100)	$\Delta$
Direct	5.52	–
+ Decision value on imagined beliefs	3.49	$-$ 2.03
+ Elicited decision model (11-option)	1.68	$-$ 1.81
+ Numeric answers (= DIVA)	0.83	$-$ 0.85
+ DP lookahead (= DIVA-DP)	0.48	$-$ 0.35
DIVA with cost-blind EIG	5.03	+4.20

Table 5: Ablation study. DP regret on 496 controlled scenarios, averaged over the same 20 LLMs.

Figure 6: Anatomy of the information trap. Regret of the first decision by error type on (a) knowledge-required and (b) controlled environments. DIVA-U equals DIVA in (b), where all reliabilities are stated.

Method	DP regret		Episode regret
Method	Know.	Ctrl.	Know.	Ctrl.
VoI-Ask ( Dong et al., 2026 )	1.99	4.06	2.40	11.16
DIVA	1.29		2.09
DIVA-U	0.65	0.86	2.77	1.80
DIVA-DP	1.07	0.54	1.86	1.58

Table 6: Myopic and planned acquisition. Regret averaged over the knowledge-required (Know.) or controlled (Ctrl.) environments and 20 LLMs.

Method	Check strengths	Knowledge-required		Controlled
Method	Check strengths	Records	SEC	Controlled
Best LLM baseline		1.49	1.10	4.04
Never		1.31	3.12	4.67
DIVA	One shared	0.72	1.23	2.07
DIVA	From LLM	0.77	1.80	0.86
DIVA-U	One shared	0.52	0.63	1.50

Table 7: What the LLM’s knowledge contributes. DP regret of the first decision, averaged over 20 LLMs. Best LLM baseline: Best of Table 3 per environment.

Domain	Environment	Source	Decision	Checks	Test
[1] Knowledge-required environments
Credit	Taiwan	Credit-card default ( Yeh and Lien, 2009 )	Approve / decline	6 (Binary)	2,000
Credit	German	German credit ( Hofmann, 1994 )	Approve / decline	6 (Binary)	1,000
Clinical	CDC	CDC diabetes indicators ( Centers for Disease Control and Prevention, 2017 )	Skip / refer	6 (Binary)	2,000
Clinical	Heart	UCI heart disease ( Janosi et al., 1989 )	Skip / refer	6 (Binary)	1,000
SEC filings	Ret-1d, Ret-5d	S&P 500 SEC filings	Buy / pass	$\leq$ 6 (Terciles)	3,437 / 3,433

Table 8: Environments , with SEC filings from 2025–2026 and the controlled domains of equity investment, loan approval, chest-pain triage, and litigation. Checks: Candidate checks per scenario. Test: Number of test scenarios.

Appendix figures & tables38 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Knowledge-required		Controlled
Parameter	Records (Taiwan, German, CDC, Heart)	SEC filings (Return and volatility targets)
Checks	6, one uninformative in half of the scenarios	Up to 5 from the filing and 1 uninformative	6 (10 or 12 in many-item scenarios), 1 uninformative
Known findings	0, 1, or 2, with probabilities 0.4, 0.4, and 0.2
Payoffs $(g,l)$	$\{1,2,3\}\times\{2,3,5,8\}$	$\{1,2,3\}\times\{1,2,3,5,8\}$ , with $g/(g{+}l)$ within 0.08 of the reference belief	$\{1,2,3\}\times\{2,3,5,8\}$
Cost regime $\rho$	$\{0,0.05,0.1,0.2,0.4\}$	$\{0,0.005,0.01,0.02,0.04\}$	$\{0,0.05,0.1,0.2,0.4\}$
Cost factor	$\{0.5,1,1,1.5,2\}$	$\{0.5,1,2\}$	$\{0.5,1,1,1.5,2\}$

Appendix

Table 9: Scenario construction. Each check costs

\rho\min(g,l)

times a listed cost factor. Known findings: Outcomes observed before the first decision. VOI

>0

and DP acquires: Shares of the first 300 test scenarios where some check has positive VOI and where acquiring first is DP-optimal, as ranges over the environments.

Term	Meaning
Episode	One decision case
State ( $Y$ )	The outcome of interest, unknown when deciding
Action, payoffs ( $a$ , $g$ , $l$ )	The two actions, the gain of acting if benign, and the loss of acting if adverse
Check, outcome, cost ( $e$ , $v$ , $c_{e}$ )	A piece of evidence that can be acquired, what it shows, and its price
Known findings ( $\mathcal{K}$ )	Outcomes already observed before deciding
Base rate ( $\hat{\pi}$ )	Probability of the adverse case before any finding

Appendix

Table 10: Notation and terms.

Family	Size	Hugging Face checkpoint	Precision	License
Qwen3	0.6B	Qwen/Qwen3-0.6B	BF16	Apache-2.0
	1.7B	Qwen/Qwen3-1.7B	BF16	Apache-2.0
	4B	Qwen/Qwen3-4B	BF16	Apache-2.0
	8B	Qwen/Qwen3-8B	FP8	Apache-2.0
	14B	Qwen/Qwen3-14B	BF16	Apache-2.0
	32B	Qwen/Qwen3-32B-AWQ	AWQ 4-bit	Apache-2.0

Appendix

Table 11: Model checkpoints served with vLLM, with their precision and license. Later tables write Gemma for Gemma-2/3, Llama for Llama-3.1/3.3, Mistral 24B for Mistral-Small, and DeepSeek-R1 for R1-Distill-Qwen.

Category	Method	How the first action is chosen
Prompting	Direct	The LLM answers with one option letter (A check or stop) without rationale, and the letter of highest probability, averaged over option orders, is taken.
	CoT ( Wei et al., 2022 )	The LLM reasons step by step about what it still needs, given the costs, and ends with “ANSWER: X” (Up to 1,024 tokens).
	SC ( Wang et al., 2023 )	Majority vote over five CoT samples (One request, temperature 0.7).
	ReAct ( Yao et al., 2023 )	The LLM writes one Thought and one Action, either check[X] or a final decision (Up to 512 tokens per step).
Decision-aware prompting	Ask-P ( Zhong et al., 2026 )	The LLM chooses between proceeding and asking for one more costly item, and if it asks with probability $\geq 0.5$ , it acquires the item it names (Averaged over three orders), else it stops.
	VoI-P	The LLM states each check’s net value in JSON after brief reasoning, and the largest positive one is acquired.

Appendix

Table 12: Baseline methods. How each baseline of Table 3 chooses its first action. ∗ : Only in Appendix F.1 .

Figure 7: Error types of a first decision. A first decision that differs from the DP-optimal one is classified by sequential tests: Each test either assigns an error type (Down) or passes to the next test (Right).

Env.	Never	Always ask	DIVA- $\kappa^{\ddagger}$	DIVA- $\beta^{\ddagger}$	DIVA	DIVA-U	Stop opt.	Stop DIVA	Stop DIVA-U	Stop VoI-Ask
Taiwan	1.02	4.65	0.43	0.40	0.52	0.41	0.65	0.69	0.56	0.93
German	0.91	4.93	0.49	0.51	0.64	0.51	0.69	0.75	0.57	0.98
CDC	0.36	4.96	0.22	0.20	0.26	0.20	0.76	0.84	0.70	0.75
Heart	2.81	4.28	1.70	1.07	1.64	1.08	0.61	0.85	0.72	0.86
Ret-1d	2.69	0.68	0.92	0.61	0.95	0.64	0.34	0.32	0.01	0.65
Ret-5d	1.14	0.85	0.74	0.64	0.80	0.77	0.52	0.39	0.02	0.71

Appendix

Table 13: Stopping behavior on knowledge-required environments. DP regret and first-decision stop rates (All test scenarios, 20 LLMs), with DIVA-

\beta^{\ddagger}

and DIVA-

\kappa^{\ddagger}

. VoI-Ask: First 300 test scenarios.

Figure 8: Acquisition behaviors on controlled environments. Gray: LLMs, orange: BED-LLM, blue: DIVA.

Method (Avg. across 20 LLMs)	Knowledge-required environments								Controlled environments
	Credit		Clinical		SEC filings (2025–2026)				Finance	Credit	Clinical	Legal
	Taiwan	German	CDC	Heart	Ret-1d	Ret-5d	Vol-5d	Vol-20d	Finance	Credit	Clinical	Legal
Prompting
Direct	31.24	31.23	30.69	34.40	6.73	5.47	6.00	6.10	28.60	23.92	30.04	31.29
CoT ( Wei et al., 2022 )	35.27	34.02	33.06	34.61	7.65	6.28	7.11	7.14	27.55	24.63	28.44	31.39
SC ( Wang et al., 2023 )	39.55	38.40	37.29	37.44	8.22	6.95	7.67	7.45	29.11	25.75	31.29	33.20

Appendix

Table 14: Episode regret on test scenarios 0–149 (First 100 per controlled domain), averaged over 20 LLMs. Final action: From the reference model’s belief (Knowledge-required) or the policy’s own belief (Controlled). –: Not run.

Reference
Method	Records				SEC filings
Method	Taiwan	German	CDC	Heart	Ret-1d	Ret-5d	Vol-5d	Vol-20d
Never	0.89	1.06	0.31	2.96	2.64	1.76	3.71	4.38
Generic decision model (One strength for every check)
Generic DIVA	0.65	0.51	0.19	1.52	1.12	0.82	1.39	1.57
Generic DIVA-U	0.44	0.45	0.12	1.06	0.61	0.65	0.65	0.60

Appendix

Table 15: Generic decision model. DP regret of DIVA and DIVA-U with one strength for every check, selected on development data, vs. the elicited decision model, first 300 test scenarios, averaged over 20 LLMs.

Reference
Method	Taiwan	German	CDC	Heart
Never	1.02	0.91	0.36	2.81
Threshold heur.	1.02	0.91	0.38	3.07
Supervised ref.	0.05	0.11	0.04	0.29
Shuffled likelihoods
Knowledge-free DIVA	0.64	0.61	0.28	1.71

Appendix

Table 16: Controls on real records. DP regret on all test scenarios, averaged over 20 LLMs. Knowledge-free: Elicited likelihoods shuffled across checks.

Variant (Avg. across 20 LLMs)	Credit		Clinical		SEC filings (2025–2026)				Avg.
Variant (Avg. across 20 LLMs)	Taiwan	German	CDC	Heart	Ret-1d	Ret-5d	Vol-5d	Vol-20d	Avg.
DIVA w/ EIG (Fixed threshold)	4.88	4.01	2.04	6.07	2.20	1.09	3.38	3.81	3.43
DIVA w/ EIG ‡ (Tuned on labels)	0.38	0.23	0.12	1.43	0.73	0.46	1.43	1.49	0.78
DIVA	0.52	0.64	0.26	1.64	0.95	0.80	2.26	2.56	1.20
DIVA-U	0.41	0.51	0.20	1.08	0.64	0.77	0.78	0.73	0.64

Appendix

Table 17: Decision value vs. EIG on the same decision model. DP regret on the full test sets, averaged over 20 LLMs. ‡ : Cost trade-off and threshold tuned on development scenarios with ground-truth decision values.

Held-out source	Selected ( $\lambda$ , $\sigma$ )	DIVA (Point)	DIVA-U ( $\lambda{=}0.75$ )	DIVA-U (Selected)
Taiwan	(0.5, 1.0)	0.53	0.44	0.32
German	(0.5, 1.0)	0.64	0.52	0.34
CDC	(0.5, 1.0)	0.81	0.70	0.40
Heart	(0.5, 1.0)	0.56	0.36	0.36
SEC filings	(0.5, 1.0)	0.53	0.26	0.24

Appendix

Table 18: Leave-one-source-out transfer. Regret relative to Never on each held-out data source.

Family	Size	Prompting	Reference		Ablation					Ours	Verbal beliefs
Family	Size	Direct	Never	Always ask	DIVA w/ EIG	DIVA w/ EIG ‡	DIVA w/ ES	DIVA w/ ES ‡	DIVA (Elic.)	DIVA	Direct	DIVA
Qwen3	0.6B	5.85	5.83	1.53	3.03	2.89	3.07	2.93	5.30	2.83	6.05	3.63
	1.7B	5.77	5.83	1.37	3.00	2.89	2.59	2.69	4.74	1.39	5.97	2.35
	4B	3.36	5.83	1.14	2.89	2.28	2.04	1.86	3.27	0.22	3.76	1.11
	8B	3.10	5.83	1.11	2.88	2.35	2.03	1.85	3.68	0.14	3.08	1.28
	14B	2.86	5.83	1.10	2.88	2.37	2.02	1.85	4.43	0.13	2.58	0.90

Appendix

Table 19: Multi-state environments. Regret on held-out scenarios (Scenarios 500–999 per domain), normalized by half the payoff range. Verbal beliefs (Scenarios 500–799): Best and second best marked within the group.

Records	LLM policies ( $\times$ 100, normalized by $l$ )					Reference DP policy (Payoff points)
Records	Never	DIVA	B / W	DIVA-U	B / W	Never	Ref. DP	Difference ( $\pm$ SE)
Taiwan	24.39	23.91	0 / 4	23.78	0 / 4	0.710	0.735	0.025 $\pm$ 0.021
German	16.56	16.08	0 / 2	15.44	0 / 5	0.394	0.428	0.034 $\pm$ 0.031
CDC	35.48	35.55	2 / 1	35.61	0 / 1	1.112	1.141	0.029 $\pm$ 0.015
Heart	1.35	0.64	1 / 6	0.34	0 / 5	0.124	0.222	0.098 $\pm$ 0.031
Taiwan (5,000)	25.18	24.80	0 / 6	24.56	0 / 8	25.94 †	27.04 †	1.10 $\pm$ 0.25 †

Appendix

Table 20: Realized utility on real records , net of costs (Higher is better), averaged over 20 LLMs. B / W: LLMs significantly better / worse than never acquiring. Ref. DP: The DP policy of the reference model. Its Never row decides at once with the reference model’s belief. † : Multiplied by 100 and normalized by

l

Prompting
Method (Avg. across 20 LLMs)	Credit		Clinical		SEC filings (2025–2026)				Avg.
Method (Avg. across 20 LLMs)	Taiwan	German	CDC	Heart	Ret-1d	Ret-5d	Vol-5d	Vol-20d	Avg.
Direct	6.68	5.43	5.26	7.25	2.12	1.79	1.84	1.73	4.01
CoT ( Wei et al., 2022 )	7.72	6.98	6.62	7.98	1.94	1.76	2.08	2.12	4.65
SC ( Wang et al., 2023 )	7.73	7.15	6.89	7.91	1.79	1.72	2.01	1.85	4.63
ReAct ( Yao et al., 2023 )	9.13	7.76	8.12	8.48	2.48	2.22	2.28	2.45	5.37

Appendix

Table 21: Myopic regret on knowledge-required environments. Regret of the first decision against the myopic optimum, on the same scenarios and LLMs as Table 3 . Avg.: Mean over the eight environments.

Table 30

Selection	Knowledge-required			Controlled
Selection	LLM	DIVA-U	DP	LLM	DIVA-U	DP
LLM	3.49	0.91	0.38	5.26	1.50	1.04
EIG	3.97	1.08	0.53	4.90	1.14	0.67
DIVA-U	2.23	0.65	0.25	2.90	0.86	0.28
DP optimum	1.92	0.38	0.00	2.55	0.64	0.00

Appendix

Table 25: Which check vs. when to stop. DP regret when stopping by the column’s rule, else acquiring the row’s top check. LLM: Direct policy. DP: DP optimum. DIVA-U equals DIVA on the controlled environments.

Method (Avg. across 20 LLMs)	Knowledge-required environments								Controlled environments
	Credit		Clinical		SEC filings (2025–2026)				Finance	Credit	Clinical	Legal
	Taiwan	German	CDC	Heart	Ret-1d	Ret-5d	Vol-5d	Vol-20d	Finance	Credit	Clinical	Legal
Prompting
Direct	6.47	5.15	5.23	6.30	1.26	1.21	1.24	1.08	4.69	4.35	6.22	5.80
Direct + tuned stop	3.43	2.68	2.50	4.30	1.81	1.35	2.33	2.59	4.39	4.39	5.49	5.57
Salience	–	–	–	–	–	–	–	–	5.48	4.51	6.87	6.71

Appendix

Table 26: Additional baselines. DP regret on the first 300 test scenarios, averaged over 20 LLMs. –: Not run. DeLLMa and UoT are adapted to acquire-or-stop. Tuned stop: Stop threshold of each LLM, cross-fitted over scenario groups. Cost-aware EIG: EIG minus

\eta c/l

on the decision model, with

\eta=2

from development data.

Decision model	Knowledge-required environments								Controlled environments
	Credit		Clinical		SEC filings (2025–2026)				Finance	Credit	Clinical	Legal
	Taiwan	German	CDC	Heart	Ret-1d	Ret-5d	Vol-5d	Vol-20d	Finance	Credit	Clinical	Legal
DIVA	0.48	0.65	0.26	1.71	1.01	0.93	2.45	2.80
DIVA-U	0.41	0.52	0.22	1.12	0.67	0.71	0.83	0.74	0.86	0.97	0.73	0.87
DIVA, $\pm$ 25%	0.52	0.70	0.31	1.78	1.23	1.06	2.55	3.02
DIVA-U, $\pm$ 25%	0.42	0.56	0.24	1.24	0.68	0.71	0.85	0.75	1.00	0.99	0.86	0.97

Appendix

Table 27: Misspecified payoffs and costs. DP regret when DIVA decides with

g

l

, and every cost multiplied by independent factors drawn uniformly from

[1-p,1+p]

and is scored with the true values, averaged over 20 LLMs.

Policy		Credit		Clinical		SEC filings (2025–2026)				Avg.
Policy		Taiwan	German	CDC	Heart	Ret-1d	Ret-5d	Vol-5d	Vol-20d	Avg.
	DIVA	0.48	0.65	0.26	1.71	1.01	0.93	2.45	2.80	1.29
	DIVA-U	0.41	0.52	0.22	1.12	0.67	0.71	0.83	0.74	0.65
First decision	DIVA-PV	0.46	0.64	0.26	1.71	1.02	0.94	2.57	2.99	1.32
	Never	0.84	1.25	0.36	3.38	2.53	1.36	4.12	4.17	2.25
	DIVA	0.69	1.26	0.35	2.66	1.65	1.24	4.47	4.39	2.09

Appendix

Table 28: Coherent predictive VOI. DP regret of the first decision (Scenarios of Table 3 ) and over full episodes (Test scenarios 0–149, final action from the reference model’s belief), averaged over 20 LLMs.

Env.	Direct	Never	Always ask	DIVA- $\kappa^{\ddagger}$	DIVA- $\beta^{\ddagger}$	DIVA	DIVA-U	Stop DIVA-U	Uninformative (%, DIVA-U / Direct)
Ret-1d	1.19 / 2.00	2.61 / 2.54	0.69 / 1.73	0.91 / 1.53	0.60 / 1.60	0.93 / 1.56	0.64 / 1.66	0.01	4.9 / 7.2
Ret-5d	1.17 / 1.47	0.99 / 0.90	0.87 / 1.32	0.72 / 0.91	0.66 / 1.04	0.79 / 0.99	0.79 / 1.20	0.02	4.7 / 7.2
Vol-5d	1.08 / 1.61	3.35 / 3.23	0.81 / 3.57	2.18 / 3.56	0.81 / 3.57	2.20 / 3.49	0.79 / 3.51	0.02	7.6 / 4.8
Vol-20d	1.05 / 1.61	3.78 / 3.69	0.77 / 3.95	2.43 / 4.02	0.76 / 3.92	2.48 / 3.88	0.72 / 3.72	0.03	9.9 / 3.9

Appendix

Table 29: Held-out SEC filings. DP / myopic regret beyond the first 1,000 scenarios per target, over 20 LLMs.

Policy	As built				Uniform				Far from belief
Policy	Ret-1d	Ret-5d	Vol-5d	Vol-20d	Ret-1d	Ret-5d	Vol-5d	Vol-20d	Ret-1d	Ret-5d	Vol-5d	Vol-20d
Never	2.64	1.76	3.71	4.38	0.56	0.46	1.07	1.64	0.10	0.01	0.38	0.52
Always ask	0.72	0.77	0.90	0.77	0.53	0.54	0.53	0.56	0.46	0.44	0.46	0.45
Generic DIVA-U	0.61	0.65	0.65	0.60	0.43	0.46	0.39	0.46	0.39	0.37	0.35	0.36
DIVA	1.01	0.93	2.45	2.80	0.30	0.29	0.78	1.13	0.14	0.08	0.37	0.52
DIVA-DP	0.86	0.77	2.02	2.37	0.29	0.26	0.69	0.96	0.15	0.11	0.32	0.46

Appendix

Table 30: SEC payoffs without the threshold filter. DP regret, averaged over 20 LLMs, with

(g,l)

as built (Threshold within 0.08 of the reference belief), drawn uniformly, or drawn only far from the reference belief, re-scoring stored decision models. LLM baselines saw the original payoffs and cannot be re-scored.

Env.	$\Delta$ to never acquiring			DIVA-U wins
Env.	DIVA	DIVA-U	DIVA-DP †	All	Sig.
Taiwan	$-$ 0.41 ∗	$-$ 0.48 ∗	$-$ 0.44 ∗	20/20	16/20
German	$-$ 0.41 ∗	$-$ 0.55 ∗	$-$ 0.07	20/20	17/20
CDC	$-$ 0.05	$-$ 0.09	$-$ 0.06	18/20	18/20
Heart	$-$ 1.26 ∗	$-$ 1.84 ∗	$-$ 1.01 ∗	20/20	20/20
Ret-1d	$-$ 1.64 ∗	$-$ 1.97 ∗	$-$ 1.21 ∗	16/20	15/20

Appendix

Table 31: Statistical significance. Paired DP-regret differences (

\Delta

), and the LLMs where DIVA-U beats every LLM baseline (All) and significantly (Sig.). ∗ : CI excludes zero. † : Pre-registered episodes (18 LLMs).

Method	Original	Para. 1	Para. 2	Spread
Direct	5.26	5.31	5.18	0.39
CoT ( Wei et al., 2022 )	5.16	5.21	5.14	0.41
VoI-P	5.00	4.97	4.99	0.56
PAL ( Gao et al., 2023 )	4.39	4.29	4.62	0.44
DIVA	0.86	0.73	0.93	0.23
DIVA-U (Records)	0.57	0.58	–	0.06

Appendix

Table 32: Prompt paraphrases. DP regret averaged over 20 LLMs on the tabular records (DIVA-U only) and the controlled environments. Spread: Range over the prompts per LLM, averaged over LLMs.

Family	Size	Direct	CoT	SC	ReAct	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-U	DIVA-DP
Qwen3	0.6B	0.89	1.29	0.94	10.38	1.42	0.91	0.70	0.89	1.80	5.56	0.90	0.67	0.82
	1.7B	4.97	6.91	6.40	10.34	4.34	4.36	0.77	10.07	6.91	5.51	0.67	0.57	0.60
	4B	10.22	10.46	10.37	10.50	10.77	11.28	0.70	4.45	2.73	5.15	0.34	0.33	0.32
	8B	6.56	8.79	9.75	8.90	3.18	9.16	0.71	5.10	2.86	5.23	0.48	0.43	0.37
	14B	6.33	7.68	7.28	8.71	9.55	6.54	0.77	8.30	2.24	5.00	0.31	0.32	0.26
	32B	3.63	6.01	5.84	5.89	8.42	3.57	0.80	7.27	2.11	5.34	0.67	0.68	0.57

Appendix

Table 33: DP regret per LLM: Taiwan. Same first 300 test scenarios as Table 3 . Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-U	DIVA-DP
Qwen3	0.6B	1.06	1.13	1.19	7.95	1.06	1.11	1.05	1.45	1.39	4.77	1.09	0.99	0.95
	1.7B	2.46	5.29	5.29	8.03	2.18	4.03	2.02	8.32	4.06	4.41	0.61	0.49	0.44
	4B	8.92	8.93	9.67	8.16	10.05	10.77	1.06	2.47	3.42	4.69	0.85	0.87	0.79
	8B	3.76	8.25	8.45	8.51	2.41	10.06	1.06	5.38	2.34	4.44	0.68	0.46	0.51
	14B	4.74	6.97	6.94	7.73	9.83	4.56	1.06	7.95	1.70	4.47	0.54	0.42	0.37
	32B	2.83	5.12	4.86	5.13	8.08	4.61	1.06	6.30	1.29	4.61	0.78	0.70	0.69

Appendix

Table 34: DP regret per LLM: German. Same first 300 test scenarios as Table 3 . Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-U	DIVA-DP
Qwen3	0.6B	0.31	1.31	1.00	9.26	0.31	1.00	1.87	0.31	1.49	5.38	1.41	1.64	1.68
	1.7B	2.88	5.80	6.23	8.95	0.46	4.81	0.45	7.76	2.12	4.82	0.24	0.22	0.25
	4B	9.19	8.58	9.20	8.70	9.16	9.91	0.31	0.31	2.78	4.75	0.16	0.19	0.16
	8B	4.84	8.75	8.48	9.03	4.00	6.50	0.72	0.31	1.43	4.81	0.24	0.14	0.21
	14B	6.17	7.58	7.59	7.96	8.21	3.81	1.28	0.38	1.05	4.73	0.11	0.08	0.07
	32B	1.91	6.73	6.88	6.26	8.31	4.96	1.36	0.31	0.84	4.68	0.11	0.08	0.06

Appendix

Table 35: DP regret per LLM: CDC. Same first 300 test scenarios as Table 3 . Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-U	DIVA-DP
Qwen3	0.6B	2.96	4.13	3.55	8.33	2.96	3.28	2.97	2.98	4.21	4.45	2.21	1.45	1.94
	1.7B	4.01	8.15	8.67	8.33	7.02	5.74	2.96	7.63	5.38	3.96	2.22	1.14	2.03
	4B	8.66	8.46	8.20	7.65	10.24	9.26	2.98	5.34	3.71	4.05	1.59	1.06	1.27
	8B	7.64	8.02	7.76	8.35	5.95	8.90	2.76	7.20	3.20	4.50	1.80	1.32	1.52
	14B	6.87	7.68	7.57	8.11	9.14	7.08	3.13	7.37	4.54	4.06	1.72	1.05	1.35
	32B	4.00	5.54	4.78	5.90	7.25	5.33	2.75	7.09	3.34	3.98	1.54	1.04	1.36

Appendix

Table 36: DP regret per LLM: Heart. Same first 300 test scenarios as Table 3 . Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-U	DIVA-DP
Qwen3	0.6B	2.64	2.37	2.42	1.53	2.63	2.54	1.96	0.84	2.52	0.71	2.42	0.74	2.35
	1.7B	2.22	1.76	1.76	1.55	2.64	2.25	2.29	1.36	2.38	1.22	1.61	1.11	1.41
	4B	0.71	1.03	1.07	1.24	1.15	2.45	2.64	1.47	2.16	0.98	1.04	0.91	0.99
	8B	0.82	1.08	0.93	1.07	1.73	2.51	1.41	1.58	2.27	0.67	0.70	0.61	0.55
	14B	0.81	0.83	0.75	0.81	0.82	2.02	1.94	1.19	1.73	0.59	0.68	0.52	0.55
	32B	1.32	0.61	0.59	0.84	0.91	0.82	1.94	1.22	1.53	0.66	0.59	0.60	0.58

Appendix

Table 37: DP regret per LLM: Ret-1d. Same first 300 test scenarios as Table 3 . Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-U	DIVA-DP
Qwen3	0.6B	1.76	1.70	1.70	1.60	1.69	1.74	1.56	0.76	1.65	0.73	1.67	0.68	1.57
	1.7B	1.72	1.65	1.69	1.72	1.81	1.89	1.24	1.53	1.72	1.03	1.51	1.00	1.38
	4B	0.82	1.33	1.22	1.49	1.29	1.60	1.70	1.63	1.69	0.99	1.07	0.91	0.86
	8B	0.92	1.16	1.14	1.22	1.47	1.93	1.07	1.49	1.72	0.77	0.72	0.71	0.61
	14B	0.82	1.04	0.96	1.11	0.98	1.79	1.52	1.29	1.57	0.65	0.67	0.63	0.58
	32B	1.05	0.82	0.79	0.94	1.06	1.05	1.33	1.31	1.34	0.73	0.61	0.69	0.57

Appendix

Table 38: DP regret per LLM: Ret-5d. Same first 300 test scenarios as Table 3 . Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-U	DIVA-DP
Qwen3	0.6B	3.71	2.83	3.32	1.50	3.59	3.71	1.79	3.71	3.40	0.75	3.49	0.82	3.44
	1.7B	1.11	1.21	1.06	1.06	3.64	2.69	3.42	1.65	3.41	1.00	2.76	1.02	2.67
	4B	1.29	1.17	1.11	1.06	1.22	1.76	3.31	1.60	2.89	0.86	2.41	0.79	2.03
	8B	0.93	1.08	0.99	1.01	1.73	2.40	2.64	1.69	3.09	0.78	3.01	0.76	2.54
	14B	1.54	0.94	1.02	0.94	1.28	3.76	3.44	1.65	2.60	0.90	2.44	0.84	1.88
	32B	2.05	0.91	0.98	0.90	1.10	1.17	3.26	1.64	2.47	0.98	2.08	0.92	1.65

Appendix

Table 39: DP regret per LLM: Vol-5d. Same first 300 test scenarios as Table 3 . Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-U	DIVA-DP
Qwen3	0.6B	4.38	3.49	3.84	1.17	4.30	4.38	2.41	4.38	4.16	0.80	4.06	0.81	4.05
	1.7B	0.99	1.05	0.75	0.86	3.84	2.68	4.25	1.55	3.75	1.03	3.23	0.98	3.12
	4B	0.85	0.78	0.74	0.79	1.01	1.91	4.13	1.56	3.02	0.76	2.62	0.70	2.31
	8B	0.62	0.80	0.69	0.68	1.63	3.00	3.12	1.52	3.88	0.68	3.27	0.63	2.55
	14B	1.34	0.67	0.65	0.79	0.80	3.98	4.24	1.42	3.15	0.69	2.89	0.82	2.30
	32B	1.92	0.70	0.73	0.94	0.86	0.95	4.24	1.55	2.56	0.85	2.29	0.79	1.92

Appendix

Table 40: DP regret per LLM: Vol-20d. Same first 300 test scenarios as Table 3 . Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Reason.	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-DP
Qwen3	0.6B	4.67	5.77	5.55	6.42	5.98	4.40	4.74	4.33	4.67	4.80	3.15	2.77	2.51
	1.7B	5.99	6.36	6.06	6.58	5.86	7.08	4.39	4.39	5.25	5.03	2.92	1.46	0.97
	4B	6.00	5.52	5.50	5.64	3.65	6.04	6.22	4.24	5.39	4.49	2.58	0.86	0.52
	8B	5.76	5.37	5.29	5.72	3.02	5.64	6.28	3.90	5.45	4.72	2.45	0.66	0.33
	14B	5.57	5.35	4.98	5.16	2.24	5.53	5.28	4.11	6.57	4.08	2.44	0.57	0.29
	32B	4.48	3.97	4.08	3.74	1.04	5.06	3.69	4.08	6.40	4.35	2.47	0.60	0.27

Appendix

Table 41: DP regret per LLM on controlled environments, averaged over the four domains, first 300 test scenarios per domain. Reason.: Native reasoning (Reasoning models only). Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Reason.	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-DP
Qwen3	0.6B	4.63	5.99	5.14	6.71	6.02	4.63	4.80	4.60	4.63	5.00	3.08	3.36	3.29
	1.7B	5.36	5.81	5.41	6.73	5.25	7.49	4.27	4.60	5.23	5.49	2.92	1.53	1.08
	4B	5.59	5.12	5.27	5.54	3.03	6.02	5.92	4.54	5.50	4.76	2.41	0.85	0.54
	8B	5.22	4.47	4.90	5.33	2.68	4.98	5.47	4.60	5.21	4.13	2.30	0.80	0.51
	14B	4.50	5.25	4.89	4.72	2.45	4.45	4.88	4.50	5.92	4.38	2.18	0.61	0.41
	32B	3.57	3.82	3.63	3.58	0.84	4.26	4.06	4.47	5.99	4.10	2.21	0.41	0.24

Appendix

Table 42: DP regret per LLM: Finance. Controlled, first 300 test scenarios. Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Reason.	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-DP
Qwen3	0.6B	4.54	4.84	4.93	5.21	5.43	3.48	4.39	3.58	4.54	4.46	2.48	2.92	2.65
	1.7B	4.93	5.24	5.92	5.04	4.97	5.95	4.50	4.24	4.64	4.53	2.33	1.80	1.24
	4B	4.95	4.29	4.22	4.58	3.43	4.90	4.89	4.24	4.10	3.82	1.88	0.90	0.57
	8B	4.64	4.56	3.98	4.47	2.66	4.62	4.66	3.88	4.81	3.84	1.76	0.69	0.31
	14B	4.33	3.92	4.10	4.23	2.13	4.38	4.47	3.97	5.45	3.71	1.88	0.75	0.32
	32B	3.52	3.72	3.49	2.66	0.96	3.98	3.12	4.35	5.21	4.35	1.95	0.91	0.35

Appendix

Table 43: DP regret per LLM: Credit. Controlled, first 300 test scenarios. Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Reason.	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-DP
Qwen3	0.6B	4.52	7.01	6.63	6.51	6.00	4.52	4.77	4.27	4.52	4.72	3.48	1.74	1.25
	1.7B	6.84	6.84	6.89	7.10	6.39	9.01	3.80	4.13	6.56	4.82	3.38	1.13	0.71
	4B	7.19	5.73	6.02	6.33	2.77	7.22	7.11	4.15	6.95	4.71	3.21	0.94	0.53
	8B	6.54	5.75	5.70	6.30	2.92	5.22	7.28	2.65	6.63	4.82	2.90	0.58	0.20
	14B	6.93	6.22	5.48	6.11	1.83	6.87	5.46	3.25	8.03	3.75	2.98	0.45	0.22
	32B	6.07	4.27	5.07	4.16	0.78	6.01	3.74	2.89	7.45	3.57	3.02	0.57	0.26

Appendix

Table 44: DP regret per LLM: Clinical. Controlled, first 300 test scenarios. Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Reason.	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Always ask	DIVA	DIVA-DP
Qwen3	0.6B	4.98	5.22	5.51	7.25	6.48	4.98	5.00	4.87	4.98	5.01	3.56	3.05	2.86
	1.7B	6.83	7.55	6.03	7.45	6.82	5.86	4.99	4.58	4.59	5.26	3.04	1.38	0.84
	4B	6.28	6.91	6.50	6.11	5.35	6.04	6.94	4.03	5.03	4.69	2.82	0.75	0.43
	8B	6.65	6.68	6.58	6.80	3.83	7.72	7.73	4.46	5.15	6.10	2.83	0.58	0.30
	14B	6.52	6.03	5.46	5.57	2.54	6.43	6.31	4.71	6.88	4.47	2.72	0.47	0.22
	32B	4.76	4.09	4.15	4.54	1.59	5.97	3.85	4.63	6.96	5.38	2.70	0.49	0.25

Appendix

Table 45: DP regret per LLM: Legal. Controlled, first 300 test scenarios. Baselines are cited in Table 3 .

Family	Size	Direct	CoT	SC	ReAct	Ask-P	VoI-P	VoI-Ask	BED-LLM	PAL	Never	Always ask	DIVA	DIVA-DP
Qwen3	0.6B	15.43	18.20	16.94	38.96	15.52	16.65	15.43	15.43	15.91	15.40	40.41	5.62	5.46
	1.7B	19.28	25.68	29.61	36.04	23.52	15.34	14.78	17.43	17.41	14.46	41.55	5.17	5.13
	4B	34.52	31.76	34.15	29.84	26.07	30.88	12.02	14.41	13.22	11.66	35.67	1.64	1.44
	8B	32.44	27.49	30.07	28.77	18.39	31.25	11.96	13.22	13.03	11.53	35.12	1.19	0.88
	14B	31.05	29.44	30.95	27.13	33.44	13.18	10.44	27.25	12.05	10.38	35.43	0.92	0.70
	32B	19.55	24.78	27.48	22.82	32.34	7.44	6.99	27.28	6.94	6.17	35.91	0.96	0.62

Appendix

Table 46: Controlled episode regret per LLM , averaged over domains (Table 4 ). As each LLM takes the final action, Never varies by LLM, unlike in the other per-LLM tables. Baselines are cited in Table 3 .

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 242 papers

Sep 25, 2026

LAVOIR: Teaching a Single-Pass Decision Encoder When and What to Ask with Amortized Value of Information

"System One" decision models such as TypeSafe's Jev and its open counterpart Laya answer typed questions about a text in a single forward pass with calibrated probabilities, but they cannot ask for missing information: when a first message does not say what separates two departments, they guess. We present LAVOIR (Laya with Value-Of-Information Routing), which places the candidate pieces of missing information (slots) in the input next to the answer options, so that one forward pass returns both the decision distribution and, for every slot, the expected gain in the probability of the correct decision if the user were asked about it. VOI targets need no human labels: gold decisions come from schema rules, an LLM only verbalizes messages and answers, a model from another family checks every text, and pairing each message with several profiles makes regression on realized gains estimate the expected gain. A Gini-impurity cap bounds the predicted value by what a calibrated model can still gain. In a controlled study, decisions on seen schemas are statistically indistinguishable from the Bayes ceiling. The final model's question policy matches a greedy oracle VOI policy on seen schemas (AUC 0.799 vs. 0.797), and with at most 0.5 questions per conversation it is 14.1 points more accurate than never asking. On real ABCD conversations, one real exchange raises accuracy by 8.3 points where LAVOIR asks and leaves it unchanged where it does not; on SGD the cap lowers the asking rate from 93% to 8.6%. On Laya's twelve benchmarks LAVOIR is above Laya's reported scores on seven, and it answers a question in 31 ms (median, GH200).

Furkan Yilmaz, Habibe Aleyna Tasdemir, Muhammed Faruk Gozay

LLM Decision-Making Active Information Gathering Value of Information Information Gain

Ask PDF

Mar 6, 2026

InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioning

LLMs are increasingly deployed in high-stakes domains such as medical triage and legal assistance, often as document-grounded QA systems in which a user provides a description, relevant sources are retrieved, and an LLM generates a prediction. In practice, initial user queries are often underspecified, and a single retrieval pass is insufficient for reliable decision-making, leading to incorrect and overly confident answers. While follow-up questioning can elicit missing information, existing methods typically depend on implicit, unstructured confidence signals from the LLM, making it difficult to determine what remains unknown, what information matters most, and when to stop asking questions. We propose InfoGatherer, a framework that gathers missing information from two complementary sources: retrieved domain documents and targeted follow-up questions to the user. InfoGatherer models uncertainty using Dempster-Shafer belief assignments over a structured evidential network, enabling principled fusion of incomplete and potentially contradictory evidence from both sources without prematurely collapsing to a definitive answer. Across legal and medical tasks, InfoGatherer outperforms strong baselines while requiring fewer turns. By grounding uncertainty in formal evidential theory rather than heuristic LLM signals, InfoGatherer moves towards trustworthy, interpretable decision support in domains where reliability is critical.

Maksym Taranukhin, Shuyue Stella Li, Evangelos Milios +3

University of British Columbia · Vector Institute · University of Washington +2

Document QA LLM Decision-Making LLM Uncertainty Estimation Clarification Question Generation Clinical Decision Support

Ask PDF

Oct 19, 2026

When to Stop Asking: Factor Coverage Bias and LLM Determinability Judgments with Underspecified Information in Medicine

Clinical decision-making often begins with underspecified information and requires not only gathering missing information but also knowing when to stop. This calls for judging determinability (whether the available information is sufficient for a decision) by considering whether missing information could change the conclusion, not simply the amount of available decision-relevant information. Reliance on this amount as a cue for determinability, which we term factor coverage bias, could undermine the trustworthiness of large language models (LLMs) through premature conclusions or excessive information gathering, such as unnecessary invasive tests. Yet whether LLMs exhibit this bias remains insufficiently studied. To address this gap, we construct a controlled pairwise benchmark in which LLMs identify the determinable case within each determinable--undeterminable pair. We add a decision-relevant factor to either side while keeping each case's determinability unchanged, and similarly add irrelevant text to distinguish this bias from context-length effects. Adding a relevant factor made LLMs more likely to select the augmented case as determinable even when it remained undeterminable, an effect that persisted with increased reasoning effort, while adding irrelevant text made them less likely to select the augmented case, an effect that reasoning substantially reduced. These findings identify factor coverage bias as a failure mode distinct from context-length effects and a potential risk to LLM trustworthiness. Our work provides an evaluation framework for LLM trustworthiness under underspecification.

Yusuke Watanabe, Yohei Kobashi, Taisei Tosaki +4

Kyoto University, Department of Biomedical Data Intelligence · Kyoto University, Department of Cardiovascular Medicine · The University of Tokyo

LLM Reliability Language Model Bias Evaluation Clinical Language Model Evaluation

Ask PDF

Abstract

LLM Decision-Making

Explore similar work

LAVOIR: Teaching a Single-Pass Decision Encoder When and What to Ask with Amortized Value of Information

InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioning

When to Stop Asking: Factor Coverage Bias and LLM Determinability Judgments with Underspecified Information in Medicine