Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking

Yepeng Huang; Jiawen Zhang; Michelle Dai; Xiaorui Su; Shanghua Gao; Zi Wang; Marinka Zitnik

	CRAFT_MD	MedQA-dev	MedQA-test
Limited information	0.69	0.6	0.58
Full information	0.91	0.88	0.86
Proportion*	74%	62%	61%
* Proportion of questions where the model selects the correct choice in at least 2/4 repeats (limited information, no interaction).

Appendix

Table 1: Performance of Qwen3-30B-A3B-Thinking-FP8 in the full MediQ dataset, with limited or full information provided to the LLM in prompt, and no interaction with user allowed.

Benchmark	Multi-turn	Proactive	Obj. Query	Timing/Order	Target Ctrl.	Domains
QuestBench [ 15 ]	✗	✓	✓	✗	✗	Logic, planning, math
AR-Bench [ 40 ]	✓	✓	$\triangle$	$\triangle$	✗	Puzzles, numbers
Active Task Disamb. [ 12 ]	✓	✓	✗	✗	✗	20Q, code
MediQ [ 17 ]	✓	✓	✗	$\triangle$	✗	Clinical QA
CRAFT-MD [ 10 ]	✓	✓	✗	$\triangle$	✗	Clinical diagnosis
LLMs Get Lost [ 13 ]	✓	✗	✗	$\triangle$	✗	Code, DB, math, NLG

Appendix

Table 2: Comparison with existing benchmarks for interactive information seeking. ✓,

\triangle

, and ✗ denote full, partial or judge-mediated, and no direct support, respectively. Multi-turn : evaluation involves multiple interaction turns. Proactive : the model must actively ask questions or choose information-gathering actions. Obj. Query : objective labels are provided for what information should be queried. Timing/Order : the benchmark evaluates when to ask, when to stop, or the order of acquired information. Target Ctrl. : multiple compatible target instances are evaluated under the same initial information to control target-specific bias.

Metric	Notation	Description	Used in
(A) Uncertainty awareness (problem-level)
Degree accuracy	$\mathbb{I}\{\hat{k}=k\}$	Predicted degree of underspecification matches ground truth.	L
Under-prediction rate	$\mathbb{I}\{\hat{k}<k\}$	Predicts fewer missing variables than required.	L
Over-prediction rate	$\mathbb{I}\{\hat{k}>k\}$	Predicts more missing variables than required.	L
MSS accuracy	$\mathbb{I}\{\hat{S}=S\}$	Predicted variable set equals the MSS.	L
Jaccard similarity	$\|\hat{S}\cap S\|/\|\hat{S}\cup S\|$	Overlap between predicted set and MSS.	L

Appendix

Table 3: Glossary of evaluation metrics. Used in: L =Logic-Q-MT, G =GeneReg-MT, M =GSME-Q-MT/-Ext, C =ClinGuide-MT, 20Q =20 Questions.

k/\hat{k}

: true/predicted degree of underspecification;

S

: ground-truth MSS (

\hat{S}

: predicted);

\hat{Q}

: variables queried over

T

turns,

\hat{Q}^{*}

: those in

S

;

Q

: order-dependent sufficient sequence (ClinGuide-MT);

\mathcal{H}_{t}

: history through turn

t

;

\hat{y}/y

: predicted/true target;

p_{t}

: belief over 20Q candidates.

Figure 10: Performance of LLMs in Logic-Q-MT task-solving with different oracle types. coop stands for cooperative, rand stands for random, and adv stands for adversarial. With a budget of 10 turns, 1 query per turn allowed.

p

-values from two-sided paired T-test.

Figure 11: Correlation between performances on tasks with True target variable values and with False target variable values for LLMs in Logic-Q-MT task-solving. Runs are with adversarial oracles and different budget (1 turn x 4 queries, 4 turns x 1 query, 10 turns x 1 query) and alternative minimal sufficient set variable forbiddance statuses (forbid, allow). Due to (realistic) budget constraints, commercial models are only run with forbid.

r

from Pearson’s correlation.

Figure 12: Correlation between performances on tasks with True target variable values and with False target variable values for LLMs in Logic-Q-MT task-solving. Runs are with cooperative oracles and different budget (1 turn x 4 queries, 4 turns x 1 query, 10 turns x 1 query) and alternative minimal sufficient set variable forbiddance statuses (forbid, allow). Due to (realistic) budget constraints, commercial models are only run with forbid.

r

from Pearson’s correlation.

Figure 13: Performance of Qwen3-30B-A3B-Thinking-FP8 in Logic-Q-MT task-solving, with or without thinking traces appended in conversation history to the next turn. Runs are with different budget (1 turn x 4 queries, 4 turns x 1 query, 10 turns x 1 query), oracle type (adversarial, random, cooperative), and alternative minimal sufficient set variable forbiddance (forbid, allow).

Oracle	Common (%)	Thing (%)
Qwen3-30B-A3B-Instruct-FP8	95.4 [93.6, 97.1]	95.6 [93.8, 97.2]
GPT-5-mini	92.7 [90.5, 94.9]	91.9 [89.5, 94.3]

Appendix

Table 4: Within-oracle consistency of the 20Q oracles: percentage of questions on which three responses sampled at temperature 0.6 all agree, with confidence intervals in brackets.

Oracle	Answerer	Agreement (%)	“Not sure” share (%)
GPT-5-mini	Qwen3-30B-A3B-Thinking-FP8	97.7 [96.6, 98.7]	86.8
GPT-5-mini	GPT-5	97.3 [96.6, 98.4]	80.6
GPT-5.4	Qwen3-30B-A3B-Thinking-FP8	94.1 [92.7, 96.5]	95.6
GPT-5.4	GPT-5	94.0 [92.8, 96.2]	90.8

Appendix

Table 5: Within-oracle consistency of the ClinGuide-MT oracles when replayed on identical inputs (system prompt, patient context, and model query), reported separately for each evaluated answerer model, with confidence intervals in brackets. The last column is the share of inconsistent cases involving a shift to or from “Not sure”.

Target	Question	Oracle responses	Human
T-Rex	Is T-Rex a living thing?	no / no / yes	yes
Christmas tree	Is Christmas tree a living thing?	no / pass / pass	pass
Mount Rushmore	Is Mount Rushmore man-made?	yes / pass / pass	pass

Appendix

Table 6: Representative 20Q cross-oracle disagreements. Oracle responses are listed in the order Qwen3-30B-A3B-Instruct-FP8 / GPT-5-mini / GPT-5.4; the human label comes from an independent annotator. Disagreements concentrate on category boundaries rather than reversals between substantive answers.

Answerer	GPT-5-mini vs. GPT-5.4 agreement (%)
Qwen3-30B-A3B-Thinking-FP8	94.0 [92.4, 96.4]
GPT-5	94.0 [93.1, 95.9]

Appendix

Table 7: Cross-oracle agreement in ClinGuide-MT: percentage of identical inputs on which the two oracle choices (GPT-5-mini and GPT-5.4) return the same response, with confidence intervals in brackets.

Dataset	Qwen3-30B-A3B-Instruct-FP8	GPT-5-mini	GPT-5.4
Common	0.907	0.953	0.960
Thing	0.933	0.940	0.960

Appendix

Table 8: Human–oracle agreement on 20Q: fraction of 300 independently annotated questions (150 per dataset) on which the human label matches each oracle’s response.

Answerer	Correct (%)	Clear mismatch (%)	Borderline (%)
GPT-5	91	5	4
Qwen3-30B-A3B-Thinking-FP8	85	9	6

Appendix

Table 9: Manual audit of 200 ClinGuide-MT oracle responses. Borderline denotes questions with meaningful clinical overlap with a desired question, where either matching or returning “Not sure” can be reasonable.

Model question	Oracle-matched question	Human verdict
Hair-loss type	Hair-loss duration under one year	Mismatch: type $\neq$ duration
Ataxia with falling	Cause of gait abnormality	Mismatch: symptom $\neq$ mechanism
Specific question about mania	Broader composite psychiatric screen	Borderline: partial overlap, not a clear paraphrase

Appendix

Table 10: Representative ClinGuide-MT oracle errors from the manual audit of 200 responses.

Dataset	# Problems					# Tasks					Source
Dataset	Total	$k{=}1$	$k{=}2$	$k{=}3$	$k{=}4$	Total	$k{=}1$	$k{=}2$	$k{=}3$	$k{=}4$	Source
Logic-Q-MT	600	200	200	200	0	1200	400	400	400	0	[ 36 ]
GeneReg-MT	800	–	–	–	–	2326	451	626	729	520	38 GRNs [ 11 ]
GSME-Q-MT	2956	1067	884	709	296	2956	1067	884	709	296	[ 15 ]
GSME-Q-MT-Ext	524	134	144	123	123	524	134	144	123	123	Synthetic
ClinGuide-MT	370	–	–	–	–	1595	521	477	370	227	59 algorithms [ 7 , 30 ]

Appendix

Table 11: Dataset statistics by degree of underspecification. Columns under Problems and Tasks report the total count and the count for each

k

; “–” denotes not applicable.

Figure 14: Complexity Across Task Settings.

\mathcal{A}_{S}\leftarrow\{X=\omega^{\star}(X):X\in\mathcal{U}\setminus S\}

Appendix

Algorithm 3 Constructing Minimal

k

-Sufficient GSME-Q-MT

Pool	# Targets	$\|\mathcal{Y}_{\mathrm{eval}}\|$
Animals	27	116
Places	21	71
Food	26	92
Objects	37	126
Common	111	405
Thing	200	188

Appendix

Table 12: Statistics of 20Q candidate pools. The target column reports the number of hidden targets used for evaluation, while the candidate column reports the size of the candidate-space surrogate used for offline question-quality estimation.

$\|\Delta\mathrm{nEIG}\|$	Human– $\mathrm{nEIG}$ agreement	$p$ -value vs. $0.5$
$<0.05$	72.3%	$<10^{-4}$
$0.05$ – $0.10$	86.5%	$<10^{-7}$
$0.10$ – $0.15$	81.3%	$<10^{-4}$

Appendix

Table 13: Human–

\mathrm{nEIG}

agreement stratified by the absolute difference in normalized expected information gain between the two candidate questions. Each cell reports the fraction of pairs in which the higher-

\mathrm{nEIG}

question matched the blinded annotator’s preference;

p

-values compare each agreement rate against the chance rate of

0.5

.

Figure 15: Performance and turns used of LLMs in Logic-Q-MT task-solving. Note that the proportion of interactions with final sufficiency = 1 is equal to the proportions that finish on sufficient turn (green) and that finish after sufficient turn (light green). LLMs are ordered by final sufficiency for

k=3

tasks. “Random” randomly selects a variable that has not been queried from the valid variables per turn.

Figure 16: Performance and turns used of LLMs in GeneReg-MT dynamic steady-state task-solving. Similar to Fig. 15 the proportion of samples with final sufficiency = 1 is equal to the proportions that finish on sufficient turn (green) and that finish after sufficient turn (light green). The high performance of random baseline is due to the low number of valid variables per task. LLMs are ordered by final sufficiency for

k=4

tasks. We do not include a random baseline as there are fewer than 10 valid variables in many tasks, which naturally leads to near-perfect performance of random baselines.

Figure 17: Performance and turns used of LLMs in GeneReg-MT dynamic marker task-solving. Similar to Fig. 15 the proportion of samples with final sufficiency = 1 is equal to the proportions that finish on sufficient turn (green) and that finish after sufficient turn (light green). The high performance of random baseline is due to the low number of valid variables per task. More concretely, for

k=1,2,3,4

, there are 86/100, 86/100, 84/100, 84/100 tasks with 10 or fewer valid variables, respectively, to query for. Some “final sufficiency turn” values are missing in the lower panel, because there is no final sufficient interaction for that model and that

k

. LLMs are ordered by final sufficiency for

k=4

tasks. We do not include a random baseline as there are fewer than 10 valid variables in many tasks, which naturally leads to near-perfect performance of random baselines.

Model	$\boldsymbol{k=1}$				$\boldsymbol{k=2}$				$\boldsymbol{k=3}$
Model	Suff. $\uparrow$	Acc. $\uparrow$	Turn	OverQ $\downarrow$	Suff. $\uparrow$	Acc. $\uparrow$	Turn	OverQ $\downarrow$	Suff. $\uparrow$	Acc. $\uparrow$	Turn	OverQ $\downarrow$
GPT-5	1.000	1.000	2.128	0.585	0.975	0.988	3.870	0.785	0.955	0.973	4.720	0.815
GPT-5-mini	0.983	0.990	3.488	0.715	0.908	0.938	4.628	0.785	0.858	0.935	5.530	0.790
Gemini-3-Flash	0.945	0.970	1.303	0.213	0.603	0.808	2.275	0.293	0.405	0.675	2.815	0.270
Qwen3-4B-Thinking	0.788	0.858	2.813	0.555	0.668	0.815	4.265	0.578	0.560	0.710	4.725	0.470
Qwen3.5-9B	0.890	0.913	2.355	0.558	0.840	0.880	3.790	0.680	0.750	0.843	4.508	0.660

Appendix

Table 14: Logic-Q-MT results across different underspecification levels

k

. Suff. denotes final sufficiency, Acc. denotes answer accuracy, Turn denotes the average number of turns, and OverQ denotes over-questioning rate. Gemini-3.1-Pro is evaluated on a reduced set of conditions and is therefore excluded from the cross-model analyses reported in the main text.

Model	$k$	Steady state				Marker
Model	$k$	Suff. $\uparrow$	Acc. $\uparrow$	# Turn	OverQ $\downarrow$	Suff. $\uparrow$	Acc. $\uparrow$	# Turn	OverQ $\downarrow$
GPT-5	$k=1$	0.956	0.964	1.004	0.036	0.895	0.925	0.925	0.020
	$k=2$	0.887	0.927	1.981	0.063	0.760	0.865	1.765	0.040
	$k=3$	0.802	0.849	2.960	0.079	0.600	0.820	2.330	0.065
	$k=4$	0.628	0.769	3.431	0.081	0.515	0.790	2.860	0.045
GPT-5-mini	$k=1$	0.861	0.916	1.159	0.227	0.815	0.895	1.065	0.135

Appendix

Table 15: GeneReg-MT results. Suff. denotes final sufficiency, Acc. denotes answer accuracy, # Turn denotes the average number of turns, and OverQ denotes over-questioning rate. Gemini-3.1-Pro is evaluated on a reduced set of conditions and is therefore excluded from the cross-model analyses reported in the main text.

Model	$k$	GSME-Q-MT				GSME-Q-MT-Ext
Model	$k$	Suff. $\uparrow$	Acc. $\uparrow$	# Turn	OverQ $\downarrow$	Suff. $\uparrow$	Acc. $\uparrow$	# Turn	OverQ $\downarrow$
Qwen3-4B-Thinking	$k=1$	0.989	0.988	$0.99{\pm}0.12$	0.004	0.813	0.813	$0.84{\pm}0.42$	0.030
	$k=2$	0.986	0.985	$1.97{\pm}0.22$	0.001	0.681	0.639	$1.42{\pm}0.89$	0.028
	$k=3$	0.983	0.985	$2.95{\pm}0.37$	0.001	0.585	0.520	$1.78{\pm}1.37$	0.008
	$k=4$	0.981	0.980	$3.93{\pm}0.53$	0.001	0.417	0.342	$1.76{\pm}1.76$	0.024
Qwen3-30B-A3B-Thinking-FP8	$k=1$	0.985	0.984	$0.99{\pm}0.12$	0.004	0.463	0.485	$0.46{\pm}0.50$	0.000

Appendix

Table 16: GSME-Q-MT and its extension results. Suff. denotes final sufficiency, Acc. denotes answer accuracy, # Turn denotes the average number of turns, and OverQ denotes over-questioning rate.

Model	$k$	Final acc. $\uparrow$			Order correctness $\uparrow$			Query correctness $\uparrow$			Query coverage $\uparrow$			Avg. turn used
Model	$k$	#d=0	#d=10	#d=20	#d=0	#d=10	#d=20	#d=0	#d=10	#d=20	#d=0	#d=10	#d=20	#d=0	#d=10	#d=20
Qwen3-4B-Thinking	$k=1$	0.503	0.436	0.426	0.916	0.739	0.749	0.575	0.384	0.384	0.916	0.739	0.749	3.084	5.269	5.284
	$k=2$	0.461	0.350	0.333	0.917	0.874	0.883	0.700	0.493	0.514	0.743	0.618	0.594	3.249	5.159	5.145
	$k=3$	0.414	0.322	0.276	0.857	0.892	0.873	0.785	0.618	0.619	0.701	0.572	0.558	3.538	4.754	4.905
	$k=4$	0.339	0.317	0.304	0.779	0.819	0.858	0.828	0.674	0.684	0.681	0.532	0.531	3.868	4.894	4.758
Qwen3-30B-A3B-Thinking-FP8	$k=1$	0.560	0.486	0.451	0.783	0.616	0.601	0.546	0.409	0.397	0.783	0.616	0.601	2.296	2.987	2.950

Appendix

Table 17: ClinGuide-MT results. “d” represents the set of distractor variables added in each problem, all obtained from other ClinGuide-MT problems and filtered. #d refers to the number of distracting variable sets.

Examiner	Guesser	Common				Things
Examiner	Guesser	Acc. $\uparrow$	#avg turns	avg. norm IG $\uparrow$	avg. final $H$ $\downarrow$	Acc. $\uparrow$	#avg turns	avg. norm IG $\uparrow$	avg. final $H$ $\downarrow$
GPT-5-mini	gpt-oss-20B-medium	0.1712	18.7928	0.1563	1.7203	0.1100	19.3550	0.1644	1.6599
	Qwen3-4B-Thinking	0.2252	18.6847	0.1557	1.6206	0.1350	19.1400	0.1561	1.6941
	Qwen3-30B-A3B-Thinking-FP8	0.2793	18.0000	0.1673	1.7864	0.2650	18.0950	0.1607	1.8684
	Gemini-3-Flash	0.4775	16.2883	0.1768	1.2130	0.3300	17.5550	0.2050	1.3496
	GPT-5-mini	0.4234	17.1982	0.2030	1.4651	0.3800	17.5900	0.1829	1.6172

Appendix

Table 18: 20 Questions results on Common and Things domains.

Dataset	Construct	Feature	Templates used
Logic-Q-MT	Knows what is missing	MSS relative mention rate	ask about {var} , ask is alice {var} , ask if alice is {var} , ask whether alice is {var} , ask if she is {var} , ask whether she is {var} , consider querying {var} , should ask about {var} , should ask if alice is {var} , should ask whether alice is {var} , need to ask about {var} , need to ask if alice is {var} , need to ask whether alice is {var} , if i ask {var} , if i ask about {var} , if i ask is alice {var} , if i ask if alice is {var} , if we ask {var} , if we ask about {var} , if we ask is alice {var} , if we ask if alice is {var} , next question is {var} , next question is is alice {var} , question is alice {var} , question is is alice {var} , question should be {var} , question should be is alice {var} , question would be {var} , question would be is alice {var} , best question is {var} , best question is is alice {var} , best question would be {var} , best question would be is alice {var} , choose {var} as the next question , choose asking whether alice is {var} , choose to ask whether alice is {var} , need to know {var} , need to know whether alice is {var}
GeneReg-MT	Knows what is missing	MSS relative mention rate	ask about {var} , ask {var} , ask for {var} , ask for the value of {var} , ask the value of {var} , should ask about {var} , should ask {var} , should ask for {var} , need to ask about {var} , need to ask {var} , need to ask for {var} , next question is {var} , next question should be {var} , next question {var} , question should be {var} , best question is {var} , want to ask about {var} , want to ask {var} , let me ask {var} , let me ask about {var} , lets ask {var} , lets ask about {var} , go with {var} , going with {var} , choose {var} , choosing {var} , choose to ask {var} , choose to query {var} , query {var} , query about {var} , query the value of {var} , should query {var} , need to query {var}
Both datasets	Knows something is missing	Awareness of underspecification	need more information , need more info , need more facts , need another question , need additional information , need additional facts , more information is needed , more info is needed , more facts are needed , more information is required , more facts are required , dont have enough information , do not have enough information , dont have enough info , do not have enough info , dont have enough facts , do not have enough facts , this is ambiguous , the situation is ambiguous , not yet enough , not yet sufficient , not yet determined , cannot yet determine , cannot yet conclude , cannot yet answer , can not yet determine , can not yet conclude , cant yet determine , cant yet conclude , still need to ask , still need to check , still need to determine , still need to find out , need to ask one more , need to ask another

Appendix

Table 19: Pattern templates used in the first-turn CoT analysis. The GeneReg-MT query-planning templates are intentionally narrower than the Logic-Q-MT templates to reduce false positives from regulatory-rule enumeration.

Figure 18: Turn-1 CoT predictors in Logic-Q-MT. Each panel shows standardized log-odds ratios with 95% confidence intervals from a pooled model with model and

k

fixed effects and standard errors clustered by interaction. All predictors are standardized before fitting, so each value is the log odds ratio for a one-standard-deviation increase in the predictor. Left: final sufficiency over all interactions with at least one turn. Right: second-turn hit among interactions whose first-turn query misses the MSS and that continue to a second turn. MSS relative mention rate is the strongest CoT predictor in both panels (

\beta_{z}=0.528

for final sufficiency;

\beta_{z}=0.365

for second-turn hit), while awareness of underspecification remains near zero (

\beta_{z}=0.005

for final sufficiency and

\beta_{z}=0.010

for second-turn hit).

Figure 19: Stability of MSS relative mention rate in Logic-Q-MT. Each cell refits the same outcome model within one model-by-

k

stratum, omitting fixed effects inside the stratum. Color indicates the standardized log-odds ratio for MSS relative mention rate; asterisks mark nominal

p<0.05

. Controls that are constant within a cell are dropped from that cell’s regression; cells are marked NA when they have fewer than 50 usable interactions, no outcome variation, no variation in MSS relative mention rate, or a separated fit.

Figure 20: Turn-1 CoT predictors in GeneReg-MT. The specification matches Logic-Q-MT conceptually but is fit separately for steady-state and marker tasks and uses a narrower gene-query template set to reduce false positives from regulatory-rule enumeration. MSS relative mention rate remains positive in both task families for final sufficiency (

\beta_{z}=0.340

for steady state;

\beta_{z}=1.016

for marker) and for second-turn hit after a first-turn miss (

\beta_{z}=0.274

for steady state;

\beta_{z}=0.590

for marker).

Figure 21: Stability of MSS relative mention rate in GeneReg-MT. Each cell is a separate model-task-

k

regression with the same controls as the pooled model and heteroskedasticity-robust standard errors. Many recovery cells are NA because the outcome is defined only for interactions that both miss on turn 1 and continue to turn 2; the appendix text reports the exact NA criteria and summary counts.

Figure 22: Proportions of interactions that end at 10 turns for models in Logic-Q-MT task-solving, with no budget in prompt but capped at 10 turns.

Figure 23: Performance of representative models with interactions ending on turn 10 in Logic-Q-MT task-solving, with no budget in prompt but capped at 10 or 20 turns.

Figure 24: Recall with regard to the MSS in predicting missing variables. Same models and settings as Fig. 4 ; Jaccard similarity and recall are reported as a complement to accuracy. (a)

k

not provided; (b)

k

provided. Recall is lower when

k

is provided for

k{=}2,3

because forcing exactly

k

selections decreases the chance of overlap with a

k

-MSS, compared with being allowed to predict a larger set of variables.

Model	$k=1$	$k=2$	$k=3$
Qwen3-30B-A3B-Thinking-FP8	0.720	0.505	0.405
Qwen3-30B-A3B-Thinking	0.710	0.495	0.330
Qwen3-4B-Thinking	0.770	0.340	0.125
Qwen3-Next-80B-A3B-Thinking-FP8	0.550	0.390	0.290
Qwen3.5-122B-A10B-FP8	0.835	0.465	0.265
Qwen3.5-27B	0.785	0.330	0.200

Appendix

Table 20: Per-model accuracy of predicting the degree of underspecification

k

in Logic-Q-MT (multiple-choice

k

-prediction; cf. Fig. 3 a). Each cell reports exact-match accuracy over 200 problems per

k

; predictions of “not ambiguous,” “not sure,” and unparseable responses are counted as incorrect.

Model	$k$	$k$ not provided			$k$ provided			Full Info.
Model	$k$	Acc.	Jac.	Rec.	Acc.	Jac.	Rec.	Acc.
Qwen3-30B-A3B-Thinking-FP8	1	0.250	0.314	0.390	0.320	0.320	0.320	0.885
	2	0.070	0.238	0.302	0.165	0.323	0.402	0.948
	3	0.040	0.250	0.283	0.135	0.410	0.522	0.976
Qwen3-4B-Thinking	1	0.230	0.288	0.350	0.320	0.320	0.320	0.855
	2	0.065	0.222	0.252	0.120	0.280	0.360	0.959

Appendix

Table 21: Performance of LLMs on Logic-Q-MT missing variable and full-information problem-solving. We report missing-variable prediction performance when

k

is not provided or provided, and full-information problem-solving correctness as a reference upper bound for sequential task-solving.

Figure 25: Performance of LLMs in Logic-Q-MT task-solving with varying budget in prompt across models. Budgets are set in the form of “number of turns allowed” x “number of queries per turn allowed”. “No budget” refers to the setting where no budget is mentioned in the prompt. To avoid infinite loops, we cap the number of turns in the “No budget” setup at 10. This cap does not significantly affect final sufficiency (Fig. 23 ). With adversarial oracles.

Figure 26: Performance of LLMs in GeneReg-MT marker and steady state identification task-solving with varying budget in prompt across models. Same setup as in Fig. 25 .

Figure 27: Proportion of tasks that have reached sufficiency at turn

t

per model, across each k, for Logic-Q-MT.

Figure 28: Scatterplot underlying every cell of Fig. 5 , plus a control row, in Logic-Q-MT. Each panel plots one of four behavioral features against final sufficiency; one dot per model, columns vary the true degree of underspecification

k

, the dashed grey line is the OLS fit, and the upper-left annotation reports the Pearson

r

. The first three rows correspond directly to the cells of the heatmap. The bottom row, avg number of turns - avg turn to sufficiency, is a control: it counts the turns each model uses past its first-sufficient turn, isolating the “keeps asking after it could stop” component from the raw turn count. Its correlation with final sufficiency strengthens with

k

(

r=0.32\to 0.67

), mirroring the avg-number-of-turns row above and showing that on harder problems the benefit of longer interactions persists even after netting out when sufficiency is first reached.

Figure 29: Heatmap of correlations between per-model behavioral features with average final sufficiency in GeneReg-MT dynamic steady state identification. Each cell is the Pearson

r

between the behavioral feature and average final sufficiency, computed across models within a fixed

k

.

Figure 30: Heatmap of correlations between per-model behavioral features with average final sufficiency in GeneReg-MT dynamic marker identification. Each cell is the Pearson

r

between the behavioral feature and average final sufficiency, computed across models within a fixed

k

.

Figure 31: Scatterplot underlying every cell of Fig. 29 , plus a control row, in GeneReg-MT dynamic steady state identification. Each panel plots one of four behavioral features against final sufficiency; one dot per model, columns vary the true degree of underspecification

k

, the dashed grey line is the OLS fit, and the upper-left annotation reports the Pearson

r

. The first three rows correspond directly to the cells of the heatmap. The bottom row, avg number of turns - avg turn to sufficiency, is a control: it counts the turns each model uses past its first-sufficient turn, isolating the “keeps asking after it could stop” component from the raw turn count.

Figure 32: Scatterplot underlying every cell of Fig. 30 , plus a control row, in GeneReg-MT dynamic marker identification. Each panel plots one of four behavioral features against final sufficiency; one dot per model, columns vary the true degree of underspecification

k

, the dashed grey line is the OLS fit, and the upper-left annotation reports the Pearson

r

. The first three rows correspond directly to the cells of the heatmap. The bottom row, avg number of turns - avg turn to sufficiency, is a control: it counts the turns each model uses past its first-sufficient turn, isolating the “keeps asking after it could stop” component from the raw turn count.

Figure 33: Recovery from a wrong first-turn query in

k=2,3

Logic-Q-MT tasks. Bars show final sufficiency for each model, partitioned by whether the model queries a correct variable in the first and second turns. For every model, the first-turn miss + second-turn hit (recovery) group performs within 10 percentage points of the first-turn hit group. A

\star

marks models for which this near-parity is statistically supported by a one-sided Miettinen–Nurminen non-inferiority test with a 10pp margin (

p<0.05

).

Figure 34: Recovery from a wrong first-turn query in

k=2,3,4

GeneReg-MT dynamic steady state identification. Bars show final sufficiency for each model, partitioned by whether the model queries a correct variable in the first and second turns.

\blacktriangle

indicates that the first-turn miss + second-turn hit (recovery) group performs within 10 percentage points of the first-turn hit group. The number on each bar represents the number of interactions in each group. Groups with fewer than 10 interactions are not shown (a dash). No non-inferiority tests are performed due to the lack of power. Because some interactions emit zero or only one query, the miss+hit and miss+miss counts need not sum to first-turn miss; for the same reason, the totals of first-turn hit and first-turn miss vary slightly across models.

Figure 35: Recovery from a wrong first-turn query in

k=2,3,4

GeneReg-MT dynamic marker identification. Bars show task-completion rate for each model, partitioned by whether the model queries a correct variable in the first and second turns.

\blacktriangle

indicates that the first-turn miss + second-turn hit (recovery) group performs within 10 percentage points of the first-turn hit group. The number on each bar represents the number of interactions in each group. Groups with with fewer than 10 interactions are not shown (a dash). No non-inferiority tests are performed due to the lack of power. Because some interactions emit zero or only one query, the miss+hit and miss+miss counts need not sum to first-turn miss; for the same reason, the totals of first-turn hit and first-turn miss vary slightly across models.

Figure 36: Sensitivity of the recovery criterion in Logic-Q-MT. For each threshold, we compute how many models satisfy the corresponding recovery criterion when comparing the “first-turn miss + second-turn hit” group against the “first-turn hit” group. The left panel uses a relative threshold based on the ratio of mean final sufficiency, while the right panel uses an absolute threshold based on the difference in mean final sufficiency. Results are shown for

k=2

and

k=3

.

Figure 37: Sensitivity of the recovery criterion in GeneReg-MT. For each threshold, we count how many models satisfy the recovery criterion when comparing the “first-turn miss + second-turn hit” group against the “first-turn hit” group. In each subfigure, the left panel uses a relative threshold based on the ratio of mean final sufficiency, while the right panel uses an absolute threshold based on the difference in mean final sufficiency. Results are shown for

k=2,3,4

.

Variant	Qwen3-4B-Thinking	Qwen3-30B-A3B-Thinking-FP8	gpt-oss-20B-high	GPT-5-mini
Logic-Q-MT
P1: reorder items	$+0.019$ (n.s.)	$+0.053$ (n.s.)	$+0.010$ (n.s.)	$-0.005$ (n.s.)
P2: drop query-selection guidance	$+0.076$ (.002)	$-0.109$ (.006)	$-0.070$ (.011)	$-0.040$ (.011)
P3: drop insufficiency reminder	$-0.141$ ( $<$ .001)	$-0.014$ (n.s.)	$-0.027$ (n.s.)	$-0.005$ (n.s.)
GeneReg-MT
P1: reorder items	$-0.018$ (n.s.)	$-0.040$ (n.s.)	$-0.039$ (n.s.)	$-0.012$ (n.s.)

Appendix

Table 22: Prompt sensitivity of final sufficiency in Logic-Q-MT and GeneReg-MT. Each cell reports the signed change in final sufficiency under a prompt variant,

\Delta=\mathrm{variant}-\mathrm{original}

, with the corresponding

p

-value in parentheses: a problem-level cluster bootstrap (10,000 resamples, Holm-corrected over the model family) tests whether the variant moves final sufficiency more than a rerun of the original prompt under a different seed. n.s. denotes

p>0.05

. The Logic-Q-MT default prompt already contains the query-selection guidance, so P2 removes it, whereas the GeneReg-MT default does not, so P2 adds it.

Model	Final sufficiency				Number of turns
Model	Original	Force-ask	Change (pp)	$p$ (McNemar)	Original	Force-ask	Change	$p$ (Wilcoxon)
Qwen3-4B-Thinking	0.560	0.657	$+9.7$	0.003	4.725	5.192	$+0.467$	$<5\times 10^{-4}$
Qwen3-30B-A3B-Thinking-FP8	0.275	0.532	$+25.7$	$<5\times 10^{-4}$	3.743	4.910	$+1.167$	$<5\times 10^{-4}$
gpt-oss-20B-high	0.818	0.825	$+0.7$	0.824	6.470	6.678	$+0.208$	0.187
GPT-5-mini	0.858	0.887	$+2.9$	0.175	5.530	5.647	$+0.117$	0.471

Appendix

Table 23: Effect of preventing premature answers on Logic-Q-MT at

k=3

(400 episodes per model). Any attempt to answer before issuing

k

queries is rejected and the model is asked to query again; query selection remains unconstrained, and the model may stop freely once

k

queries have been issued. Final sufficiency is compared with exact McNemar tests and the average number of turns with Wilcoxon signed-rank tests, both paired at the episode level.

Class	Problems
All three identical	19
Two alternatives identical, both different from the original	43
All three pairwise different	538 (89.7%)

Appendix

Table 24: Overlap classes of the three canonical MSS draws per Logic-Q-MT problem: the original designated MSS and two independently resampled alternatives ( altgt1 , altgt2 ).

Model	Comparison	$\Delta_{\mathrm{suff}}(\mathrm{alt}-\mathrm{orig})$	$\Delta_{\mathrm{suff}}(\mathrm{repl}-\mathrm{orig})$	$\|\Delta_{\mathrm{alt}}\|-\|\Delta_{\mathrm{repl}}\|$	95% CI	$p_{\mathrm{boot}}$	$p_{\mathrm{Holm}}$
Qwen3-4B-Thinking	vs altgt1	$+0.037$	$-0.015$	$+0.022$	$[-0.057,+0.088]$	n.s.	n.s.
Qwen3-4B-Thinking	vs altgt2	$+0.035$	$-0.015$	$+0.020$	$[-0.057,+0.088]$	n.s.	n.s.
Qwen3-30B-A3B-Thinking-FP8	vs altgt1	$+0.072$	$+0.027$	$+0.045$	$[-0.018,+0.100]$	0.090	n.s.
Qwen3-30B-A3B-Thinking-FP8	vs altgt2	$-0.005$	$+0.027$	$-0.022$	$[-0.062,+0.050]$	n.s.	n.s.
gpt-oss-20B-high	vs altgt1	$+0.065$	$+0.017$	$+0.047$	$[-0.005,+0.088]$	0.042	n.s.
gpt-oss-20B-high	vs altgt2	$+0.052$	$+0.017$	$+0.035$	$[-0.015,+0.077]$	0.117	n.s.

Appendix

Table 25: Sensitivity of final sufficiency to the designated MSS in Logic-Q-MT at

k=3

.

\Delta

columns are signed (run minus original):

\Delta_{\mathrm{suff}}(\mathrm{alt}-\mathrm{orig})

compares the alternative-MSS variant with the original run, and

\Delta_{\mathrm{suff}}(\mathrm{repl}-\mathrm{orig})

compares a rerun of the original configuration with a different random seed. The test statistic is

|\Delta_{\mathrm{alt}}|-|\Delta_{\mathrm{repl}}|

, i.e., whether the redraw effect exceeds run-to-run noise, assessed by a problem-level cluster bootstrap (10,000 resamples) with the 95% percentile confidence interval shown;

p_{\mathrm{boot}}

is the raw one-sided bootstrap

p

-value and

p_{\mathrm{Holm}}

its Holm adjustment over the 8 tests within each metric. n.s. denotes

p>0.2

. Statistics are computed from unrounded values and rounded independently, so displayed columns may disagree in the last digit.

Model	Comparison	$\Delta_{\mathrm{turns}}(\mathrm{alt}-\mathrm{orig})$	$\Delta_{\mathrm{turns}}(\mathrm{repl}-\mathrm{orig})$	$\|\Delta_{\mathrm{alt}}\|-\|\Delta_{\mathrm{repl}}\|$	95% CI	$p_{\mathrm{boot}}$	$p_{\mathrm{Holm}}$
Qwen3-4B-Thinking	vs altgt1	$-0.355$	$-0.087$	$+0.268$	$[-0.013,+0.485]$	0.032	n.s.
Qwen3-4B-Thinking	vs altgt2	$-0.557$	$-0.087$	$+0.470$	$[+0.192,+0.683]$	0.001	0.006
Qwen3-30B-A3B-Thinking-FP8	vs altgt1	$-0.040$	$-0.058$	$-0.018$	$[-0.188,+0.167]$	n.s.	n.s.
Qwen3-30B-A3B-Thinking-FP8	vs altgt2	$-0.048$	$-0.058$	$-0.010$	$[-0.198,+0.183]$	n.s.	n.s.
gpt-oss-20B-high	vs altgt1	$-0.345$	$-0.007$	$+0.338$	$[-0.125,+0.542]$	0.090	n.s.
gpt-oss-20B-high	vs altgt2	$-0.270$	$-0.007$	$+0.263$	$[-0.177,+0.475]$	0.173	n.s.

Appendix

Table 26: Sensitivity of the number of turns to the designated MSS in Logic-Q-MT at

k=3

. Same protocol as Table 25 :

\Delta

columns are signed (run minus original), the test statistic

|\Delta_{\mathrm{alt}}|-|\Delta_{\mathrm{repl}}|

is assessed by a problem-level cluster bootstrap (10,000 resamples), and Holm adjustment is over the 8 tests within each metric. n.s. denotes

p>0.2

.

Draw	% forward-derivable	Mean MSS depth	Median MSS depth
original	100.0%	4.400	4.0
altgt1	59.0%	3.550	4.0
altgt2	60.5%	3.499	3.0

Appendix

Table 27: Forward-derivability and derivation depth of the designated MSS by draw at

k=3

. % forward-derivable is the fraction of designated MSSs whose sufficiency admits an explicit forward-chaining (unit-propagation) derivation; MSS depth is the length of that derivation. Mean and median depths are computed over forward-derivable draws only, as depth is undefined for MSSs whose sufficiency holds only by refutation; for the original draw this includes all problems.

Figure 38: Answer accuracy on GSME across different values of

k

. The original GSME setting is nearly saturated across models, whereas the DAG-generated complex GSME setting reveals a sharp degradation in the multi-turn setting.

Figure 39: Analysis of multi-turn degradation in GSME. Accuracy change from single-turn to multi-turn querying, measured as multi-turn accuracy minus single-turn accuracy. Negative values indicate degradation under multi-turn interaction.

Figure 40: ClinGuide-MT final accuracy by MSS coverage and MSS ordering in the queried variables (a-f:

k=3,4

, and

0,10,20

distractors).

Figure 41: The effect of query coverage, order correctness, and query correctness on final accuracy for ClinGuide-MT Problem (a-f:

k=3,4

, and

0,10,20

distractors).

Model	$\boldsymbol{k=3}$		$\boldsymbol{k=4}$
Model	Closed-book	Open-book	Closed-book	Open-book
GPT-5-mini	55.7%	88.4%	54.6%	85.0%
gpt-oss-20B-high	28.1%	56.5%	29.5%	54.2%
Qwen3-30B-A3B-Thinking-FP8	38.1%	71.1%	33.9%	64.3%
Qwen3-4B-Thinking	32.2%	78.4%	31.7%	80.2%

Appendix

Table 28: ClinGuide-MT final accuracy in the closed-book and open-book settings for

k=3

and

k=4

tasks. In the open-book setting, the model is given all 59 diagnostic decision trees from the source textbook [ 7 ] in their original wording, including the tree relevant to the case.

Model	$k$	Closed-book (decision trees $=0$ )				Open-book (all 59 decision trees)
Model	$k$	All/Correct	All/Incorrect	Partial/Correct	Partial/Incorrect	All/Correct	All/Incorrect	Partial/Correct	Partial/Incorrect
Qwen3-30B-A3B-Thinking-FP8	3	69.6% (N=23)	50.0% (N=40)	38.2% (N=251)	25.0% (N=28)	94.0% (N=184)	–	50.0% (N=168)	–
Qwen3-30B-A3B-Thinking-FP8	4	83.3% (N=6)	59.1% (N=22)	29.6% (N=152)	37.8% (N=37)	97.8% (N=93)	–	41.5% (N=123)	28.6% (N=7)
Qwen3-4B-Thinking	3	63.6% (N=22)	40.0% (N=45)	29.2% (N=271)	22.2% (N=27)	94.4% (N=195)	88.9% (N=18)	60.8% (N=143)	40.0% (N=5)
Qwen3-4B-Thinking	4	–	38.1% (N=21)	35.3% (N=150)	16.0% (N=50)	93.6% (N=110)	100.0% (N=7)	67.0% (N=103)	50.0% (N=6)
GPT-5-mini	3	82.0% (N=61)	61.7% (N=60)	52.9% (N=210)	25.8% (N=31)	97.9% (N=243)	93.3% (N=15)	68.9% (N=103)	80.0% (N=5)

Appendix

Table 29: ClinGuide-MT final accuracy stratified by MSS coverage and query order for

k\in\{3,4\}

in the closed-book setting (no diagnostic decision trees provided) and the open-book setting (all 59 diagnostic decision trees provided). Column headers abbreviate the strata: All/Partial denotes full versus partial MSS coverage, and Correct/Incorrect denotes whether the queries follow the order induced by the diagnostic pathway. Each cell reports final accuracy with the stratum size

N

in parentheses; “–” marks strata with fewer than five samples, which are not reported because of insufficient statistical power.

Figure 42: Accuracy versus offline question informativeness on the 20Q tasks. Bars show final guessing accuracy, and lines show the average offline question-quality score of the questions asked by each guesser model.

Figure 43: Remaining entropy of the candidate distribution across turns. Each curve tracks the average posterior uncertainty after each turn for a given guesser model, dataset, and examiner.

Figure 44: Pass mass across turns in the 20Q tasks. Pass mass denotes the posterior support mass of candidates for which the current question would receive a pass response, meaning that the question is ambiguous or not cleanly answerable by yes or no for those candidates.

Figure 45: Representative failure modes in 20 questions.

Do LLMs Know What to Ask and When? Evaluating Multi-Turn Information Seeking

Abstract

Figures & tables

Appendix

LLM Evaluation

Knowing When Not to Answer: Cross-Domain and Multi-Turn Generalization of Latent Underspecification Signals

Not All Information Is Worth Asking For: Decision-Informed Information Seeking with LLMs

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap

Abstract

LLM Evaluation

Explore similar work

Knowing When Not to Answer: Cross-Domain and Multi-Turn Generalization of Latent Underspecification Signals

Not All Information Is Worth Asking For: Decision-Informed Information Seeking with LLMs

Found in Conversation: LLMs Teach Themselves to Close the Multi-Turn Gap