KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues

Junhao Chen; Yu Huang; Siyuan Li; Rui Yao; Hanqian Li; Hanyu Zhang; Jungang Li; Jian Chen

cs.CLSep 26, 2025

KnowMT-Bench: Benchmarking Knowledge-Intensive Long-Form Question Answering in Multi-Turn Dialogues

Authors: Junhao Chen, Yu Huang, Siyuan Li, Rui Yao, Hanqian Li, Hanyu Zhang, Jungang Li, Jian Chen,

Organizations: Hong Kong University of Science and Technology (Guangzhou) · The University of Osaka · Hong Kong University of Science and Technology

Abstract

Multi-Turn Long-Form Question Answering (MT-LFQA) is a key application paradigm of Large Language Models (LLMs) in knowledge-intensive domains. However, existing benchmarks are limited to single-turn dialogue, while multi-turn dialogue benchmarks typically assess other orthogonal capabilities rather than knowledge-intensive factuality. To bridge this critical gap, we introduce KnowMT-Bench, a new benchmark designed to systematically evaluate MT-LFQA for LLMs across knowledge-intensive fields, including medicine, finance, and law. To faithfully assess the model's real-world performance, KnowMT-Bench employs a dynamic evaluation setting where models generate their own multi-turn dialogue histories given logically progressive question sequences. Our experiments on a diverse suite of LLMs show a clear degradation in both factual capability and information delivery efficiency in multi-turn dialogue. We further probe the underlying causes and investigate mitigation strategies spanning domain-specific finetuning, active context denoising, and RAG. The results show domain-dependent gains from finetuning, while active context denoising and RAG more consistently mitigate these issues, with RAG notably reversing this performance degradation. These findings underscore the importance of our benchmark for evaluating LLMs' conversational factual capabilities in real-world applications. Code and data are publicly available.\footnote{https://github.com/Sven-LI-sankyuu/KnowMT-Bench}

Figures & tables

Benchmark	# Avg. Tokens	MT	OD
CoQA ( Reddy et al., 2019 )	2.52	✓	✗
QASA ( Lee et al., 2023 )	50	✗	✗
K-QA ( Manes et al., 2024 )	119.89	✗	✓
MedLFQA ( Jeong et al., 2024 )	132.86	✗	✓
FintextQA ( Chen et al., 2024b )	75 *	✗	✓
KnowMT-Bench (ours)	95.15	✓	✓

Table 1: Comparison of KnowMT-Bench with existing QA benchmarks. # Avg. Tokens refer to the token counts of the ground truth answer, computed by the GPT-4o tokenizer. * : FintextQA is partially open-sourced; we report the 75-word value from their paper. MT denotes Multi-Turn, and OD refers to Open-Domain.

Figure 1: Single- vs. Multi-Turn LFQA on Llama-3.3-70B-Instruct; correct , irrelevant , and faulty statements.

Figure 2: Overview of the two-stage natural language inference-based evaluation. Here, GTA refers to G round T ruth A nswer, and the detailed prompts for evaluation are provided in Appendix J .

Panel of Figure 3 — Figure 3: LLMs’ performance on factual capability and information delivery efficiency.

Panel of Figure 4 — Figure 4: Effects of dialogue length on multi-turn performance across dialogue turns and context length.

Figure 5: Impact of manually annotated gold context on Qwen-2.5-7B .

Figure 6: Performance of GPT-4o under different types of controlled dialogue-history noise.

	Finance				Non-finance
Model	$\mathbf{S_{f}}$ (%) $\uparrow$	$\mathbf{S_{h}}$ (%) $\downarrow$	$\mathbf{D_{f}}$ $\downarrow$	$\mathbf{D_{h}}$ $\uparrow$	$\mathbf{S_{f}}$ (%) $\uparrow$	$\mathbf{S_{h}}$ (%) $\downarrow$	$\mathbf{D_{f}}$ $\downarrow$	$\mathbf{D_{h}}$ $\uparrow$
Qwen-2.5-7B	16.50	2.29	376.19	616.21	21.87	4.26	361.73	575.38
	14.60	2.12	578.12	888.26	19.52	4.84	524.34	815.99
Fin-R1	14.51	2.19	568.49	895.72	22.52	4.76	487.90	831.48
	13.39	3.53	511.55	825.01	20.08	4.85	467.26	802.78
	Medical				Non-medical

Table 2: Results across different domains. Underlined values denote improvements over the base model under the same domain and turn setting. White rows indicate single-turn results, while gray rows indicate multi-turn results.

w/o RAG
Setting	$\mathbf{S_{f}}$ (%) $\uparrow$	$\mathbf{S_{h}}$ (%) $\downarrow$	$\mathbf{D_{f}}$ $\downarrow$	$\mathbf{D_{h}}$ $\uparrow$
Original	19.86	3.52	367.15	590.67
Original	17.68	3.82	544.48	843.06
w/ RAG
Base	40.77	1.87	193.70	575.75
Base	40.19	2.27	304.82	888.61

Table 3: Comparison of w/o versus w/ RAG on Qwen-2.5-7B . White rows indicate single-turn settings , while gray rows indicate multi-turn settings . Bold numbers highlight the best results of the multi-turn RAG setting. Underlined values denote improvements over the baseline under the same turn setting.

Strategy	$\mathbf{S_{f}}$ (%) $\uparrow$	$\mathbf{S_{h}}$ (%) $\downarrow$	$\mathbf{D_{f}}$ $\downarrow$	$\mathbf{D_{h}}$ $\uparrow$
Baselines
Original	22.91	2.77	365	683
Original	22.05	3.12	434	743
Interventions
+ Simple Prompt	21.89	3.05	417	764
+ Summarization	24.05	3.28	259	419

Table 4: Performance comparison of Qwen2.5-32B with prompt-based mitigation strategies. Gray and white rows indicate as above. Underlined values denote improvements to the multi-turn baseline.

Appendix figures & tables27 assets

Supplementary material from the paper’s appendix.

Appendix

Benchmark	# Avg. Length	# Avg. Turns	Multi-turn	Long-Form	Open-Domain
QuAC ( Choi et al., 2018 )	14.6 t	7.2	✓	✗	✗
CoQA ( Reddy et al., 2019 )	2.52 t	15.97	✓	✗	✗
DoQA ( Campos et al., 2020 )	12.99 t	4.48	✓	✗	✗
Doc2Dial ( Feng et al., 2020 )	21 t	12	✓	✗	✗
MultiDoc2Dial ( Feng et al., 2021 )	21.6 t	6.36	✓	✗	✗
ConvFinQA ( Chen et al., 2022 )	-	3.67	✓	✗	✗

Appendix

Table 5: Conversational QA and LFQA benchmark. Columns show average ground-truth length, average number of turns, and whether dialogs are multi-turn, answers are long-form, and the task is open-domain (✓=yes, ✗=no). Notation: t = tokens, w = words, c = Chinese characters, -=not provided in the original paper.

Figure 7: Overview of the dataset construction process

Metric	Dimension	Meaning	Direction
$\mathbf{S_{f}}$	Factuality	Factual F1 over supported facts and generated claims	Higher is better
$\mathbf{S_{h}}$	Reliability	Hallucination F1 over contradicted facts and claims	Lower is better
$\mathbf{D_{f}}$	Efficiency	Token cost per correctly entailed fact	Lower is better
$\mathbf{D_{h}}$	Efficiency	Token cost per contradicted fact	Higher means sparser errors

Appendix

Table 6: Summary of the main evaluation metrics.

Panel of Figure 8 — Figure 8: Financial topics and turn-count distribution.

Model	Accuracy	F1	Precision	Recall
Qwen-2.5-14B	0.83	0.84	0.84	0.83
GPT-4o	0.81	0.82	0.84	0.81
DeepSeek-V3	0.76	0.78	0.84	0.76

Appendix

Table 7: Performance comparison on the Factual Consistency Judgment stage.

Model	Acc.	F1	Prec.	Rec.	SMAPE(%)	OR(%)
DS-V3	0.85	0.85	0.87	0.85	13.12	6.34
L3-8B	0.84	0.85	0.86	0.84	20.66	2.54
GPT-4o	0.83	0.83	0.83	0.83	19.70	7.52
Q2.5-14B	0.81	0.81	0.81	0.81	18.10	5.90

Appendix

Table 8: Performance breakdown per model, as judged by the Qwen-2.5-14B evaluator. OR refers to Omission Rate.

Model	DS-V3	L3-8B	GPT-4o	Q2.5-14B
DS-V3	-	.761 ns	.445 ns	.183 ns
L3-8B	.761 ns	-	.674 ns	.347 ns
GPT-4o	.445 ns	.674 ns	-	.602 ns
Q2.5-14B	.183 ns	.347 ns	.602 ns	-

Appendix

Table 9: Pairwise significance tests on model accuracy for evaluator impartiality. We report p-values from Mann-Whitney U tests. Note: ns denotes p

\geq

0.01.

Model	DS-V3	L3-8B	GPT-4o	Q2.5-14B
DS-V3	-	.477 ns	.237 ns	.104 ns
L3-8B	.477 ns	-	.086 ns	.030 ns
GPT-4o	.237 ns	.086 ns	-	.841 ns
Q2.5-14B	.104 ns	.030 ns	.841 ns	-

Appendix

Table 10: Pairwise significance tests (p-values from Mann-Whitney U tests) on model SMAPE for Decomposer. Note: ns denotes p

\geq

0.01.

Figure 9: Human validation of our evaluation pipeline. ( left ) Alignment between LLM-based and human-annotated atomic decompositions. ( right ) Alignment between LLM-based and human annotations for factual consistency judgments.

OR	Model (Mode)	$\mathbf{S_{f}}$	GPT-4o Eval (Rank)	GPT-4o Decomp (Rank)
1	GPT-4o (Single)	28.99%	38.06% (1)	29.54% (1)
2	GPT-4o (Multi)	28.08%	37.86% (2)	29.24% (2)
3	DeepSeek-V3 (Single)	24.33%	33.82% (3)	24.51% (3)
4	Qwen-2.5-72B (Single)	24.30%	32.37% (5)	24.47% (4)
5	Llama-3.3-70B (Single)	24.06%	32.95% (4)	24.41% (5)
6	Llama-3.3-70B (Multi)	21.37%	29.26% (6)	21.50% (6)

Appendix

Table 11: Model Ranking (

\mathbf{S_{f}}

) Consistency across Evaluators and Decomposer Variants. OR refers to Original Rank.

Model	Setting	Configuration	$\mathbf{S_{f}}$ (%)	$\mathbf{S_{h}}$ (%)	$\mathbf{D_{f}}$	$\mathbf{D_{h}}$
GPT-4o	Single	Original	28.99	2.29	226	505
GPT-4o	Single	w/ GPT-4o Eval	38.06	3.36	179	434
GPT-4o	Single	w/ GPT-4o Decomp	29.54	2.55	226	507
GPT-4o	Multi	Original	28.08	2.47	255	553
GPT-4o	Multi	w/ GPT-4o Eval	37.86	3.11	214	532
GPT-4o	Multi	w/ GPT-4o Decomp	29.24	2.26	253	495

Appendix

Table 12: Cross-Model Validation – Detailed Metrics for Evaluator and Decomposer Variants.

Dataset	Logical	Goal	Transition	Overall	Confusion
	Coherence	Clarity	Naturalness	Mean	Factor (CF)
StructFlowBench ( Li et al., 2025 )	4.78	4.46	4.18	4.47	0.82
MT-Bench-101 ( Bai et al., 2024 )	4.50	4.52	4.50	4.51	0.74
KnowMT-Bench (Ours)	4.72	4.46	4.12	4.43	0.78

Appendix

Table 13: Results of the Complex Scenario Suitability Study. We compare KnowMT-Bench against StructFlowBench and MT-Bench-101 across three qualitative dimensions and the composite Confusion Factor (CF). All scores are on a 1-5 scale except for CF, which is a ratio.

	Factuality			Hallucination			Efficiency
Model	$\mathbf{R_{f}}$ (%) $\uparrow$	$\mathbf{P_{f}}$ (%) $\uparrow$	$\mathbf{S_{f}}$ (%) $\uparrow$	$\mathbf{R_{m}}$ (%) $\downarrow$	$\mathbf{P_{fc}}$ (%) $\downarrow$	$\mathbf{S_{h}}$ (%) $\downarrow$	$\mathbf{D_{f}}$ $\downarrow$	$\mathbf{D_{h}}$ $\uparrow$
Single-turn
Gemini-2.5-Pro	56.95	18.10	23.40	11.30	3.33	3.03	674	1614
Gemini-3.1-Pro	57.37	24.27	28.24	7.71	3.39	2.37	280	706
GPT-4o	41.94	32.22	28.99	6.97	3.47	2.29	226	505
GPT-4o-mini	39.49	30.72	26.89	6.75	3.99	2.38	255	545

Appendix

Table 14: Results for all models. Sections explicitly indicate Single-turn , Multi-turn , and the separate Diff block (Multi vs. Single). Bold values mark, within each block , the best score per column (for

\uparrow

higher is better; for

\downarrow

lower is better). Green in Diff indicates improvement and red indicates degradation.

Figure 10: Comparison between standard multi-turn and a concatenated-history setting.

Figure 11: Performance degradation under controlled noise, comparing standard multi-turn and concatenated-history settings.

Figure 12: Bidirectional history substitution experiment between a strong model (GPT-4o) and a weaker model (Qwen-2.5-7B). It also underscores that our dynamic evaluation setting more faithfully captures real model performance.

Figure 13: Model performance mapped onto the four-quadrant framework. The x-axis captures overall factuality, derived from the correlation between

\mathbf{S_{f}}

and

\mathbf{S_{h}}

. The y-axis measures performance against the efficiency trade-off baseline, derived from the correlation between

\mathbf{D_{f}}

and

\mathbf{D_{h}}

	N-gram Based Metrics				Our Proposed Core Metrics
Model	$\mathbf{BLEU-4}$ $\uparrow$	$\mathbf{R-1}$ $\uparrow$	$\mathbf{R-2}$ $\uparrow$	$\mathbf{R-L}$ $\uparrow$	$\mathbf{S_{f}}$ (%) $\uparrow$	$\mathbf{S_{h}}$ (%) $\downarrow$	$\mathbf{D_{f}}$ $\downarrow$	$\mathbf{D_{h}}$ $\uparrow$
Single-turn
Gemini-2.5-Pro	1.51	13.33	3.72	12.43	23.40	3.03	674	1614
Gemini-3.1-Pro	3.76	20.44	6.87	18.65	28.24	2.37	280	706
GPT-4o	5.43	25.98	8.65	23.53	28.99	2.29	226	505
GPT-4o-mini	4.79	24.98	7.89	22.55	26.89	2.38	255	545

Appendix

Table 15: Comparison of model performance using traditional n-gram-based metrics and our proposed core metrics. This table provides the detailed numerical results that complement the scatter plot analysis. Green indicates improvement, while red indicates degradation.

Metric	Correct & significant	Correct & not significant	Opposite & significant	Opposite & not significant
$S_{f}$	9/13	3/13	0/13	1/13
$S_{h}$	3/13	8/13	0/13	2/13
$D_{f}$	11/13	1/13	0/13	1/13
$D_{h}$	10/13	0/13	0/13	3/13

Appendix

Table 16: Overall results of the Wilcoxon signed-rank tests for Figure 3. Here, “Correct” means that the observed difference between Multi-turn and Single-turn is aligned with the pre-defined expected direction: Multi-turn

<

Single-turn for

\mathbf{S_{f}}

(degradation in factuality) and Multi-turn

>

Single-turn for

\mathbf{S_{h}}

and

\mathbf{D_{f}}

(degradation in hallucination and efficiency for correct facts) as well as for

\mathbf{D_{h}}

(an improvement, i.e., larger token cost per hallucinated fact). “Significant” means one-sided

p<0.05

. Most models move in the expected direction and are significant on

\mathbf{S_{f}}

\mathbf{D_{f}}

, and

\mathbf{D_{h}}

, whereas for

\mathbf{S_{h}}

the direction is usually correct but the significance is weaker, mainly because

\mathbf{S_{h}}

values are close to 0 for many models under both settings.

Model	MT mean	ST mean	Diff (M-S)	Fit expectation	$p$ -value (one-sided)	Significance
Gemini-2.5-Pro	0.22	0.23	-0.01	Yes	0.02
Gemini-3.1-Pro	0.28	0.28	-0.00	Yes	0.33
GPT-4o	0.28	0.29	-0.01	Yes	0.08
GPT-4o-mini	0.26	0.27	-0.01	Yes	0.02
DeepSeek-R1	0.20	0.22	-0.02	Yes	0.01	**
DeepSeek-V3	0.19	0.24	-0.05	Yes	0.00

Appendix

Table 17: Wilcoxon signed-rank test results for

\mathbf{S_{f}}

(one-sided). The expected direction is Multi-turn

<

Single-turn. The column “Aligned with expectation?” indicates whether the mean difference between Multi-turn and Single-turn follows this expected direction. The “Significance” column uses *, ** and *** to denote one-sided

p<0.05

p<0.01

and

p<0.001

, respectively. MT and ST denote multi-turn and single-turn.

Model	MT mean	ST mean	Diff (M-S)	Fit expectation	$p$ -value (one-sided)
Gemini-2.5-Pro	0.04	0.03	0.01	Yes	0.00
Gemini-3.1-Pro	0.02	0.02	-0.00	No	0.54
GPT-4o	0.02	0.02	0.00	Yes	0.24
GPT-4o-mini	0.03	0.02	0.00	Yes	0.08
DeepSeek-R1	0.04	0.04	0.00	Yes	0.17
DeepSeek-V3	0.03	0.03	0.00	Yes	0.22

Appendix

Table 18: Wilcoxon signed-rank test results for

\mathbf{S_{h}}

(one-sided). The expected direction is Multi-turn

>

p<0.05

p<0.01

and

p<0.001

, respectively. MT and ST denote multi-turn and single-turn.

Model	MT mean	ST mean	Diff (M-S)	Fit expectation	$p$ -value (one-sided)	Significance
Gemini-2.5-Pro	697.13	673.89	23.24	Yes	1.00
Gemini-3.1-Pro	302.54	280.41	22.12	Yes	0.00
GPT-4o	255.22	226.20	29.02	Yes	0.00
GPT-4o-mini	343.61	255.35	88.26	Yes	0.00
DeepSeek-R1	626.28	604.11	22.17	Yes	0.01	**
DeepSeek-V3	443.92	339.39	104.52	Yes	0.00

Appendix

Table 19: Wilcoxon signed-rank test results for

\mathbf{D_{f}}

(one-sided). The expected direction is Multi-turn

>

Single-turn, corresponding to a degradation in efficiency for correct facts (more tokens per correct fact). The column “Aligned with expectation?” indicates whether the mean difference between Multi-turn and Single-turn follows this expected direction. The “Significance” column uses *, ** and *** to denote one-sided

p<0.05

p<0.01

and

p<0.001

, respectively. MT and ST denote multi-turn and single-turn.

Model	MT mean	ST mean	Diff (M-S)	Fit expectation	$p$ -value (one-sided)
Gemini-2.5-Pro	1403.42	1613.53	-210.10	No	1.00
Gemini-3.1-Pro	733.68	705.55	28.13	Yes	0.00
GPT-4o	552.83	504.97	47.86	Yes	0.00
GPT-4o-mini	782.07	545.17	236.90	Yes	0.00
DeepSeek-R1	1175.41	1363.99	-188.58	No	1.00
DeepSeek-V3	877.23	737.92	139.31	Yes	0.00

Appendix

Table 20: Wilcoxon signed-rank test results for

\mathbf{D_{h}}

(one-sided). The expected direction is Multi-turn

>

Single-turn, which corresponds to a larger token cost per contradicted fact and hence sparser hallucinations per token (an improvement from a robustness perspective). The column “Aligned with expectation?” indicates whether the mean difference between Multi-turn and Single-turn follows this expected direction. The “Significance” column uses *, ** and *** to denote one-sided

p<0.05

p<0.01

and

p<0.001

, respectively. MT and ST denote multi-turn and single-turn.

Setting	$\mathbf{S_{f}}$ (%) $\uparrow$	$\mathbf{S_{h}}$ (%) $\downarrow$	$\mathbf{D_{f}}$ $\downarrow$	$\mathbf{D_{h}}$ $\uparrow$
QwQ-32B	21.34	2.76	526	918
QwQ-32B-2048	20.24	3.82	699	1545
QwQ-32B	19.31	3.91	468	810
QwQ-32B-2048	17.61	3.92	807	1532

Appendix

Table 21: Performance comparison of QwQ-32B and QwQ-32B-2048. White and gray rows indicate the single-turn and multi-turn setting, respectively.

Panel of Figure 14 — Figure 14: Correlation between $\mathbf{S_{f}}$ and traditional lexical metrics. While a positive trend is observed, models optimized for CoT reasoning tend to be undervalued by n-gram metrics, potentially due to subtle stylistic differences in their final answers.

Figure 15: Visualization of contextual token importance. Green indicates tokens that positively contribute to the model’s response, while red indicates tokens with a negative influence.

Setting	Mode	$S_{f}$ (%)	$S_{h}$ (%)	$D_{f}$	$D_{h}$
Original	Single-turn	19.86	3.52	367	591
Topic Drift	Single-turn	19.50	3.43	367	590
Original	Multi-turn	17.68	3.82	544	843
Topic Drift	Multi-turn	20.48	4.81	536	833

Appendix

Table 22: Impact of Topic Drift on Qwen-2.5-7B. We compare the standard setting against a "Topic Drift" setting where an unrelated question is prepended to the context.

Knowledge-Intensive QA

Knowledge-intensive question answering combines language models with external knowledge to answer factual, multi-step, and domain-specific questions.

See all 73 papers

Mar 6, 2026

MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?

Large Language Models (LLMs) have demonstrated impressive capabilities across various specialist domains and have been integrated into high-stakes areas such as medicine. However, as existing medical-related benchmarks rarely stress-test the long-context memory, interference robustness, and safety defense required in practice. To bridge this gap, we introduce MedMT-Bench, a challenging medical multi-turn instruction following benchmark that simulates the entire diagnosis and treatment process. We construct the benchmark via scene-by-scene data synthesis refined by manual expert editing, yielding 400 test cases that are highly consistent with real-world application scenarios. Each test case has an average of 22 rounds (maximum of 52 rounds), covering 5 types of difficult instruction following issues. For evaluation, we propose an LLM-as-judge protocol with instance-level rubrics and atomic test points, validated against expert annotations with a human-LLM agreement of 91.94%. We test 17 frontier models, all of which underperform on MedMT-Bench (overall accuracy below 60.00%), with the best model reaching 59.75%. MedMT-Bench can be an essential tool for driving future research towards safer and more reliable medical AI. The benchmark is available in https://openreview.net/attachment?id=aKyBCsPOHB&name=supplementary_material

Lin Yang, Yuancheng Yang, Xu Wang +2

ByteDance

Long-Context Language Model Evaluation Clinical Language Model Evaluation Multi-Turn Dialogue Evaluation LLM Safety Benchmarks

Ask PDF

Jul 11, 2026

Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

Evaluating large language models (LLMs) as multi-turn conversational partners requires probing capabilities that single-turn benchmarks miss: persona consistency, evolving intent tracking, emotional dynamics, and goal completion. We introduce EYT-Bench, a human-centered benchmark built around a three-party decoupled design: a persona-grounded user simulator, a target model that separates intent perception from response generation, and an independent third-party LLM judge with optional multi-judge ensembling. Personas are sampled from public human-curated corpora, Nemotron-Personas-USA and PersonaMem-v2, rather than synthesized, reducing LLM-induced persona bias. EYT-Bench also introduces two trajectory-level metrics: embedding-based intent drift and final-intent completion rate (FICR), inspired by tau-bench. In a 17-target x 200-dialogue evaluation, EYT-Bench reveals four findings: (i) state-of-the-art closed- and open-source models are statistically close on subjective dimensions (empathy / persona / anthropomorphism vary within <= 0.3), but differ by up to 9x on objective intent tracking; (ii) reasoning ("thinking on") sharply improves objective tracking on long-context personas (+0.47-0.50 latent-intent accuracy on Gemma-4) while leaving subjective scores nearly unchanged; (iii) persona format dominates trajectory spread, with FICR saturating above 0.95 on Nemotron-USA but spreading from 0.53 to 0.88 on PersonaMem-v2; and (iv) the warm-up effect is robust on 16/17 models (one outlier, GPT-5.5, reverses the effect), with stable rankings across alpha in [0.05, 0.15]. A cross-judge ablation using deepseek-v4-pro confirms that target rankings and final-intent satisfaction are preserved across judges.

Jinglan Gong, Jiefan Lu, Hewei Guo +5

SenseTime Research · University of Science and Technology of China · Tsinghua University

Language Model Generation Evaluation Persona Consistency Intent Classification Multi-Turn Dialogue Evaluation Multi-Turn Dialogue Management

Ask PDF

May 26, 2026

Beyond Questions: Evaluating the Stated Knowledge of LLMs

Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge benchmarks typically rely on predefined questions (e.g., "What is the birth date of M.L. King?"), evaluating only knowledge that benchmark designers explicitly choose to query, a problematic availability bias. In this paper, we introduce open knowledge evaluation, a new paradigm for benchmarking the stated knowledge of LLMs. Instead of asking narrow questions, it evaluates models on the knowledge they choose to surface in response to open-ended elicitation prompts (e.g., "Tell me everything you know about M.L. King"). This shifts the focus from predefined answer retrieval toward characterizing the knowledge models naturally express. We instantiate this paradigm with BeQu (Beyond Questions), a benchmark of 10,000 entities paired with reference corpora for statement verification. Using BeQu, we evaluate a broad range of language models and analyze the effects of reasoning effort, model scale, prompt format, and knowledge domain. Data and leaderboard are available on this work's GitHub repository and at ANONYMIZED.

Luca Giordano, Simon Razniewski

ScaDS.AI Dresden/Leipzig & TU Dresden, Germany

LLM Evaluation Factual Knowledge in Language Models

Ask PDF

Abstract

Knowledge-Intensive QA

Explore similar work

MedMT-Bench: Can LLMs Memorize and Understand Long Multi-Turn Conversations in Medical Scenarios?

Enjoy Your Talk: A Human-Centered Benchmark for Multi-Turn Dialogue with Decoupled User Simulation, Target Modeling, and Judging

Beyond Questions: Evaluating the Stated Knowledge of LLMs