ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models

Mayank Kulkarni; Rakesh Vaideeswaran

cs.AIOct 13, 2026

ListQA: A Benchmark for Evaluating List-Formatted Factual Knowledge Retrieval in Large Language Models

Authors: Mayank Kulkarni, Rakesh Vaideeswaran

Organizations: Amazon AGI

Abstract

Many real-world questions demand not a single fact but an organized collection of them, yet existing factual knowledge benchmarks almost exclusively target single answer retrieval. We introduce ListQA, a benchmark of 9,045 human-curated, cross-validated questions (audited error rate of 1.7%) that require LLMs to recall multiple facts and compose them into structured lists of 3-10 elements. Questions range from flat lists ("Which NCAA teams went undefeated between 2000 and 2024?") to hierarchical lists with sub-attributes ("...and what was their record and result?"), spanning eight categories across 60+ countries. An evergreen design (explicit temporal constraints or historically immutable facts) keeps ground truth valid without periodic updates. For evaluation, we frame element matching as a linear sum assignment problem: optimal bipartite matching paired with LLM-based semantic grading jointly captures factual accuracy, hallucination, and format compliance. We benchmark 34 models (8 families, 42 configurations across standard and thinking enabled inference) and find: the best standard-inference model scores below 37% LLM-Judge-F1, multi-faceted questions are harder across the board, and providing the source Wikipedia page lifts scores dramatically, confirming that models can organize facts but struggle to retrieve them from parameters alone. Extended analyses cover base vs. instruction-tuned checkpoints and token efficiency. We release the dataset and evaluation code.

Figures & tables

Single-answer factual QA
Benchmark	ME	SO	EG	K+IF
TriviaQA [ 3 ]	✗	✗	✗	✗
Natural Questions [ 4 ]	✗	✗	✗	✗
PopQA [ 5 ]	✗	✗	✗	✗
SimpleQA [ 6 ]	✗	✗	✓	✗
FACT-Bench [ 7 ]	✗	✗	✗	✗

Table 1: Comparison of ListQA with representative benchmarks across the following dimensions: ME : multi-element retrieval. SO : structured output. EG : evergreen design. K+IF : knowledge + instruction following. Shading indicates partial coverage.

Panel of Figure 2 — Figure 2: Overview of the ListQA dataset. Top left: Distribution across 8 topical categories, split by SimpleListQA and ComplexListQA . Top center: Geographic coverage spanning 60+ countries across 6 continents. Top right: Temporal distribution; bars after the dashed line (2017–2024) reflect the deliberate recency emphasis. Bottom: Key dataset statistics.

Figure 3: Overview of the LLM-Judge-F1 evaluation pipeline. Given a reference list and model prediction, (1) individual list elements are extracted via regex parsing, (2) pairwise ROUGE-1 scores form a cost matrix for optimal one-to-one element assignment via the Jonker-Volgenant algorithm, and (3) each matched pair is independently graded by an LLM judge as CORRECT or INCORRECT. Unmatched predicted elements are automatically graded INCORRECT. Precision and recall are computed over elements judged CORRECT.

Figure 4: (a) LLM-Judge-F1 scores across inference settings. Hatched bars show LLM-Judge-F1-Ordered , a stricter metric penalizing incorrect sequencing for select samples (details in App. J ). (b) Improvement from thinking over standard inference. Thinking universally improves factual accuracy, but its effect on Format-Correctness may be architecture and mode family-dependent. Exact scores and thinking levels are reported in App. K .

Closed-source
			Parametric			Oracle RAG
Model	Context	Src. budget	All	S	C	All	S	C	$\Delta$ All
Claude Opus 4.6	Full		34.6	43.0	22.4	84.7	86.0	82.8	+50.1
Claude Sonnet 4.6	Full		28.0	35.5	16.9	83.0	85.0	79.9	+55.0
GPT-5.4 mini	Full		27.0	35.2	15.1	75.8	79.8	70.0	+48.8
Gemini 2.5 Flash Lite	Full		20.4	27.2	10.4	70.6	74.8	64.3	+50.2

Table 3: Oracle RAG vs. parametric-only LLM-Judge-F1. All = Overall, S = SimpleListQA , C = ComplexListQA . Context / Src. budget : context window (tokens) and maximum number of source characters provided to open-weight models (App. L ); closed-source models received the full page.

Appendix figures & tables31 assets

Supplementary material from the paper’s appendix.

Appendix

Validation outcomes
No issues found	3,957	39%
Fixed	5,088	51%
Discarded	987	10%
Error types identified
Typo / grammar	3,565	44%
Vague prompt	2,502	31%

Appendix

Table 4: Validation round outcomes and error type distribution. The same sample may be associated with one or more error types

Global (15)		Europe		Asia		Africa		Australasia		S. America
Spain	120	Albania	2	Cambodia	6	Angola	4	New Zealand	15	General	21
Italy	120	Lithuania	1	Sri Lanka	1	Kenya	5	Papua N.G.	3	Chile	9
Germany	119	Andorra	1	Maldives	4	Seychelles	2	Cook Islands	1	Colombia	14
Portugal	120	Austria	3	Thailand	12	Madagascar	3	General	1	Bolivia	3
France	120	Greece	1	Uzbekistan	2	Algeria	7			Argentina	6
India	120	Spain	1	S. Korea	10	Ethiopia	5			Venezuela	4

Appendix

Table 5: Geographic distribution of samples across continents and countries (excl. USA

Figure 5: Left: Distribution of list elements per answer. Most answers contain 3 elements; the distribution is right-skewed. Right: Questions per unique Wikipedia table. Over half of tables contribute a single question, with a long tail extending to 28.

Component	Purpose
\n	Anchors to a newline boundary, preventing false matches on
	numbered references appearing mid-sentence.
\d+	Matches one or more digits, accommodating lists of arbitrary
	length (1–9, 10–99, 100+) without imposing an upper bound.
.	Matches the literal period delimiter, distinguishing numbered
	list items from other digit-newline patterns.

Appendix

Table 6: Component analysis of the top-level list element regex

\texttt{R}_{1}

Component	Purpose
ˆ	With re.MULTILINE , anchors to line start, matching only
	bullets that begin a line.
*	Matches the literal asterisk bullet marker followed by a space.
(.*?) (1st)	Non-greedy capture of the category/key name, stopping at the
	first colon-space delimiter.
:	Literal colon-space separator between category and value.

Appendix

Table 7: Component analysis of the sub-element extraction regex

\texttt{R}_{2}

	LLM-Judge-F1 $\uparrow$			Recall $\uparrow$			Precision $\uparrow$			Format $\uparrow$			ROUGE-F1 $\uparrow$
Model	All	S	C	All	S	C	All	S	C	All	S	C	All	S	C
Gemini 3.1 Pro ∗ †	60.7	67.7	50.3	59.6	59.9	59.4	63.7	66.5	61.8	97.7	98.5	97.2	68.5	68.8	68.3
Gemini 3 Flash ∗ †	56.4	63.7	45.8	59.7	59.4	60.2	57.8	54.9	62.1	91.9	87.3	98.8	63.7	62.6	65.5
Gemini 3.1 Flash Lite ∗ †	40.1	48.4	27.9	44.8	44.1	45.7	38.1	28.5	52.1	39.9	3.8	92.7	38.8	33.2	47.0
Gemini 2.5 Flash †	38.3	47.1	25.4	43.0	42.2	44.2	52.3	45.9	61.8	70.3	86.8	46.2	43.3	47.8	36.8
Claude Opus 4.6 †	37.9	46.2	25.7	43.2	42.3	44.5	50.8	45.9	58.0	66.8	69.0	63.7	38.6	40.6	35.6

Appendix

Table 8: Full results across all five metrics. All = Overall, S = SimpleListQA , C = ComplexListQA . [ † = thinking-enabled inference], [ ∗ = Preview Models]. All values are percentages. Models sorted by overall LLM-Judge-F1 . Recall: Relaxed-Recall; Precision: Relaxed-Precision; Format: Format-Correctness

Model Family	Models	Infrastructure
GPT	GPT-5.4
	GPT-5.4 mini	OpenAI API
	GPT-5.4 nano
Claude	Claude Opus 4.7, 4.6, 4.5
	Claude Sonnet 4.6, 4.5	Amazon Bedrock
	Claude Haiku 4.5

Appendix

Table 9: Inference infrastructure for each evaluated model.

Model	Setting	Perfect	Wrong Facet	Too Few Elem.	No List	Mixed Facet
GPT-5.4 mini	Std	63.9	21.8	14.0	0.0	0.2
GPT-5.4 mini	Think	60.1 $\downarrow$	17.8 $\downarrow$	21.7 $\uparrow$	0.0	0.3 $\uparrow$
Gemini 2.5 Flash	Std	76.2	9.3	13.4	0.6	0.5
Gemini 2.5 Flash	Think	70.2 $\downarrow$	18.5 $\uparrow$	9.8 $\downarrow$	1.2 $\uparrow$	0.2 $\downarrow$
Claude Opus 4.6	Std	77.4	7.5	12.1	1.5	1.5
Claude Opus 4.6	Think	66.7 $\downarrow$	16.2 $\uparrow$	12.4 $\uparrow$	4.4 $\uparrow$	0.2 $\downarrow$

Appendix

Table 10: Format error distribution (% of 9,045 samples) under standard (Std) vs. thinking-enabled (Think) inference for models from Fig. 4 (b). Perfect indicates accurate format compliance.

		LLM-Judge-F1		Blended-F1
Model	Level	Original Prompt	Order-Aware Prompt	Order-Aware Prompt
Gemini 3.1 Pro †	Low	60.7	61.2	59.4
Gemini 3.1 Flash Lite †	Minimal	40.1	40.2	38.4
GPT-5.4-mini †	Medium	36.2	36.6	35.0

Appendix

Table 11: F1 metrics on the full dataset (9,045 samples) under different prompt and scoring configurations. † Thinking enabled.

	Overall		Single-faceted		Multi-faceted
Model	Unord.	Ord.	Unord.	Ord.	Unord.	Ord.
Gemini 3.1 Pro †	55.5	43.1	62.7	47.8	45.2	36.4
Gemini 3.1 Flash Lite †	35.5	23.6	43.1	27.9	24.6	17.3
GPT-5.4-mini †	32.8	22.1	41.2	26.4	20.7	16.0
Claude Opus 4.6 †	36.2	26.0	43.7	30.7	25.5	19.3

Appendix

Table 12: LLM-Judge-F1 on the ordered subset only (1,319 samples) under the order-aware prompt. Unord. : standard LLM-Judge-F1 (order-agnostic). Ord. : LLM-Judge-F1-Ordered (penalizes wrong sequence). Results are shown for single-faceted and multi-faceted questions within the ordered subset . † Thinking enabled.

		LLM-Judge-F1 $\uparrow$						Format $\uparrow$
		Overall		S		C
Model	Think Level	Score	$\Delta$	Score	$\Delta$	Score	$\Delta$	Score	$\Delta$
Gemini 3.1 Pro †	Low	60.7	—	67.7	—	50.3	—	97.7	—
Gemini 3 Flash †	High	56.4	—	63.7	—	45.8	—	91.9	—
Gemini 3.1 Flash Lite †	Minimal	40.1	—	48.4	—	27.9	—	39.9	—
Paired comparisons (same model, thinking vs. standard)

Appendix

Table 13: Effect of thinking-enabled inference on ListQA .

\Delta

denotes the change relative to the same model under standard inference (results in Tab. 2 ). Models with paired standard-inference baselines are grouped below the line.

Figure 6: Left: LLM-Judge-F1 on timeless vs. time-constrained questions. Every model scores higher on timeless questions. Right: The gap (timeless

-

time-constrained), sorted by magnitude.

Figure 7: LLM-Judge-F1 on questions with exactly 3 reference elements vs. 4 or more elements in the reference answer. Every model performs better on shorter lists, though the gap is modest.

Figure 8: Left: LLM-Judge-F1 per category (sample counts in parentheses). Right: Weighted contribution to overall LLM-Judge-F1 score (category proportion in parentheses). The radar shape is consistent across models in both views, differing primarily in scale.

Figure 9: Left: LLM-Judge-F1 per continent (sample counts in parentheses). Center: Weighted contribution to overall score (continent proportion in parentheses). North America dominates the overall score due to its 78% share of the dataset. Right: Weighted contribution excluding North America (renormalized proportions).

		LLM-Judge-F1 $\uparrow$			Format $\uparrow$
Model	Setting	All	S	C	All	S	C
Gemma 4 31B	Base	9.1	12.6	4.0	51.7	51.7	51.8
	IT	16.0	22.0	7.3	54.3	45.8	66.7
	IT + Think	18.7	25.3	9.0	74.9	73.9	76.5
Qwen 3.5 9B	Base	4.9	7.2	1.5	55.9	54.6	57.8
	IT	8.3	12.0	2.9	62.0	65.5	56.8

Appendix

Table 14: Base vs. instruction-tuned (IT) performance. Base models are evaluated few-shot; IT models are evaluated zero-shot, with and without thinking enabled at inference. All values are percentages. All = Overall, S = SimpleListQA , C = ComplexListQA .

Model	Setting	Perfect	Wrong Facet	Too Few	No List	Mixed
Gemma 4 31B	Base	43.6	23.3	3.6	13.9	15.6
Gemma 4 31B	IT	53.9	14.4	29.0	2.0	0.7
Qwen 3.5 9B	Base	37.7	20.5	6.3	1.6	34.0
Qwen 3.5 9B	IT	55.4	19.8	11.0	2.8	11.0
Qwen 3.5 35B-A3B	Base	42.5	23.0	10.2	5.8	18.6
Qwen 3.5 35B-A3B	IT	49.8	26.0	20.2	0.6	3.5

Appendix

Table 15: Format error distribution (% of 9,045 samples) for Base vs. IT models.

Model	N	Elem.	Hum. P	Hum. R	Hum. F1	Judge F1	Pearson $r$	Spearman $\rho$
GPT-5.4	200	1,077	0.346	0.372	0.349	0.328	0.974	0.974
Qwen 3.5 122B-A10B	191	1,330	0.155	0.191	0.164	0.150	0.951	0.922
Claude Opus 4.7	180	906	0.385	0.383	0.374	0.346	0.958	0.951
Overall	571	3,313	–	–	–	–	0.965	0.956

Appendix

Table 16: Sample-level agreement between LLM-Judge-F1 (GPT-OSS-120B) and human F1, by model.

	N	Elem.	Hum. F1	Judge F1	Pearson $r$	Spearman $\rho$
By category
Entertainment	75	380	0.293	0.269	0.958	0.974
Food	73	462	0.348	0.322	0.954	0.954
Lifestyle	75	431	0.168	0.171	0.997	0.974
Movies	73	456	0.338	0.318	0.981	0.968
Places of Interest	70	410	0.351	0.298	0.924	0.895

Appendix

Table 17: Sample-level agreement between LLM-Judge-F1 and human F1, by topic category and question type.

Comparison	Agreement	Cohen’s $\kappa$
Human vs. LLM-Judge-F1	92.8%	0.793
Human vs. ROUGE-1 $\geq$ 0.5	89.3%	0.735

Appendix

Table 18: Element-level agreement with human labels (

N=3{,}313

Model	ROUGE-1	Embedding	% Identical
Claude Opus 4.6	0.346	0.3451	91.3
Claude Sonnet 4.5	0.253	0.2517	94.1

Appendix

Table 19: LLM-Judge-F1 under ROUGE-1 vs. embedding-based matching (9,045 samples). % Identical : samples with the same score under both strategies.

Judge Model	LLM-Judge-F1	Zero-scored (%)
GPT-OSS-120B	0.346	39.2
Qwen 3.5 27B	0.362	37.8
Qwen 3.5 9B	0.345	39.7
Qwen 3.5 4B	0.354	38.5

Appendix

Table 20: Basic statistics for each judge model. Scores are LLM-Judge-F1 (0–1) across 9,045 samples from Claude Opus 4.6 under standard inference.

Judge Pair	MAD $\downarrow$	Signed Diff	Spearman $\rho$ $\uparrow$
GPT-OSS vs. Qwen 3.5 27B	0.022	$-$ 0.015	0.973
GPT-OSS vs. Qwen 3.5 9B	0.024	$+$ 0.002	0.966
GPT-OSS vs. Qwen 3.5 4B	0.022	$-$ 0.008	0.972

Appendix

Table 21: Pairwise agreement between judge models. MAD: mean absolute difference (lower = more agreement). Signed Diff: mean signed difference (positive = first judge scores higher). Spearman

\rho

: rank correlation (higher = more agreement).

Model	Reported	Mean (5 runs)	Min / Max	% Identical
Claude Opus 4.6	0.3460	0.3469	0.3465 / 0.3473	86.3
Claude Sonnet 4.6	0.2800	0.2803	0.2794 / 0.2811	88.0
Claude Sonnet 4.5	0.2530	0.2531	0.2526 / 0.2536	89.9

Appendix

Table 22: Repeatability of LLM-Judge-F1 across five independent re-scoring runs with GPT-OSS-120B (temperature 0.7). Reported : score in Tab. 2 . % Identical : percentage of the 9,045 samples that receive the same score in all five runs.

Figure 10: LLM-Judge-F1 vs. average output tokens for non-thinking models. Marker color indicates model family; marker size is proportional to parameter count for open-weight models (fixed size for closed-source). Higher and further left is better.

Figure 11: LLM-Judge-F1 vs. average total generation tokens per sample for models with both thinking and non-thinking configurations. Arrows point from non-thinking (circle) to thinking (diamond) variants.

Figure 12: Comparison of thinking vs. non-thinking inference. Left: LLM-Judge-F1; annotations show the delta. Center: Average total generation tokens; annotations show the cost multiplier. Right: Cost ratio vs. LLM-Judge-F1 delta with best-fit line.

Figure 13: Reasoning vs. output token allocation for models with explicit token separation. Stacked bars show average tokens per sample; percentages indicate the fraction spent on reasoning. The black line shows LLM-Judge-F1.

Figure 14: Generation cost vs. LLM-Judge-F1 by prompt type, grouped by model family. blue = single-faceted, orange = multi-faceted. Arrows connect the same model’s single-faceted (source) and multi-faceted (target) results. Diamond markers indicate thinking-enabled models.

Figure 15: Generation cost vs. LLM-Judge-F1 by prompt type, grouped by model family. blue = single-faceted, orange = multi-faceted. Arrows connect the same model’s single-faceted (source) and multi-faceted (target) results. Diamond markers indicate thinking-enabled models.

Figure 16: Per-sample LLM-Judge-F1 distributions, sorted by overall F1 descending. Vertical markers indicate overall ( red ), single-faceted ( blue ), and multi-faceted ( orange ) mean LLM-Judge-F1 .

Factual QA

See all 25 papers

May 26, 2026

Beyond Questions: Evaluating the Stated Knowledge of LLMs

Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge benchmarks typically rely on predefined questions (e.g., "What is the birth date of M.L. King?"), evaluating only knowledge that benchmark designers explicitly choose to query, a problematic availability bias. In this paper, we introduce open knowledge evaluation, a new paradigm for benchmarking the stated knowledge of LLMs. Instead of asking narrow questions, it evaluates models on the knowledge they choose to surface in response to open-ended elicitation prompts (e.g., "Tell me everything you know about M.L. King"). This shifts the focus from predefined answer retrieval toward characterizing the knowledge models naturally express. We instantiate this paradigm with BeQu (Beyond Questions), a benchmark of 10,000 entities paired with reference corpora for statement verification. Using BeQu, we evaluate a broad range of language models and analyze the effects of reasoning effort, model scale, prompt format, and knowledge domain. Data and leaderboard are available on this work's GitHub repository and at ANONYMIZED.

Luca Giordano, Simon Razniewski

ScaDS.AI Dresden/Leipzig & TU Dresden, Germany

LLM Evaluation Factual Knowledge in Language Models

Ask PDF

Oct 21, 2025

KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

We present

\textbf{Korean SimpleQA (KoSimpleQA)}

, a benchmark for evaluating factuality in large language models (LLMs) with a focus on Korean cultural knowledge. KoSimpleQA is designed to be challenging yet easy to grade, consisting of 938 short, fact-seeking questions with unambiguous answers. We conduct a comprehensive evaluation across a diverse set of open-source LLMs of varying sizes that support Korean, and find that even the strongest model generates correct answer only 31.6% of the time, underscoring the challenging nature of KoSimpleQA. Notably, performance rankings on KoSimpleQA differ substantially from those on the English SimpleQA, highlighting the unique value of our dataset. Furthermore, we observe that reasoning helps mitigate the cross-lingual knowledge gap in LLMs, which refers to disparities in their ability to manifest knowledge across languages. KoSimpleQA can be found at https://github.com/naver-ai/KoSimpleQA.

Donghyeon Ko, Kyubyung Chae, Yeguk Jin +6

Naver Cloud · Graduate School of Data Science, Seoul National University

Multilingual Language Model Evaluation LLM Evaluation Factual Knowledge in Language Models Cultural Knowledge in Language Models

Ask PDF

Aug 24, 2026

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models

Aggregate factuality scores hide where a language model succeeds, which relations it confuses, and whether an answer survives innocuous changes to the question or decoder. We introduce PROOF, a profile-oriented benchmark for factual coverage in instruction-tuned language models. PROOF converts a frozen Wikidata snapshot into 18,486 English multiple-choice questions grounded in 11,779 semantic facts, 101 classes, 392 properties, and 14 domains. Each question has an explicit "I don't know" option, a "No correct option" control, and nine controlled formulations; 1,849 questions are no-correct-option traps. We evaluate 18 open-weight model deployments on 166,374 prompts each and separately perturb decoding on a fixed 10% subset. Base factual accuracy ranges from 6.58% to 57.59% (chance: 8.64%), yet every model has a 19.3-36.4 percentage-point spread across domains. Paired facts reveal direction-dependent retrieval, usually favoring subject-to-object queries, with the pattern reversing for one model. We find no consistent temporal penalty after exact-stratum adjustment. Neutral wording changes accuracy by as much as 26.5 percentage points, while adversarial formulations break up to 79.4% of answers that were initially correct. Direct switching to an injected false label varies from 0.04% to 27.5%, showing that accuracy loss and hint following are distinct. Selected-token confidence often indicates severe overconfidence, and decoder perturbations move accuracy by up to 15.7 percentage points and domain profiles by 16.8 points. PROOF therefore measures factual coverage as a structured, intervention-aware profile rather than a single claim about what a model "believes."

Andrei Chetvergov, Mikhail Solovev, Timofei Sivoraksha +4

Factual Knowledge in Language Models LLM Evaluation LLM Reliability Factual QA Adversarial Robustness of Language Models

Ask PDF

Abstract

Factual QA

Explore similar work

Beyond Questions: Evaluating the Stated Knowledge of LLMs

KoSimpleQA: A Korean Factuality Benchmark with an Analysis of Reasoning LLMs

PROOF: Profiling Reliability of Object-Level Facts in Large Language Models