Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Models

Pei-Fu Guo; Ya-An Tsai; Chun-Chia Hsu; Kai-Xin Chen; Yun-Da Tsai; Kai-Wei Chang; Nanyun Peng; Mi-Yen Yeh

cs.CLMar 13, 2026

Beyond Facts: Benchmarking Distributional Reading Comprehension in Large Language Models

Authors: Pei-Fu Guo, Ya-An Tsai, Chun-Chia Hsu, Kai-Xin Chen, Yun-Da Tsai, Kai-Wei Chang, Nanyun Peng, Mi-Yen Yeh,

Organizations: National Taiwan University · University of California, Los Angeles · Academia Sinica, Taiwan · NTU AI-CoRE

Abstract

While most reading comprehension benchmarks for LLMs focus on factual information that can be answered by localizing specific textual evidence, many real-world tasks require understanding distributional information, such as population-level trends and preferences expressed across collections of text. We introduce Text2DistBench, a reading comprehension benchmark for evaluating LLMs' ability to infer distributional knowledge from natural language. Built from real-world YouTube comments about movie and music entities, the benchmark provides models with entity metadata and associated comments, and requires them to answer distributional questions, such as estimating the proportions of positive and negative comments, or identifying the most and second most frequent topics discussed among viewers. To support reliable and long-term evaluation, the construction pipeline of Text2DistBench is fully automated and continuously updated to incorporate newly emerging entities over time. Experiments across multiple LLMs show that while models substantially outperform random baselines, performance varies widely across different distribution types and characteristics. These findings highlight both the capabilities and limitations of current LLMs in distributional reading comprehension and demonstrate the value of Text2DistBench as a practical and scalable testbed for future research.

Figures & tables

Figure 1: Factual vs. Distributional Knowledge.

Distribution	Question Example (Task Type: Most Frequent)
Marginals	$P(S)$ : What overall attitude do most viewers express? Positive or Negative. $P(T)$ : Which aspect of the movie is discussed most often? Actor, Storyline, Visual, or Audio.
Conditionals	$P(S\mid T)$ : Among the comments that talk about the actor, what attitude do viewers express most commonly? Positive or Negative. $P(T\mid S)$ : Among the comments that express a positive attitude, which aspect of the movie is mentioned most often? Actor, Storyline, Visual, or Audio.
Joint	$P(S,T)$ : Considering both (1) which aspect (Actor, Storyline, Visual, Audio) is being talked about and (2) whether the attitude is positive or negative. Which (aspect, sentiment) combination appears most often in the comments?

Table 1: Question Examples of different distribution type.

Question	Question Example (Distribution Type: Marginals)
Estimation	Movie Information: {meta_data} YouTube Viewer Comments: {comments} What aspects of the movie do viewers talk about? Summarize how frequently each aspect appears in the comments using percentages. Output your answer in the following format: { "Actor":"<int>%", "Storyline":"<int>%", "Visual":"<int>%", "Audio":"<int>%" }
Most Frequent	Movie Information: {meta_data} YouTube Viewer Comments: {comments} Which aspect of the movie is discussed most often? Actor, Storyline, Visual, or Audio.
Second Frequent	Movie Information: {meta_data} YouTube Viewer Comments: {comments} Which aspect of the movie is discussed the second most often? Actor, Storyline, Visual, or Audio.

Table 2: Question Examples of different task type

Figure 2: Text2DistBench Generation Pipeline. The construction process consists of four stages: (1) selecting valid opinion entities; (2) annotating comment topic and sentiment; (3) estimating empirical opinion distributions; and (4) generating distributional reading-comprehension questions.

	Estimation (1-TVD) $\uparrow$				Most Frequent (ACC) $\uparrow$				Second Frequent (ACC) $\uparrow$
Model	M	C	J	Avg	M	C	J	Avg	M	C	J	Avg
GPT-5.1	0.930	0.865	0.847	0.881	0.861	0.877	0.778	0.839	0.834	0.805	0.556	0.732
Gemini-2.5-Pro	0.903	0.869	0.805	0.859	0.805	0.886	0.722	0.804	0.778	0.835	0.667	0.760
Grok-4-Fast	0.911	0.856	0.838	0.868	0.889	0.858	0.667	0.805	0.833	0.728	0.611	0.724
Claude-Sonnet-4.5	0.907	0.818	0.828	0.851	0.833	0.809	0.611	0.751	0.805	0.728	0.500	0.678
Llama-3.3-70B	0.855	0.836	0.744	0.812	0.778	0.819	0.667	0.755	0.805	0.728	0.278	0.604

Table 3: Text2DistBench Results We report performance across distribution ( M arginals, C onditionals, J oint) and task types. Random baselines are computed from 50 runs of random guessing for Most/Second Frequent questions and from a uniform distribution for Estimation questions. Higher values indicate better performance.

Figure 3: Sensitivity to target distribution characteristics. Each cell reports the Spearman correlation between model performance and distribution statistics. (*) indicates statistical significance ( $pvalue<0.05$ ).

Figure 4: Effect of scaling reasoning effort. Performance of GPT-5.1 under different reasoning-effort settings, with GPT-4.1 shown as a non-reasoning baseline for comparison.

Figure 5: Prior belief from factual information. Each subplot compares prior (metadata-only) and posterior (with comments) performance for one task. The diagonal denotes equal prior and posterior performance, with points above it indicating improvements after observing viewer comments. Colored vertical dashed lines indicate random baselines for each distribution type.

Model	Esti- mation	Most Frequent	Second Frequent
GPT-5.1	0.142	0.220*	0.246*
Gemini-2.5-Pro	0.215*	0.221*	0.316*
Grok-4-Fast	0.186*	0.341*	0.181*
Claude-Sonnet-4.5	0.069	0.274*	0.158*
Llama-3.3-70B	0.219*	0.154*	0.122
Qwen3-32B	0.255*	0.304*	0.077

Table 4: Spearman correlation between model prior and posterior performance. (*) indicates statistical significance

pvalue<0.05

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Entities	#Marginals	#Conditionals	#Joints	#Total (per Task)	#Total (3 Tasks)
20 (10 Movie + 10 Music)	40	120	20	$180$	540

Appendix

Table 5: Benchmark Statistics Summary.

Attribute	Description
Actor	Performance, emotion, chemistry, casting quality.
Storyline	Plot, pacing, themes, character arcs, dialogue.
Visual	Cinematography, lighting, color, design, VFX, costumes.
Audio	Soundtrack, score, songs, sound effects, audio mix.

Appendix

Table 6: Movie Comment Topics

Attribute	Description
Song	Melody, harmony, rhythm, structure, arrangement.
Singer	Tone, technique, emotion, appearance.
Lyrics	Themes, message, storytelling, rhymes.
Visual	Cinematography, lighting, concept, choreography, VFX.

Appendix

Table 7: Music Comment Topics

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,408 papers

UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs
Jun 4, 2026Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo +4LLM Evaluation Language Model Generation Evaluation
University of British Columbia · 2Independent Researcher
Beyond Questions: Evaluating the Stated Knowledge of LLMs
May 26, 2026Luca Giordano, Simon RazniewskiLLM Evaluation Factual Knowledge in Language Models
ScaDS.AI Dresden/Leipzig & TU Dresden, Germany
CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases
Aug 27, 2026Sil Hamilton, Albert Yu Sun, Oscar J. Romero +4LLM Evaluation LLM Information Extraction
Epiq AI Labs · Cornell University

Jun 4, 2026J/K move · Enter open · S save

UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs

Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo, Ellie Dingqiao Wen+3

University of British Columbia · 2Independent Researcher

LLM Evaluation Language Model Generation Evaluation Synthetic Benchmark Generation

Open paper AskPDF arXiv

We introduce UnpredictaBench, an evaluation that tests the ability of large language models (LLMs) to capture true underlying distributions. As LLMs are increasingly used as substitutes for other entities (e.g., for humans in economic simulations), the tendency of many models to collapse towards a single plausible answer means a failure to capture the unpredictability of real systems. Recent work on improving output diversity is insufficient for this setting: simulation requires samples that are calibrated to a target distribution, not merely varied outputs. UnpredictaBench isolates a simplified but fundamental version of this problem: sampling outcomes from individual target distributions, including canonical statistical distributions, distributions induced by stochastic programs, and natural-language scenarios that describe random processes. We introduce 448 such problems together with KS@N, a general-purpose evaluation metric that quantifies how well a model outputs approximate black-box target distributions via the Kolmogorov-Smirnov statistical test. This is the rate at which we fail to reject model samples of size N against ground-truth samples, with larger N indicating greater difficulty. Tested across open and proprietary models, we find a large spread in distributional capabilities. For instance, when models generate samples of size 100 (KS@100, our standard metric), scores range from near 0 to over 20%. No model is able to achieve over 40% at KS@100, showing significant headroom in distributional sampling as a capability. Although adding reasoning can somewhat increase scores, we find no immediate solution for this issue. UnpredictaBench shows that even simple distributional simulation remains challenging, making it a necessary first step toward using LLMs as stand-ins for complex systems. Project website and resources are available at https://unpredictabenchmark.github.io/.

Abstract

LLM Evaluation

Explore similar work

UnpredictaBench: A Benchmark for Evaluating Distributional Randomness in LLMs

Beyond Questions: Evaluating the Stated Knowledge of LLMs

CorporateBench: Large-Scale Q&A Benchmarking with Temporal Knowledge Bases