While most reading comprehension benchmarks for LLMs focus on factual information that can be answered by localizing specific textual evidence, many real-world tasks require understanding distributional information, such as population-level trends and preferences expressed across collections of text. We introduce Text2DistBench, a reading comprehension benchmark for evaluating LLMs' ability to infer distributional knowledge from natural language. Built from real-world YouTube comments about movie and music entities, the benchmark provides models with entity metadata and associated comments, and requires them to answer distributional questions, such as estimating the proportions of positive and negative comments, or identifying the most and second most frequent topics discussed among viewers. To support reliable and long-term evaluation, the construction pipeline of Text2DistBench is fully automated and continuously updated to incorporate newly emerging entities over time. Experiments across multiple LLMs show that while models substantially outperform random baselines, performance varies widely across different distribution types and characteristics. These findings highlight both the capabilities and limitations of current LLMs in distributional reading comprehension and demonstrate the value of Text2DistBench as a practical and scalable testbed for future research.
Figures & tables
Figure 1: Factual vs. Distributional Knowledge.
Distribution
Question Example (Task Type: Most Frequent)
Marginals
P(S) : What overall attitude do most viewers express? Positive or Negative. P(T) : Which aspect of the movie is discussed most often? Actor, Storyline, Visual, or Audio.
Conditionals
P(S∣T) : Among the comments that talk about the actor, what attitude do viewers express most commonly? Positive or Negative. P(T∣S) : Among the comments that express a positive attitude, which aspect of the movie is mentioned most often? Actor, Storyline, Visual, or Audio.
Joint
P(S,T) : Considering both (1) which aspect (Actor, Storyline, Visual, Audio) is being talked about and (2) whether the attitude is positive or negative. Which (aspect, sentiment) combination appears most often in the comments?
Table 1: Question Examples of different distribution type.
Question
Question Example (Distribution Type: Marginals)
Estimation
Movie Information: {meta_data} YouTube Viewer Comments: {comments} What aspects of the movie do viewers talk about? Summarize how frequently each aspect appears in the comments using percentages. Output your answer in the following format: { "Actor":"<int>%", "Storyline":"<int>%", "Visual":"<int>%", "Audio":"<int>%" }
Most Frequent
Movie Information: {meta_data} YouTube Viewer Comments: {comments} Which aspect of the movie is discussed most often? Actor, Storyline, Visual, or Audio.
Second Frequent
Movie Information: {meta_data} YouTube Viewer Comments: {comments} Which aspect of the movie is discussed the second most often? Actor, Storyline, Visual, or Audio.
Table 2: Question Examples of different task type
Figure 2: Text2DistBench Generation Pipeline. The construction process consists of four stages: (1) selecting valid opinion entities; (2) annotating comment topic and sentiment; (3) estimating empirical opinion distributions; and (4) generating distributional reading-comprehension questions.
Estimation (1-TVD) ↑
Most Frequent (ACC) ↑
Second Frequent (ACC) ↑
Model
M
C
J
Avg
M
C
J
Avg
M
C
J
Avg
GPT-5.1
0.930
0.865
0.847
0.881
0.861
0.877
0.778
0.839
0.834
0.805
0.556
0.732
Gemini-2.5-Pro
0.903
0.869
0.805
0.859
0.805
0.886
0.722
0.804
0.778
0.835
0.667
0.760
Grok-4-Fast
0.911
0.856
0.838
0.868
0.889
0.858
0.667
0.805
0.833
0.728
0.611
0.724
Claude-Sonnet-4.5
0.907
0.818
0.828
0.851
0.833
0.809
0.611
0.751
0.805
0.728
0.500
0.678
Llama-3.3-70B
0.855
0.836
0.744
0.812
0.778
0.819
0.667
0.755
0.805
0.728
0.278
0.604
Table 3: Text2DistBench Results We report performance across distribution ( M arginals, C onditionals, J oint) and task types. Random baselines are computed from 50 runs of random guessing for Most/Second Frequent questions and from a uniform distribution for Estimation questions. Higher values indicate better performance.
Figure 3: Sensitivity to target distribution characteristics. Each cell reports the Spearman correlation between model performance and distribution statistics. (*) indicates statistical significance ( pvalue<0.05 ).
Figure 4: Effect of scaling reasoning effort. Performance of GPT-5.1 under different reasoning-effort settings, with GPT-4.1 shown as a non-reasoning baseline for comparison.
Figure 5: Prior belief from factual information. Each subplot compares prior (metadata-only) and posterior (with comments) performance for one task. The diagonal denotes equal prior and posterior performance, with points above it indicating improvements after observing viewer comments. Colored vertical dashed lines indicate random baselines for each distribution type.
Model
Esti- mation
Most Frequent
Second Frequent
GPT-5.1
0.142
0.220*
0.246*
Gemini-2.5-Pro
0.215*
0.221*
0.316*
Grok-4-Fast
0.186*
0.341*
0.181*
Claude-Sonnet-4.5
0.069
0.274*
0.158*
Llama-3.3-70B
0.219*
0.154*
0.122
Qwen3-32B
0.255*
0.304*
0.077
Table 4: Spearman correlation between model prior and posterior performance. (*) indicates statistical significance pvalue<0.05 .
We introduce UnpredictaBench, an evaluation that tests the ability of large language models (LLMs) to capture true underlying distributions. As LLMs are increasingly used as substitutes for other entities (e.g., for humans in economic simulations), the tendency of many models to collapse towards a single plausible answer means a failure to capture the unpredictability of real systems. Recent work on improving output diversity is insufficient for this setting: simulation requires samples that are calibrated to a target distribution, not merely varied outputs. UnpredictaBench isolates a simplified but fundamental version of this problem: sampling outcomes from individual target distributions, including canonical statistical distributions, distributions induced by stochastic programs, and natural-language scenarios that describe random processes. We introduce 448 such problems together with KS@N, a general-purpose evaluation metric that quantifies how well a model outputs approximate black-box target distributions via the Kolmogorov-Smirnov statistical test. This is the rate at which we fail to reject model samples of size N against ground-truth samples, with larger N indicating greater difficulty. Tested across open and proprietary models, we find a large spread in distributional capabilities. For instance, when models generate samples of size 100 (KS@100, our standard metric), scores range from near 0 to over 20%. No model is able to achieve over 40% at KS@100, showing significant headroom in distributional sampling as a capability. Although adding reasoning can somewhat increase scores, we find no immediate solution for this issue. UnpredictaBench shows that even simple distributional simulation remains challenging, making it a necessary first step toward using LLMs as stand-ins for complex systems. Project website and resources are available at https://unpredictabenchmark.github.io/.
Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Liang Luo +4
University of British Columbia · 2Independent Researcher
Parametric knowledge in large language models (LLMs) is a cornerstone of their success, yet remains poorly understood. Existing knowledge benchmarks typically rely on predefined questions (e.g., "What is the birth date of M.L. King?"), evaluating only knowledge that benchmark designers explicitly choose to query, a problematic availability bias. In this paper, we introduce open knowledge evaluation, a new paradigm for benchmarking the stated knowledge of LLMs. Instead of asking narrow questions, it evaluates models on the knowledge they choose to surface in response to open-ended elicitation prompts (e.g., "Tell me everything you know about M.L. King"). This shifts the focus from predefined answer retrieval toward characterizing the knowledge models naturally express. We instantiate this paradigm with BeQu (Beyond Questions), a benchmark of 10,000 entities paired with reference corpora for statement verification. Using BeQu, we evaluate a broad range of language models and analyze the effects of reasoning effort, model scale, prompt format, and knowledge domain. Data and leaderboard are available on this work's GitHub repository and at ANONYMIZED.
LLMs are increasingly able to answer complex questions about enterprise-scale document collections. But evaluation is hard: companies don't want to share internal communications, and synthetic datasets have been overly simple. We present CorporateBench (CB), a human-validated multi-task Q&A benchmark whose scale approaches the conditions LLMs encounter in corporate communication networks, with evaluation corpora surpassing 230,000 documents. CB evaluates LLMs across two dimensions (information extraction and knowledge base querying) through four synthetically generated firms ranging from 12 to 10,000 employees. Each corpus is sampled from a temporally evolving knowledge base describing a consistent world, guaranteeing cross-document logical consistency even across hundreds of thousands of documents. We evaluate five LLMs on CB, revealing increasingly poor performance as input size approaches realistic scales. CB provides LLM developers a metric for corporate communication reasoning, filling a crucial gap in the benchmarking ecosystem.