CruxBench: A Benchmark of Information Discovery

Hui Dai; Lina Piao; Nick Merrill; Nadja Flechner; Ezra Karger; Haifeng Xu

cs.CLSep 26, 2026

CruxBench: A Benchmark of Information Discovery

Authors: Hui Dai, Lina Piao, Nick Merrill, Nadja Flechner, Ezra Karger, Haifeng Xu

Organizations: The University of Chicago · Forecasting Research Institute · Federal Reserve Bank of Chicago

Abstract

Benchmarks for large language models (LLMs) typically evaluate the accuracy of answers against fixed reference labels. But a central step in many complex real-world tasks is identifying which questions are worth asking in the first place: decomposing a difficult problem into subquestions -- which we call cruxes -- whose answers provide key steps on the path toward solving the target problem. To evaluate this capability of information discovery, we introduce CruxBench, a benchmark that grades LLM-generated questions by their Value of Information (VOI): how much a model-proposed crux updates beliefs about a target forecasting question. CruxBench enjoys a rare combination of three key properties: it is (1) contamination-resistant by construction, since ground truth is generated by future world events; (2) open-ended, admitting unbounded and complex text-based submissions rather than one correct numeric answer; and (3) grounded, with informativeness measured against quantified changes in real-world beliefs. We evaluate a diverse set of eight models on 293 target forecasting questions and find that VOI correlates highly with independent measures of model capability (r=0.90) and captures cruxes' usefulness for answering target questions. However, information discovery remains challenging even for frontier LLMs, which only narrowly outperform a random-timing baseline.

Figures & tables

Figure 1: CruxBench pipeline overview. Given an ultimate question $U_{i}$ and the information available at the start of evaluation window $j$ , a generator proposes a crux $Q_{ij}$ that is expected to resolve within the window. A separate resolver identifies the crux’s resolution time $t_{ij}$ using evidence available during the window. We measure the market-belief update around $t_{ij}$ using KL divergence and average across questions to obtain the generator’s VOI score.

Model	$\widehat{\mathrm{VOI}}^{\mathrm{KL}}$ (bits)	95% CI	# resolved (rate)	Knowledge cutoff
Claude-Opus-4.6	0.016 ∗∗	[0.009, 0.023]	279 (95.2%)	Aug 2025
Gemini-3.1-Pro-Preview	0.014 ${}^{}\phantom{}$	[0.007, 0.020]	274 (93.5%)	Jan 2025
GPT-5.4	0.013 ${}^{}\phantom{}$	[0.007, 0.019]	268 (91.5%)	Aug 2025
DeepSeek-V3.1	0.012	[0.003, 0.020]	262 (89.4%)	Aug 2025 †
Qwen3-235B-A22B	0.012	[0.005, 0.018]	271 (92.5%)	Jul 2025 †
Llama-3.3-70B	0.010	[0.005, 0.016]	252 (86.0%)	Dec 2023

Table 1: Performance on CruxBench (

n=293

). “Resolvable” reports the number and percentage of cruxes with resolvable=true . Random baseline is the per-window mean of

\widehat{\mathrm{VOI}}^{\mathrm{KL}}

over all hours in the active window (random-timing baseline); oracle is the per-window maximum (best-possible pivot in hindsight). Bold indicates the best-performing model. Stars denote uncorrected one-sided paired

t

-tests against the random baseline:

{}^{*}\,p<0.05

{}^{**}\,p<0.01

. Cutoffs marked † use the model’s public release date as a proxy where an official training-data cutoff is not published.

Figure 2:

\widehat{\mathrm{VOI}}^{\mathrm{KL}}

correlates with model capability. (a) vs. Epoch AI capability score. (b) vs. each generator’s forecasting performance, measured by predicting

P(U)

at window start and scored by

-\mathrm{Brier}(P(U))

(left) and

\mathrm{AUC}(P(U))

(right). Brier score is sign-flipped so that higher is better.

Figure 3:

\widehat{\mathrm{VOI}}^{\mathrm{KL}}

correlates with downstream forecasting utility for separate forecasters. For each generator, three forecasters ( Claude-Haiku-4.5 , Gemini-3-Flash-Preview , GPT-5-Mini ) predict

P(U\mid Q)

given the crux information; results are averaged across the three.

Figure 4: Semantic similarity is only a weak proxy for crux equivalence. (a) Mean cosine similarity between cruxes generated by each pair of models for the same ultimate question (mean ${=}0.75$ , $n=293$ ). (b) Outcome disagreement, measured as $1-P(\mathrm{same\ Yes/No\ resolution})$ , and the median absolute difference in resolution dates, grouped by cosine-similarity of crux pairs.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 5: Empirical verification of the martingale property. Each blue point is the mean rolling-window posterior $p_{a}$ within a $p_{b}$ decile ( $\pm 1\,\text{SE}$ ); dashed grey: 45° null $E[p_{a}\mid p_{b}]=p_{b}$ ; red: OLS fit. On $n=2{,}088$ crux-resolution pairs across 8 generators ( $W=24\text{h}$ ), $\hat{\beta}=1.00$ , $\hat{\alpha}=+0.0002$ , $R^{2}=0.96$ , and decile-binned $E[p_{a}\mid p_{b}]$ lies within $1.0\text{pp}$ of the diagonal in every bin: Polymarket prices satisfy the martingale property in the mean.

Stage	Markets
Raw Polymarket dump (2026-03-25)	108,101
price-history end $\geq$ 2025-09-01	87,124
has resolution	86,819
dedup by question text	84,043
UMA status = resolved	84,035
has Yes outcome	30,524

Appendix

Table 2: Per-stage retention markets from the raw Polymarket dump to the final dataset (

|\mathcal{D}|=293

Model	Proposed Crux	KL
Gemini-3.1-Pro-Preview	Will authorities announce the arrest or formal charging of at least one suspect in connection with the October 2025 Louvre jewel heist between Oct. 21 and Nov. 20, 2025?	0.1348
Claude-Opus-4.6	Will French police publicly identify or arrest at least one suspect in the Louvre crown jewels heist by Nov. 15, 2025?	0.1348
GPT-5.4	Will French authorities publicly announce at least one arrest or formal charging of a suspect in connection with the October 19, 2025 Louvre crown-jewels theft by 2025-11-20?	0.1348
Llama-3.3-70B	Will the French authorities announce the recovery of at least one stolen Louvre jewel or the arrest of at least one suspect in connection with the heist between 2025-10-21 and 2025-11-20?	0.1348
Qwen3-235B-A22B	Will French law enforcement announce the arrest of at least one suspect in connection with the Louvre crown jewels heist between Oct. 21, 2025, and Nov. 20, 2025?	0.1348
DeepSeek-V3.1	Will at least one suspect involved in the Louvre crown jewels heist be formally arrested and charged by Nov. 15, 2025?	0.0022

Appendix

Table 3: Cruxes generated for “Will any of the stolen Louvre jewels be returned by December 31?” with their realized

\mathrm{KL}(\hat{p},\hat{p}^{\prime})

Figure 6: Price history of the “stolen Louvre jewels” ultimate. The shaded band is the 30-day evaluation window. Markers show each crux’s resolution time, and fanned horizontally where several share a timestamp.

Model	Proposed Crux	KL
GPT-5.4	Will ADOR or NewJeans publicly announce between Oct. 31 and Nov. 30, 2025 that they have begun, resumed, or scheduled formal talks specifically aimed at restarting NewJeans group activities?	0.5023
Qwen3-235B-A22B	Will ADOR announce new group activities for NewJeans between Oct. 31 and Nov. 30, 2025?	0.5023
Llama-3.3-70B	Will NewJeans or their members make a public statement about their plans for 2025 between Oct. 31 and Nov. 30, 2025?	0.5023
Llama-3.1-8B-Instruct	Will NewJeans’ members contradict their recent court statement on their relationship with ADOR by engaging in any form of public activity as NewJeans, including interviews, social-media posts, or public appearances, between Oct. 31 and Nov. 20, 2025?	0.5023
Claude-Opus-4.6	Will NewJeans, or their representatives, file a formal appeal of the October 30, 2025 Seoul Central District Court ruling by Nov. 30, 2025?	0.0539
Gemini-3.1-Pro-Preview	Will ADOR, HYBE, or NewJeans officially announce any new public group activities, such as a music release, fan event, or live performance, between Oct. 31 and Nov. 30, 2025?	0.0007

Appendix

Table 4: Cruxes generated for “Will NewJeans perform again in 2025?” with their realized

\mathrm{KL}(\hat{p},\hat{p}^{\prime})

Figure 7: Price history of the “NewJeans’ return” ultimate. The shaded band is the 30-day evaluation window. Markers show each crux’s resolution time and are fanned horizontally where several share a timestamp.

Role	Model	Provider	Identifier
Generator	Claude-Opus-4.6 [ 3 ]	Anthropic	claude-opus-4-6
Generator	Gemini-3.1-Pro-Preview [ 17 ]	Google	gemini-3.1-pro-preview
Generator	GPT-5.4 [ 33 ]	OpenAI	gpt-5.4
Generator	DeepSeek-V3.1 [ 11 ]	Together	deepseek-ai/DeepSeek-V3.1
Generator	Qwen3-235B-A22B [ 35 ]	Together	Qwen/Qwen3-235B-A22B-Instruct-2507-tput
Generator	Llama-3.3-70B [ 26 ]	Together	meta-llama/Llama-3.3-70B-Instruct-Turbo

Appendix

Table 5: Models used in our experiments: eight generators produce candidate cruxes, one resolver assigns binary outcomes and initial resolution timestamps, a verifier checks and aligns those timestamps, and three forecasters are used in the downstream forecasting experiment in Section 4.5 .

Model	$\widehat{\mathrm{VOI}}^{\mathrm{KL}}$		95% CI	Rank	$\widehat{\mathrm{VOI}}^{\Delta H}$	95% CI	Rank	$\|\Delta p\|$		95% CI	Rank	$\Delta p^{2}$		95% CI	Rank
Claude-Opus-4.6	0.016	∗∗	[0.009, 0.023]	1	0.017	[0.006, 0.029]	1	0.027	∗∗	[0.021, 0.034]	1	0.004	${}^{}\phantom{}$	[0.002, 0.006]	1
Gemini-3.1-Pro-Preview	0.014	${}^{}\phantom{}$	[0.007, 0.020]	2	0.011	[-0.001, 0.023]	3	0.024	${}^{}\phantom{}$	[0.019, 0.030]	2	0.003	${}^{}\phantom{}$	[0.001, 0.005]	2
GPT-5.4	0.013	${}^{}\phantom{}$	[0.007, 0.019]	3	0.009	[-0.002, 0.020]	6	0.023		[0.017, 0.028]	3	0.003		[0.001, 0.005]	3
DeepSeek-V3.1	0.012		[0.003, 0.020]	4	0.014	[0.004, 0.025]	2	0.020		[0.014, 0.026]	5	0.003		[0.000, 0.005]	4
Qwen3-235B-A22B	0.012		[0.005, 0.018]	5	0.007	[-0.003, 0.017]	8	0.021		[0.016, 0.027]	4	0.003		[0.001, 0.004]	5
Llama-3.3-70B	0.010		[0.005, 0.016]	6	0.009	[-0.001, 0.019]	7	0.019		[0.014, 0.024]	7	0.002		[0.001, 0.004]	6

Appendix

Table 6: Additional metric results by model (

n=293

\widehat{\mathrm{VOI}}^{\mathrm{KL}}

and

\widehat{\mathrm{VOI}}^{\Delta H}

are measured in bits. Bold indicates the best-performing model. Significance is relative to the random baseline:

{}^{*}\,p<0.05

{}^{**}\,p<0.01

Figure 8: Different metrics vs. averaging window

W

. Dashed black curve is the per-hour random baseline.

Figure 9: Window-size robustness across

W\in[1,48]

h. (a) Pearson

r

between each metric’s per-generator mean and the Epoch AI capability score (

\star

means

p<0.05

). (b) Spearman

\rho

between the generator ranking at the reference

W=24

h and at every other

W

Figure 10: Where KL has lower Monte Carlo sampling variance than $\Delta H$ . Heatmap over the swept $(F_{p},F_{q},\alpha)$ grid: red cells are $(F_{p},F_{q},\alpha)$ at which $\mathrm{Var}(\widehat{\mathrm{KL}})<\mathrm{Var}(\widehat{\Delta H})$ (KL optimal); blue otherwise. Cell winners come from an $F$ -ratio test on the two replication-level variance estimates and are stable at our $R$ , though variance ratios sit within a few percent of 1. The KL advantage concentrates in the low- $\alpha$ regimes ( $\alpha\in[0.1,0.6]$ ). The boundary cases at $\alpha\in\{0,1\}$ are degenerate — both estimators reduce to a constant.

Combination ( $F_{p}/F_{q}$ )	$n_{\alpha}$	KL win rate (%)	Median Var(H)/Var(KL)
beta(2,2) / beta(2,2)	11	81.8	1.06
beta(2,5) / beta(5,2)	11	81.8	1.09
beta(5,2) / beta(2,5)	11	81.8	1.14
beta(5,5) / beta(5,5)	11	72.7	1.13
jeffreys / uniform	11	72.7	1.02
beta(2,5) / uniform	11	63.6	1.02

Appendix

Table 7: Per-combination win rate for KL. Fraction of

\alpha

-values (per row

n_{\alpha}=11

) at which

\mathrm{Var}(\widehat{\mathrm{KL}})<\mathrm{Var}(\widehat{\Delta H})

, with median variance ratio

\mathrm{Var}(\widehat{\Delta H})/\mathrm{Var}(\widehat{\mathrm{KL}})

. Under a two-sided sign test of

H_{0}:P(\mathrm{KL\ wins})=0.5

, the strongest rows (

81.8\%

) yield

p\approx 0.065

. Variance ratios cluster within

[0.98,1.14]

, so per-combination effects are small. Across all 11 combinations,

9

favor KL (sign test,

p\approx 0.065

), indicating a directional advantage.

Model	$\hat{\alpha}$	TPR	FPR	$n_{+}$	$n_{-}$
Gemini-3.1-Pro-Preview	$+0.123$	$0.405$	$0.282$	$79$	$195$
GPT-5.4	$+0.118$	$0.444$	$0.327$	$72$	$196$
Claude-Opus-4.6	$+0.113$	$0.436$	$0.323$	$78$	$201$
GPT-3.5-Turbo	$+0.042$	$0.222$	$0.180$	$81$	$194$
DeepSeek-V3.1	$-0.014$	$0.181$	$0.195$	$72$	$190$
Llama-3.3-70B	$-0.034$	$0.380$	$0.414$	$71$	$181$

Appendix

Table 8: Empirical estimate of crux informativeness, by model.

\hat{\alpha}=\mathrm{TPR}-\mathrm{FPR}

(Youden’s

J

);

n_{+},n_{-}

are the per-row counts of

U=1

and

U=0

Figure 11: Operationalization-axis match rate by cosine similarity. For each question pair we record whether the two cruxes share the same numeric threshold (when both emit one) and the same source kind (Appendix K.1 ). Error bars are Wilson 95% CIs on each binomial proportion. Even at $\cos>0.8$ , threshold agreement is only $0.16$ and source-kind agreement only $0.68$ .

Model	Data avail.	Temporal	Observability	Total
Claude-Opus-4.6	1	0	2	3
Gemini-3.1-Pro-Preview	2	0	3	5
GPT-5.4	3	2	1	6
DeepSeek-V3.1	4	1	3	8
Qwen3-235B-A22B	5	0	1	6
Llama-3.3-70B	8	0	3	11

Appendix

Table 9: Unresolvable cruxes by failure mode and model (

n=50

). Counts of cruxes each model marked unresolvable, classified into three failure modes: data availability , temporal , and observability . Data availability dominates overall (

25/50=50\%

), but the distribution is model-dependent: Llama-3.3-70B and Qwen3-235B-A22B most often fail due to data availability, while Llama-3.1-8B-Instruct ’s failures are predominantly temporal. Gemini-3.1-Pro-Preview and Claude-Opus-4.6 show a larger share of observability failures.

LLM Evaluation

Large language model evaluation measures what models can do, how reliably they do it, and whether benchmark scores predict real-world performance.

See all 2,427 papers

Not All Information Is Worth Asking For: Decision-Informed Information Seeking with LLMs
Oct 23, 2026Seunghan Lee, Jun Seo, Junhyeok Kang +9LLM Decision-Making Value of Information
LG AI Research
Can LLMs Anticipate Their Cruxes?
Oct 14, 2026David Reber, James Reber, Ari Holtzman +1LLM Decision-Making LLM Evaluation
Department of Computer Science, University of Chicago · Independent Researcher · Data Science Institute, University of Chicago +1
Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility
Aug 10, 2026Siyang Wu, Yibo Jiang, Bryon AragamPrompt Sensitivity LLM Evaluation
Data Science Institute University of Chicago Chicago, USA · Department of Computer Science University of Chicago Chicago, USA · Booth School of Business University of Chicago Chicago, USA

Oct 23, 2026J/K move · Enter open · S save

Not All Information Is Worth Asking For: Decision-Informed Information Seeking with LLMs

Seunghan Lee, Jun Seo, Junhyeok Kang, Jaehoon Lee+8

LG AI Research

LLM Decision-Making Value of Information

Open paper AskPDF arXiv

Large language models (LLMs) are increasingly used to support decisions under incomplete information, where deciding what to look up and when to stop matters as much as the final answer. However, LLMs fall into an information trap: They pay for checks, i.e., costly pieces of evidence, that cannot change their decision or are not worth their cost, and they keep asking when the evidence is already sufficient. To this end, we propose Decision-Informed Value-of-information Acquisition (DIVA), a simple yet effective procedure for LLM information seeking based on the decision value of evidence. DIVA 1) lets the LLM specify a small decision model, i.e., how likely a bad outcome is and how each check would shift this probability, through short numeric questions and 2) values each candidate check by how much its imagined outcomes improve the decision net of its cost, stopping when no check is worth its cost. Its variant, DIVA-U, further accounts for the LLM's uncertain estimates of how strongly each check shifts this probability. DIVA is training-free and applies to any LLM. Across four domains and 20 LLMs, DIVA-U has lower regret, i.e., value lost relative to the best possible decision, than every LLM baseline on real-world data, mainly by stopping once evidence suffices, and DIVA cuts the regret of the best LLM baseline by 75% in synthetic scenarios where every check's reliability is given. Code: github.com/seunghan96/diva.

Abstract

LLM Evaluation

Explore similar work

Not All Information Is Worth Asking For: Decision-Informed Information Seeking with LLMs

Can LLMs Anticipate Their Cruxes?

Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibility