Can LLMs Anticipate Their Cruxes?

David Reber; James Reber; Ari Holtzman; Victor Veitch

cs.CLOct 14, 2026

Can LLMs Anticipate Their Cruxes?

Authors: David Reber, James Reber, Ari Holtzman, Victor Veitch

Organizations: Department of Computer Science, University of Chicago · Independent Researcher · Data Science Institute, University of Chicago · Department of Statistics, University of Chicago

Abstract

To use language models to help us make decisions we would like to understand the rationale behind the language model's outputs. In principle, the language model provides its rationale as part of its text output. However, the sheer volume of text makes this hard to use in practice. In this paper, we are interested in extracting the cruxes that lead a model to recommend one action over another. We take a crux to be a sub-question that a model's rollouts disagree on, where changing its value may have altered the conclusion. A crux is therefore a property of the distribution of a model's rollouts, not of any one rollout. We consider three strategies of increasing computational cost. First, we just ask a model to identify, a priori, what the cruxes of a decision will be. Second, we have the model actually go through the process of generating a response and then extract the cruxes that arose in its reasoning. Finally, we generate many responses and extract the cruxes by comparing the reasoning across the rollouts. We make this last procedure computationally tractable with cheap calls to a reranker, an estimated 86 times less compute than a judge. We then conduct an extensive empirical evaluation to assess whether the computationally inexpensive options are sufficient. They are not. Across domains and models, we find that a model's a priori assessment of the cruxes has little relation to the factors that are actually identified as important a posteriori. Using a single rollout helps, but is still inadequate for identifying the full set of cruxes. Accordingly, for high-stakes decisions (and to understand the space of language model decision making) the many rollout procedure is unavoidable.

Figures & tables

Panel of Figure 1 — Figure 1: A crux is an assumption that forecasts disagree on, and whose answer could have changed their conclusions.

Panel of Figure 2 — Figure 2: In a basketball game and a bankruptcy question, the cruxes a model predicted matched the questions its forecasts varied on far less often than the permutation baseline, which reassigns the same cruxes at random to the a priori and multiple-rollout sets. Left: Iowa State vs. Kentucky, a game in March Madness, the American college basketball tournament. Right: Frontier Group bankruptcy, an event on Metaculus, a public forecasting platform. Each diagram splits the clusters of paraphrased cruxes ( Section 5.1 ), formed from 25 cruxes per set, into those holding a priori cruxes only (orange), both kinds (green, the overlap) and multiple-rollout cruxes only (blue). Below each diagram stand one a priori crux, as a question, and one multiple-rollout crux, as one forecast’s answer ( contribution 1 extracts the multiple-rollout cruxes; contribution 2 groups similar paraphrases at low cost). A turnover is a possession lost before the team can shoot, and a forced turnover is one the defense causes. The Altman Z-score is a bankruptcy-risk score computed from a company’s accounts. Below the circles each diagram prints its overlap at 25 cruxes and its permutation baseline ( contribution 3 runs this comparison across games, events and models).

Panel of Figure 3 — Figure 3: In March Madness the test asks whether a model, reading every scouting report its forecasts were written from, can foresee where those forecasts will disagree. A priori cruxes (left) against multiple-rollout cruxes (right), for one March Madness game, Iowa State vs. Kentucky. The a priori prompt predicts cruxes from the reports, and the multiple-rollout prompt finds them in each pair of the game’s rollouts. The a priori and multiple-rollout cruxes are then pooled and clustered, and their overlap is compared with a baseline that shuffles the two labels among the same cruxes (bottom row). Each crux is a yes/no question with both answers, and the figure quotes the multiple-rollout crux by the assumption one of the two rollouts took, that Iowa State forced 23 turnovers in an earlier game.

Figure 4: Cruxes are often shared across rollouts, so the number of distinct cruxes grows about linearly with the number of rollouts even though the number of rollout pairs grows quadratically. Left: the median number of distinct crux clusters at the same-resolution level ( Section 5.1 ) against the number of rollouts, each game or event averaged over up to 20 random subsets (bands, 10th to 90th percentile), with the number of rollout pairs for reference. March Madness reaches 28 clusters at nine rollouts and shows no sign of saturation. Metaculus, with 12 of the 29 or 30 rollouts per event and all their cruxes, appears to converge to 5 cruxes. Middle and right: March Madness (59 games, $N=9$ ) and Metaculus (15 events, $N=12$ ), where the colours now mark quantities, not sources, and every curve is a share of what all $N$ rollouts give. Blue is coverage, the share of the clusters from all $N$ rollouts that $n$ rollouts already find. The dashed line is the coverage of as many cruxes drawn at random from the pool. Pink is the number of clusters from $n$ rollouts as a share of the number from all $N$ . In Metaculus that ratio passes 100%, and the axis stops at 150%.

Figure 5: Naming more cruxes, up to 25 on each side, still does not close the gap between predicted and discovered cruxes. Bands give the share of clusters holding a priori cruxes only (top, orange), multiple-rollout only (middle, blue) or both (bottom, green, the overlap). The dashed line is the permutation baseline, the overlap when the two labels are shuffled. Left: March Madness, 59 games. Right: Metaculus, 15 events, baseline 49.5% at 5 cruxes and 70.9% at 25. Same-resolution matching throughout; lighter bands show $\pm 1$ SE, and the numbers at the right give each band’s share at 25 cruxes.

Appendix figures & tables15 assets

Supplementary material from the paper’s appendix.

Appendix

Figure 6: Whether two cruxes must ask the same question or need only concern the same dispute, predicted cruxes match discovered ones less often than the permutation baseline. Stacked-area decomposition at the three levels, same question, same resolution and same dispute, for March Madness (59 games, top row) and Metaculus (15 events, bottom row), in the colours of Figure 5 . The numbers at the right of each panel give its three shares at 25 cruxes. At 25 cruxes the March Madness overlap is 5.6%, 13.4% and 12.8% against baselines of 33.4%, 39.2% and 41.2%, and the Metaculus overlap is 21.9%, 39.9% and 39.7% against 58.0%, 70.9% and 68.9%.

Source	Level	Overlap (%)	Baseline (%)	Gap	$p$	Below
March Madness	same question	$5.6\pm 0.9$	33.4	27.8	$<10^{-10}$	59 / 59
	same resolution	$13.4\pm 1.6$	39.2	25.9	$<10^{-10}$	55 / 59
	same dispute	$12.8\pm 1.5$	41.2	28.4	$<10^{-10}$	57 / 59
Metaculus	same question	$21.9\pm 3.6$	58.0	36.1	$6.1\times 10^{-5}$	14 / 15
	same resolution	$39.9\pm 4.0$	70.9	31.0	$1.2\times 10^{-4}$	13 / 15
	same dispute	$39.7\pm 4.4$	68.9	29.3	$2.4\times 10^{-4}$	12 / 15

Appendix

Table 1: At every level of strictness the gap below the permutation baseline is significant in both sources, and nearly every game and event falls below its own baseline. Overlap and baseline at

k=25

at each level of strictness, each level against its own baseline, for March Madness (59 games) and Metaculus (15 events). Overlaps carry

\pm

1 SE across games or events. Gaps are in points, and

p

is from a one-sided Wilcoxon signed-rank test across games or events. The last column counts games or events below their own baseline; the rest equal it, apart from one March Madness game at the same-resolution level that lies above.

Figure 7: In March Madness, however coarsely cruxes are grouped, predicted cruxes match discovered ones less often than the permutation baseline, though halves of the discovered cruxes match each other. Left: Coarsening the clustering does not close the gap below the baseline. The overlap and its baseline across cuts of the tree, from broad categories on the left to one crux per cluster on the right (March Madness, same-resolution matching, 25 cruxes per set). The horizontal axis is $\rho$ , the number of clusters divided by the number of pooled cruxes, and the top axis gives the clusters per game. The dotted line marks the cut used in the main text, a mean of 20.5 clusters per game ( $\rho=0.41$ ), and the grey band the middle half of games. Right: Two halves of either set share several times the overlap the two sets share with each other. Overlap against the number of cruxes drawn from each set, for two halves of the a priori cruxes, two halves of the multiple-rollout cruxes, and the a priori cruxes against the multiple-rollout cruxes. The dashed line gives the permutation baseline for the a priori against multiple-rollout comparison (March Madness, same-resolution matching). At 25 cruxes two halves of the multiple-rollout cruxes sit just above the dashed baseline, two halves of the a priori cruxes well above it, and the two sets far below it. So the discovered cruxes are not too scattered for a set of cruxes to match them.

Figure 8: Predicted cruxes match discovered ones less often than the permutation baseline in nearly every game and event, so the mismatch is the typical case. Top: the gap of each game and event at 25 cruxes, its own baseline minus its overlap, with a line at zero. In March Madness 55 of 59 games fall below their own baseline, 3 equal it and 1 lies above it; in Metaculus 13 of 15 events fall below, 2 equal it and none lies above. Bottom: six March Madness games chosen at random (seed 42), as each set grows to 25 cruxes, in the colours of Figure 5 and with each game’s own baseline as the dashed line. The shared band rises and falls with set size and differs from game to game. Same-resolution matching throughout.

Source	Cruxes	Factor	Claim	Value
March Madness	predicted, original prompt	1.9	66.4	31.7
	predicted, same-day control	1.8	66.5	31.7
	predicted, specific prompt	0.1	3.3	96.6
	predicted, exemplar prompt	0.2	2.8	97.0
	predicted, ladder, factor prompt	9.3	68.1	22.6
	predicted, ladder, claim prompt	1.9	77.2	20.9

Appendix

Table 2: Asked for specific cruxes, the predictor states values at least as often as the discovered cruxes do, while its usual cruxes mostly do not. Share of cruxes at each level of specificity, in percent, for the cruxes as written and for each prompt. The factor prompt of the ladder failed its check, so the ladder separates claims from values only.

Source	Prompt	$k=25$	$k=12$
March Madness	original (headline, one seed)	25.8	14.8
	same-day control	24.7	15.0
	specific	16.4 ( $p=1.4\times 10^{-5}$ )	10.1 ( $p=0.0012$ )
	exemplar	12.1 ( $p=1.4\times 10^{-8}$ )	7.8 ( $p=5.2\times 10^{-6}$ )
	ladder, factor	27.4 ( $p=0.10$ )	16.1 ( $p=0.57$ )
	ladder, claim	23.0 ( $p=0.40$ )	12.5 ( $p=0.043$ )

Appendix

Table 3: In March Madness, asking for specific cruxes narrows the gap by about a third only by lowering the permutation baseline, and no prompt closes it. Gap in points under each prompt, at 25 and 12 cruxes per set. March Madness

p

values are two-sided Wilcoxon signed-rank tests of control minus arm over 59 games. The

p

values are normal approximations. In Metaculus the drift between the original and the same-day control (7.8 points at

k=25

) exceeds every difference between arms. Every row, the original included, uses the seed of the arm runs for the baseline’s label shuffles. That seed is the same for every game or event. The runs of Table 1 instead give each game or event its own seed. With that seeding the original prompt’s gaps are 25.9 and 15.0 in March Madness and 31.0 and 18.8 in Metaculus.

Source	Arm	$k$	Original	Arm
March Madness	multiple-rollout cruxes, rewritten one level vaguer	25	25.8	26.9 ( $p=0.57$ )
		12	14.8	16.8
	both sets rewritten toward factors	25	25.8	15.3 ( $p=1.3\times 10^{-6}$ )
		12	14.8	11.0 ( $p=0.008$ )
	both sets restricted to values (24 games)	25	23.9	22.4
	both sets restricted to values (53 games)	12	15.6	15.1

Appendix

Table 4: Making cruxes vaguer, or comparing only cruxes of equal specificity, leaves predicted cruxes matching discovered ones less often than the permutation baseline. Gap in points when cruxes are rewritten to be vaguer, and when both sets keep only cruxes at the same levels of specificity. The

p

values are two-sided Wilcoxon signed-rank tests of arm against original over games or events. The original column uses the full set of multiple-rollout cruxes, none rewritten or removed, on the same games or events. Both columns shuffle the two labels with one fixed seed for all games or events. So the original column gives 25.8 and 14.8 in March Madness and 31.4 in Metaculus. The main text gives each game or event its own seed and reports 25.9, 15.0 and 31.0.

Cruxes per set	Strict	Moderate	Broad	Text, same pools
25	20.3	24.3	20.1	22.9
12	13.9	18.2	16.0	13.7

Appendix

Table 5: Matching cruxes on the evidence behind them, instead of on their wording, still leaves predicted cruxes matching discovered ones less often than the permutation baseline. Gap in points when cruxes are matched on their written evidence under three wordings of the matching instruction, beside matching on the cruxes’ own text on the same pools (March Madness, 59 games). The full set of multiple-rollout cruxes gives 25.9 points at 25 cruxes.

Part of the prompt	Pairwise (multiple-rollout) extraction	Single-forecast (single-rollout) extraction
Task statement	none; the rubric leads straight into the steps	“You are given ONE argument: a single forecast for the question below, with its reasoning. Identify the cruxes of this argument: the factual premises about the world that its forecast turns on, so that if a premise were false, the forecast would change.”
Distillation	“Distill any differences into cruxes”	“Distill the premises its forecast depends on into cruxes”
option_A and option_B	Argument A’s premise and Argument B’s premise, contradicting along the crux dimension	the premise this argument relies on and the contrary premise, “so that if it were true the forecast would change”
Analysis step	analyze both arguments, do the same analysis for Argument B, then for each dimension note whether the arguments differ	analyze the one argument on the same four dimensions, with no comparison step
Maximum and order	none stated	“Return at most 30 cruxes, most important first: the premises whose reversal would move this forecast the most”
Minimum output	“You MUST return at least one crux if the arguments rely on different world-facts, even if they reach the same final answer”	no minimum

Appendix

Table 6: The single-forecast extraction prompt asks what one forecast rests on where the pairwise prompt asks where two forecasts disagree. Every part of the two prompts that differs, in descending order of how much the change matters; all else is shared.

Source	Cruxes	Overlap vs baseline	Gap	Below / above / tied, $p$
March Madness	single-rollout	18.8 vs 25.1	6.4	47 / 11 / 1, $4\times 10^{-9}$
March Madness	a priori	14.6 vs 41.4	26.8	56 / 0 / 3, $10^{-17}$
Metaculus	single-rollout	48.5 vs 44.0	$-4.5$	2 / 9 / 4, 0.94
Metaculus	a priori	35.1 vs 56.9	21.9	11 / 1 / 3, $5\times 10^{-4}$

Appendix

Table 7: Cruxes from one forecast come far closer to the permutation baseline than predicted cruxes, so scoring one set of cruxes against a pooled one does not explain the gap. Overlap against baseline at

k=25

, same resolution, the single-rollout cruxes and the a priori cruxes scored on the same stripped multiple-rollout cruxes. Gaps are in points, the counts are games or events below, above and at their own baseline, and

p

is from the one-sided exact Wilcoxon test.

Source	Cruxes, pool	obs	base	gap	$p$	obs	base	gap	$p$
		In pool				Covers pool
MM	a priori, headline	32.7	65.3	32.5	$10^{-12}$	23.2	65.3	42.1	$2\times 10^{-18}$
MM	single-rollout, stripped pool	36.7	57.7	21.0	$3\times 10^{-14}$	28.6	37.8	9.2	$8\times 10^{-8}$
MM	a priori, stripped pool	41.1	68.3	27.3	$10^{-9}$	25.7	68.4	42.7	$2\times 10^{-18}$
MM	per-forecast pooled, headline	40.9	52.2	11.3	$3\times 10^{-12}$	55.0	52.1	$-2.9$	0.95
MM	split half	68.8	68.7	$-0.1$	0.73	65.9	68.5	2.6	0.035

Appendix

Table 8: Predicted cruxes cover far less of the discovered cruxes than the permutation baseline would, and in March Madness cruxes from one forecast fall short too. Containment at

k=25

, same resolution, each direction against its own baseline. In pool is the share of the set’s cruxes whose cluster holds a pooled crux, covers pool the share of the pooled cruxes whose cluster holds one of the set’s; gaps are in points,

p

is one-sided exact Wilcoxon, and MM is March Madness.

Model	$k$	Overlap (%)	Baseline (%)	Gap	Below / above / equal	$p$
qwen3:32b	25	$38.3\pm 2.2$	52.3	$13.9\pm 2.0$	39 / 5 / 15	$1.4\times 10^{-9}$
gpt-oss:120b	12	11.5	18.6	7.1	48 / 10 / 1	$2\times 10^{-10}$
gemma4:31b	12	$14.4\pm 1.8$	30.3	$15.9\pm 1.3$	55 / 2 / 2	$4\times 10^{-16}$

Appendix

Table 9: For each of three open-weight models, predicted cruxes match discovered ones less often than the permutation baseline in most games. The runs repeat the March Madness experiment at the same-resolution level; the qwen3:32b row holds no repeated cruxes, and the gpt-oss:120b and gemma4:31b rows are rescored with repeated cruxes removed. Gaps are in points, and the counts are games below, above and equal to their own baseline, where equal means that the game’s overlap matches the mean of its shuffles. The model gpt-oss:120b is quoted at

k=12

because its extraction yields fewer than 25 cruxes in 17 games. The model gemma4:31b is also quoted at

k=12

, the largest length at which most of its games are full, since its extraction yields fewer than 12 cruxes in 17 games. A game with fewer than

k

cruxes on one side is scored on all the cruxes it has ( Section 5.2 ).

	March Madness	Metaculus
Forecaster	gpt-5.4-nano	gpt-5-nano, with web search
Predictor	gpt-5.4-nano-2026-03-17	gpt-5.4
Extractor	gpt-5.4-nano	gpt-5.4-mini
Scouting reports	gpt-5.4-mini, with web search	none
Matching	Qwen3-Reranker-8B	Qwen3-Reranker-8B

Appendix

Table 10: In March Madness one model forecasts, predicts and extracts, while in Metaculus each role goes to a different model. The model behind each role, behind the March Madness scouting reports and behind the matching of cruxes.

Experiment	Model	Calls	Cost
March Madness
Scouting reports	gpt-5.4-mini + web search	not recorded	not recorded
Forecasts, 62 teams	gpt-5.4-nano (batch)	17,019 stored	not recorded
Forecasts, First Four	gpt-5.4 (batch)	3,483	$\sim$ $100
Multiple-rollout extraction	gpt-5.4-nano	68,076	$\sim$ $290
A priori prediction	gpt-5.4-nano	2,341	not recorded

Appendix

Table 11: API and compute costs for all experiments. Costs are billed amounts where a record exists, dashboard readings for the March 2026 runs and token counts at OpenAI list prices of September 2026 for the later batch runs, and estimates where a figure is given without a record; cells with no per-stage record read not recorded, and batch rates apply to the runs marked batch, at half the standard rates (OpenAI list prices, 2026-09-22). Where the model column names a snapshot, that is the snapshot the run record names; elsewhere the column gives the undated model name the calls were sent with. Which teams meet in later rounds depends on who wins the earlier games, so March Madness forecasts were made for every pairing of the tournament’s teams. None of the 59 games draws on the forecasts for the First Four, the tournament’s play-in round.

Post	Question	Domain
42117	Will AfD win the most seats in the Baden-Württemberg Landtag election?	elections
42116	Will CDU win the most seats in the Baden-Württemberg Landtag election?	elections
42231 (or 40005)	Will any political party or coalition acquire a supermajority in the 2026 Hungarian parliamentary elections?	elections
42118	Will the official result of the Lamag by-election (Sabah), scheduled for January 24, 2026, be overturned before May 1, 2026?	elections
42232 (or 42148)	Will Ali Khamenei cease to be supreme leader of Iran before April 1, 2026?	geopolitics
42119	Will any state actor use chemical weapons in Ukraine before May 1, 2026?	geopolitics

Appendix

Table 12: The 15 Metaculus events, their post identifiers and one domain label each.

LLM Decision-Making

LLM decision-making studies how language models choose, act, seek information, and allocate resources under goals, uncertainty, constraints, and social pressures.

See all 246 papers

Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs
Feb 16, 2026Luise Ge, Yongyan Zhang, Yevgeniy VorobeychikLLM Decision-Making Framing Effects in Language Models
Washington University in St. Louis
Truncated Reasoning Under a Budget: Evaluating System-1 Models as Decision-Makers
Oct 19, 2026Zhongyin Wang, Zhaoqian Xue, Junting Mu +4LLM Decision-Making Efficient Language Model Reasoning
University of Illinois Urbana-Champaign · University of Arizona · University of California, Irvine +3
Superficial Beliefs in LLM Decision-Making
Jun 9, 2026Gabriel Freedman, Francesca ToniLLM Decision-Making Metacognition in Language Models
Department of Computing Imperial College London

Feb 16, 2026J/K move · Enter open · S save

Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs

Luise Ge, Yongyan Zhang, Yevgeniy Vorobeychik

Washington University in St. Louis

LLM Decision-Making Framing Effects in Language Models Large Reasoning Models Human-AI Decision Making

Open paper AskPDF arXiv

The use of large language models either as decision support systems, or in agentic workflows, is rapidly transforming the digital ecosystem. However, the understanding of LLM decision-making under uncertainty remains limited. We study LLM risky choices along two dimensions: (1) prospect representation (based on an explicit representation or outcome history) and (2) decision rationale (explanation). Our study, which involves 20 frontier and open LLMs, is complemented by a matched human subjects experiment, which provides one reference point, while an expected payoff maximizing rational agent model provides another. We find that LLMs cluster into two categories: reasoning models (RMs) and conversational models (CMs). RMs tend towards rational behavior, are insensitive to the order of prospects, gain/loss framing, and explanations, and behave similarly whether prospects are explicit or presented via a history of outcomes. CMs are significantly less rational, slightly more human-like, sensitive to prospect ordering, framing, and explanation, and exhibit a large description-history gap. Paired comparisons of open LLMs suggest that a key factor differentiating RMs and CMs is training for mathematical reasoning.

Abstract

LLM Decision-Making

Explore similar work

Mind the (DH) Gap! A Contrast in Risky Choices Between Reasoning and Conversational LLMs

Truncated Reasoning Under a Budget: Evaluating System-1 Models as Decision-Makers

Superficial Beliefs in LLM Decision-Making