Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3

Natapong Nitarach

cs.CLMar 29, 2026

Model Capability Dominates: Inference-Time Optimization Lessons from AIMO 3

Authors: Natapong Nitarach

Abstract

Majority voting over multiple LLM attempts improves mathematical reasoning, but correlated errors limit the effective sample size. A natural fix is to assign different reasoning strategies to different voters. The approach, Diverse Prompt Mixer, is tested on the AIMO 3 competition: 3 models, 23+ experiments, 50 IMO-level problems, one H100 80 GB, 5-hour limit. Every prompt-level intervention fails. High-temperature sampling already decorrelates errors; weaker strategies reduce accuracy more than they reduce correlation. Across an 8-point capability gap at equal N=8 and every optimization tested, model capability dominates. The gap between the best majority-vote score (42/50) and pass@20 (~45.5) is selection loss, not prompt loss. A verifier-based selector could close it. Prompt engineering cannot.

Figures & tables

Figure 1: Model capability dominates. Per-attempt accuracy

\hat{p}

vs. expected majority-vote score under Binomial voting at

N{=}3,8,16,32

. Seven data points across four model families. At equal

N{=}8

, the 8-point gap between gpt-oss-120b (

\hat{p}{=}0.69

, score 39.3) and gpt-oss-20b (

\hat{p}{=}0.61

, score 31.0) dwarfs every prompt optimization (

\pm

2 points). Scaling

N

beyond compute budget backfires: gpt-oss-20b drops from 31.0 (

N{=}8

) to 26 (

N{=}32

) as per-attempt time shrinks.

Isolated strategies (8 $\times$ each):
Configuration	Orig	Small	Back	Class	$N$	LB Score
Baseline (21 runs)	8	0	0	0	8	39.3 mean
Conservative	5	1	1	1	8	40
Aggressive	3	2	2	1	8	40
Equal	2	2	2	2	8	38
8 $\times$ Small Cases	0	8	0	0	8	37

Table 1: Prompt diversity configurations on gpt-oss-120b . More diversity = worse performance. Code-first: 41, 38, 34 across three runs (mean 37.7, below baseline).

Figure 2: Prompt diversity vs. score on gpt-oss-120b . Blue circles: individual baseline runs (

N{=}21

). Black diamonds: configuration means. Shaded band: baseline

\pm 1\sigma

. More diversity monotonically degrades performance.

Temperature	LB Score	$\Delta$ from baseline
$T{=}0.5$	38	$-1.3$
$T{=}0.8$	40	$+0.7$
$T{=}1.0$	39.3	— (baseline, 21-run mean)
$T{=}1.2$ , min_p $=$ 0.03	37	$-2.3$

Table 2: Temperature ablation on gpt-oss-120b .

T{=}1.0

is optimal; both lower and higher temperatures degrade performance.

Figure 3: Per-problem

\hat{\rho}

vs.

\hat{p}

across four models. Circles: Qwen (

N{=}16

); squares: gpt-oss-120b (

N{=}8

); filled diamonds: gpt-oss-20b (

N{=}8

); hollow triangles/diamonds: Nemotron-Super/ gpt-oss-20b (

N{=}3

, forced

\hat{\rho}{=}-0.500

). All 19 computable points show

\hat{\rho}<0

. Mean

\hat{\rho}=-0.122

for

N{\geq}7

. No correlation headroom for diversity strategies.

Model	Total / Active	Quant	$N$	$\hat{p}$	Local /10	LB Score
gpt-oss-120b	116.8B / 5.1B	FP8	8	0.69	10	39.3 (21 runs)
gpt-oss-20b	20B / 3.6B	FP8	8	0.61	9	31.0 (3-run: 35,28,30 1 )
gpt-oss-20b	20B / 3.6B	FP8	32	0.52	—	26
gpt-oss-20b	20B / 3.6B	FP8	3	0.54	—	28.3 (3-run: 26,28,31)
Nemotron-Super-120B -NVFP4	120B / 12B	NVFP4	3	0.47	—	23
Qwen3.5-35B-A3B	35B / 3B	BF16	16	0.46	8	23

Table 3: Model comparison. At equal

N{=}8

, the 8-point gap between gpt-oss-120b (39.3) and gpt-oss-20b (31.0) dwarfs all prompt optimizations (

\pm

2 points). Scaling

N

to 32 on gpt-oss-20b backfires: per-attempt time shrinks,

\hat{p}

drops from 0.61 to 0.52, score drops from 31.0 to 26.

Experiment	Change	$N$	Local Score
Baseline	—	8	8/10
Long prompts	From GPT-OSS	16	7/10
$N{=}16$ , short	Double attempts	16	8/10
$N{=}16$ , long	Double + long prompts	16	7/10
Entropy voting	Add logprobs $=$ 5	16	8/10
top_k=20	Qwen recommended	16	7/10

Table 4: Qwen3.5-35B-A3B ablation. Nothing improves beyond baseline.

Figure 4: Qwen3.5-35B-A3B ablation on 10 local problems. Blue: baseline (

8/10

). Orange: underperform (

7/10

). Red: crashed. Nothing improves beyond baseline.

ID	Change	LB Score	$\Delta$	Category
Baseline	— (21-run mean)	39.3	—	—
#1	Conservative mix (5+1+1+1)	40	+0.7	Diversity
#2	Aggressive mix (3+2+2+1)	40	+0.7	Diversity
#3	Equal mix (2+2+2+2)	38	$-$ 1.3	Diversity
E1	8 $\times$ Small Cases	37	$-$ 2.3	Strategy
E2	8 $\times$ Work Backwards	39	$-$ 0.3	Strategy

Table 5: Complete ablation on gpt-oss-120b . No experiment reliably exceeds baseline mean. E12 (41) was not replicated in confirmation runs. gpt-oss-20b at equal

N{=}8

scores 31.0; scaling to

N{=}32

drops to 26 as per-attempt time shrinks.

Figure 5: Complete ablation across all experiments. Blue: baseline (39.3). Orange: interventions. Yellow-orange: diversity mixer. Teal: N-ablation. Red/Purple: cross-model. Green dashed: baseline mean. No experiment reliably exceeds baseline.

Competition	Winner	This Work	$N$	Key Innovation
AIMO-1 (2024)	29/50	—	48	Brute-force voting ( XTX Investments, 2024 ; Numina Team, 2024 )
AIMO-2 (2025)	34/50	—	64	Custom training + GenSelect ( Frieder et al., 2024 ; Moshkov et al., 2025 )
AIMO-3 (2026)	46 *	42	8	Off-the-shelf model ( Frieder et al., 2025 )

Table 6: Competition progression. From high-

N

to high-

\hat{p}

: as models improve, inference-time tricks yield diminishing returns. * Top leaderboard score.

Figure 6: AIMO competition progression. Orange bars: winner/top LB scores (29

\to

\to

46). Blue bar: this work (42). Red line: voters

N

(48

\to

\to

8). High-

N

voting gives way to high-

\hat{p}

capability.

Figure 7: Left: score distributions. Baseline (

\mu{=}39.3

\sigma{=}1.7

, blue) vs. Mixer (

\mu{\approx}39.0

\sigma{\approx}2.0

, orange). Red dashed line: target score 42. Right: cumulative probability of

\max\geq 42

over

K

submissions. Baseline:

p{\approx}0.056

per run. Mixer:

p{\approx}0.037

. Dotted line at 42 submissions used.

Appendix figures & tables3 assets

Supplementary material from the paper’s appendix.

Appendix

Parameter	Value	Notes
Model & Serving
Model	gpt-oss-120b	116.8B total, 5.1B active (MoE)
Model path	/kaggle/input/gpt-oss-120b/	Kaggle model hub
Quantization	FP8 (weights + KV cache)	kv_cache_dtype=fp8_e4m3
Engine	vLLM 0.11.x	pip install vllm==0.11.2
GPU	NVIDIA H100 80 GB	Kaggle competition kernel

Appendix

Table A1: Complete baseline configuration.

#	Configuration	LB	Notes
0	Baseline 8 $\times$ original	42	Best ever
1	Conservative (5+1+1+1)	40
2	Aggressive (3+2+2+1)	40
3	Equal (2+2+2+2)	38	Worst diversity
4	Nemotron-14B ( $N$ =16)	3	Format mismatch
5	Qwen3.5-35B-A3B	23	Cross-model

Appendix

Table A2: Complete submission log (42 entries).

\dagger

Public notebook with identical config.

Parameter	Qwen3.5-35B-A3B	Nemotron-Super-120B
Total params	35B	120B
Active params	3B (MoE)	12B (MoE+Mamba-2)
Quantization	BF16	NVFP4
Engine	SGLang	vLLM 0.17.0rc0
$N$ (attempts)	8 / 16	3 / 4
Temperature	1.0	1.0

Appendix

Table A3: Cross-model validation configurations.

Explore similar work

Know Thy Reasoner: Not All Language Models Explore Alike
Apr 12, 2026Moulik Choraria, Argyrios Gerogiannis, Anirban Das +4LLM Reasoning Efficient Language Model Reasoning
UIUC · Capital One · Stony Brook University
Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve
Aug 4, 2026Denys Pushkin, Albert Q. Jiang, Aryo Lotfi +2LLM Prompting CoT Reasoning
EPFL · Apple · Mistral AI
Beyond Compression: Diagnosing How Post-Training Changes Mathematical Reasoning
Sep 29, 2026Hongyang Li, Yiming Zhu, Xiao Li +3Language Model Distillation RL Post-Training
University of Luxembourg · Seafill Open-Source Community · Université Paris-Saclay

Apr 12, 2026J/K move · Enter open · S save

Know Thy Reasoner: Not All Language Models Explore Alike

Moulik Choraria, Argyrios Gerogiannis, Anirban Das, Supriyo Chakraborty+3

UIUC · Capital One · Stony Brook University

LLM Reasoning Efficient Language Model Reasoning Reasoning Diversity

Open paper AskPDF arXiv

Compute scaling for LLM reasoning trades off exploring solution approaches (breadth) against refining promising ones (depth), yet why a given trade-off works, and why it often fails to transfer across models, remains unclear. We argue that the optimal strategy depends on the model's diversity profile, the spread of probability mass across solution approaches, and that this must be characterized before any exploration strategy is adopted. We formalize this with a framework decomposing reasoning uncertainty, deriving when depth-based refinement outperforms parallel sampling, and validate it across three model families at both inference and training. Our central finding is that the diversity regime dictates the strategy: low-diversity aligned models benefit from depth-based refinement with lightweight intrinsic signals, whereas high-diversity base models are often harmed by it, and instead need breadth or stronger signals to compensate.

Abstract

Explore similar work

Know Thy Reasoner: Not All Language Models Explore Alike

Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve

Beyond Compression: Diagnosing How Post-Training Changes Mathematical Reasoning