Two Sample Sizes in Test-Time LLM Ensembling: Buffered ERM and Minimax Theory

Zhi Zhang; Arash A. Amini

math.STOct 15, 2026

Two Sample Sizes in Test-Time LLM Ensembling: Buffered ERM and Minimax Theory

Authors: Zhi Zhang, Arash A. Amini

Organizations: Department of Statistics and Data Science, UCLA

Abstract

Test-time scaling can improve the performance of large language models by generating multiple answers to a question and aggregating them. Weighted ensembles extend this idea by drawing answers from several models, with the weights determining how often each model is used. Learning these weights from a training dataset involves two sample sizes: $Q$ labeled questions and $N$ generations per model and question. We study their interplay through the clean risk, the population error of the ensemble's infinite-generation voting rule. Under a margin condition with exponent $β$ , we show that empirical risk minimization (ERM) with a calibrated margin buffer achieves excess risk of order $Q^{-1/2}+N^{-β/2}$ , up to logarithmic factors, for fixed numbers of models and candidate answers. To investigate the optimality of this dependence on $Q$ and $N$ , we analyze a simplified submodel with two models and two possible answers per question. We characterize its minimax excess risk up to constants by a Hellinger modulus for a binomial-mixture experiment. Polynomial approximation and spectral analysis reveal three regimes. When questions are relatively scarce, the $Q^{-1/2}$ term governs learning and the general buffered ERM is optimal. In an intermediate regime, pooling generation counts across questions improves upon estimating each answer distribution separately, with only logarithmic gains from additional questions. With sufficiently many questions, the minimax excess risk reaches an $N^{-β}$ identification floor that persists even with infinitely many questions. The submodel's lower bounds also apply to general ensembles, matching the buffered-ERM upper bound up to logarithmic factors when questions are scarce and in parts of the intermediate regime. Questions remain about the transitions between regimes and the construction of matching pooled estimators for general ensembles.

Figures & tables

	Answer (empirical frequencies in %)
Distribution	70	147	273	$294^{\star}$	495	810	1225	1820	2730	4116	7350	12870	44100	Observed [1pt]support
Gemma 3 12B	4	12	0	20	8	4	24	4	4	4	4	8	4	12
Llama 4 Maverick	0	0	4	44	0	52	0	0	0	0	0	0	0	3
Equal mixture	2	6	2	32	4	28	12	2	2	2	2	4	2	13

Table 1: Empirical answer frequencies for AIME 2024 I, Problem 6, with

N=25

generations per model at temperature

0.8

. Both models were asked to reason step by step and return a boxed final answer; all

50

responses contained complete boxed answers and finished below their token caps. Every observed answer value is shown. The star marks the gold answer, zeros denote unobserved answers, and boldface marks each empirical mode. The last row is the equal-weight mixture.

Figure 1: The scalar binomial channel and its two inferential targets, shown for the family in ( 39 ) at

\eta=1/2

and with

n=12

. Orange and blue distinguish the negative and positive latent mass of

G_{\eta}

, whose signed difference is

\theta(G_{\eta})

. The channel maps the continuous mixing law to the discrete count law

K_{n}G_{\eta}

. The two statistical tasks are to estimate this imbalance or decide which side is larger; neither requires reconstructing

G_{\eta}

itself.

Panel of Figure 2 — Figure 2: A numerical realization of the dual pair in ( 48 ) for $\beta=2$ and $m=n=7$ , corresponding to margin constant $L=2$ . Since the dual witness is bang–bang, the Lebesgue densities of $G_{7,+}$ and $G_{7,-}$ occupy complementary shells beneath the common envelope $2|z|$ . The measures agree on every moment through degree seven, so their seven-trial binomial-mixture laws coincide even though their sign-imbalance functionals differ.

Language Model Ensembles

Language model ensembles combine predictions or reasoning from multiple models to improve accuracy, reliability, coverage, or efficiency.

See all 137 papers

Apr 1, 2026

Test-Time Scaling Makes Overtraining Compute-Optimal

Modern LLMs scale at test-time, e.g. via repeated sampling, where inference cost grows with model size and the number of samples. This creates a trade-off that pretraining scaling laws, such as Chinchilla, do not address. We present Train-to-Test (

T^2

) scaling laws that jointly optimize model size, training tokens, and number of inference samples under fixed end-to-end budgets.

T^2

modernizes pretraining scaling laws with pass@

k

modeling used for test-time scaling, then jointly optimizes pretraining and test-time decisions. Forecasts from

T^2

are robust over distinct modeling approaches: measuring joint scaling effect on the task loss and modeling impact on task accuracy. Across eight downstream tasks, we find that when accounting for inference cost, optimal pretraining decisions shift radically into the overtraining regime, well-outside of the range of standard pretraining scaling suites. We validate our results by pretraining heavily overtrained models in the optimal region that

T^2

scaling forecasts, confirming their substantially stronger performance compared to pretraining scaling alone. Finally, as frontier LLMs are post-trained, we show that our findings survive the post-training stage, making

T^2

scaling meaningful in modern deployments.

Nicholas Roberts, Sungjun Cho, Zhiqi Gao +7

University of Wisconsin-Madison · Stanford University

Scaling Laws Language Model Scaling Laws Test-Time Scaling LLM Training Language Model Pretraining

Ask PDF

Sep 16, 2026

Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling

Test-time scaling can improve large language model reasoning by generating and combining multiple candidate responses. In sampling-based methods, the inference budget is often described by the number of generated candidates, N. However, N tells us how many candidates are generated, not how they are executed. The same candidate budget can be produced in one batched generation call or split across several sequential calls with smaller batch sizes. We first study the effect of increasing N on reasoning accuracy using Phi-3-mini and Qwen2.5-1.5B on 500 GSM8K prompts. As expected, increasing N from 1 to 8 improves accuracy by 8.4 percentage points for Phi-3-mini and 18.4 points for Qwen2.5-1.5B. However, accuracy alone does not show the systems cost of using a larger candidate budget. We therefore fix N = 8 and compare four generation schedules: 1x8, 2x4, 4x2, and 8x1, where axb denotes a generation calls with b candidates per call. We measure latency, throughput, GPU-hours, and gross GPU-device energy while keeping the total candidate count fixed. On A100 GPUs, eight serial calls use 4.64-4.86x as much gross GPU-device energy and have 5.77-6.12x the P95 latency of one batched call with eight candidates. The same pattern appears across three independently scheduled A100 nodes per model and in short-output SciQ/V100 experiments. These results show that candidate count alone is not enough to describe the systems cost of multi-candidate test-time scaling. When candidates are independent and memory allows it, fewer generation calls with larger batch sizes are more efficient. Evaluations should therefore report not only candidate count and accuracy, but also generation schedule and GPU-level systems metrics.

Mobina Kashaniyan, Ali Jannesari

Department of Computer Science Iowa State University, USA

LLM Inference Efficiency LLM Inference LLM Inference Scheduling Test-Time Scaling Parallel LLM Inference

Ask PDF

Aug 4, 2026

Interpretable Adaptive Sampling for LLM Test-Time Scaling

Test-time scaling improves LLM reasoning by generating and aggregating multiple candidate answers, yet many pipelines use fixed per-query budgets that spend the same compute on easy and difficult prompts. These fixed budgets are also difficult to inspect because they do not explain why a given prompt receives a particular number of samples. We propose adaptive} test-time scaling with a lightweight fuzzy controller that maps interpretable signals, including estimated prompt complexity and model confidence, to a per-query sampling budget. The controller assigns fewer samples to easier or more confident prompts and more samples to harder or less certain prompts, making inference-time compute inspectable rather than fixed or opaque. We evaluate under a fair-alignment protocol with matched decoding settings and controlled answer selection, and compare against best-of-

N

, compute-aware scaling, and self-certainty-based baselines on question-answering and mathematical reasoning tasks. Across models and datasets, adaptive fuzzy control improves over several standard baselines and remains close to a selector-matched full-budget control while reducing the average number of samples. These findings suggest that interpretable adaptive sampling is a practical direction for more efficient test-time reasoning in large language models.

Mobina Kashaniyan, Ali Jannesari

Iowa State University Ames, IA, USA

LLM Inference Efficiency Test-Time Scaling Efficient Language Model Reasoning Adaptive Sampling

Ask PDF

Abstract

Language Model Ensembles

Explore similar work

Test-Time Scaling Makes Overtraining Compute-Optimal

Sample Count Is Not Enough: Candidate-Generation Strategy Shapes the Energy and Performance of LLM Test-Time Scaling

Interpretable Adaptive Sampling for LLM Test-Time Scaling