Variance

Follow this topic Ask about this topic

Momentum

9 papers in the last four weeks, with none the four weeks before. 0.1% of all new papers.

Jul 6Week of Sep 21

Latest papers 70

Sep 30, 2026cs.LG

Cheap to Draw, Expensive to Trust: Certifying Test-Time Scaling Curves

Sampling several answers and keeping the one a verifier scores highest is one of the simplest ways to buy accuracy at test time. Its effect is reported as a scaling curve: accuracy against the number

k

of sampled answers. The curve is cheap to draw and expensive to trust. A budget read off it is chosen after looking at every point, so only a band that covers all budgets at once protects the choice, and on a 100-question benchmark a fixed exact-binomial design needs 192,000 generated answers to certify 64 budgets to within

\pm1/32

at 95%. Most of that cost pays for the wrong uncertainty. A benchmark is a fixed list of questions; at budget 64, about three quarters of the variance of a selected answer's correctness lies between questions, and an audit that revisits every question need not pay for it. We derive the minimax cost of certifying the whole curve, up to logarithmic factors. It has three parts: calibrating the tail of the score distribution, telling the questions apart, and within-question noise summed along the curve. At a single benchmark the last part sharpens to the variance of one answer's influence under the best allocation of answers to questions, which every valid audit pays and an audit that learns the allocation attains, up to a logarithm, as the precision grows. A paired audit built on an exponential inequality for two independent draws at the same question needs no pilot. On 185 held-out score pools it uses 0.74 times the answers of the cheapest competing certified audit at 64 budgets and 0.53 times at 1,024, and on a newly generated MMLU-Pro study it certified the curve with 79,133 answers, within 0.6% of what a cost law fitted beforehand predicted from the study's within-question variance. The same paths certify pass@

k

and majority voting, and the bands extend to populations of questions and to answers that depend on earlier ones.

Sohail, Sarkar, Shakuntala Baichoo

Neel · PMCC AI Lab, Peter Munk Cardiac Centre, University Health Network, Toronto, Ontario, Canada

Model Auditing Test-Time Scaling Variance Curves AI Trustworthiness

Sep 28, 2026cs.CV

Handwritten Text Recognition Lives in the High-Pixel Variance Subspace

In self-supervised pretraining for Handwritten Text Recognition (HTR), pixel reconstruction methods outperform contrastive methods, unlike in natural-image classification. We argue that this difference follows from where discriminative signal lies in pixel space: for HTR, it is concentrated in high-variance directions and largely absent from low-variance ones. This predicts that objectives preserving high-variance pixel content will transfer best. We test six SSL methods from three families (pixel-grounded MIM, JEPA, and contrastive) under matched encoder, data, and evaluation protocols on six handwriting benchmarks across five languages. With full labels, pixel-groundrounded SSL achieves the lowest CER on every benchmark and both frozen probes, exposes per-position character information that other families recover only through the readout, and is the only family to benefit from pretraining on real handwriting. Pixel-grounded representations are also more label efficient. Across datasets, encoder alignment with the high-variance pixel subspace predicts CER within every method. With a pretrained LLM decoder, a frozen pixel-grounded encoder is competitive with fully fine-tuned supervised baselines; full fine-tuning achieves the lowest mean CER and ranks first or second on every benchmark. These results show that the value of pixel reconstruction depends on where discriminative signal lies in the input.

Carlos Garrido-Munoz, Jorge Calvo-Zaragoza

University of Alicante, Spain

Handwritten Text Recognition Handwriting Self-Supervised Learning Character Error Rate Frozen Encoder Text Analysis and Detection+2

Sep 24, 2026cs.CL

Two Emojis of Difference: What Multilingual Affective Generation Benchmarks Actually Measure

We audit a multilingual affective generation benchmark eight instruction-tuned LLMs producing emoji summaries for 17,100 Bangla, English and Hindi sentences, with 6,960 human judgements and find its headline conclusions to be artefacts of the measurement instrument rather than properties of the systems. Treating annotators as a random rather than a fixed factor, no system differs significantly from any other (

F(7,14)=0.59

,

p=0.76

), although the conventional analysis declares 19 of 28 pairwise differences significant. Annotator identity explains far more rating variance than system identity, and the winning system changes whenever any single annotator is removed. The ordering that does emerge tracks output length: mean emoji count explains 78.7% of between-system variance, and a within-item length-matched comparison over 2,599 pairs reverses the leaderboard. We further show that cross-provider anisotropy differences vanish under mean-centring, that per-language token costs change sign with the normalising unit, and that multi-view row-wise splits inflate macro-F1 by

3.1

points and change the top-ranked system. In place of preference scoring we propose emoji-affect decodability, a reference-based probe whose rankings are stable to

\pm0.003

macro-F1 across seeds.

Fardeen Sadab, Adib Sakhawat

Department of Computer Science and Engineering Islamic University of Technology, Dhaka, Bangladesh

Multilingual Benchmark Human Annotators Variance

Sep 22, 2026cs.LG

Gaussian Flow-Matching Schedules: Implications for Sampling and Training

Flow-matching schedules affect both sampling dynamics and the variance of the regression target. For centered commuting Gaussians, we show that a direction-dependent schedule decomposes into two independent design choices: a variance path, which fully determines the intermediate laws and probability flow, and a factorization, which leaves this flow unchanged while controlling irreducible regression variance. On the sampling side, we analyze finite-step Euler accuracy and derive a necessary drift bound for exact N -step sampling, connecting the geodesic and the logarithmic path. On the training side, for any fixed path, we derive closed-form factorizations that either minimize time-averaged regression variance or make it constant along the path.

Arsène Claustre, Hugo Negrel, Claire Boyer +2

DI-ENS · CNRS, ENS Paris · DMA, CFM +6

Time Discretization Variance Schedulers Factor Job Shop Scheduling

Sep 21, 2026stat.ME

The Informational Content in Lepto-Variance and Its Relation to Higher Moments

Lepto-regression is defined as the machine learning process of constructing a Regression Tree of a target feature on itself. It is a novel, model-free method potentially revealing information on important sample structure properties. But it is yet not clear what the informational content of lepto-variance is and how it is related to other well-known statistics of a sample. One significant finding is that 58% of the historical US stock return variability is 1-bit lepto-variance that can not be explained by any financial factor. The central question investigated in this paper is to use small normal N(0, 1) drawn samples to explore how the 1-bit sample lepto-variance and lepto-ratio relate to sample variance, skewness and excess kurtosis. Using a large sample simulation, the lepto ratio of a normal is found to converge to 36.3%. For smaller normally distributed simulated N(0, 1) samples, while lepto-variance itself is highly correlated to sample variance, lepto-variance as a fraction of total variance is highly correlated to excess kurtosis. Both lepto-variance and lepto-ratio are orthogonal to sample skew. Another finding is that while lepto-ratio is strongly correlated to lepto-variance it remains orthogonal to sample variance.

Vassilis Polimenis

University of Limassol, Limassol, Cyprus

Variance Skewness Ratio Ultrasound

Sep 18, 2026cs.CL

How Many Humans Are 32 LLM Judges Worth?

A panel's human-equivalent size is target-specific. Matching a fixed 32-judge panel to empirical human label distributions on three ChaosNLI tasks yields two distinct effective sizes: distributional-error matching gives

ν_{\mathrm{MSE}}=2.304

,

3.750

, and

3.445

, whereas spectral matching gives

ν_H=4.242

,

6.459

, and

6.499

, a gap of

1.72

--

1.89\times

; a binary-error diagnostic credits the same panels with only

1.971

--

2.227

effective votes. Extrapolating the distributional-error curve at fixed squared mean residual, mean member variance, and normalized mean covariance gives asymptotes of

2.392

,

3.990

, and

3.655

, with 32 judges already reaching

94.0

--

96.3\%

. An exact spectral identity explains the gap: error depends on member energy and on the orientation of residual variation relative to averaging, information that the participation ratio (PR) discards. A realizable hard-label construction confirms that higher spectral diversity can coexist with worse distribution recovery even under equal member energies and nonnegative correlations, and the consensus direction retains

γ_{\mathrm{co}}=43.8\%

,

33.7\%

, and

35.9\%

of centered residual variance. An external check on CC-1000, a 1,000-item Civil Comments subset with a different panel, gives

ν_H=2.84

. For panel choice, we establish an existence result and one feasible path: exhaustive enumeration at

k\in\{5,7\}

shows that panels beating the accuracy-top-

k

baseline on both accuracy and

ν_H

always exist, and greedily swapping at most two members reaches

24.8

--

56.0\%

higher

ν_H

at

0.10

--

1.10

percentage points higher accuracy. Our dataset and code are available at https://github.com/Chao1208/32judges-votes.

Chao Li, Yingying Yu, Yunfeng Li

Tsinghua University · University College London · Hebei University of Economics and Business

Judges Large Language Model Judges Choice Variance

Sep 16, 2026cs.LG

Revisiting Distributed Sign-Based Variance Reduction

Sign-based methods reduce communication costs in distributed environments, but aggregating local signs can introduce bias when data are heterogeneous. As a result, existing sign-based variance reduction methods fail to obtain the optimal convergence rates. In this paper, we solve this problem and obtain optimal rates for both nonconvex stochastic and finite-sum optimization. We first give a counterexample showing that majority voting can fail to approach stationary points even with exact local gradients. Motivated by this limitation, we propose tracking the global gradient at the server through unbiased compression of recursive gradient increments. As a result, we can obtain the convergence rates of

O(\sqrt{d/K}+\sqrt d (a/(nK))^{1/3})

for the

\ell_1

-norm and

O(\sqrt{a/K}+\sqrt a/(nK)^{1/3})

for the

\ell_2

-norm. Here,

K

is the iteration number,

n

is the number of workers,

d

is the dimension, and

a=1+ω

, with

ω

denoting the compressor's relative variance. For finite-sum problems with

M

components, we combine periodic exact gradient refreshes with compressed component-gradient differences. The resulting total sample complexities are

O(M+d\sqrt{aM}ε^{-2})

and

O(M+a\sqrt M\ epsilon^{-2})

for

\ell_1

and

\ell_2

gradient norms at most

ε

, matching the corresponding bounds in centralized settings.

Wei Jiang, Zechao Li, Lijun Zhang

School of Computer Science and Engineering, Nanjing University of Science and Technology, China · National Key Laboratory for Novel Software Technology,Nanjing University, China · School of Artificial Intelligence, Nanjing University, China

Decentralized Optimization Variance Reduction Stochastic Convex Optimization Convergence Optimal Sample Complexity Variance+1

Sep 15, 2026cs.LG

Same Flow, Different Paths: Variance Reduction in Flow Matching

In flow matching (FM), a velocity model

v_θ

is trained using a predefined path

g_t

that connects data and noise samples (e.g.,

g_t(x_0, x_1) = (1 - t) x_0 + t x_1

). In this work, we study the choice of this path from an optimization perspective by analyzing the variance of stochastic gradients. We consider the class

G(p_t,v^\star_t)

of paths that induce the same marginal distributions

p_t

and marginal velocity field

v^\star_t

, and therefore the same FM objective. Our main finding is that the choice of path

g_t

can fundamentally change the convergence rate of SGD, even when the FM objective remains exactly the same. (i) For a linear velocity model and one-dimensional Gaussian data, we derive a tight bound on the SGD iteration complexity up to logarithmic factors and find an analytically optimal path that minimizes this bound among linear paths inducing the same FM problem. (ii) We then extend the variance analysis to general FM problems and formulate path selection at a fixed

θ

as the variance-minimization problem PathOpt

_θ

, constrained to

g_t\in G(p_t,v^\star_t)

. We show that this constraint is essential: reducing variance without it can lead to slower convergence. (iii) Since the constraint

g_t \in G(p_t,v^\star_t)

cannot generally be verified directly, we derive an equivalent formulation with constraints that can be estimated from samples, allowing paths to be found numerically. Our theoretical results are supported by experiments with Gaussian data, Gaussian mixture models, and real datasets.

Alexander Tyurin

AXXX, Moscow, Russia · Applied AI Institute, Moscow, Russia

Conditional Flow Matching Variance Reduction Stochastic Gradient Descent Variance Shortest Paths Path

Sep 8, 2026math.OC

Oracle Complexity of Stochastic Fixed-Point Equations with Nonexpansive Maps

We study the oracle complexity of computing a point with small fixed-point residual

\|T(x)-x\| \leq ε

, for a general norm

\|\cdot\|

and a self-map

T

of a compact convex set. We study this problem in the setting where

T

is nonexpansive with respect to the same norm

\|\cdot\|

and accessed via an unbiased stochastic oracle with bounded variance

σ^2

. We provide an algorithm that solves such instances for any norm with a weak Rademacher type

q > 1

, with high probability. The algorithm is based on a recursive anchoring technique. For type-

2

spaces, such as

\ell_p

-spaces for

p \in [2, \infty]

, our algorithm attains stochastic oracle complexity

\tilde O(σ^2 ε^{-3} + ε^{-1})

. We further prove a near-matching lower bound (i.e., matching up to poly-log factors) for such

\ell_{\infty}

-norm instances in high dimensions. Our lower bound holds against any randomized algorithm that succeeds with constant probability. It further extends to settings with ``sparse'' noise, where variance measured with respect to any

\ell_p

norm is of the same order, ruling out the possibility of improving oracle complexity as a function of

\varepsilon

by measuring variance in a non-matching

\ell_p

norm.

Jelena Diakonikolas, Cristóbal Guzmán, David Martínez-Rubio

Department of Computer Sciences, University of Wisconsin-Madison · Institute for Mathematical and Computational Engineering, Faculty of Mathematics and School of Engineering, Pontificia Universidad Católica de Chile, Santiago, Chile · IMDEA Software Institute, Madrid, Spain

First Order Oracle Complexity Fixed-Point Iteration Spectral Norm Stochastic Convex Sets Variance

Sep 2, 2026stat.ML

Posterior Tempering Explains Variance Inflation in Linear and Generalized Linear Thompson Sampling

We study a variant of the Thompson Sampling (TS) algorithm, called

α

-TS, for solving stochastic generalized linear bandit problems. Existing analyses of TS require inflating the posterior variance to derive near-optimal regret guarantees. We formalize the idea of variance inflation by introducing

α

-TS that uses a fractional or

α

-posterior instead of the standard posterior. Our main contribution is to identify general regularity conditions on the prior and reward distributions that enable a regret analysis of

α

-TS without assuming any tractable approximation of the posterior distribution, unlike previous works. For a specific choice of

α\propto d^{-1}

, our general regret bound yields the best known regret bound of

O(d^{3/2}\sqrt{T}\log T)

for both the exponential and sub-Gaussian families of reward distributions. We further provide an

α

-dependent lower bound showing that the regret constant depends on the product

αd

, and that when

α\propto d^{-1}

the regret scales as

Ω(d^{3/2}\sqrt{T})

, explaining the origin of the

d^{3/2}

factor in the upper bound. Our proof technique adapts and combines recent advancements in the analysis of linear bandit problems with first- and second-order posterior concentration theory from the Bayesian statistics literature.

Prateek Jaiswal, Debdeep Pati, Anirban Bhattacharya +1

Daniels School of Business, Purdue University. · Department of Statistics, University of Wisconsin-Madison. · Department of Statistics, Texas A&M University.

Thompson Sampling \Widetilde{\Mathcal{O}}(\Sqrt{T})$ Regret Variance Regret Posterior

Sep 1, 2026cs.LG

Generative Diffusion Surrogates with Analytical Variance Schedule

Stochastic transport describes physical systems in which an initially structured distribution spreads under unresolved forcing, scattering, or heterogeneous media. Useful surrogates for such systems should be probabilistic, time-resolved, and able to represent non-Gaussian distributional structure. Generative diffusion models, which corrupt data with Gaussian noise and learn a reverse flow back to structured states, have these properties. Their noise schedules, however, are usually chosen heuristically: image and audio generation---the canonical use cases---provide no physical clock. In transport, by contrast, the variance, or mean-square displacement, is often known from macroscopic theory or empirical scaling even when the full distribution is not. Here we prescribe the forward noising rate as the time derivative of this variance, turning generative time into a calibrated transport clock. The variance path is enforced by construction, while the learned score field represents how non-Gaussian structure inherited from entrance data is smoothed along that path, requiring no intermediate-time physical transport data. For ballistic-to-diffusive transport in turbulent plasmas, the surrogate matches test-particle distributions, reproduces the laboratory-measured variance scale, and tracks the simulated kurtosis evolution without schedule tuning, enabling calibrated emulation and likelihood-based inference.

Patrick Reichherzer, Gianluca Gregori, David N. Hosking +1

Department of Physics, University of Oxford, Oxford OX1 3PU, UK. · Department of Astrophysical Sciences, Princeton University, Princeton NJ 08544, USA. · Max Planck Institute for Security and Privacy, Bochum 44799, Germany. +2

Diffusion Dynamics Diffusion Models Latent Transport Variance Non-Gaussianity

Jul 31, 2026cs.IR

QASP: Query-Adaptive Robust Vector Search Policy

A fundamental challenge of vector search is achieving consistently high recall while minimizing computational costs. Fixed search parameters cause significant performance variance across queries, and conventional evaluation on average recall masks these per-query disparities. We introduce QASP (Query-Adaptive robust vector Search Policy), which predicts the complete recall progression curve per query via a single upfront supervised regression, from which a search policy is derived for any recall target; this avoids iterative model invocations during search or separate predictors per target. By predicting normalized recall values with scale-invariant features and pre-search inference, QASP generalizes across recall targets, index configurations, and datasets. Its fine-grained progress predictions further enable a lightweight reactive complement that adjusts search depth based on predicted-versus-observed deviations without additional inference. We prove that QASP requires a finite training sample independent of dataset size and dimensionality, that its loss exceeds the irreducible lower bound of any fixed policy by a vanishing margin, and that its data access savings over fixed probing grow exponentially in intrinsic dimensionality. Experimentally, QASP achieves significantly lower recall variance and deviation from target, higher query satisfaction rate, and scales to large data and hierarchical indices without retraining, achieving 99% recall with 80% less data access.

Hakan Ferhatosmanoglu, Kushal Kumar, Tal Wagner +1

Amazon Vancouver, Canada

Vector Database Search Algorithms Residual Vector Quantization Vectors Variance Semi-Supervised Learning

Jul 30, 2026cs.LG

Understanding Submodular Information Measure Based Objectives for Representation Learning: A Variance and Separation Perspective

Submodular Information Measures (SIMs) have recently emerged as a powerful framework for representation learning and multimodal learning. In particular, the SCORE framework~\cite{majee2024score} demonstrated that SIMs can serve as effective objectives for supervised contrastive learning. Despite their empirical success, however, the geometric and statistical properties induced by different submodular information measures remain poorly understood. In this work, we develop a unified theoretical framework connecting SIMs to classical concepts in representation learning and statistical pattern recognition. We show that Total Information (TI) objectives characterize intra-class structure: Graph Cut TI recovers within-class variance, LogDet TI recovers generalized variance and covariance volume, and Facility Location TI induces imbalance-aware separation that emphasizes rare and confusable classes. We further show that Mutual Information (MI) objectives capture complementary notions of inter-class structure: Graph Cut MI is closely related to centroid separation and Fisher-style discrimination, LogDet MI captures covariance-aware separation through Mahalanobis distance, and Facility Location MI measures nearest-mode representational overlap. We validate these theoretical characterizations using controlled synthetic experiments that independently vary variance, covariance, class imbalance, class separation, and multimodal overlap. Across all settings, the empirical behavior closely matches the proposed theory. Our results provide the first unified geometric and statistical understanding of submodular information measures and offer principled guidance for selecting and designing SIM-based objectives for representation learning.

Rishabh Iyer, Truong Pham, Anay Majee

The University of Texas at Dallas, Richardson, TX, USA · Adobe, San Jose, CA, USA

Mutual Information Representational Similarity Analysis Representation Learning Submodular Functions Multimodal Learning Objectives+3

Jul 29, 2026cs.AI

Dimensionality and Measurement Precision in HLE's Multiple-Choice Subset

Humanity's Last Exam (HLE) is widely used to evaluate frontier language models. HLE organizes its questions into eight subject-domain categories, whose subscores are often interpreted as evidence of distinct capabilities. However, no study has assessed whether these labels correspond to empirically separable latent constructs, nor whether the benchmark effectively differentiates between models of similar ability. We evaluate 29 LLMs on the text-only multiple-choice subset of HLE (

J = 428

items) and apply psychometric methods to assess both the dimensionality of the benchmark and the distribution of its measurement precision. Fitting a two-parameter logistic IRT model, we find convergent evidence that HLE measures a single general reasoning factor: McDonald's

ω_h = 0.998

, domain labels explain only 3.5% of item response variance, within- and between-domain residual correlations are nearly identical (Cohen's

d = 0.016

), and domain-specific ability estimates are near-redundant with the total score (

r \geq 0.81

). A separate analysis of the test information function reveals that measurement precision concentrates at moderate ability levels and drops sharply above

θ= 0

, where frontier models sit. These findings suggest that HLE's domain subscores do not warrant distinct capability interpretations and that the benchmark's ability to discriminate among the strongest models is limited.

Mayank Sharma, Savira Nadela, Tyler Matteson

Stanford University

Frontier Models Assessment Precision Variance

Jul 19, 2026cs.SE

Teach it to stop, not just to click

Agentic computer-use RL is reported in single runs, and those numbers mislead. Using verifier-guided repair of a 35B computer-use agent (CUA) across five oracle-graded environments, we show a repaired policy's success rate is dominated by upstream variance: a variance-components decomposition across three cells (crossed data-draw

\times

seed grid, bootstrap CIs) finds evaluation variance negligible (

σ_{\mathrm{eval}} \approx 0

) and the training-seed effect small everywhere (

\leq 10\%

); instead it splits between the data draw and run-to-run nondeterminism, the data draw's share rising to dominant (

48\%

) on the hardest cell. There the run-to-run distribution is bimodal (Hartigan dip

p=0.07

,

k=10

), so a single run has roughly a 30% chance of the failure mode and mean

\pm

std is the wrong summary. On that footing, two findings hold. First, repairability is two-tier in how constrained the corrective action is: a single fixed token installs reliably (done-detection

0.97\pm0.06

), while open-ended corrections are only partial -- spatial-coordinate clicks (grounding

0.53\pm0.35

) and a generative field-fill (

0.14\pm0.04

). Second, the frame-level repair transfers to task success only when the corrective action is the task's sole remaining blocker (LinkedIn 8/20 vs. base 0/15, Fisher

p=0.006

). We caught two of our own over-claims -- a sample-efficiency curve and a 'grounding cannot be bought' boundary -- only by replicating across seeds; a stress test makes the stakes external: a single-run improvement of the size this field publishes would have the wrong sign roughly one-third of the time in a comparable regime. We release a library (cua_reliability) for routine k-seed reporting. The apparatus is, to our knowledge, the first multimodal segment-aggregated on-policy self-distillation (SA-OPSD) update on a real 35B CUA policy.

Barada Sahu, Shivesh Pandey

Cabal AI · Para AI

Agentic Improvement Variance

Jul 16, 2026cs.LG

Accelerating A/B-Tests with Counterfactual Estimation: Reducing Variance through Policy Overlap

Online controlled experiments are the gold standard for hypothesis testing in online platforms. Notwithstanding their ubiquity, they are notoriously expensive to run, and issues of variance hamper statistical power in assessing treatment effects. While standard variance reduction techniques leverage model-based control variates to reduce outcome noise, they remain agnostic to potential structural relationships between competing policies. In this work, we identify a critical inefficiency in the standard A/B-testing protocol: when a treatment and control policy agree on an action, the resulting outcome contributes noise but no signal regarding the treatment effect -- unnecessarily inflating confidence intervals. We propose a novel experimental protocol that exploits this policy overlap to accelerate experimentation. The key insight is to frame the randomised treatment assignment mechanism as a meta-policy, and leverage

Δ

-Off-Policy Estimation methods to obtain unbiased estimates for average treatment effects. We prove analytically that our approach recovers standard A/B-testing practices in the general case, but that its variance scales with the divergence between policies rather than raw outcome variance. Hence, we dominate the standard Difference-in-Means estimator whenever policies have common support, and the improvement is strict whenever the overlap region contributes non-zero residual variance. Empirical results corroborate these theoretical insights -- holding promise for significant impact on the real-world evaluation of recommender systems, information retrieval pipelines, and large language model interfaces.

Olivier Jeunen

Independent Researcher Antwerp, Belgium

Variance Reduction Heterogeneous Treatment Effects Variance Learning-Augmented Algorithms Counterfactual Learning Evaluation Benchmarks

Jul 12, 2026cs.CV

Diversify Diffusion with Temperature Sampling and Variance-Corrective Time Shifting

Diffusion models faithfully reproduce their training distribution, but also inherit its imbalances and leave rare or under-represented modes hard to reach. A natural inference-time remedy is to sample from the high-temperature target

p^{(γ)}_0(x) \propto p_0(x)^γ

for

0 < γ< 1

, which flattens dominant modes and lifts rare ones. However, naive score scaling while correctly reweighting modes also inflates the per-mode variance, breaking the reverse diffusion process and degrading sample quality. We introduce variance-corrective time shifting, a training-free fix that queries the network at a shifted timestep and scales the resulting score by

γ

, canceling the variance inflation while preserving the mode reweighting. The correction turns simple temperature sampling into a practical diversity knob for pretrained diffusion and flow-matching backbones with no retraining, and we demonstrate consistent gains at minimal cost to sample quality and condition fidelity across DiT, Stable Diffusion and Motion Diffusion models. We further show that the timing of the temperature intervention enables coarse-to-fine control: high-noise stages drive compositional diversity across modes, while low-noise stages drive local appearance variation under a fixed composition.

Peizhuo Li, Emre Aksan, Alexandru-Eugen Ichim +2

ETH Zurich · Google

Diffusion Sampling Diffusion Models Diffusion Dynamics Diversity Variance

Jul 11, 2026cs.CL

CAFE: A Compound-AI Factorial Evaluation Framework

We introduce CAFE (Compound-AI Factorial Evaluation), an open-source platform that brings design of experiments to the evaluation of compound AI systems (CAIS). Such systems expose many interchangeable choices - e.g. which retriever, model, or prompt - and practitioners rarely know which of them most affects answer quality. With CAFE, a practitioner registers each swappable component of a pipeline as a factor to build a factorial design over the chosen factors, run the resulting configurations, and score the answers on a shared rubric using a configurable LLM judge together with human raters. From these ratings it attributes answer-quality variance to the components and their interactions with mixed-effects models and reports effect sizes, significance, the best configuration, cost and latency trade-offs, and judge-human reliability. Whereas existing tools mostly either search for a good configuration or score outputs in isolation, CAFE also explains which component drives quality and whether an observed difference is significant. We validate CAFE on a retrieval-augmented question-answering (QA) pipeline over the HotpotQA benchmark dataset, where it recovers planted factor effects and stays calibrated under a permutation null. CAFE is released as a Python package and as a Web application.

Fabian Lukassen, Christoph Weisser, Thomas Kneib +1

University of Göttingen · Bielefeld University of Applied Sciences and Arts (HSBI)

Artificial Intelligence Evaluation Artificial Intelligence Systems Factor Effect Sizes Variance

Jul 5, 2026cs.CL

Language Models Represent and Transform Concepts with Shared Geometry

How concepts are represented in neural networks is a fundamental question in machine learning. The dominant view treats concept representations as stationary geometric objects. Yet concepts appear in context, and context transforms them. Drawing from neural population geometry, we formalize concept representations as point-cloud manifolds and contextual transformations as vector fields, and instantiate this framework in large language models. Across six model families of varying scales, we find that context moves each concept differently. The variance in these displacements is semantically organized, correlating with lexical concreteness and density. Importantly, both the concepts being transformed and this variance structure are shared across models: displacement structure transported from one model predicts held-out displacements in others significantly above chance. Together, these findings show that models share a common geometry not only in how concepts are represented, but more importantly in how context transforms them, a structure with richer organization than prior work has recognized.

Zhimin Hu, Lanhao Niu, Sashank Varma

1Georgia Institute of Technology · University of Edinburgh

Language Modeling Transform Concepts Geometric Constraints Variance

Jul 3, 2026cs.SD

Trajectory Variance: An Unsupervised Measure of Developmental Vocal Plasticity in Birdsong

How much does a vocalization change over the course of development? We propose trajectory variance, a per-vocalization plasticity score that answers this question without type labels. A displacement model learns to predict age-conditioned shifts in autoencoder latent space; the variance of its predictions across target ages quantifies how much each vocalization would change if produced at different developmental stages. Evaluated on three zebra finches (183K-274K vocalizations, 40-101 days post-hatch), trajectory variance separates learned song syllables from innate calls (Cohen's d = 0.29-0.57, AUC = 0.58-0.67, after controlling for duration), while no nonparametric baseline achieves consistent separation. Trajectory variance also correlates with spectral flatness across all three birds (r = -0.48 to -0.75): more plastic vocalizations tend to have more tonal, structured spectra.

Kanghwi Lee

Institute of Neuroinformatics, University of Zurich and ETH Zurich, Switzerland

Animal Vocalizations Vocalizations Developmental Trajectories Variance Bird Unsupervised

Jun 30, 2026cs.LG

GRPO, Dr. GRPO, and DAPO Are Three Operations on One Number: The Group-Standard-Deviation Identity

Three of the most popular methods for training language models to reason look like three different tricks. They are not. All three adjust a single number: standard deviation, reflecting how much a prompt's sampled answers disagree. When such a model is trained, it answers each problem many times, and an automatic checker marks every answer right or wrong. The standard deviation of those marks measures the disagreement: largest when the answers split evenly between right and wrong, and zero when they all agree. Group Relative Policy Optimization (GRPO) divides by this number, GRPO Done Right (Dr. GRPO) drops the division, and Decoupled Clip and Dynamic Sampling Policy Optimization (DAPO) discards the groups where it is zero. Each is presented as its own fix, yet this paper proves they are three settings of one dial. That dial is not cosmetic: for right-or-wrong rewards, the disagreement is exactly the size of the training update, the group-standard-deviation identity. A split group teaches the most, while a unanimous group teaches nothing and falls silent. The same result says which problems deserve the most weight and how many tries each one needs. This paper confirms the intuition on a large real difficulty dataset (Big-Math) and in a controlled training run. What looks like a harmless normalization step is the dial that decides where learning happens and how strongly.

Yong Yi Bay, Kathleen A. Yearick

PhD, University of Illinois at Urbana-Champaign

Variance Disagreement Language Modeling Logical Connectives

Jun 27, 2026stat.ML

Variance Reduction for Stochastic Gradient Generalized Non-reversible Langevin Monte Carlo Algorithms

We study the leading-order fluctuation of stochastic gradient Euler-Maruyama estimators for generalized non-reversible Langevin dynamics. Under structural assumptions tailored to the small-stepsize central limit theorem and under an unbiased stochastic gradient oracle, we prove that the empirical average over a horizon of order the inverse squared stepsize satisfies a central limit theorem in the vanishing-stepsize regime. The limiting variance is characterized through the Poisson equation of the limiting full-gradient diffusion. We then rewrite this constant in an operator form that links it to the continuous-time asymptotic variance and, under standard operator-theoretic assumptions, derive a sufficient condition under which an anti-symmetric perturbation strictly reduces the leading-order fluctuation constant relative to the reversible baseline. We also identify bounded smooth predictive observables that re directly covered by the main theorem. As a separate Gaussian calculation beyond the bounded-test-function regime, we obtain closed-form formulas for quadratic Hamiltonians and linear observables. The framework covers non-reversible Langevin dynamics and augmented-state examples including Hessian-free high-resolution dynamics and a positive-definite subclass of gradient-adjusted underdamped Langevin dynamics that allow stochastic gradients. Numerical experiments on basic examples and Bayesian linear regression using synthetic data, and Bayesian logistic regression using real data support the predicted Gaussian fluctuations and show that the non-reversible schemes consistently reduce the root mean squared error (RMSE) relative to their reversible baselines.

Bingye Ni, Xiaoyu Wang, Yingli Wang +1

FinTech Thrust, Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, People's Republic of China · School of Mathematical Science, Fudan University, Shanghai, People's Republic of China · Department of Mathematics, Florida State University, Tallahassee, Florida, United States of America

Langevin Dynamics Variance Reduction Stochastic Gradient Descent Step Accuracy Fluctuation-Dissipation Theorem Variance+1

Jun 26, 2026cs.NE

Analysis of Parameter Settings for the Bat Algorithm Using Variance Evolution

Parameter settings in evolutionary algorithms and metaheuristics are important because such parameter values can influence the performance of algorithms under evaluation. For a given algorithm, there are many different numerical experiments to show that the algorithm can work well in practice; however, in most cases there is no theoretical analysis of parameter settings. In this work, we show that theoretical analysis using the theory of dynamical systems and evolution of population variance can give some good results in terms of parameter ranges for the bat algorithm. We also show that results from numerical experiments are consistent with theoretical bounds. Such analyses can provide good insights from different perspectives about the algorithmic characteristics such as variance evolution, transition between exploration and exploitation as well as convergence behaviour.

Xin-She Yang, Mehmet Karamanoglu

Faculty of Science and Technology, Middlesex University, The Burroughs, London NW4 4BT, United Kingdom.

Evolutionary Optimization Genetic Algorithms Variance Bandits Convergence

Jun 21, 2026cs.LG

Escaping the Variance Trap: Jacobian-Free Dynamics for Root-Finding Bilevel Optimization

Many central machine learning tasks, from entropy tuning in reinforcement learning to equilibrating generative adversarial networks, are fundamentally stochastic root-finding problems rather than loss minimization. Yet, they are frequently forced into a minimization framework via squared residuals, introducing a critical flaw we identify as the Variance Trap. Standard bilevel minimization algorithms require estimating hypergradients involving implicit Jacobians; in stochastic settings, these terms act as noise amplifiers, destabilizing convergence. We formalize Root-Finding Bilevel Optimization (RF-BO) as a distinct problem class that bypasses this pathology. We propose a Jacobian-free solution using Two-Time-Scale Stochastic Approximation (TTSA) that updates directly along the root error, structurally avoiding variance amplification. We provide the first non-asymptotic convergence guarantees for TTSA in this setting under Markovian noise. Extensive experiments demonstrate the decisive advantage of this paradigm: compared to squared-residual and implicit-gradient baselines, our framework achieves a 2.6% top-1 accuracy gain in SimCLR, 17

\times

faster convergence in non-linear ODE control where baselines fail, significantly improved entropy stability in reinforcement learning, and an 11.1% quality improvement in generative modeling.

Zhiyu Li, Xi Xuan, Davide Carbone

University of Science and Technology of China · City University of Hong Kong · Laboratoire de Physique de l’École Normale Supérieure, Université PSL, CNRS, Sorbonne Université, Université de Paris

Bilevel Optimization Stochastic Approximation Jacobian Convergence Variance

Jun 18, 2026cs.LG

On the Variance of Temporal Difference Learning and its Reduction Using Control Variates

We analyze the variance of temporal difference (TD) learning using the phased setting with tabular representation, and show that one of the mechanisms behind its ability to reduce variance is by effectively aggregating over a larger number of independent trajectories. Based on this insight, we demonstrate that (1) the variance of TD is asymptotically bounded from above by Monte Carlo (MC) estimators, and (2) shorter horizon updates incurs less variance for a fixed number of samples. Beyond TD, we show that Direct Advantage Estimation (DAE), a method for estimating the advantage function, can be seen as a type of regression-adjusted control variate, which achieves a tighter bound on the variance compared to TD in the large-sample limit. Finally, we numerically illustrate the behaviors of these estimators with carefully designed environments.

Hsiao-Ru Pan, Bernhard Schölkopf

Max Planck Institute for Intelligent Systems, Tübingen · ELLIS Institute Tübingen · ETH Zürich

Temporal Difference Variance Reduction Variance Step-Level Advantage Estimation Large Deviation Principle

Jun 17, 2026stat.AP

Ensuring Trustworthy Online A/B Testing: Addressing Five Key Questions on CUPED

A/B testing has become the gold standard for data-driven decision-making in large-scale online experimentation, providing critical guidance for feature launch, pricing optimization, and user experience enhancement. To maximize statistical sensitivity, many technology companies routinely employ Controlled-experiment Using Pre-Experiment Data (CUPED), a technique that achieves substantial variance reduction while preserving the unbiasedness of estimating the average treatment effect. Despite its widespread adoption, several critical methodological and practical nuances of CUPED remain underexplored. This paper systematically addresses five frequently encountered yet overlooked questions regarding the application of CUPED. First, we provide a comparative analysis of various post-CUPED estimators to identify the optimal adjustment specification. Second, we evaluate the validity of regression-based adjustments and delineate robust variance estimation methods tailored for such frameworks. Finally, we extend our investigation to complex but common scenarios, including multi-arm experiments and two-stage sampling designs. Our findings reveal that in these settings, naive reliance on standard variance estimators can lead to severely misleading inferences. By offering rigorous theoretical insights and extensive experimental validation, this work deepens the conceptual understanding of CUPED. Notably, the recommended methodologies have been successfully deployed and integrated into ByteDance's experimentation platform.

Yu Zhang, Bokui Wan, Yongli Qin +2

ByteDance Beijing, China

Heterogeneous Treatment Effects Variance Question

Jun 16, 2026stat.ML

Bounded Difference Concentration for Infinitely Exchangeable Sequences with Applications to AI Benchmark Uncertainty

We consider the concentration properties of functions of infinitely exchangeable random variables. By conditioning on the de Finetti directing measure, we show that the deviation of any function with bounded-difference constants

c_1, \dots, c_n

decomposes into a conditional sampling fluctuation and a latent mixture fluctuation. When this latent mixture is

σ_{\mathrm{mix}}^2

-subgaussian, we establish a concentration inequality with an effective variance proxy of

\frac{1}{4}\sum_i c_i^2 + σ_{\mathrm{mix}}^2

. Crucially, we demonstrate that for zero-sum linear contrasts, such as the difference between a subsample mean and a full population mean, the latent mixture term cancels exactly. This cancellation yields a tight, mixture-free Hoeffding-type bound that provides a direct de Finetti mechanism for the infinite-extendibility limit of recent finite-exchangeable concentration results. We apply this framework to quantify uncertainty in composite AI benchmarks, such as MMLU, where question items naturally exhibit exchangeable dependence across domains. Our results provide both a domain-stratified hierarchical model for bounding the uncertainty of accuracy scores, and a distribution-free, cost-saving statistical guarantee for accurately estimating full benchmark scores from random subsets.

Fangyuan Lin, Spencer Frei, Victor H. de la Pena

Department of Statistics, Columbia University · Google DeepMind

Concentration Bounds Artificial Intelligence Benchmarks Uncertainty Score Variance Uncertainty Latent Dynamics

Jun 15, 2026cs.LG

Uncertainty Quantification of Engineering Structures by Polynomial Chaos Expansion and Multivariate Active Learning

In many engineering applications, a single high-fidelity model produces multiple quantities of interest (QoIs) under the same input parameters, e.g. finite element models of complex physical systems. To alleviate the high computational cost of direct model evaluations, surrogate models are widely used to construct efficient approximations of model responses. Naturally, the accuracy of surrogates strongly depends on the quality of the experimental design (ED). However, a single ED may not provide an adequate representation for all outputs simultaneously, especially when different outputs exhibit varying sensitivities to the input variables. A straightforward solution is to perform separate sampling for each output, but this results in increased sampling complexity and computational cost. From a statistical perspective, such an approach also ignores potential correlations among all outputs and may compromise data consistency. To address this issue, an adaptive sequential sampling method for constructing polynomial chaos expansion surrogate models is generalized for vector valued QoIs. The method sequentially selects new samples from a candidate pool based on their local contribution to the output variance, while balancing distance-based exploration of the input space and exploitation of aggregated variance information across all outputs. Its performance is compared with non-sequential Latin Hypercube Sampling through several numerical examples from engineering problems. Numerical results demonstrate that the proposed strategy improves both surrogate accuracy and stability, and provides a more reliable estimation of second-order statistics.

Qitian Lu, Jafar Jafari-Asl, Panagiotis Spyridis +1

Brno University of Technology, Brno, Czech Republic · University of Rostock, Rostock, Germany

Surrogate Models Uncertainty Quantification Bayesian Quadrature Adaptive Sampling Variance Engineering+2

Jun 15, 2026cs.LG

Demystifying Variance in Circuit Discovery of LLMs

Circuit discovery is a key technique in mechanistic interpretability to pinpoint the model components that are crucial for performing a given task. Although the current state-of-the-art method (EAP-IG) performs well on the metric of (un)faithfulness, it suffers from substantial variability. This includes resampling variance, where the circuit changes when we probe with a new batch of data from the same distribution; rephrasing variance, where the discovered circuit shifts when the prompts are rephrased; and sample-wise variance, where a circuit with low population unfaithfulness exhibits large fluctuations in unfaithfulness across individual samples. This paper studies the roots of these variances. We demonstrate that CEAP, our new circuit discovery method that improves upon EAP-IG with a theoretical guarantee, can substantially lessen resampling variance. We further show that rephrasing variance arises because prompts with different templates tend to activate different circuits in the model. This leads us to argue that it may be challenging to find a comprehensive circuit that explains and controls the model's behavior on a task, which can be expressed in countless templates, suggesting that LLMs may be inherently hard to steer. We show that sparsity, which has been claimed to form more compact and interpretable task circuits, fails to solve this problem. Regarding sample-wise variance, we argue that it is largely benign: extremely poor unfaithfulness scores often stem from how unfaithfulness is defined, rather than from defects in the measured circuits. We show that the magnitude of unfaithfulness is affected by selective contribution scaling, a neural mechanism that accounts for the extremely poor scores sometimes observed.

Frank Zhengqing Wu, Francesco Tonin, Volkan Cevher

Laboratory for Information and Inference Systems (LIONS) École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland · Laboratory for Informationand Inference Systems (LIONS) École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland

Mechanistic Interpretability Circuits Model Discovery Variance Faithful Explanation Discovery

Jun 15, 2026cs.LG

Variance Reduction for Non-Log-Concave Sampling with Applications to Inverse Problems

Sampling from high-dimensional, non-log-concave distributions with unnormalized densities is a fundamental challenge in machine learning, particularly when the exact gradient of the potential is unavailable and must be approximated via stochastic gradients that exhibit high variance under a fixed budget of gradient computations per iteration. Although variance reduction techniques such as SGD with momentum, STORM, and PAGE have demonstrated improved convergence properties in non-convex optimization, their implications for sampling from non-log-concave distributions remain largely unexplored. In this work, we develop the first unified analysis of these estimators for sampling from non-log-concave distributions. We establish improved non-asymptotic convergence rates in

\varepsilon

-relative Fisher information and, under a Poincaré inequality assumption, in squared total variation distance, and further prove weak convergence to the target distribution. We extend our analysis to solving inverse problems with score-based generative priors. We empirically validate our theory and demonstrate that, under a fixed gradient computations per iteration, variance-reduction techniques consistently improve sample quality in two standard imaging applications.

M. Berk Sahin, Ahmet Ege Tanriverdi, Behzad Sharif +1

School of Electrical and Computer Engineering, Purdue University, West Lafayette, Indiana, USA · School of Electrical and Computer Engineering, University of Southern California, California, USA · School of Biomedical Engineering, Purdue University, West Lafayette, Indiana, USA

Log-Concave Distributions Variance Reduction Stochastic Convex Optimization Optimal Sample Complexity Variance Stochastic Gradient Descent+2

Jun 14, 2026hep-lat

Learning the generating functional for variance reduction in lattice QCD

The generating functional in quantum field theory provides the natural framework for constructing correlation functions as derivatives with respect to source operators. We present a methodology that leverages machine-learned normalizing flows to reduce the variance of arbitrary

N

-point correlation functions of bosonic operators in lattice gauge field theory calculations by encoding a representation of the generating functional. We show that it is possible to systematically approach noiseless estimators of correlation functions in this framework. We demonstrate this methodology with applications to calculations of glueball correlation functions and Wilson loops in Quantum Chromodynamics and Yang-Mills theory. The results show up to three orders of magnitude variance reduction.

Ryan Abbott, Yang Fu, Daniel C. Hackett +3

1Physics Department, Columbia University, New York, NY 10027, USA · Center for Theoretical Physics, Massachusetts Institute of Technology, Cambridge, MA 02139, USA · 3The NSF AI Institute for Artificial Intelligence and Fundamental Interactions +3

Effective Field Theory Variance Reduction Variance

Jun 10, 2026cs.LG

Crossing the Validation Crisis: Cross-Validation Reduces Benchmarking Variance Surprisingly Well

Modern machine learning progresses through empirical work, benchmarking new methods to evaluate relative performance. However, the statistical variability inherent to evaluation - exacerbated by the stochastic nature of many algorithms - often makes performance estimation unreliable due to the limited test samples available, leading to a validation crisis in which genuine advances are difficult to discern. In this work, we show that cross-validation improves markedly confidence when evaluating and comparing learning algorithm performances. We introduce the concept of sample gain, which quantifies the virtual data augmentation achieved by using multiple cross-validation splits to reduce benchmarking variance. Experiments on both synthetic and real-world datasets (histopathologic scans and NLP fine-tuning) demonstrate that multiple splits can substantially improve the reliability and stability of performance estimates, with diminishing returns often setting in later than expected. We also introduce a procedure to dynamically early-stop cross-validation by estimating from the first few folds if subsequent folds will bring large sample gains. Our findings highlight the value of pushing cross-validation on available samples to achieve robust and reliable benchmarking.

Célestin Eve, Gaël Varoquaux, Thomas Moreau

MIND Team, Université Paris-Saclay, Inria, CEA, Palaiseau, France · SODA Team, Inria, Palaiseau, France · Probabl

Cross Validation Variance Crossing Crisis

Jun 2, 2026cs.CL

The Unsampled Truth: Quantifying Prompt Artifacts in LM Psychometrics

When prompting language models for psychometric assessment, researchers assume that the responses reflect the injected persona and the meaning of the survey item. We test this premise using a diagnostic design that crosses five semantically distinct baseline personas with five semantically equivalent variants of each of four prompt components (persona wording, task instruction, item wording, option symbol). Measuring the 1-Wasserstein distance between the resulting response distributions and partitioning the variation among the five components allows for the separation of target effects from prompt artifacts. We apply the framework to 13 open-weight small language models (0.6B to 14B) on the Big Five Inventory and the Short Dark Triad. We find that in most models, the task instruction and option symbol displace response distributions further than paraphrasing the persona description or the item itself. For a substantial share of items, the artifact share of explained variation exceeds 50%; non-semantic changes of the prompt account for more response variation than the baseline personas. Our framework lets researchers quantify these prompt artifacts before interpreting psychometric output.

Nils Schwager, Christoph Hau, Simon Münker +1

Trier University, Trier, Germany

Psychometric Properties Psychology Prompt Engineering Truth Variance

May 31, 2026cs.LG

Optimal-Point Variance Reduction For Bayesian Optimization With Regret Guarantee

This paper studies a one-step lookahead Bayesian optimization (BO) method and its theoretical guarantee. Although the empirical effectiveness of one-step lookahead BO methods, such as entropy search, has been studied extensively, they often rely on computationally intractable approximations, and their regret guarantees remain underdeveloped. Thus, this paper proposes a one-step lookahead BO method called optimal-point variance reduction (OVR), which requires only posterior sampling and Monte Carlo approximations. We obtain a uniform error bound over an input domain for the Monte Carlo estimation in OVR. Furthermore, we show that the regularized OVR, with the slight modification to promote exploration, achieves a vanishing Bayesian expected simple regret upper bound. Finally, we demonstrate the effectiveness of OVR through numerical experiments.

Shion Takeno

1Nagoya University

Bayesian Optimization Variance Reduction Bayesian Variance Regret

May 26, 2026cs.LG

Resource-Constrained Affect Modelling via Variance Regularisation Pruning

Affective computing systems are increasingly embedded in pervasive and interactive environments, such as adaptive games, assistive technologies, and resource-constrained platforms, where computational efficiency must be balanced with reliability across diverse users. Model pruning offers an effective way to reduce computational demands, yet existing approaches typically optimise for sparsity alone, without accounting for how parameter removal impacts robustness across individuals. In this work, we introduce Variance-Regularised Pruning (VR), a pruning framework that explicitly incorporates cross-participant stability into the sparsification process. Rather than relying solely on average prediction error, VR evaluates each connection based on its joint contribution to both prediction accuracy and variability across users, prioritising parameters that remain reliable under distributional differences. We evaluate the proposed approach on the AGAIN dataset, which includes arousal annotations collected across nine affect-eliciting game environments. Experimental results demonstrate that VR maintains competitive Concordance Correlation Coefficient (CCC) performance even at 80% sparsity without additional fine-tuning, highlighting its suitability for deployment in real-world, resource-limited affect-aware systems. Overall, the proposed framework supports the development of compact, robust affective models that can operate reliably in real-world interactive environments.

Kosmas Pinitas, Konstantinos Katsifis

Mediterranean College, Athens, Greece · University of Derby, Derby, UK

Affective Computing Valence-Arousal Estimation Regularization Variance

May 25, 2026stat.ML

Statistical Inference for Stochastic Gradient Descent Beyond Finite Variance

Stochastic gradient descent (SGD) is a foundational algorithm for large-scale statistical learning and stochastic optimization. However, statistical inference based on SGD iterates remains challenging when stochastic gradients have infinite variance, as the relevant limiting distributions depend on unknown nuisance parameters. In this paper, we develop an efficient, model-agnostic methodology for constructing confidence regions from SGD trajectories that applies in both finite- and infinite-variance regimes. The procedure is based on a joint weak convergence result for the Polyak-Ruppert averaged estimator and an empirical second-moment normalizer constructed from stochastic gradients along the SGD trajectory. This joint limit yields a self-normalized statistic in which the leading tail-dependent scaling terms cancel. We then use a subsampling calibration scheme to estimate the relevant critical values, avoiding explicit estimation of tail indices, slowly varying functions, or stable-law parameters. The resulting confidence regions are straightforward to implement and are asymptotically valid under both the finite- and infinite-second-moment regimes. Simulation studies show reliable coverage in various settings, supporting the proposed method as a practical tool for uncertainty quantification in stochastic optimization.

Jose Blanchet, Peter Glynn, Wenhao Yang

Management Science and Engineering, Stanford University

Stochastic Gradient Descent Confidence Intervals Statistical Inference Variance

May 25, 2026cs.LG

Boundary Variance Inflation Causes Acquisition Bias in Gaussian Processes

Gaussian processes with stationary kernels on bounded domains exhibit inflated posterior variance near the boundary. Despite being a long-recognized artifact in geostatistics and a source of over-exploration in Bayesian optimization, the causes and effects of boundary-induced acquisition bias are underexplored. We trace the root cause to a simple geometric mechanism: the truncation of the kernel correlation neighborhood at the domain boundary creates an observation-independent distortion that worsens with dimensionality. We show how this distortion manifests across three acquisition classes: variance maximization concentrates selections at the corners, whereas negative integrated posterior variance and expected predictive information gain move selections inward to axis-aligned interior shells. These patterns arise without reference to any objective function, meaning that acquisition behavior can be dominated by kernel geometry rather than the desired task-specific uncertainty. To quantify this, we introduce a function-free selection-profile diagnostic for arbitrary acquisitions, kernels, and bounded-domain geometries.

Maria Bånkestad, Sanna Jarl, Jens Sjölund

RISE Research Institutes of Sweden, Stockholm, Sweden · Uppsala University, Uppsala, Sweden

Gaussian Process Posterior Variance Boundaries Distortion

May 25, 2026cs.LG

Pre-Registering the Detectable Effect: A Paired-MDE Budget for 4-bit Quantization Benchmarks, with a Pilot Audit

This is a planning-method note with an unpaired pilot audit. We adapt the classical paired-binary sample-size calculation (Miettinen, 1968) to quantization benchmarks, giving a conservative minimum detectable effect (MDE) bound

δ^{*} \le (z_{1-α/2}+z_{1-β})\sqrt{ρ_d/m}

in the paired item count

m

and the FP16-NF4 disagreement rate

ρ_d

. The bound turns "how reliable is my quantization claim?" into a one-line budget a benchmark designer can commit to before running. We illustrate the bound on four models and four benchmarks (

k=5

splits of

n=100

), and add a parallel MMLU prompt-template study to put the bound's quantization-noise scale alongside the prompt-noise scale. Assuming

ρ_d=0.10

(an unmeasured planning value), all observed NF4-FP16 deltas fall below the implied MDE, and most cross-split SDs lie within

\pm 1.5

pp of the binomial reference

\sqrt{p(1-p)/n}

, so much of the variance reported as "benchmark unreliability" on

n=100

subsamples is binomial sampling noise. The single borderline cell (OPT-WinoGrande,

|Δ|=3.2

pp) is below the implied MDE at

ρ_d=0.10

but above it at

ρ_d=0.05

, illustrating the planning trade-off the bound makes explicit. On MMLU, prompt-template ranges of 2-10 pp meet or exceed the largest observed quantization delta (3.2 pp), so a quantization audit that does not first fix the prompt template absorbs template variance into its noise floor. We complement the bound with a five-line pre-registration template.

Zexin Zhuang, Yanhang Li, Zhichao Fan

1Southern Methodist University · 2Northeastern University · University of Illinois Urbana-Champaign

Quantized Effect Sizes Token Budget Allocation Model Auditing Variance

May 24, 2026cs.AI

AI Cartography: Mapping the Latent Landscape of AI Benchmark Ecosystems

While aggregate leaderboard scores drive AI development, they contain substantial measurement noise whose sources and magnitudes remain unquantified, making it unclear when rankings reflect genuine capability differences versus evaluation artifacts. We introduce a framework for measuring the latent landscape in AI benchmark ecosystems. Applying Confirmatory Factor Analysis (CFA) and Generalizability Theory to 4,000+ models from the Open LLM Leaderboard, we decompose sources of ranking variance and establish: (1) structures assumed in current reporting practice underestimate the strength of relationships between benchmarks; (2) evidence of local dependence among leaderboard items, undermining uses of benchmarks as measurement instruments under current scoring systems; (3) contributor metadata explains more rank-relevant variance (

\approx9\%

) than architecture or deployment categories in this context; (4) a manifest-score "scaling law" slope has low reliability (

R_β=0.53

); by contrast, the latent general-factor size slope is highly stable across ecosystem controls (

R_g=0.97

). We are able to provide unique insights into benchmark dynamics, such as which benchmarks are a function of LLM size and which can be oppositely impacted by post-training practices. We provide actionable diagnostics to determine how benchmark rankings can be trusted and how benchmark design can be improved.

Michael Hardy, Anka Reuel, Lijin Zhang +6

Stanford University, Stanford, CA, USA · BroadMetrics, Edison, NJ, USA

Artificial Intelligence Benchmarks Leaderboard Variance

May 20, 2026cs.LG

Variance Reduction for Expectations with Diffusion Teachers

Pretrained diffusion models serve as frozen teachers feeding downstream pipelines such as text-to-3D, single-step distillation, and data attribution. The teacher gradients these pipelines consume are Monte Carlo (MC) expectations over noise levels and Gaussian noise samples; their estimator variance dominates compute cost because each draw requires expensive upstream work (rendering, simulation, encoding). We introduce CARV, a compute-aware variance-accounting framework that motivates a hierarchical MC estimator: amortize the expensive upstream computation over cheap diffusion-noise resamples, sharpened by timestep importance sampling and a stratified-inverse-CDF construction. In our text-to-3D distillation and attribution experiments, CARV delivers 2-3x effective compute multipliers (most from amortized reuse; ~25% additional from IS+stratification) without changing the objective; in single-step distillation, the same techniques cut gradient variance by an order of magnitude but do not improve downstream FID, marking the regime where MC variance is no longer the bottleneck.

Jesse Bettencourt, Xindi Wu, Matan Atzmon +2

1NVIDIA · University of Toronto · 3Princeton University

Variance Reduction Variance Monte Carlo Computational Cost

May 18, 2026cs.AI

New Insight of Variance reduce in Zero-Order Hard-Thresholding: Mitigating Gradient Error and Expansivity Contradictions

Hard-thresholding is an important type of algorithm in machine learning that is used to solve

\ell_0

constrained optimization problems. However, the true gradient of the objective function can be difficult to access in certain scenarios, which normally can be approximated by zeroth-order (ZO) methods. The SZOHT algorithm is the only algorithm tackling

\ell_0

sparsity constraints with ZO gradients so far. Unfortunately, SZOHT has a notable limitation on the number of random directions % in ZO gradients due to the inherent conflict between the deviation of ZO gradients and the expansivity of the hard-thresholding operator. This paper approaches this problem by considering the role of variance and provides a new insight into variance reduction: mitigating the unique conflicts between ZO gradients and hard-thresholding. Under this perspective, we propose a generalized variance reduced ZO hard-thresholding algorithm as well as the generalized convergence analysis under standard assumptions. The theoretical results demonstrate the new algorithm eliminates the restrictions on the number of random directions, leading to improved convergence rates and broader applicability compared with SZOHT. Finally, we illustrate the utility of our method on a ridge regression problem as well as black-box adversarial attacks.

Xinzhe Yuan, William de Vazelhes, Bin Gu +1

Harbin Institute of Technology · IASM, Harbin Institute of Technology · Mohamed bin Zayed University of Artificial Intelligence +2

Variance Reduction Adaptive Thresholding Gradient Variance

May 15, 2026cs.CV

Second-Order Multi-Level Variance Correction for Modality Competition in Multimodal Models

Autoregressive next-token training offers a unified formulation for image generation and text understanding, but it also creates strong modality competition that destabilizes optimization and limits large-batch scaling. We show that first-order optimizers such as AdamW are vulnerable to cross-modality gradient heterogeneity, while second-order preconditioning, particularly SOAP, provides a more stable basis for multimodal alignment. Building on this insight, we propose \emph{ML-FOP-SOAP}, a second-order optimization framework with Multi-Level Variance Correction. Our Fisher-Orthogonal Projection suppresses variance-induced modality conflicts, reducing the trade-off between visual generation and textual understanding. To make this practical under large gradient accumulation, we introduce a hierarchical folding strategy that captures fine-grained variance with low micro-step overhead. Experiments on Janus and Emu3 show consistent gains across both modalities and stable training at batch size 8192. Compared with AdamW, our method improves sample efficiency by up to

1.4\times

and accelerates wall-clock training by up to

1.5\times

, offering a robust optimizer for scaling multimodal foundation models.

Yishun Lu, Wes Armour

University of Oxford, Oxford, United Kingdom

Multimodal Alignment Multimodal Model Modalities Autoregressive Token Prediction Variance

May 15, 2026cs.CV

Embedding-perturbed Exploration Preference Optimization for Flow Models

Recent advancements have established Reinforcement Learning (RL) as a pivotal paradigm for aligning generative models with human intent. However, group-based optimization frameworks (e.g., GRPO) face a critical limitation: the rapid decay of intra-group variance. As the distinctiveness among samples within a group diminishes, the variance approaches zero. This eliminates the very learning signal required for optimization, rendering the process unstable and forcing the policy into premature stagnation or reward hacking. Existing strategies, such as varying the initial noise or increasing group sizes, often fail to address this fundamental issue, resulting in training instability or diminishing returns. To overcome these challenges, we propose

\textbf{Embedding-perturbed Exploration Preference Optimization (}E^2\textbf{PO)}

, a novel framework that sustains optimization through embedding-level perturbation. Our method introduces structured, embedding-level perturbations within sample groups, guaranteeing a robust variance that preserves the discriminative signal throughout the training process. Extensive experiments demonstrate that our approach significantly outperforms state-of-the-art baselines, achieving a more faithful alignment with human preference.

Sujie Hu, Chubin Chen, Jiashu Zhu +3

1Tsinghua University · ⋆Work done during the internship at AMAP · 2AMAP, Alibaba Group

Stochastic Exploration Flow Models Exploration Variance

May 15, 2026cs.LG

Position: Zeroth-Order Optimization in Deep Learning Is Underexplored, Not Underpowered

Zeroth-order (ZO) optimization, learning from finite differences of function evaluations without backpropagation, has recently regained attention in deep learning due to its memory efficiency and applicability to gray- or black-box pipelines. Yet, ZO methods are often dismissed as fundamentally unscalable because of estimator variance and unfavorable query complexity. We argue that this conclusion might be misguided: ZO optimization is underexplored, not underpowered. We show that many perceived limitations stem from myopic development practices, most notably full-space, element-wise, estimator-centric designs. We articulate six positions spanning the algorithmic, systems, and evaluation stack. First, we revisit the feasibility boundaries of estimator-centric ZO methods through variance control, variance-query tradeoffs, and directional-derivative lenses. Then, we identify three underexplored opportunities: (i) subspace and spectral views of ZO that enable interpretable variance reduction with graceful query scaling, (ii) the forward-only nature of ZO as a systems advantage for communication-efficient, pipeline-friendly, and resource-constrained training, and (iii) the need to de-obfuscate ZO evaluations from task complexity. We strongly advocate rethinking ZO optimization around its unique strengths and acting accordingly, opening a viable path toward large-scale, system-aware, and resource-efficient learning with ZO optimization.

Sijia Liu, Yicheng Lang, Soumyadeep Pal +6

1OPTML Lab, Michigan State University, USA · 2Lawrence Livermore National Laboratory, USA

Zeroth-Order Optimization Zeroth-Order Deep Learning Variance Position

May 14, 2026cs.LG

Beyond Bounded Variance: Variance-Reduced Normalized Methods for Nonconvex Optimization under Blum-Gladyshev Noise

We study nonconvex stochastic optimization under the Blum-Gladyshev (

\mathsf{BG}

-0) noise model, where the stochastic gradient variance grows quadratically with the distance from the initialization. We consider this problem under both standard smoothness and the symmetric generalized-smoothness framework, which captures objectives whose local curvature can scale with the gradient norm. We prove that normalized stochastic gradient descent with momentum, using only one stochastic gradient per iteration, converges under

\mathsf{BG}

-0 noise with oracle complexity

O(\varepsilon^{-6})

. This rate holds both for standard smoothness and for

α

-symmetric generalized smoothness, showing that generalized smoothness is rate-neutral for normalized momentum in this setting. We then study a variance-reduced normalized STORM method. Under mean-square smoothness and sharp initialization, the method achieves the minimax optimal

O(\varepsilon^{-4})

complexity, matching the lower bound. Under expected

α

-symmetric generalized smoothness, the STORM recursion couples gradient-dependent smoothness with distance-dependent noise, leading to complexity

O(\varepsilon^{-(4+α)})

for

α\in(0,1)

and

O(\varepsilon^{-5})

for

α=1

. When the distance-growth parameter in the noise model vanishes, our guarantees recover the standard bounded-variance rates:

O(\varepsilon^{-4})

for momentum,

O(\varepsilon^{-3})

for variance reduction, and

O(\varepsilon^{-2})

in the deterministic case. To our knowledge, these are the first convergence guarantees for normalized methods in non-convex stochastic optimization under

\mathsf{BG}

-0 noise without bounded domains, increasing batch sizes, or explicit anchoring, covering both standard and generalized smoothness regimes.

Antesh Upadhyay, Arda Fazla, Abolfazl Hashemi

School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA.

Stochastic Convex Optimization Stochastic Gradient Descent Convex Optimization Smoothness Nonconvex Variance+3

May 14, 2026cs.AI

Heuristic Pathologies and Further Variance Reduction via Uncertainty Propagation in the AIVAT Family of Techniques

How should an agent's performance in a multiagent environment be evaluated when there is a limited sample size or a high cost of running a trial? The AIVAT family of variance reduction techniques was proposed to address this challenge by introducing unbiased low-variance estimators of agents' expected payoffs. An important component of AIVAT is a heuristic value function that discriminates between potentially low- and high-value counterfactual histories. A notable gap in the literature is that there is little to no constraint or guideline on how the heuristic value function should be chosen or how uncertainty in its output should be handled. In our first contribution, we parameterize the heuristic value function to highlight AIVAT's potential vulnerabilities: a) the sample variance can be set pathologically low by directly applying gradient descent on the sample variance, and b) one can p-hack to draw a desired statistical conclusion via gradient descent/ascent on the test statistic. The main takeaway is that the heuristic value function should be fixed prior to observing the evaluation data! In our second contribution, we show how the heuristic uncertainty can be propagated to quantify the uncertainty of AIVAT estimates. It is then possible to further reduce the variance using inverse-variance weighted averaging, but AIVAT's unbiasedness guarantee may have to be sacrificed. In our experiments, we use a dataset of 10,000 poker hands to demonstrate our heuristic pathology and uncertainty results, with the latter yielding a 43.0% reduction in the number of samples (poker hands) needed to draw statistical conclusions.

Juho Kim, Tuomas Sandholm

Computer Science Department Carnegie Mellon University

Variance Value Functions Heuristics Monte Carlo Multi-Agent Planning

May 13, 2026cond-mat.stat-mech

Beyond Explained Variance: A Cautionary Tale of PCA

We address shortcomings of principal component analysis (PCA) for visualizing high-dimensional data lying on a nonlinear low-dimensional manifold via two-dimensional scatterplots, focusing on a fossil teeth dataset from the early mammalian insectivore Kuehneotherium. While the PCA scatterplot reported by Jolliffe and Cadima (Philosophical Transactions of the Royal Society A, 2016) shows clustering in the region where PC2 < 0, our analysis based on t-SNE and persistent homology (PH) reveals a ring-like structure with no evident clustering and intrinsic dimensionality equal to one. We further propose a generative probabilistic-geometric model in which the data are sampled uniformly from a unit circle. Under this model, pairwise cosine distances follow an arcsine distribution, in qualitative agreement with the observed U-shaped distribution, thereby independently supporting the analysis based on t-SNE and persistent homology.

Gionni Marchetti

Barcelona, Spain

Principal Component Analysis Intrinsic Dimensionality Persistent Homology Variance Security Signals

May 13, 2026cs.CV

Reducing Bias and Variance: Generative Semantic Guidance and Bi-Layer Ensemble for Image Clustering

Image clustering aims to partition unlabeled image datasets into distinct groups. A core aspect of this task is constructing and leveraging prior knowledge to guide the clustering process. Recent approaches introduce semantic descriptions as prior information, most of which typically relying on matching-based techniques with predefined vocabularies. However, the limited matching space restricts their adaptability to downstream clustering tasks. Moreover, these methods primarily focus on reducing bias to improve performance, frequently overlooking the importance of variance reduction. To address these limitations, we propose GSEC (Image Clustering based on Generative Semantic Guidance and Bi-Layer Ensemble), a framework designed to reduce bias through generative semantic guidance and mitigate variance via ensemble learning. Our method employs Multimodal Large Language Models to generate semantic descriptions and derive image embeddings via weighted averaging. Additionally, a bi-layer ensemble strategy integrates cross-modal information through BatchEnsemble in the inner layer and aligns outputs via an alignment mechanism in the outer layer. Comparative experiments demonstrate that GSEC outperforms 18 state-of-the-art methods across six benchmark datasets, while further analysis confirms its effectiveness in simultaneously reducing both bias and variance. The code is available at https://github.com/2017LI/GSEC.git.

Feijiang Li, Zhenxiong Li, Jieting Wang +3

Institute of Big Data Science and Industry, Shanxi University · Key Laboratory of Evolutionary Science Intelligence of Shanxi Province · School of Artificial Intelligence, Shanxi University

Clustering Ensemble Cross-Modal Variance

May 12, 2026cs.LG

Decomposing the Generalization Gap in PROTAC Activity Prediction: Variance Attribution and the Inter-Laboratory Ceiling

Machine-learning predictors of biochemical activity often exhibit large random-split-to-leave-one-target-out generalisation gaps that have been documented but not decomposed. We frame this as an evaluation-science question and use targeted protein degradation as the empirical test bed. PROTACs (proteolysis-targeting chimeras) are heterobifunctional small molecules that induce targeted protein degradation, with more than forty candidates currently in clinical trials; published predictors report AUROC of 0.85 to 0.91 under random-split cross-validation, while the leave-one-target-out (LOTO) protocol of Ribes et al. reduces performance to approximately 0.67. Random splits reward within-target interpolation, whereas LOTO measures the novel-target prediction that de-novo design depends on. We decompose this gap and identify inter-laboratory measurement variance as the dominant component, anchored by a within-target cross-laboratory cascade bounding the inter-laboratory contribution at 0.124 AUROC, well above the 0.05 contribution from binarisation-threshold choice. Across eight published architectures and ESM-2 protein language models up to 3B parameters, LOTO AUROC plateaus near 0.67, with a comparable plateau under SMILES-level deduplication; a 21-dimensional 2000-trial hyperparameter optimisation cannot break this ceiling, and the rank-1 single-seed configuration regresses by 0.161 AUROC under multi-seed validation, matching a closed-form selection-bias prediction (Bailey and Lopez de Prado, 2014). Few-shot k=5 stratified per-target retraining combined with ADMET features lifts 65-target LOTO AUROC from 0.668 to 0.7050, and post-hoc Platt scaling recovers raw output to within the 0.05 well-calibrated threshold. We release PROTAC-Bench (10,748 measurements, 173 targets, 65 LOTO folds), the variance-decomposition framework, the per-target calibration protocol, and the evaluation code.

Thor Klamt, Wolfgang Nejdl, Ming Tang

L3S Research Center, Leibniz Universität Hannover, Appelstraße 9a, 30167 Hannover, Germany · Institute of Data Science (Knowledge-Based Systems), Faculty of Electrical Engineering and Computer Science,2026 Leibniz Universität Hannover, Appelstraße 9a, 30167 Hannover, Germany

Protein Receiver Operating Characteristic Area Under The Curve Drug Design Ceiling Variance

May 10, 2026cs.NE

RDEx-CASK: Cauchy Mutation, Archive, and Stagnation Kick for RDEx-CSOP

We extend RDEx-CSOP with 3 changes that target stagnation & late-stage variance, plus minor parameter tuning. The second scale factor in the standard branch is sampled independently from a truncated Cauchy. A small feasible-only JADE-style archive (|A|_max = 50) is added & sampled with probability |A|/(|A|+|P|). Per-individual stagnation counter triggers, after 180 no-improvement generations, three local overrides on standard branch: pull toward the global best, lift the archive sampling floor to 0.65, & saturate CR to 0.95 when population success rate is below 0.10. The exploitation biased branch & every other RDEx component are left untouched. On CEC CSOP suite (D=30, 25 runs), RDEx-CASK is competitive with RDEx, UDE-III, & CL-SRDE in feasibility-aware quality & improves time-to-target on most problems.

Dikshant, Dikshit Chauhan, Chen Hao +3

Lattice Mutation Variance Archive Small Models

May 10, 2026cs.LG

On Variance Reduction in Learning Mean Flows

One-step generative modeling has emerged as a leading approach for amortizing the inference cost of diffusion and flow-matching models. Among distillation-free methods, MeanFlow training is notoriously unstable, with non-decreasing loss and unbounded gradient variance. In this work, we establish a theory that attributes this pathology to a misuse of the conditional velocity field. We show that the conditional velocity plays two distinct statistical roles in the loss: both as an unbiased regression target and as a Monte Carlo control variate in a Jacobi-vector product, with the original MeanFlow loss assigning the wrong coefficient to the latter. We derive the optimal coefficient in closed form and show that a family of fixes in concurrent works corresponds to different practical realizations of the same optimum. A controlled sweep of this coefficient on two-dimensional benchmarks and on a latent Diffusion Transformer recovers the predicted bias-variance ordering. Our DiT experiment also reveals a quantitative FID-MSE landscape mismatch. Specifically, although the gradient-MSE is minimized at an interior coefficient value near

β\!=\!0.94

, the coefficient that minimizes FID prefers to use conditional velocity directly at the unbiased corner. Our analysis therefore explains why MeanFlow is unstable and unifies its concurrent remedies, and shows that the variance-optimal coefficient need not coincide with the quality-optimal one.

Juanwu Lu, Ziran Wang

Purdue University

Latent Flow Velocity Field Variance Reduction Variance Generative Models

May 8, 2026cs.LG

SGD for Variational Inference: Tackling Unbounded Variance via Preconditioning and Dynamic Batching

Black-Box Variational Inference (BBVI) typically relies on Stochastic Gradient Descent (SGD) to optimize the Evidence Lower Bound (ELBO). However, the stochastic gradients in BBVI inherently exhibit unbounded variance, violating standard assumptions and instead satisfying the weaker Blum-Gladyshev (BG) condition, where variance grows quadratically with distance from the optimum. In this paper, we bridge the gap between stochastic optimization theory and the practical instances of BBVI. Focusing on the broad elliptic location-scale family of parameterized distributions, we offer two main contributions. First, we prove the existence of an ELBO solution, a foundational property usually assumed a priori in the literature. Second, we establish comprehensive convergence guarantees spanning finite-time and asymptotic regimes for Minibatch Projected SGD (PSGD) equipped with dynamic batching and preconditioning under the BG condition. Our theoretical framework demonstrates that dynamic batching combined with preconditioning systematically enables rigorous guarantees even in complex settings. We illustrate our theoretical findings with numerical results, highlighting the efficacy of our approach for modern inference tasks.

Hippolyte Labarrière, Cesare Molinari, Silvia Villa +1

MaLGa, DIBRIS Università degli Studi di Genova Genoa, Italy · MaLGa, DIMA Università degli Studi di Genova Genoa, Italy · MaLGa, DIBRIS, IIT Università degli Studi di Genova Genoa, Italy

Variational Inference Stochastic Gradient Descent Stochastic Optimization Variance Preconditioner

May 7, 2026cs.LG

The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparity

Despite the prevalence of the attention sink phenomenon in Large Language Models (LLMs), where initial tokens disproportionately monopolize attention scores, its structural origins remain elusive. This work provides a \textit{mechanistic explanation} for this phenomenon. First, we trace its root to the value aggregation process inherent in self-attention, which induces a systematic variance discrepancy. We further demonstrate that this discrepancy is drastically amplified by the activation of super neurons within Feed-Forward Network (FFN) layers. Specifically, the channel-sparse down-projections trigger a dimension disparity of the first-token representation, necessitating the formation of attention sinks as a structural anchor. Then, we validate this causal chain through two controlled interventions: (i) isolating the aggregation effect via attention mask modifications and (ii) amplifying the variance of targeted token representations. Both interventions can replicate attention sinks at arbitrary positions. Our mechanistic understanding offers a foundation for the systematic control of sink formation. Finally, as a proof of concept, we propose \textit{head-wise RMSNorm}, an architectural modification that stabilizes value aggregation outputs during pre-training. Our experiments demonstrate that restoring statistical parity across positions significantly accelerates convergence.

Siquan Li, Kaiqi Jiang, Jiacheng Sun +1

1The Chinese University of Hong Kong, Shenzhen · 2Huawei

Feed-Forward Neurons Variance

May 7, 2026cs.LG

Matrix-Decoupled Concentration for Autoregressive Sequences: Dimension-Free Guarantees for Sparse Long-Context Rewards

Sequence-level evaluations in autoregressive Large Language Models (LLMs) rely on highly dependent token generation. Establishing tight concentration bounds for these processes remains a challenge due to two fundamental bottlenecks in existing frameworks: (i) classical inequalities typically separate dependency structures from target sensitivities, leading to a scalar collapse that inflates the variance proxy to a suboptimal

\mathcal{O}(N)

for sparse terminal rewards; (ii) conversely, while certain spatial methods achieve tighter bounds, they lack the strictly causal filtration required by sequential generation, rendering them inapplicable to the autoregressive setting. To resolve both bottlenecks, we establish a sharp McDiarmid-type inequality for dependent sequences, governed strictly by the exact matrix-vector multiplication of the causal dependency resolvent and the target sensitivity vector. This Matrix-Decoupled Concentration (MDC) framework natively recovers optimal constants for Markov chains and exploits directed

d

-separation to yield order-optimal bounds for causal trees. Crucially, by exactly preserving the coordinate-wise sparsity of rewards within a strictly causal framework, MDC mathematically prevents scalar collapse, guaranteeing a dimension-free

\mathcal{O}(1)

variance proxy and providing a rigorous mathematical justification for the stability of long-context reasoning.

Pei-Sen Li

School of Mathematics and Statistics, Beijing Institute of Technology, Beijing 100872, China

Autoregressive Language Models Autoregressive Model Concentration Bounds Causal Attention Concentration Variance

May 2, 2026cs.AI

Format Sensitivity Index: Token-Controlled Prompt Wrapper Robustness and Schema Compliance in LLM Benchmarking

Prompt wrappers often differ only in formatting, yet they can change model scores enough to flip leaderboard conclusions. We study this variance under a token-controlled protocol and introduce two complementary metrics: the Format Sensitivity Index (FSI), the accuracy range induced by wrapper choice, and the Parseability Sensitivity Index (PSI), the corresponding range in answer parseability. Across 140,000 OpenRouter generations spanning 7 QA tasks, 5 wrapper families, and 4 instruct models from 7B to 72B parameters, we find that mean FSI varies by over 30x across models and is largely explained by compliance failures. A fixed-effects regression shows that parseability remains a strong predictor of accuracy even after controlling for task, model, and wrapper. We argue that reporting accuracy without wrapper variance and compliance is statistically fragile, and we give practical recommendations for both benchmarking and structured-output deployments.

Deep Pankajbhai Mehta

Adobe Inc.

Large Language Model Benchmarks Variance

May 1, 2026cs.CL

Prompt-Induced Score Variance in Zero-Shot Binary Vision-Language Safety Classification

Single-prompt first-token probabilities from zero-shot vision-language model (VLM) safety classifiers are treated as decision scores, but we show they are unreliable under semantically equivalent prompt reformulation: even when the binary label is constrained to a fixed output position, equivalent prompts can induce materially different unsafe probabilities for the same sample. Across multimodal safety benchmarks and multiple VLM families, cross-prompt variance is strongly associated with prompt-level disagreement and higher error, making it a useful fragility diagnostic. A training-free mean ensemble improves NLL on all 14 dataset-model evaluation pairs and ECE on 12/14 relative to a train-selected single-prompt baseline, and wins more head-to-head NLL comparisons than labeled temperature scaling, Platt scaling, and isotonic regression applied to the same prompt. Ranking gains are consistent against the train-selected baseline on both AUROC and AUPRC, and against the full 15-prompt distribution remain consistent on AUPRC while softening on AUROC. Labeled calibration on top of the mean provides further gains when labels are available, identifying prompt averaging as a strong label-free first stage rather than a replacement for calibration. We frame this as a reliability stress test for zero-shot VLM first-token safety scores and recommend prompt-family evaluation with mean aggregation as a standard label-free reliability baseline.

Charles Weng, Dingwen Li, Alexander Martin

Johns Hopkins University · Independent

Large Language Model Safety Zero-Shot Robustness Safety Alignment Single-Token Output Distributions Variance

Apr 25, 2026cs.LG

A Tale of Two Variances: When Single-Seed Benchmarks Fail in Bayesian Deep Learning

In limited-data settings, a single endpoint mean of an evaluation metric such as the Continuous Ranked Probability Score (CRPS) is itself a random variable, yet it is routinely reported as if it were a stable property of the method. We study when this practice fails. Using 50 independent repetitions across six regression datasets, we show that CRPS variance trajectories differ substantially across methods and are not always well described by a smooth power-law decay. Methods with a learned heteroscedastic variance head, namely MAP and Deep Ensembles, can develop pronounced, reproducible variance peaks at intermediate training sizes on real datasets, whereas MC Dropout and Bayes by Backprop typically show smooth variance contraction. These peaks have direct practical consequences: at the variance peak on Seoul Bike, the relative RMSE of a single-seed MAP estimate reaches 93.6%, and the probability of falling within

\pm 10\%

of the repeated-run mean drops to 5.9%. We show that local CRPS variance provides a direct signal of single-seed estimation error, with Spearman correlations above 0.96 on every real dataset. Power-law fit quality and monotonicity together provide compact method-level summaries of trajectory regularity. Finally, replacing the standard heteroscedastic objective with

β

-NLL substantially reduces the irregular behavior, consistent with the view that the heteroscedastic training objective contributes to the instability. Practitioners should report trajectory summaries alongside endpoint means and concentrate repeated evaluation in high-variance regions.

Qishi Zhan, Minxuan Hu, Liang He +2

Department of Mathematical and Statistical Sciences Marquette University, USA · Cornell Ann S. Bowers College of Computing and Information Science Cornell University, USA · Physics Science and Engineering Tongji University, China +2

Bayesian Neural Networks Variance Heteroskedasticity Deep Learning Single Pass

Apr 22, 2026cs.LG

Variance Is Not Importance: Structural Analysis of Transformer Compressibility Across Model Scales

We present a systematic empirical study of transformer compression through over 40 experiments on GPT-2 (124M parameters) and Mistral 7B (7.24B parameters). Our analysis covers spectral compression, block-level function replacement, rotation-based quantization, activation geometry, and adaptive early exit. We identify five structural properties relevant to compression. (1) Variance is not importance: high-variance activation directions are approximately 96 percent uncorrelated with predictive directions (measured via CCA), and projecting onto these subspaces preserves over 90 percent of variance while degrading perplexity. (2) Block linearity is conditional: transformer blocks are approximately linear (R^2 ~ 0.95 on GPT-2, 0.93 on Mistral block 31) only under the correct upstream distribution; modifying earlier blocks induces distribution shift that degrades downstream approximations. (3) The reconstruction wall: approaches that factor weights into quantized components amplify errors through cross-terms, making direct quantization strictly superior. (4) Linearity increases with depth: Mistral 7B exhibits a progression from R^2 = 0.17 (block 0) to R^2 = 0.93 (block 31), indicating a division between nonlinear feature construction and linear refinement. (5) Approximately 30 percent of tokens are computationally easy, confirmed via exit heads and KL divergence sensitivity. We demonstrate that single-block linear replacement achieves 34x compression with a 1.71 perplexity increase on the final block of Mistral 7B, while multi-block replacement fails due to residual error accumulation and distribution shift. These findings suggest fundamental limits to static post-training compression and motivate adaptive, per-token computation as a more effective direction.

Samuel Salfati

fraQtl AI Research

Importance Kullback-Leibler Divergence Variance

Apr 17, 2026cs.LG

Lower Bounds and Proximally Anchored SGD for Non-Convex Minimization Under Unbounded Variance

Analysis of Stochastic Gradient Descent (SGD) and its variants typically relies on the assumption of uniformly bounded variance, a condition that frequently fails in practical non-convex settings, such as neural network training, as well as in several elementary optimization settings. While several relaxations are explored in the literature, the Blum-Gladyshev (BG-0) condition, which permits the variance to grow quadratically with distance has recently been shown to be the weakest condition. However, the study of the oracle complexity of stochastic first-order non-convex optimization under BG-0 has remained underexplored. In this paper, we address this gap and establish information-theoretic lower bounds, proving that finding an

ε

-stationary point requires

Ω(ε^{-6})

stochastic BG-0 oracle queries for smooth functions and

Ω(ε^{-4})

queries under mean-square smoothness. These limits demonstrate an unavoidable degradation from classical bounded-variance complexities, i.e.,

Ω(ε^{-4})

and

Ω(ε^{-3})

for smooth and mean-square smooth cases, respectively. To match these lower bounds, we consider Proximally Anchored STochastic Approximation (PASTA), a unified algorithmic framework that couples Halpern anchoring with Tikhonov regularization to dynamically mitigate the extra variance explosion term permitted by the BG-0 oracle. We prove that PASTA achieves minimax optimal complexities across numerous non-convex regimes, including standard smooth, mean-square smooth, weakly convex, star-convex, and Polyak-Lojasiewicz functions, entirely under an unbounded domain and unbounded stochastic gradients.

Arda Fazla, Ege C. Kaya, Antesh Upadhyay +1

School of Electrical and Computer Engineering, Purdue University, West Lafayette, IN 47907, USA.

Stochastic Convex Optimization Stochastic Gradient Descent Convex Optimization Nonconvex First Order Oracle Complexity Optimal Sample Complexity+3

Apr 10, 2026cs.LG

Statistical Properties of the King Wen Sequence: An Anti-Habituation Structure That Does Not Improve Neural Network Training

The King Wen sequence of the I-Ching (c. 1000 BC) orders 64 hexagrams -- states of a six-dimensional binary space -- in a pattern that has puzzled scholars for three millennia. We present a rigorous statistical characterization of this ordering using Monte Carlo permutation analysis against 100,000 random baselines. We find that the sequence has four statistically significant properties: higher-than-random transition distance (98.2nd percentile), negative lag-1 autocorrelation (p=0.037), yang-balanced groups of four (p=0.002), and asymmetric within-pair vs. between-pair distances (99.2nd percentile). These properties superficially resemble principles from curriculum learning and curiosity-driven exploration, motivating the hypothesis that they might benefit neural network training. We test this hypothesis through three experiments: learning rate schedule modulation, curriculum ordering, and seed sensitivity analysis, conducted across two hardware platforms (NVIDIA RTX 2060 with PyTorch and Apple Silicon with MLX). The results are uniformly negative. King Wen LR modulation degrades performance at all tested amplitudes. As curriculum ordering, King Wen is the worst non-sequential ordering on one platform and within noise on the other. A 30-seed sweep confirms that only King Wen's degradation exceeds natural seed variance. We explain why: the sequence's high variance -- the very property that makes it statistically distinctive -- destabilizes gradient-based optimization. Anti-habituation in a fixed combinatorial sequence is not the same as effective training dynamics.

Augustin Chan

Independent Researcher

Neural Network Training Variance