Covariance Estimation

Follow this topic Ask about this topic

Covariance estimation studies how to recover the relationships and uncertainty among variables from finite, often noisy observations, supporting tasks such as scientific inference, portfolio design, sensor fusion, and machine learning. Classical approaches include shrinkage and low-rank models, while robust likelihoods, Bayesian methods, and expectation-maximization address outliers and missing data. Newer methods use online estimators and structured neural models to track changing covariance or represent large matrices efficiently, sometimes with privacy or stability guarantees. A central challenge is balancing accuracy and reliable uncertainty estimates against limited data, computational cost, distribution shifts, and privacy constraints.

Latest papers

61 papers7 papers in the last four weeks, down 22% on the four weeks before. 0.0% of all new papers.

May 26, 2026

Semiparametrically Efficient Inference for Kernel Measures of Noise Heterogeneity

We develop semiparametrically efficient inference for kernel measures of noise heterogeneity in additive noise models. In many applications, the regression function is estimated using flexible machine learning methods. Downstream procedures based on the resulting residuals can then inherit first-stage bias: regression error may induce spurious dependence between covariates and residuals, invalidating the assumptions needed for standard analysis. We construct a novel Hilbert-valued one-step estimator of the kernel covariance operator between covariates and residuals. Our estimator yields bootstrap-calibrated tests for residual independence and goodness of fit in additive noise models, while also providing asymptotically efficient confidence intervals for the kernel dependence measure under noise heterogeneity. The framework extends to settings with additional covariates, enabling inference on distributional heterogeneity of residual noise across treatment groups. Simulations show improved calibration and power relative to naive plug-in residual methods.

Jakub Wornbard, Zikai Shen, Dimitri Meunier +1

University College London

Semiparametric Inference Kernel Methods Covariance Estimation Statistical Hypothesis Testing

May 26, 2026

Gaussian Process-based learning with new MCMC-based implementation of Wishart prior on correlation matrix

In probabilstic supervised learning of an input-output relationship - as a sample function of a Gaussian Process (GP) - priors are typically specified for the hyperparameters of the kernel that parametrises the covariance function of the GP, where the induced covariance matrix of the (resulting multivariate Normal) likelihood, governs the learning and prediction. When the sought function is highly multivariate, multiple lengthscale parameters must be learnt simultaneously, making inference difficult. We develop a ``self-assembled'' Wishart prior for the covariance matrix, while undertaking Bayesian inference on the kernel hyperparameters using MCMC. The construction uses a look-back window over recent MCMC iterations to define a time-step dependent scale matrix, thereby introducing adaptiveness to the chain. Results suggest that direct prior specification on the covariance matrix can be useful for diagnosing weakly informative inputs within the GP-based learning paradigm. We support our prior development with two distinct empirical illustrations - one on synthetic data, and another on a real-world dataset.

Kane Warrior, Dalia Chakrabarty

Department of Mathematics University of York

Bayesian Inference Covariance Estimation Gaussian Process Regression

May 22, 2026

Private Adaptive Covariance Estimation via Gaussian Graphical Models

We propose PACE-GGM, a data-adaptive differentially private method for covariance estimation that concentrates its privacy budget on the most informative entries of the empirical covariance matrix, rather than perturbing all entries. This applies in the natural setting where the modeler supplies separate bounds for each variable, so that individual entries can be measured with less noise than the full matrix. In each round, our method selects a poorly approximated entry, measures it using the Gaussian mechanism, and then reconstructs a full covariance matrix using a maximum-entropy reconstruction objective, leading to a Gaussian graphical model structure. Experiments on diverse real-world datasets demonstrate consistent improvements in estimation error with respect to the Gaussian mechanism and other baselines, particularly in high-dimensional and low-to-moderate privacy regimes.

Cecilia Ferrando, Miguel Fuentes, Brett Mullins +2

Manning College of Information and Computer Sciences University of Massachusetts Amherst

Differential Privacy Covariance Estimation

May 18, 2026

Self-Distillation is Optimal Among Spectral Shrinkage Estimators in Spiked Covariance Models

Self-distillation has emerged as a promising technique for improving model performance in modern machine learning systems. We develop the statistical foundations of self-distillation in spiked covariance models, by introducing and analyzing a broad class of estimators, namely spectral shrinkage estimators. We establish that for spiked covariance matrices with

s

spikes,

s

-step self-distillation achieves optimal performance among spectral shrinkage estimators, outperforming well-known estimators in statistics and machine learning. Moreover, we show that

s

steps are necessary for optimality: any

(s-k)

-step distilled estimator is strictly suboptimal for

1 \leq k \leq s

. For the special subclass of isotropic covariances, we show that optimally tuned Ridge regression performs best among spectral shrinkage estimators. We also study a federated approach where multiple data centers share spectral shrinkage estimators and a common server seeks to aggregate them to achieve optimal performance. In this case, we find that the best local rule again takes the form of self-distillation, though it differs from the optimal rule when data are hosted centrally on a single server. Together, our results elucidate why self-distillation improves predictive performance and provide a broader statistical framework connecting it with classical shrinkage-based methods.

Radu Lecoiu, Debarghya Mukherjee, Pragya Sur

Department of Statistics, Harvard University · Department of Mathematics & Statistics, Boston University

Self-Distillation Spectral Methods Covariance Estimation Statistical Learning Theory Shrinkage Estimation

May 17, 2026

A Hybrid Gaussian Process Regression Framework for Stable Volatility-Covariance Estimation: Evidence from Global Equity Indices

Accurate forecasting of the Volatility-Covariance Matrix (VCV) is central to regulatory capital adequacy processes such as the Internal Capital Adequacy Assessment Process (ICAAP) and the Comprehensive Capital Analysis and Review (CCAR). Traditional econometric models, including GARCH-family and Exponentially Weighted Moving Average (EWMA) approaches, suffer from parametric rigidity, distributional assumptions, and numerical instability under stress, leading to systematic underestimation of tail risk. This paper proposes and validates a novel Hybrid Gaussian Process Regression-Historical Simulation (GPR-HS) framework for estimating Value-at-Risk (VaR) and Expected Shortfall (ES) across a diversified portfolio of seven major global equity indices. The framework decouples the VCV estimation problem: individual asset volatilities are modelled dynamically using Univariate GPR with a Matern 5/2 kernel, while inter-asset correlations are estimated via stable historical covariance. A key methodological contribution is the Aggressive Noise Initialization (ANI) strategy, which sets the initial White Noise kernel variance equal to the empirical variance of the training returns, ensuring Gram matrix positive-definiteness, regularization, and conservative, regulatory-compliant forecasts. Evaluated using an expanding window forward-chaining cross-validation scheme over June 2020 -June 2025, the GPR-HS framework achieves regulatory compliance in the majority of test splits; including a 100% ES pass rate at the portfolio level, while outperforming the static Historical VaR benchmark in 71.4% of univariate cases by Quadratic Loss and 100% of cases by violation count.

Ujjwala Vadrevu

Quantitative Finance Financial Time Series Covariance Estimation Gaussian Process Regression

May 13, 2026

Covariance-aware sampling for Diffusion Models

We present a covariance-aware sampler that improves the quality of pixel-space Diffusion Model (DM) sampling in the few-step regime. We hypothesize that in the few-step regime samplers fail because they rely solely on the predicted mean of the reverse distribution, while our solution explicitly models the reverse-process covariance. Our method combines Tweedie's formula to estimate the covariance with an efficient, structured Fourier-space decomposition of the covariance matrix. Implemented as an extension of DDIM, our method requires only a minimal overhead: one extra Jacobian-Vector Product (JVP) per step. We demonstrate that for pixel-based DMs, our method consistently produces superior samples compared to state-of-the-art second order samplers (Heun, DPM-Solver++) and the recent aDDIM sampler, at an identical number of function evaluations (NFE).

Andrea Schioppa, Tim Salimans

GDM - Amsterdam · work done at GDM - Amsterdam

Diffusion Model Sampling Diffusion Sampling Pixel-Space Diffusion Models Few-Step Diffusion Sampling Covariance Estimation

May 8, 2026

Online Segmented Beamforming via Dynamic Programming

In dynamic acoustic environments characterized by time-varying interferers and moving sources, effective beamforming requires accurately identifying stationary regions over time. Traditional Capon beamformers rely on the instantaneous ensemble covariance matrix, which is inaccessible in practice. Practical implementations overcome this by estimating the sample covariance matrix (SCM) through averaging over a block of temporal samples. However, in non-stationary settings, a naive batch approach fails. Moving interferers smear the SCM, causing the beamformer to place nulls in outdated locations while failing to track newly active interferers, thereby degrading its nulling capabilities. To address this fundamental limitation, an Online Segmented Beamformer is proposed. This algorithm incorporates data-driven temporal segmentation to causally minimize output power while dynamically adapting the SCM estimation windows to local stationarity. By framing the problem through the lens of dynamic programming, the proposed method tracks abrupt environmental changes and resets covariance estimates in real-time. We validate the performance of this framework in a complex, reverberant simulated acoustic environment and in highly reverberant real world experiments, demonstrating its superiority over fixed-window adaptive methods.

Manan Mittal, Ryan M. Corey, Diego Cuji +2

Stony Brook University1 · University of Illinois Chicago2 · University of Massachusetts Darthmouth3

Dynamic Programming Covariance Estimation

May 1, 2026

Divergence is Uncertainty: A Closed-Form Posterior Covariance for Flow Matching

Flow matching has become a leading framework for generative modeling, but quantifying the uncertainty of its samples remains an open problem. Existing approaches retrain the model with auxiliary variance heads, maintain costly ensembles, or propagate approximate covariance through many integration steps, trading off training cost, inference cost, or accuracy. We show that none of these trade-offs is necessary. By extending Tweedie's formula from the denoising setting to the flow matching interpolant, we derive an exact, closed-form expression for the posterior covariance at every point along the generative trajectory. The result depends on a single quantity, namely the divergence of the learned velocity field, which can be computed post-hoc on any pre-trained flow matching model, requiring no retraining and no architectural modification. For one-step generators such as MeanFlow, the same formula yields the end-to-end generation uncertainty in a single forward pass, eliminating the multi-step variance propagation required by all prior methods. Experiments on MNIST confirm that the resulting per-pixel uncertainty maps are semantically meaningful, concentrating on digit boundaries where inter-sample variation is highest, and that the scalar uncertainty score tracks actual prediction error, all at roughly

10^4 \times

less total compute than ensembling or Monte Carlo dropout.

Jiarui Xing, Song Wang, Jian Wang

Yale University · Harvard Medical School

Flow Matching Uncertainty Quantification Bayesian Inference Flow-Based Generative Modeling Covariance Estimation Uncertainty Propagation

Apr 25, 2026

Inference of Online Newton Methods with Nesterov's Accelerated Sketching

Reliable decision-making with streaming data requires principled uncertainty quantification of online methods. While first-order methods enable efficient iterate updates, their inference procedures still require updating proper (covariance) matrices, incurring

O(d^2)

time and memory complexity, and are sensitive to ill-conditioning and noise heterogeneity of the problem. This costly inference task offers an opportunity for more robust second-order methods, which are, however, bottlenecked by solving Newton systems with

O(d^3)

complexity. In this paper, we address this gap by studying an online Newton method with Hessian averaging, where the Newton direction at each step is approximately computed using a sketch-and-project solver with Nesterov's acceleration, matching

O(d^2)

complexity of first-order methods. For the proposed method, we quantify its uncertainty arising from both random data and randomized computation. Under standard smoothness and moment conditions, we establish global almost-sure convergence, prove asymptotic normality of the last iterate with a limiting covariance characterized by a Lyapunov equation, and develop a fully online covariance estimator with non-asymptotic convergence guarantees. We also connect the resulting uncertainty quantification to that of exact and sketched Newton methods without Nesterov's acceleration. Extensive experiments on regression models demonstrate the superiority of the proposed method for online inference.

Haoxuan Wang, Xinchen Du, Sen Na

School of Industrial and Systems Engineering, Georgia Institute of Technology.

Stochastic Approximation Second-Order Optimization Randomized Sketching Covariance Estimation

Apr 23, 2026

Refining Covariance Matrix Estimation in Stochastic Gradient Descent Through Bias Reduction

We study online inference and asymptotic covariance estimation for the stochastic gradient descent (SGD) algorithm. While classical methods (such as plug-in and batch-means estimators) are available, they either require inaccessible second-order (Hessian) information or suffer from slow convergence. To address these challenges, we propose a novel, fully online de-biased covariance estimator that eliminates the need for second-order derivatives while significantly improving estimation accuracy. Our method employs a bias-reduction technique to achieve a convergence rate of

n^{(α-1)/2} \sqrt{\log n}

, outperforming existing Hessian-free alternatives.

Ziyang Wei, Wanrong Zhu, Jingyang Lyu +1

Department of Statistics University of Chicago Chicago, IL 60637, USA · Department of Statistics University of California, Irvine Irvine, CA 92617, USA · Department of Statistics University of Wisconsin–Madison Madison, WI 53706, USA

Covariance Estimation Stochastic Gradient Descent

Apr 14, 2026

On Higher-Order Geometric Refinements of Classical Covariance Asymptotics: An Approach via Intrinsic and Extrinsic Information Geometry

Classical Fisher-information asymptotics describe the covariance of regular efficient estimators through the local quadratic approximation of the log-likelihood, and thus capture first-order geometry only. In curved models, including mixtures, curved exponential families, latent-variable models, and manifold-constrained parameter spaces, finite-sample behavior can deviate systematically from these predictions. We develop a coordinate-invariant, curvature-aware refinement by viewing a regular parametric family as a Riemannian manifold

(Θ,g)

with Fisher--Rao metric, immersed in

L^2(μ)

through the square-root density map. Under suitable regularity and moment assumptions, we derive an

n^{-2}

correction to the leading

n^{-1}I(θ)^{-1}

covariance term for score-root, first-order efficient estimators. The correction is governed by a tensor

P_{ij}

that decomposes canonically into three parts, an intrinsic Ricci-type contraction of the Fisher--Rao curvature tensor, an extrinsic Gram-type contraction of the second fundamental form, and a Hellinger discrepancy tensor encoding higher-order probabilistic information not determined by immersion geometry alone. The extrinsic term is positive semidefinite, the full correction is invariant under smooth reparameterization, and it vanishes identically for full exponential families. We then extend the picture to singular models, where Fisher information degenerates. Using resolution of singularities under an additive normal crossing assumption, we describe the resolved metric, the role of the real log canonical threshold in learning rates and posterior mean-squared error, and a curvature-based covariance expansion on the resolved space that recovers the regular theory as a special case. This framework also suggests geometric diagnostics of weak identifiability and curvature-aware principles for regularization and optimization.

Malik Amir, Sourangshu Ghosh

Information Geometry Covariance Estimation Singular Learning Theory

Apr 12, 2026

Online Covariance Estimation in Averaged SGD: Improved Batch-Mean Rates and Minimax Optimality via Trajectory Regression

We study online covariance matrix estimation for Polyak--Ruppert averaged stochastic gradient descent (SGD). The online batch-means estimator of Zhu, Chen and Wu (2023) achieves an operator-norm convergence rate of

O(n^{-(1-α)/4})

, which yields

O(n^{-1/8})

at the optimal learning-rate exponent

α\rightarrow 1/2^+

. A rigorous per-block bias analysis reveals that re-tuning the block-growth parameter improves the batch-means rate to

O(n^{-(1-α)/3})

, achieving

O(n^{-1/6})

. The modified estimator requires no Hessian access and preserves

O(d^2)

memory. We provide a complete error decomposition into variance, stationarity bias, and nonlinearity bias components. A weighted-averaging variant that avoids hard truncation is also discussed. We establish the minimax rate

Θ(n^{-(1-α)/2})

for Hessian-free covariance estimation from the SGD trajectory: a Le Cam lower bound gives

Ω(n^{-(1-α)/2})

, and a trajectory-regression estimator--which estimates the Hessian by regressing SGD increments on iterates--achieves

O(n^{-(1-α)/2})

, matching the lower bound. The construction reveals that the bottleneck is the sublinear accumulation of information about the Hessian from the SGD drift.

Yijin Ni, Xiaoming Huo

H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology

Polyak-Ruppert Averaging Covariance Estimation Minimax Estimation Stochastic Gradient Descent

Apr 4, 2026

Polynomial-Time Optimal Group Selection via the Double-Commutator Eigenvalue Problem

The algebraic diversity framework generalizes temporal averaging over multiple observations to algebraic group action on a single observation for second-order statistical estimation. The central open problem in this framework is

\textit{group selection}

: given an

M

-dimensional observation with unknown covariance structure, find the finite group whose spectral decomposition best matches the covariance. Naive enumeration of all subgroups of the symmetric group

S_M

requires exponential time in

M

. We prove that this combinatorial problem reduces to a generalized eigenvalue problem derived from the double commutator of the covariance matrix, yielding a polynomial-time algorithm with complexity

O(d^2M^2 + d^3)

, where

d

is the dimension of a generator basis. The minimum eigenvector of the double-commutator matrix directly constructs the optimal group generator in closed form, with no iterative optimization. The reduction is exact: the double-commutator minimum eigenvalue is zero if and only if the optimal generator lies in the span of the basis, and its magnitude provides a certifiable optimality gap when it does not. This problem does not appear in the standard catalogs of computational complexity (Garey and Johnson, 1979) and represents a new class linking group theory, matrix analysis, and statistical estimation. We establish connections to independent component analysis (JADE), structured matrix nearness problems, and simultaneous matrix diagonalization, and we show that the double-commutator formulation is the unique approach that is simultaneously polynomial-time, closed-form, and certifiable. We extend the framework to non-Abelian symmetry recovery via a Sequential GEVP with deflation, and add two identifiability theorems characterizing the commutant-lattice ambiguity and the dichotomy on whether

\mathrm{Aut}(\mathbf{R})

recovers a generative subgroup or only a supergroup.

Mitchell A. Thornton

Richardson, TX 75080 USA

Covariance Estimation Spectral Methods

Apr 3, 2026

State estimations and noise identifications with intermittent corrupted observations via Bayesian variational inference

This paper focuses on the state estimation problem in distributed sensor networks, where intermittent packet dropouts, corrupted observations, and unknown noise covariances coexist. To tackle this challenge, we formulate the joint estimation of system states, noise parameters, and network reliability as a Bayesian variational inference problem, and propose a novel variational Bayesian adaptive Kalman filter (VB-AKF) to approximate the joint posterior probability densities of the latent parameters. Unlike existing AKF that separately handle missing data and measurement outliers, the proposed VB-AKF adopts a dual-mask generative model with two independent Bernoulli random variables, explicitly characterizing both observable communication losses and latent data authenticity. Additionally, the VB-AKF integrates multiple concurrent multiple observations into the adaptive filtering framework, which significantly enhances statistical identifiability. Comprehensive numerical experiments verify the effectiveness and asymptotic optimality of the proposed method, showing that both parameter identification and state estimation asymptotically converge to the theoretical optimal lower bound with the increase in the number of sensors.

Peng Sun, Ruoyu Wang, Xue Luo

School of Mathematical Sciences, Beihang University, Beijing, P. R. China 102206. · School of Mathematical Sciences, Beihang University, Beijing, P. R. China 102206, and Key Laboratory of Mathematics, Informatics and Behavioral Semantics (LMIB), Beihang University, Beijing, P. R. China 100191.

Robust Estimation Covariance Estimation Variational Inference

Apr 1, 2026

Model Merging via Data-Free Covariance Estimation

Model merging provides a way of cheaply combining individual models to produce a model that inherits each individual's capabilities. While some merging methods can approach the performance of multitask training, they are often heuristically motivated and lack theoretical justification. A principled alternative is to pose model merging as a layer-wise optimization problem that directly minimizes interference between tasks. However, this formulation requires estimating per-layer covariance matrices from data, which may not be available when performing merging. In contrast, many of the heuristically-motivated methods do not require auxiliary data, making them practically advantageous. In this work, we revisit the interference minimization framework and show that, under certain conditions, covariance matrices can be estimated directly from difference matrices, eliminating the need for data while also reducing computational costs. We validate our approach across vision and language benchmarks on models ranging from 86M parameters to 7B parameters, outperforming previous data-free state-of-the-art merging methods

Marawan Gamal Abdel Hameed, Derek Tam, Pascal Jr Tikeng Notsawo +2

Mila & DIRO, Universit´e de Montr´eal · University of Toronto & Vector Institute

Multi-Task Learning Language Model Merging Model Merging Covariance Estimation

Mar 30, 2026

Machine Learning-Assisted High-Dimensional Matrix Estimation

Efficient estimation of high-dimensional matrices-including covariance and precision matrices-is a cornerstone of modern multivariate statistics. Most existing studies have focused primarily on the theoretical properties of the estimators (e.g., consistency and sparsity), while largely overlooking the computational challenges inherent in high-dimensional settings. Motivated by recent advances in learning-based optimization method-which integrate data-driven structures with classical optimization algorithms-we explore high-dimensional matrix estimation assisted by machine learning. Specifically, for the optimization problem of high-dimensional matrix estimation, we first present a solution procedure based on the Linearized Alternating Direction Method of Multipliers (LADMM). We then introduce learnable parameters and model the proximal operators in the iterative scheme with neural networks, thereby improving estimation accuracy and accelerating convergence. Theoretically, we first prove the convergence of LADMM, and then establish the convergence, convergence rate, and monotonicity of its reparameterized counterpart; importantly, we show that the reparameterized LADMM enjoys a faster convergence rate. Notably, the proposed reparameterization theory and methodology are applicable to the estimation of both high-dimensional covariance and precision matrices. We validate the effectiveness of our method by comparing it with several classical optimization algorithms across different structures and dimensions of high-dimensional matrices.

Wan Tian, Hui Yang, Zhouhui Lian +2

Advanced Institute of Information Technology, Peking University · Wangxuan Institute of Computer Technology, Peking University, China, 100871 · Guanghua School of Management, Peking University, Beijing, China, 100871 +3

Learning to Optimize Alternating Direction Method of Multipliers Algorithm Unrolling Covariance Estimation

Mar 20, 2026

Minimax and Adaptive Covariance Matrix Estimation under Differential Privacy

Estimating covariance matrices is fundamental to a wide range of statistical applications. This paper studies minimax and adaptive estimation of high-dimensional covariance matrices under

ρ

-zero-concentrated differential privacy (

ρ

-zCDP) over three nested classes: the pointwise-decay class

\mathcal{H}_α

, the row-tail class

\mathcal{G}_α

, and the separated-block class

\mathcal{F}_α

. We consider both squared operator norm loss and normalized squared Frobenius norm loss. For

\mathcal{H}_α

and

\mathcal{G}_α

, we develop center--outer dyadic estimators tailored to the refined geometry of the two classes, while for

\mathcal{F}_α

, we develop a blockwise tridiagonal estimator. The resulting minimax-optimal rates reveal a nontrivial interplay among the smoothness

α

, the loss, the geometry of the covariance class, and the privacy constraint. In contrast to the non-private setting, privacy distinguishes covariance classes that share the same leading non-private rate and induces a polynomial dependence on the ambient dimension. We further develop procedures that adapt to the unknown decay parameter over all three covariance classes under both losses, at the cost of at most polylogarithmic factors. To establish minimax lower bounds, we develop a novel differentially private van Trees inequality that connects Fisher information with the

ρ

-zCDP constraint and may be useful for other private estimation problems. We also construct carefully designed prior distributions to obtain matching minimax lower bounds.

T. Tony Cai, Yicheng Li

Department of Statistics and Data Science, The Wharton School, University of Pennsylvania · KLATASDS-MOE, School of Statistics, East China Normal University

Covariance Estimation Parameter Estimation Minimax Estimation Differential Privacy

Mar 20, 2026

K-GMRF: Kinetic Gauss-Markov Random Field for First-Principles Covariance Tracking on Lie Groups

Tracking non-stationary covariance matrices is fundamental to vision yet hindered by existing estimators that either neglect manifold constraints or rely on first-order updates, incurring inevitable phase lag during rapid evolution. We propose K-GMRF, an online, training-free framework for covariance tracking that reformulates the problem as forced rigid-body motion on Lie groups. Derived from the Euler-Poincaré equations, our method interprets observations as torques driving a latent angular velocity, propagated via a structure-preserving symplectic integrator. We theoretically prove that this second-order dynamics achieves zero steady-state error under constant rotation, strictly superior to the proportional lag of first-order baselines. Validation across three domains demonstrates robust tracking fidelity: (i) on synthetic ellipses, K-GMRF reduces angular error by 30x compared to Riemannian EMA while maintaining stability at high speeds; (ii) on SO(3) stabilization with 20% dropout, it decreases geodesic error from 29.4° to 9.9°; and (iii) on OTB motion-blur sequences, it improves loU from 0.55 to 0.74 on BlurCar2 with a 96% success rate. As a fully differentiable symplectic module, K-GMRF provides a plug-and-play geometric prior for data-constrained scenarios and an interpretable layer within modern deep architectures.

ZhiMing Li

School of Computer Science and Technology, Tianjin University

Covariance Estimation Visual Object Tracking Dynamical Systems

Mar 19, 2026

Recovering Sparse Neural Connectivity from Partial Measurements: A Covariance-Based Approach with Granger-Causality Refinement

Inferring the connectivity of neural circuits from incomplete observations is a fundamental challenge in neuroscience. We present a covariance-based method for estimating the weight matrix of a recurrent neural network from sparse, partial measurements across multiple recording sessions. By accumulating pairwise covariance estimates across sessions where different subsets of neurons are observed, we reconstruct the full connectivity matrix without requiring simultaneous recording of all neurons. A Granger-causality refinement step enforces biological constraints via projected gradient descent. Through systematic experiments on synthetic networks modeling small brain circuits, we characterize a fundamental control-estimation tradeoff: stimulation aids identifiability but disrupts intrinsic dynamics, with the optimal level depending on measurement density. We discover that the ``incorrect'' linear approximation acts as implicit regularization -- outperforming the oracle estimator with known nonlinearity at all operating regimes -- and provide an exact characterization via the Stein--Price identity.

Quilee Simeon

Massachusetts Institute of Technology

Brain Network Analysis Sparse Recovery Functional Connectivity Recurrent Neural Networks Covariance Estimation

Mar 17, 2026

High-dimensional estimation with missing data: Statistical and computational limits

We consider computationally-efficient estimation of population parameters when observations are subject to missing data. In particular, we consider estimation under the realizable contamination model of missing data in which an

ε

fraction of the observations are subject to an arbitrary (and unknown) missing not at random (MNAR) mechanism. When the true data is Gaussian, we provide evidence towards statistical-computational gaps in several problems. For mean estimation in

\ell_2

norm, we show that in order to obtain error at most

ρ

, for any constant contamination

ε\in (0, 1)

, (roughly)

n \gtrsim d e^{1/ρ^2}

samples are necessary and that there is a computationally-inefficient algorithm which achieves this error. On the other hand, we show that any computationally-efficient method within certain popular families of algorithms requires a much larger sample complexity of (roughly)

n \gtrsim d^{1/ρ^2}

and that there exists a polynomial time algorithm based on sum-of-squares which (nearly) achieves this lower bound. For covariance estimation in relative operator norm, we show that a parallel development holds. Finally, we turn to linear regression with missing observations and show that such a gap does not persist. Indeed, in this setting we show that minimizing a simple, strongly convex empirical risk nearly achieves the information-theoretic lower bound in polynomial time.

Kabir Aladin Verchand, Ankit Pensia, Saminul Haque +1

Department of Data Sciences and Operations, University of Southern California · Department of Statistics and Data Science, Carnegie Mellon University · Department of Computer Science, Stanford University

Learning with Missing Data Covariance Estimation Linear Regression

Mar 12, 2026

A Hybrid Neural-Assisted Unscented Kalman Filter for Unmanned Ground Vehicle Navigation

Modern autonomous navigation for unmanned ground vehicles relies on different estimators to fuse inertial sensors and GNSS measurements. However, the constant noise covariance matrices often struggle to account for dynamic real-world conditions. In this work we propose a hybrid estimation framework that bridges classical state estimation foundations with modern deep learning approaches. Instead of altering the fundamental unscented Kalman filter equations, a dedicated deep neural network is developed to predict the process and measurement noise uncertainty directly from raw inertial and GNSS measurements. We present a sim2real approach, with training performed only on simulative data. In this manner, we offer perfect ground truth data and relieves the burden of extensive data recordings. To evaluate our proposed approach and examine its generalization capabilities, we employed a 160-minutes test set from three datasets each with different types of vehicles (off-road vehicle, passenger car, and mobile robot), inertial sensors, road surface, and environmental conditions. We demonstrate across the three datasets a position improvement of

12.7\%

compared to the adaptive model-based approach. Thus, offering a scalable and a more robust solution for unmanned ground vehicles navigation tasks.

Gal Versano, Itzik Klein

Autonomous Navigation and Sensor Fusion Lab, Hatter Department of Marine Technologies, Charney School of Marine Sciences, University of Haifa, Israel.

Robot Navigation Unscented Kalman Filtering Covariance Estimation Multimodal Sensor Fusion

Mar 11, 2026

Heavy-Tailed Principal Component Analysis

Principal Component Analysis (PCA) is a cornerstone of dimensionality reduction, yet its classical formulation relies critically on second-order moments and is therefore fragile in the presence of heavy-tailed data and impulsive noise. While numerous robust PCA variants have been proposed, most either assume finite variance, rely on sparsity-driven decompositions, or address robustness through surrogate loss functions without a unified treatment of infinite-variance models. In this paper, we study PCA for high-dimensional data generated according to a superstatistical dependent model of the form

\mathbf{X} = A^{1/2}\mathbf{G}

, where

A

is a positive random scalar and

\mathbf{G}

is a Gaussian vector. This framework captures a wide class of heavy-tailed distributions, including multivariate

t

and sub-Gaussian

α

-stable laws. We formulate PCA under a logarithmic loss, which remains well defined even when moments do not exist. Our main theoretical result shows that, under this loss, the principal components of the heavy-tailed observations coincide with those obtained by applying standard PCA to the covariance matrix of the underlying Gaussian generator. Building on this insight, we propose robust estimators for this covariance matrix directly from heavy-tailed data and compare them with the empirical covariance and Tyler's scatter estimator. Extensive experiments, including background denoising tasks, demonstrate that the proposed approach reliably recovers principal directions and significantly outperforms classical PCA in the presence of heavy-tailed and impulsive noise, while remaining competitive under Gaussian noise.

Mario Sayde, Christopher Khater, Jihad Fahs +1

Electrical and Computer Engineering Department American University of Beirut Riad El-Solh, Beirut 1107 2020, Lebanon

Principal Component Analysis Robust Estimation Heavy-Tailed Distributions Covariance Estimation

Mar 9, 2026

Outlier-robust Autocovariance Least Square Estimation via Iteratively Reweighted Least Square

The autocovariance least squares (ALS) method is a computationally efficient approach for estimating noise covariances in Kalman filters without requiring specific noise models. However, conventional ALS and its variants rely on the classic least mean squares (LMS) criterion, making them highly sensitive to measurement outliers and prone to severe performance degradation. To overcome this limitation, this paper proposes a novel outlier-robust ALS algorithm, termed ALS-IRLS, based on the iteratively reweighted least squares (IRLS) framework. Specifically, the proposed approach introduces a two-tier robustification strategy. First, an innovation-level adaptive thresholding mechanism is employed to filter out heavily contaminated data. Second, the outlier-contaminated autocovariance is formulated using an

ε

-contamination model, where the standard LMS criterion is replaced by the Huber cost function. The IRLS method is then utilized to iteratively adjust data weights based on estimation deviations, effectively mitigating the influence of residual outliers. Comparative simulations demonstrate that ALS-IRLS reduces the root-mean-square error (RMSE) of noise covariance estimates by over two orders of magnitude compared to standard ALS. Furthermore, it significantly enhances downstream state estimation accuracy, outperforming existing outlier-robust Kalman filters and achieving performance nearly equivalent to the ideal Oracle lower bound in the presence of noisy and anomalous data.

Jiahong Li, Fang Deng

College of Robotics, Beijing Union University, Beijing 100101, China · Key Laboratory of Intelligent Control and Decision of Complex Systems, and also with the School of Automation, Beijing Institute of Technology, Beijing 100081, China

Covariance Estimation Robust Estimation Kalman Filtering

Mar 9, 2026

Joint Return and Risk Modeling with Deep Neural Networks for Portfolio Construction

Portfolio construction traditionally relies on separately estimating expected returns and covariance matrices using historical statistics, often leading to suboptimal allocation under time-varying market conditions. This paper proposes a joint return and risk modeling framework based on deep neural networks that enables end-to-end learning of dynamic expected returns and risk structures from sequential financial data. Using daily data from ten large-cap US equities spanning 2010 to 2024, the proposed model is evaluated across return prediction, risk estimation, and portfolio-level performance. Out-of-sample results during 2020 to 2024 show that the deep forecasting model achieves competitive predictive accuracy (RMSE = 0.0264) with economically meaningful directional accuracy (51.9%). More importantly, the learned representation effectively captures volatility clustering and regime shifts. When integrated into portfolio optimization, the proposed Neural Portfolio strategy achieves an annual return of 36.4% and a Sharpe ratio of 0.91, outperforming equal weight and historical mean-variance benchmarks in terms of risk-adjusted performance. These findings demonstrate that jointly modeling return and covariance dynamics can provide consistent improvements over traditional allocation approaches. The framework offers a scalable and practical alternative for data-driven portfolio construction under nonstationary market conditions.

Keonvin Park

Interdisciplinary Program in Artificial Intelligence, Seoul National University, Seoul, 08829, South Korea

Quantitative Finance Portfolio Optimization Covariance Estimation Multivariate Time Series Forecasting

Mar 3, 2026

ACE-Merging: Data-Free Model Merging with Adaptive Covariance Estimation

Model merging aims to combine multiple task-specific expert models into a single model while preserving generalization across diverse tasks. However, interference among experts, especially when they are trained on different objectives, often leads to significant performance degradation. Despite recent progress, resolving this interference without data access, retraining, or architectural modification remains a fundamental challenge. This paper provides a theoretical analysis demonstrating that the input covariance of each task, which is a key factor for optimal merging, can be implicitly estimated from the parameter differences of its fine-tuned model, even in a fully data-free setting. Building on this insight, we introduce \acem, an Adaptive Covariance Estimation framework that effectively mitigates inter-task interference. Our approach features a principled, closed-form solution that contrasts with prior iterative or heuristic methods. Extensive experiments on both vision and language benchmarks demonstrate that \acem sets a new state-of-the-art among data-free methods. It consistently outperforms existing baselines; for example, \acem achieves an average absolute improvement of 4% over the previous methods across seven tasks on GPT-2. Owing to its efficient closed-form formulation, \acem delivers superior performance with a modest computational cost, providing a practical and theoretically grounded solution for model merging.

Bo Xu, Haotian Wu, Hehai Lin +4

The Hong Kong University of Science and Technology (Guangzhou) · University of Science and Technology of China

Model Merging Language Model Merging Covariance Estimation

Feb 24, 2026

Environment-Aware Learning of Smooth GNSS Covariance Dynamics for Autonomous Racing

Ensuring accurate and stable state estimation is a challenging task crucial to safety-critical domains such as high-speed autonomous racing, where measurement uncertainty must be both adaptive to the environment and temporally smooth for control. In this work, we develop a learning-based framework, LACE, capable of directly modeling the temporal dynamics of GNSS measurement covariance. We model the covariance evolution as an exponentially stable dynamical system where a deep neural network (DNN) learns to predict the system's process noise from environmental features through an attention mechanism. By using contraction-based stability and systematically imposing spectral constraints, we formally provide guarantees of exponential stability and smoothness for the resulting covariance dynamics. We validate our approach on an AV-24 autonomous racecar, demonstrating improved localization performance and smoother covariance estimates in challenging, GNSS-degraded environments. Our results highlight the promise of dynamically modeling the perceived uncertainty in state estimation problems that are tightly coupled with control sensitivity.

Y. Deemo Chen, Arion Zimmermann, Thomas A. Berrueta +1

Department of Computing and Mathematical Sciences, California Institute of Technology, Pasadena, CA.

Contraction Theory Covariance Estimation Robot State Estimation Autonomous Racing

Feb 18, 2026

Machine Learning Argument of Latitude Error Model for LEO Satellite Orbit and Covariance Correction

Low Earth orbit (LEO) satellites are leveraged to support new position, navigation, and timing (PNT) service alternatives to GNSS. These alternatives require accurate propagation of satellite position and velocity with a realistic quantification of uncertainty. It is commonly assumed that the propagated uncertainty distribution is Gaussian; however, the validity of this assumption can be quickly compromised by the mismodeling of atmospheric drag. We develop a machine learning approach that corrects error growth in the argument of latitude for a diverse set of LEO satellites. The improved orbit propagation accuracy extends the applicability of the Gaussian assumption and modeling of the errors with a corrected mean and covariance. We compare the performance of a time-conditioned neural network and a Gaussian Process on datasets computed with an open source orbit propagator and publicly available Vector Covariance Message (VCM) ephemerides. The learned models predict the argument of latitude error as a Gaussian distribution given parameters from a single VCM epoch and reverse propagation errors. We show that this one-dimensional model captures the effect of mismodeled drag, which can be mapped to the Cartesian state space. The correction method only updates information along the dimensions of dominant error growth, while maintaining the physics-based propagation of VCM covariance in the remaining dimensions. We therefore extend the utility of VCM ephemerides to longer time horizons without modifying the functionality of the existing propagator.

Alex Moody, Penina Axelrad, Rebecca Russell

University of Colorado Boulder Boulder, CO · The Charles Stark Draper Laboratory, Inc. Cambridge, MA

Uncertainty Propagation Covariance Estimation LEO Satellite Networks

Nov 3, 2025

The Optimization Landscape of Carathéodory Decomposition of Toeplitz Covariances

Toeplitz covariance estimation is a classical signal processing problem, yet the geometry of the Gaussian maximum-likelihood objective remains only partially understood. Recent Newton-type, majorization-minimization, and gradient-based methods often solve the nonconvex problem globally with enough samples, but also reveal a difficult computational landscape. We study this phenomenon through an overparameterized Carathéodory representation, which parameterizes the covariance by the frequencies and amplitudes of steering vectors. A simple example shows that, for any uniform frequency grid, optimizing only the amplitudes can leave a positive error even in the population setting, which motivates optimizing the frequencies as well. Our main result proves that this nonlinear parameterization adds no stationary points: for any sample covariance, every stationary point with a positive definite covariance gives a stationary point of the Toeplitz maximum-likelihood problem. In the population setting, every such point therefore recovers the true covariance. We also prove that gradient descent converges to the true covariance in the population setting, and to the unique maximum likelihood estimate when the sample covariance is close enough to the true covariance. These results do not require overparameterization. Numerically, overparameterization speeds up gradient descent and gives a more accurate estimate for a fixed number of iterations.

Daniel Busbib, Ami Wiesel

School of Computer Science and Engineering, The Hebrew University of Jerusalem, Jerusalem 91904, Israel

Nonconvex Optimization Matrix Optimization Covariance Estimation

Feb 12, 2025

Non-asymptotic Estimation of Sparse Partial Correlation via Joint Partial Regression

Partial correlation analysis is widely used in modern multivariate analysis. However, prior methods lack non-asymptotic guarantees and have slow asymptotic estimation rates. We present a method for simultaneously estimating sparse high-dimensional partial correlation matrices and inverse covariance with non-asymptotic rates. The method is a two-stage estimation which exploits the connection between the inverse covariance matrix and linear regression estimation method while enforcing positive semi-definiteness simultaneously. An efficient proximal splitting algorithm for numerically computing the estimate is also derived, and we provide a fast implementation of the algorithm. The effectiveness of the proposed method is demonstrated on both synthetic and real-world data.

Samuel Erickson, Tobias Rydén

Division of Decision and Control Systems KTH Royal Institute of Technology Stockholm, Sweden · Lynx Asset Management AB Stockholm, Sweden

Covariance Estimation

Oct 29, 2024

Conditional Moment Kernels: Joint Modeling of Expected Returns and Covariances in Unbalanced Panels

We propose the conditional moment kernel (CMK), a joint model of the conditional mean and covariance matrix of asset returns given their characteristics. A single kernel function of the characteristics gives the means and second moments of all returns. The implied covariance matrices are positive semidefinite for any number and composition of assets, so the model applies directly to unbalanced panels. Its estimation is a convex program whose size does not depend on the number of assets. A floor at the scale of the idiosyncratic noise keeps the estimated systematic covariance away from zero where the data cannot resolve it, and a trace penalty selects the number of factors. A closed form, the CMK estimator, approximates the solution. For fixed features and hyper-parameters, we prove identification, consistency and finite-sample bounds. In out-of-sample tests on US stocks from 1972 to 2021, the CMK estimator fits the conditional moments significantly better than instrumented principal component analysis (IPCA) with the same characteristics. Almost all of the gain is in the covariance matrix, and nonlinear functions of the characteristics add to it. The implied unconditionally mean--variance efficient portfolios earn annualized Sharpe ratios of 4.0 to 5.1 before costs, close to those of IPCA and declining over time.

Damir Filipovic, Paul Schneider

École Polytechnique Fédérale de Lausanne and Swiss Finance Institute · Università della Svizzera italiana and Swiss Finance Institute

Quantitative Finance Covariance Estimation Kernel Methods