Design Guidelines for Nonlinear Kalman Filters via Covariance Compensation

Shida Jiang; Jaewoong Lee; Shengyu Tao; Scott Moura

eess.SYMar 24, 2026

Design Guidelines for Nonlinear Kalman Filters via Covariance Compensation

Authors: Shida Jiang, Jaewoong Lee, Shengyu Tao, Scott Moura

Organizations: Department of Civil and Environmental Engineering, University of California, Berkeley, CA 94720, USA.

Abstract

Nonlinear extensions of the Kalman filter (KF), such as the extended Kalman filter (EKF) and the unscented Kalman filter (UKF), are indispensable for state estimation in complex dynamical systems, yet the conditions for a nonlinear KF to provide robust and accurate estimations remain poorly understood. This work proposes a theoretical framework that identifies the causes of failure and success in certain nonlinear KFs and establishes guidelines for their improvement. Central to our framework is the concept of covariance compensation: the deviation between the covariance predicted by a nonlinear KF and that of the EKF. With this definition and detailed theoretical analysis, we derive three design guidelines for nonlinear KFs: (i) invariance under orthogonal transformations, (ii) sufficient covariance compensation beyond the EKF baseline, and (iii) selection of compensation magnitude that favors underconfidence. Both theoretical analysis and empirical validation confirm that adherence to these principles significantly improves estimation accuracy, whereas fixed parameter choices commonly adopted in the literature are often suboptimal. The codes and the proofs for all the theorems in this paper are available at https://github.com/Shida-Jiang/Guidelines-for-Nonlinear-Kalman-Filters.

Figures & tables

Figure 1: The selected sigma points in different variants of unscented Kalman filters when $n=2,\bm{x}\sim(\bm{0},I_{2\times 2})$ .

Figure 2: The sensitivity of the estimated covariance $P_{z}^{\textnormal{est}}$ and $P_{xz}^{\textnormal{est}}=[P_{xz}(1)\quad P_{xz}(2)]^{T}$ to rotation in different nonlinear KFs.

Figure 3: Estimation error of a simple system under different magnitudes of covariance estimation fluctuations and covariance compensation.

Applications	Tracking	Navigation	Generator
Number of states	6	2	4
Linear state transition function?	Yes	Yes	No
Number of measurements	2	1	1
Linear measurement functions?	No	No	No
Number of inputs	3	2	3
Number of iterations	30	100	100

Table I: Nonlinear systems investigated in this paper.

Figure 4: The actual and estimated geometric mean of the state estimation RMSE under different magnitudes of covariance compensation. (a) 3D target tracking, (b) terrain-referenced navigation, (c) generator state estimation.

Kalman Filtering

Kalman filtering recursively estimates hidden states by balancing model predictions with noisy measurements.

See all 73 papers

Unscented KalmanNet: Structure-Preserving Deep Learning with Calibrated Posterior Uncertainty under Incomplete Physics and Unknown Noise
Aug 4, 2026Minhyeok Ko, Abdollah ShafieezadehKalman Filtering Unscented Kalman Filtering
Department of Civil and Construction Engineering and Management, The University of Texas at Tyler, Tyler, TX, 75799, USA · Department of Civil, Environmental, and Geodetic Engineering, The Ohio State University, Columbus, OH, 43210, USA
The Iterative Equivariant Filter
Sep 11, 2026Pieter van Goor, James Richard ForbesEquivariant Filtering Robot State Estimation
School of Aerospace, Mechanical, and Mechatronic Engineering The University of Sydney NSW 2006, Australia · Dept. of Mechanical Engineering McGill University Montreal, QC, Canada
Schur-Neural KF: Learned Schur-Consistent Corrections to the Extended Kalman Filter
Sep 26, 2026Min Kim, Lianghao Cao, Soon-Jo Chung +1Kalman Filtering Extended Kalman Filtering
The authors are with the California Institute of Technology (Caltech), Pasadena, CA 91125 USA.

Aug 4, 2026J/K move · Enter open · S save

Unscented KalmanNet: Structure-Preserving Deep Learning with Calibrated Posterior Uncertainty under Incomplete Physics and Unknown Noise

Minhyeok Ko, Abdollah Shafieezadeh

Department of Civil and Construction Engineering and Management, The University of Texas at Tyler, Tyler, TX, 75799, USA · Department of Civil, Environmental, and Geodetic Engineering, The Ohio State University, Columbus, OH, 43210, USA

Kalman Filtering Unscented Kalman Filtering Covariance Estimation Robot State Estimation Uncertainty Calibration

Open paper AskPDF arXiv

Nonlinear state estimation requires sequentially fusing model-based predictions with noisy measurements. Under imperfect dynamics and unknown, time-varying noise statistics, this fusion can degrade in both accuracy and statistical consistency. Existing learning-aided filters largely treat accuracy and uncertainty estimation separately, limiting their ability to correct model-mismatch-induced bias while retaining an explicit, calibrated posterior covariance. This paper introduces Unscented KalmanNet (UKN), a model-based deep learning architecture that extends the Unscented Kalman Filter (UKF) with learned mechanisms for these two sources of filtering error while preserving explicit posterior covariance propagation. NoiseNet learns time-varying process and measurement covariances as bounded multiplicative corrections to baseline covariances, guaranteeing positive definiteness, while GainNet learns a bounded residual correction to the analytical UKF gain to compensate for model-mismatch-induced bias. A calibration-aware training objective couples state error with posterior covariance and innovation consistency terms through adaptive weighting, jointly optimizing accuracy and calibration. UKN is benchmarked against UKF, KalmanNet, and Bayesian KalmanNet on three synthetic systems and real-flight UZH-FPV data. It achieves the lowest state-estimation error in all four examples and reduces RMSE by 26.4-49.7% compared with UKF in the synthetic cases. Leave-one-sequence-out cross-validation over 11 flights shows 22.4% and 34.3% reductions in mean position and velocity RMSE, respectively. UKN also yields the lowest fold-to-fold variability, with dimension-normalized NEES and empirical coverage closest to nominal values among covariance-reporting filters. These results show that structured learned adaptation improves estimation accuracy while retaining calibrated uncertainty.

Abstract

Kalman Filtering

Explore similar work

Unscented KalmanNet: Structure-Preserving Deep Learning with Calibrated Posterior Uncertainty under Incomplete Physics and Unknown Noise

The Iterative Equivariant Filter

Schur-Neural KF: Learned Schur-Consistent Corrections to the Extended Kalman Filter